Fixed a bug that prevented alerts from being generated for doc level monitors that use wildcard characters in index names.

[AWSHurneyt]

Issue #, if available:

Description of changes:

1. Fixed a bug that prevented alerts from being generated for doc level monitors that use wildcard characters in index names.
2. Fixed integration tests that examine this use case.

CheckList:
[x] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[AWSHurneyt]

The BWC and security github checks are expected to be failing on main at the moment.

[lezzago]

Did we figure out exactly why the previous tests did not catch this bug?

[AWSHurneyt]

Some findings regarding the test that failed to pick up this bug

The test execute monitor with wildcard index that generates alerts and findings test used the * wildcard character to configure the index for the test monitor. However, it still didn't pick up this bug; alerts and findings were still generated.

This will need further deep diving, but preliminary troubleshooting so far suggests the issue may actually relate to the names used for the test indices; test1, test2, and test* in the monitor configuration itself.

(For reference, the test documents contained the value "test_field" : "us-west-2".)

Even without the fix included in this PR, the test passes when the query is defined as query = "NOT (test_field:\"us-west-1\")" or query = "test_field:\"us-west-2\"". However, I noticed that the test was using test1 and test2 for the index names whereas my manual testing used test-index1 and test-index2. I adjusted the test to add the -index to the index names, and it started to fail as expected; but only for query = "NOT (test_field:\"us-west-1\")". After applying the fix, it passed as expected for both query test cases.

[lezzago]

Lets create an issue to follow up on investigation

[opensearch-trigger-bot]

The backport to 2.0 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.0 2.0
# Navigate to the new working tree
cd .worktrees/backport-2.0
# Create a new branch
git switch --create backport/backport-894-to-2.0
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 8c033b93fab9d329eaabf76b684d172c6eb502ea
# Push it to GitHub
git push --set-upstream origin backport/backport-894-to-2.0
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.0

Then, create a pull request where the base branch is 2.0 and the compare/head branch is backport/backport-894-to-2.0.

[opensearch-trigger-bot]

The backport to 2.1 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.1 2.1
# Navigate to the new working tree
cd .worktrees/backport-2.1
# Create a new branch
git switch --create backport/backport-894-to-2.1
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 8c033b93fab9d329eaabf76b684d172c6eb502ea
# Push it to GitHub
git push --set-upstream origin backport/backport-894-to-2.1
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.1

Then, create a pull request where the base branch is 2.1 and the compare/head branch is backport/backport-894-to-2.1.

[opensearch-trigger-bot]

The backport to 2.2 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.2 2.2
# Navigate to the new working tree
cd .worktrees/backport-2.2
# Create a new branch
git switch --create backport/backport-894-to-2.2
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 8c033b93fab9d329eaabf76b684d172c6eb502ea
# Push it to GitHub
git push --set-upstream origin backport/backport-894-to-2.2
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.2

Then, create a pull request where the base branch is 2.2 and the compare/head branch is backport/backport-894-to-2.2.

[opensearch-trigger-bot]

The backport to 2.3 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.3 2.3
# Navigate to the new working tree
cd .worktrees/backport-2.3
# Create a new branch
git switch --create backport/backport-894-to-2.3
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 8c033b93fab9d329eaabf76b684d172c6eb502ea
# Push it to GitHub
git push --set-upstream origin backport/backport-894-to-2.3
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.3

Then, create a pull request where the base branch is 2.3 and the compare/head branch is backport/backport-894-to-2.3.

[opensearch-trigger-bot]

The backport to 2.4 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.4 2.4
# Navigate to the new working tree
cd .worktrees/backport-2.4
# Create a new branch
git switch --create backport/backport-894-to-2.4
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 8c033b93fab9d329eaabf76b684d172c6eb502ea
# Push it to GitHub
git push --set-upstream origin backport/backport-894-to-2.4
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.4

Then, create a pull request where the base branch is 2.4 and the compare/head branch is backport/backport-894-to-2.4.

[AWSHurneyt]

In follow-up revision, refactor this from .filter() to .must().

[eirsep]

THE percolateQueryBuilder sub-query is still being added as a "query.filter()"
we should add that also as a MUSTxAND condition
Otherwise we are liable to having a change there where we add a new subquery with query.filter() which makes the percolate query optional in an implicit OR with the newly added sub-query

@lezzago @getsaurabh02