[BUG] ExecuteMonitor inserting metadata doc during dry run #758

petardz · 2023-01-16T18:36:59Z

Issue #, if available:

Description of changes:

Fixed ExecuteMonitor Action inserting metadata doc during dryrun

CheckList:
[ ] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

lezzago · 2023-01-17T19:29:48Z

alerting/src/main/kotlin/org/opensearch/alerting/transport/TransportExecuteMonitorAction.kt

@@ -130,7 +130,7 @@ class TransportExecuteMonitorAction @Inject constructor(
                                docLevelMonitorQueries.initDocLevelQueryIndex(monitor.dataSources)
                                log.info("Central Percolation index ${ScheduledJob.DOC_LEVEL_QUERIES_INDEX} created")
                            }
-                            val (metadata, _) = MonitorMetadataService.getOrCreateMetadata(monitor)
+                            val (metadata, _) = MonitorMetadataService.getOrCreateMetadata(monitor, skipIndex = true)


This should only be true if the dryrun param is true

even if monitorId is null ?

I think if monitorId is null we definitely can't save metadata (there's no monitor to tie it to).

Dryrun can be true for existing monitor and we take that into account before indexing metadata inside DocumentLevelMonitorRunner here

Oh I missed that part of the logic, good call out.

lezzago · 2023-01-17T19:39:22Z

alerting/src/main/kotlin/org/opensearch/alerting/util/DocLevelMonitorQueries.kt

-                    if (e.message?.contains("Limit of total fields") == true) {
+                    val unwrappedException = ExceptionsHelper.unwrapCause(e) as Exception
+                    log.debug("exception after rollover queryIndex index: $targetQueryIndex exception: ${unwrappedException.message}")
+                    if (unwrappedException.message?.contains("Limit of total fields") == true) {


Will this be changed to false from true ever?

codecov-commenter · 2023-01-17T23:25:06Z

Codecov Report

Merging #758 (95d3be7) into main (8084943) will increase coverage by 0.27%.
The diff coverage is 83.33%.

@@             Coverage Diff              @@
##               main     #758      +/-   ##
============================================
+ Coverage     75.27%   75.55%   +0.27%     
  Complexity      116      116              
============================================
  Files           125      125              
  Lines          6828     6835       +7     
  Branches       1024     1027       +3     
============================================
+ Hits           5140     5164      +24     
+ Misses         1159     1134      -25     
- Partials        529      537       +8

Impacted Files	Coverage Δ
.../org/opensearch/alerting/MonitorMetadataService.kt	`65.38% <66.66%> (+0.30%)`	⬆️
.../opensearch/alerting/DocumentLevelMonitorRunner.kt	`77.81% <100.00%> (+0.23%)`	⬆️
...lerting/transport/TransportExecuteMonitorAction.kt	`82.08% <100.00%> (ø)`
...rch/alerting/opensearchapi/OpenSearchExtensions.kt	`50.00% <0.00%> (-1.62%)`	⬇️
...lin/org/opensearch/alerting/alerts/AlertIndices.kt	`74.27% <0.00%> (+0.41%)`	⬆️
...destinationmigration/DestinationConversionUtils.kt	`70.00% <0.00%> (+1.11%)`	⬆️
...in/org/opensearch/alerting/MonitorRunnerService.kt	`81.63% <0.00%> (+2.04%)`	⬆️
.../kotlin/org/opensearch/alerting/util/IndexUtils.kt	`72.91% <0.00%> (+2.08%)`	⬆️
.../alerting/transport/TransportIndexMonitorAction.kt	`57.89% <0.00%> (+2.96%)`	⬆️
...ing/model/destination/DestinationContextFactory.kt	`78.57% <0.00%> (+3.57%)`	⬆️
... and 1 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

Signed-off-by: Petar Dzepina <[email protected]>

alerting/src/test/kotlin/org/opensearch/alerting/MonitorDataSourcesIT.kt

* execute monitor bugfix Signed-off-by: Petar Dzepina <[email protected]> * added IT Signed-off-by: Petar Dzepina <[email protected]> * fixed created retval when skipIndex=true Signed-off-by: Petar Dzepina <[email protected]> --------- Signed-off-by: Petar Dzepina <[email protected]> (cherry picked from commit ce7094a)

* execute monitor bugfix Signed-off-by: Petar Dzepina <[email protected]> * added IT Signed-off-by: Petar Dzepina <[email protected]> * fixed created retval when skipIndex=true Signed-off-by: Petar Dzepina <[email protected]> --------- Signed-off-by: Petar Dzepina <[email protected]> (cherry picked from commit ce7094a) Co-authored-by: Petar Dzepina <[email protected]>

…h-project#758) * execute monitor bugfix Signed-off-by: Petar Dzepina <[email protected]> * added IT Signed-off-by: Petar Dzepina <[email protected]> * fixed created retval when skipIndex=true Signed-off-by: Petar Dzepina <[email protected]> --------- Signed-off-by: Petar Dzepina <[email protected]>

* Added exception check once the .opendistro-alerting-config index is b… (#650) * Added exception check once the .opendistro-alerting-config index is being created During .opendistro-alerting-config index creation, if ResourceAlreadyExists exception is being raised, the flow will check first if the index is in yellow state and then it will re-try to index monitor Signed-off-by: Stevan Buzejic <[email protected]> * Formating of the file fixed Signed-off-by: Stevan Buzejic <[email protected]> Signed-off-by: Stevan Buzejic <[email protected]> * refactored DeleteMonitor Action to be synchronious (#628) (#630) * refactored DeleteMonitor Action to be synchronious Signed-off-by: Petar Dzepina <[email protected]> Signed-off-by: Petar Dzepina <[email protected]> Co-authored-by: Petar Dzepina <[email protected]> * [Backport 2.x] QueryIndex rollover when field mapping limit is reached (#729) Signed-off-by: Petar Dzepina <[email protected]> * Mappings fix backport 2.x (#730) * Added support for "nested" mappings (#645) * example Signed-off-by: Petar Dzepina <[email protected]> * fixed updating mappings for queryIndex Signed-off-by: Petar Dzepina <[email protected]> Signed-off-by: Petar Dzepina <[email protected]> * mappings traversal bug fix (#669) Signed-off-by: Petar Dzepina <[email protected]> Signed-off-by: Petar Dzepina <[email protected]> * Added unwrapping exception from core; added more debug logs (#728) Signed-off-by: Petar Dzepina <[email protected]> * Add DataSources test for future backports Signed-off-by: Ashish Agrawal <[email protected]> * fix percolator mapping error when having field name 'type' (#726) Signed-off-by: Raj Chakravarthi <[email protected]> * [BUG] ExecuteMonitor inserting metadata doc during dry run (#758) * execute monitor bugfix Signed-off-by: Petar Dzepina <[email protected]> * added IT Signed-off-by: Petar Dzepina <[email protected]> * fixed created retval when skipIndex=true Signed-off-by: Petar Dzepina <[email protected]> --------- Signed-off-by: Petar Dzepina <[email protected]> * fix for ERROR alert state generation in doc-level monitors (#768) Signed-off-by: Subhobrata Dey <[email protected]> * Adjusting max field index setting dynamically for query index (#776) * added adjusting max field index setting dynamicly for query index Signed-off-by: Petar Dzepina <[email protected]> * Multiple indices support in DocLevelMonitorInput (#784) Signed-off-by: Petar Dzepina <[email protected]> * Doc transform 2.x backport (#853) * conflict resovle - backport from main to 2.x Signed-off-by: Petar Dzepina <[email protected]> * fixed module class names Signed-off-by: Petar Dzepina <[email protected]> --------- Signed-off-by: Petar Dzepina <[email protected]> * Update config index schema if needed at the start of each monitor execution (#849) * Update config index schema if needed at the start of each monitor execution Signed-off-by: Ashish Agrawal <[email protected]> * Mappings parsing fix (#851) * fixed mappings parsing when field name named "properties" exists in mappings Signed-off-by: Petar Dzepina <[email protected]> * message typo fix Signed-off-by: Petar Dzepina <[email protected]> --------- Signed-off-by: Petar Dzepina <[email protected]> * [Backport 2.x] Notification security fix (#861) * Notification security fix (#852) * added injecting whole user object in threadContext before calling notification APIs so that backend roles are available to notification plugin Signed-off-by: Petar Dzepina <[email protected]> * compile fix Signed-off-by: Petar Dzepina <[email protected]> * refactored user_info injection to use InjectSecurity Signed-off-by: Petar Dzepina <[email protected]> * ktlint fix Signed-off-by: Petar Dzepina <[email protected]> --------- Signed-off-by: Petar Dzepina <[email protected]> (cherry picked from commit e0b7a5a) * remove unneeded import Signed-off-by: Ashish Agrawal <[email protected]> --------- Signed-off-by: Ashish Agrawal <[email protected]> Co-authored-by: Petar Dzepina <[email protected]> Co-authored-by: Ashish Agrawal <[email protected]> * Fixed a bug that prevented alerts from being generated for doc level monitors that use wildcard characters in index names. (#894) Signed-off-by: AWSHurneyt <[email protected]> * fixed security tests (#484) (#794) * fixed security tests Signed-off-by: Raj Chakravarthi <[email protected]> (cherry picked from commit c51940f) --------- Signed-off-by: Stevan Buzejic <[email protected]> Signed-off-by: Petar Dzepina <[email protected]> Signed-off-by: Ashish Agrawal <[email protected]> Signed-off-by: Raj Chakravarthi <[email protected]> Signed-off-by: Subhobrata Dey <[email protected]> Signed-off-by: AWSHurneyt <[email protected]> Co-authored-by: Stevan Buzejic <[email protected]> Co-authored-by: Surya Sashank Nistala <[email protected]> Co-authored-by: Petar Dzepina <[email protected]> Co-authored-by: RAJ CHAKRAVARTHI <[email protected]> Co-authored-by: Subhobrata Dey <[email protected]> Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com> Co-authored-by: AWSHurneyt <[email protected]> Co-authored-by: RAJ CHAKRAVARTHI <[email protected]>

petardz requested a review from a team January 16, 2023 18:37

petardz changed the title ~~Monitor execute bugfix~~ ExecuteMonitor inserting metadata doc during dry run Jan 17, 2023

petardz changed the title ~~ExecuteMonitor inserting metadata doc during dry run~~ [BUG] ExecuteMonitor inserting metadata doc during dry run Jan 17, 2023

lezzago requested changes Jan 17, 2023

View reviewed changes

petardz added 3 commits January 18, 2023 00:51

execute monitor bugfix

9e92fb5

Signed-off-by: Petar Dzepina <[email protected]>

added IT

e61b19c

Signed-off-by: Petar Dzepina <[email protected]>

fixed created retval when skipIndex=true

95d3be7

Signed-off-by: Petar Dzepina <[email protected]>

petardz force-pushed the alerting-fix branch from 1122500 to 95d3be7 Compare January 17, 2023 23:51

lezzago reviewed Feb 3, 2023

View reviewed changes

alerting/src/test/kotlin/org/opensearch/alerting/MonitorDataSourcesIT.kt Show resolved Hide resolved

lezzago added backport 2.x backport 2.5 labels Feb 3, 2023

lezzago approved these changes Feb 3, 2023

View reviewed changes

AWSHurneyt approved these changes Feb 3, 2023

View reviewed changes

AWSHurneyt merged commit ce7094a into opensearch-project:main Feb 3, 2023

opensearch-trigger-bot bot mentioned this pull request Feb 3, 2023

[Backport 2.x] [BUG] ExecuteMonitor inserting metadata doc during dry run #777

Merged

opensearch-trigger-bot bot mentioned this pull request Feb 3, 2023

[Backport 2.5] [BUG] ExecuteMonitor inserting metadata doc during dry run #778

Merged

AWSHurneyt mentioned this pull request Jun 6, 2023

[Backport 2.5] Manual backports of several PRs #957

Merged

lezzago mentioned this pull request Jun 8, 2023

[Backport 2.3] Manual backports of several PRs #960

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[BUG] ExecuteMonitor inserting metadata doc during dry run #758

[BUG] ExecuteMonitor inserting metadata doc during dry run #758

Uh oh!

petardz commented Jan 16, 2023 •

edited

Loading

Uh oh!

lezzago Jan 17, 2023

Uh oh!

petardz Jan 17, 2023

Uh oh!

petardz Jan 17, 2023 •

edited

Loading

Uh oh!

lezzago Feb 3, 2023

Uh oh!

lezzago Jan 17, 2023

Uh oh!

codecov-commenter commented Jan 17, 2023 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

[BUG] ExecuteMonitor inserting metadata doc during dry run #758

[BUG] ExecuteMonitor inserting metadata doc during dry run #758

Uh oh!

Conversation

petardz commented Jan 16, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lezzago Jan 17, 2023

Choose a reason for hiding this comment

Uh oh!

petardz Jan 17, 2023

Choose a reason for hiding this comment

Uh oh!

petardz Jan 17, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lezzago Feb 3, 2023

Choose a reason for hiding this comment

Uh oh!

lezzago Jan 17, 2023

Choose a reason for hiding this comment

Uh oh!

codecov-commenter commented Jan 17, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

petardz commented Jan 16, 2023 •

edited

Loading

petardz Jan 17, 2023 •

edited

Loading

codecov-commenter commented Jan 17, 2023 •

edited

Loading