Skip to content

Correlate findings in batches #1539

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 15 commits into
base: main
Choose a base branch
from
Open

Conversation

toepkerd
Copy link
Contributor

@toepkerd toepkerd commented Jun 12, 2025

Description

Correlations now correlates findings in batches synchronously instead of starting a task for each individual finding. Also performs some prechecks before even correlating in the first place:

  1. is auto correlations setting enabled
  2. does correlation rules index exist
  3. is correlation rules index not empty

For Reviewers

The only file with meaningful changes is TransportCorrelateFindingAction.java, below is a guide to direct your attention to the most important changes:

  • Lines 153-181: prechecks before running correlations at all
  • Lines 303-313: synchronous and timeboxed correlations of findings
  • Lines 558-574: changes to onFailure and onOperation (correlation's version of onSuccess) functionality to support new batch correlations

Related Issues

#1537

Check List

  • New functionality includes testing.
  • New functionality has been documented.
  • [N/A] API changes companion pull request created.
  • Commits are signed per the DCO using --signoff.
  • [N/A] Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

build.gradle Outdated
@@ -210,7 +210,7 @@ dependencies {
implementation 'com.jayway.jsonpath:json-path:2.9.0'
implementation 'net.minidev:json-smart:2.5.2'
implementation 'net.minidev:accessors-smart:2.5.2'
compileOnly "com.google.guava:guava:32.1.3-jre"
implementation "com.google.guava:guava:32.1.3-jre"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do we need this?

pranu2502 and others added 8 commits June 17, 2025 11:17
… of Java-agent. (opensearch-project#1505)

* Revert "Fix build due to phasing off SecurityManager usage in favor of Java Agent (opensearch-project#1504)"

This reverts commit d6ddd5b.

Signed-off-by: Pranav Reddy <[email protected]>

* Using java-agent gradle plugin to phase off Security Manager in favor of Java-agent.

Signed-off-by: Pranav Reddy <[email protected]>

---------

Signed-off-by: Pranav Reddy <[email protected]>
Signed-off-by: Dennis Toepker <[email protected]>
Signed-off-by: vikhy-aws <[email protected]>
Signed-off-by: Dennis Toepker <[email protected]>
…1520)

(cherry picked from commit e4fc510)

Signed-off-by: Peter Zhu <[email protected]>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Signed-off-by: Dennis Toepker <[email protected]>
Signed-off-by: opensearch-ci-bot <[email protected]>
Co-authored-by: opensearch-ci-bot <[email protected]>
Co-authored-by: Peter Zhu <[email protected]>
Signed-off-by: Dennis Toepker <[email protected]>
* added release notes for 3.0

Signed-off-by: Riya Saxena <[email protected]>

* added release notes for 3.0

Signed-off-by: Riya Saxena <[email protected]>

* Update opensearch-security-analytics.release-notes-3.0.0.0.md

Signed-off-by: Peter Zhu <[email protected]>

---------

Signed-off-by: Riya Saxena <[email protected]>
Signed-off-by: Peter Zhu <[email protected]>
Co-authored-by: Peter Zhu <[email protected]>
Signed-off-by: Dennis Toepker <[email protected]>
…ect#1530)

* Switch guava deps from compileOnly to implementation

Signed-off-by: Craig Perkins <[email protected]>

* Use AccessController

Signed-off-by: Craig Perkins <[email protected]>

* Use getResource

Signed-off-by: Craig Perkins <[email protected]>

* Use getResourceAsStream

Signed-off-by: Craig Perkins <[email protected]>

* Try removing lead forward slash

Signed-off-by: Craig Perkins <[email protected]>

---------

Signed-off-by: Craig Perkins <[email protected]>
Signed-off-by: Dennis Toepker <[email protected]>
Signed-off-by: Dennis Toepker <[email protected]>
@@ -116,6 +125,7 @@ public TransportCorrelateFindingAction(TransportService transportService,
this.xContentRegistry = xContentRegistry;
this.detectorIndices = detectorIndices;
this.correlationIndices = correlationIndices;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

update this name

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants