[Build Tools] Custom Gradle plugin to leverage java agent

[prudhvigodithi]

Description

To Support phasing off SecurityManager usage in favor of Java Agent a custom Gradle plugin from core should help leverage the java agent.

Coming from opensearch-project/custom-codecs#235 (comment) created a custom java agent plugin that can be applied as required. The plugins can use as apply plugin: 'opensearch.java-agent'. With this we can remove the duplicate code across plugins.

I have tested by adding apply plugin: 'opensearch.java-agent' to job-scheduler and now able to run the tests depend on java agent.

Related Issues

Part of #16634 and #16753

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[prudhvigodithi]

Adding @reta @cwperks @kumargu @andrross @getsaurabh02 to please take a look and provide some feedback.

[github-actions]

❌ Gradle check result for c5a2ed7: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

✅ Gradle check result for 77ba174: SUCCESS

[codecov]

Codecov Report

Attention: Patch coverage is 45.83333% with 13 lines in your changes missing coverage. Please review.

Project coverage is 72.49%. Comparing base (1628152) to head (0c652ee).
Report is 19 commits behind head on main.

Files with missing lines	Patch %	Lines
...in/java/org/opensearch/gradle/agent/JavaAgent.java	45.83%	12 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #17900      +/-   ##
============================================
+ Coverage     72.43%   72.49%   +0.05%     
- Complexity    66789    66835      +46     
============================================
  Files          5449     5452       +3     
  Lines        309085   309170      +85     
  Branches      44979    44989      +10     
============================================
+ Hits         223899   224121     +222     
+ Misses        66906    66702     -204     
- Partials      18280    18347      +67     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[prudhvigodithi]

Following changes can be reverted once we have the changes part of the gradle plugin.

• Fix build due to phasing off SecurityManager usage in favor of Java Agent alerting#1823
• Fix build due to phasing off SecurityManager usage in favor of Java Agent notifications#1013
• Fix build due to phasing off SecurityManager usage in favor of Java Agent k-NN#2657
• Add fixes k-NN#2658
• Fix build due to phasing off SecurityManager usage in favor of Java A… geospatial#735
• Add fixes k-NN#2658
• Fix build due to phasing off SecurityManager usage in favor of Java A… anomaly-detection#1450
• Fix build due to phasing off SecurityManager usage in favor of Java Agent index-management#1404
• Fix build due to phasing off SecurityManager usage in favor of Java Agent skills#550
• Support phasing off SecurityManager usage in favor of Java Agent common-utils#814
• Support phasing off SecurityManager usage in favor of Java Agent cross-cluster-replication#1525
• Support phasing off SecurityManager usage in favor of Java Agent flow-framework#1105
• Support phasing off SecurityManager usage in favor of Java Agent query-insights#296
• Support phasing off SecurityManager usage in favor of Java Agent opensearch-system-templates#67
• Support phasing off SecurityManager usage in favor of Java Agent ml-commons#3724
• Fix build due to phasing off SecurityManager usage in favor of Java Agent observability#1916
• Support phasing off SecurityManager usage in favor of Java Agent opensearch-remote-metadata-sdk#138
• Support phasing off SecurityManager usage in favor of Java Agent skills#553
• Support phasing off SecurityManager usage in favor of Java Agent neural-search#1265
• Use java-agent Gradle plugin to support phasing off SecurityManager usage in favor of Java Agent job-scheduler#762
• Use java-agent Gradle plugin to support phasing off SecurityManager usage in favor of Java Agent ml-commons#3727
• Use java-agent Gradle plugin to support phasing off SecurityManager usage in favor of Java Agent custom-codecs#237
• Use java-agent Gradle plugin to support phasing off SecurityManager usage in favor of Java Agent flow-framework#1108
• https://github.com/opensearch-project/common-utils/pull/815/files
• Using java-agent gradle plugin to phase off Security Manager in favor of Java-agent. opensearch-remote-metadata-sdk#142
• Using java-agent gradle plugin to phase off Security Manager in favor of Java-agent. opensearch-learning-to-rank-base#157
• Using java-agent gradle plugin to phase off Security Manager in favor of Java-agent. opensearch-system-templates#68
• Using java-agent gradle plugin to phase off Security Manager in favor of Java-agent. query-insights#303
• Using java-agent gradle plugin to phase off Security Manager in favor of Java-agent. performance-analyzer#798
• Using java-agent gradle plugin to phase off Security Manager in favor of Java-agent. security-analytics#1505
• Using java-agent gradle plugin to phase off Security Manager in favor of Java-agent. anomaly-detection#1454
• Using java-agent gradle plugin to phase off Security Manager in favor of Java-agent. asynchronous-search#720
• Using java-agent gradle plugin to phase off Security Manager in favor of Java-agent. sql#3551
• Using java-agent gradle plugin to phase off Security Manager in favor of Java-agent. reporting#1086
• Using java-agent gradle plugin to phase off Security Manager in favor of Java-agent. observability#1917
• Using java-agent gradle plugin to phase off Security Manager in favor of Java-agent. geospatial#738

Thanks
@peterzhuamazon

[kumargu]

can they coexist and we revert the above tomorrow? Once we know the change in core works for all plugins?

[peterzhuamazon]

There are more, but for now we can just merge all plugin individual changes and at least bring RC1 up.

Thanks.

[prudhvigodithi]

can they coexist and we revert the above tomorrow? Once we know the change in core works for all plugins?

From the PR's a task prepareAgent was added which core already has one https://github.com/search?q=repo%3Aopensearch-project%2FOpenSearch%20prepareAgent&type=code.

So in this custom plugin I have used prepareJavaAgent, so I assume for plugins they can co-exist. But I would go with reverting the code and just use apply plugin: 'opensearch.java-agent' as required.

[prudhvigodithi]

Just testing one more thing looks like https://github.com/opensearch-project/OpenSearch/blob/main/libs/agent-sm/agent/build.gradle#L14 we already have net.bytebuddy:byte-buddy dependency here we dont need to explicitly add again. Will post my test results shortly

[prudhvigodithi]

Just testing one more thing looks like https://github.com/opensearch-project/OpenSearch/blob/main/libs/agent-sm/agent/build.gradle#L14 we already have net.bytebuddy:byte-buddy dependency here we dont need to explicitly add again. Will post my test results shortly

I can see the tests pass on my local and on a sample test PR opensearch-project/job-scheduler#761.

[github-actions]

❕ Gradle check result for 0c652ee: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.