Fix ingest pipeline cannot be executed when retry the failed index requests for update_by_query and reindex

[gaobinlong]

Description

This PR fixes the bug that ingest pipeline cannot be executed again when retrying the failed index requests for update_by_query and reindex, update_by_query and reindex are kind of AbstractAsyncBulkByScrollAction, they have retry logic when the index requests failed with 429 status code, but when build a new bulk request for retry, it doesn't honor the global pipeline specified the api request and the isPipelineResolved field was set to true in all of the failed indexRequests, so the pipeline won't be executed again when retrying the bulk request.

In addition, we need to find a way to mark the failure step if the BulkItemResponse, only if the ingest pipeline failed with 429 status code, then the isPipelineResolved field need to be reset, so I add a source field to the Failure class to mark which step has failure during the bulk processing, for other cases like the index has a read_only_allow_delete block which is also retryable, we shouldn't reset the isPipelineResolved field.

Some unit test are added for the code change, but failed to add some yaml tests or IT because pipeline execution failure with 429 status code is not easy to mock.

Related Issues

Resolves #14337.

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

❌ Gradle check result for 949410a: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❕ Gradle check result for 38f6bb5: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov]

Codecov Report

Attention: Patch coverage is 86.36364% with 6 lines in your changes missing coverage. Please review.

Project coverage is 72.61%. Comparing base (54e02a7) to head (0e4401a).
Report is 10 commits behind head on main.

Files with missing lines	Patch %	Lines
...a/org/opensearch/action/bulk/BulkItemResponse.java	82.75%	4 Missing and 1 partial ⚠️
...rc/main/java/org/opensearch/action/bulk/Retry.java	87.50%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #18003      +/-   ##
============================================
+ Coverage     72.47%   72.61%   +0.14%     
- Complexity    67034    67165     +131     
============================================
  Files          5478     5478              
  Lines        310132   310168      +36     
  Branches      45087    45091       +4     
============================================
+ Hits         224775   225238     +463     
+ Misses        66960    66516     -444     
- Partials      18397    18414      +17     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[owaiskazi19]

Looks good with few minor comments

[martin-gaievski]

Looks good to me, thank you

[github-actions]

❌ Gradle check result for 7fa102c: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 0e4401a: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 0e4401a: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

✅ Gradle check result for 0e4401a: SUCCESS

[owaiskazi19]

@andrross can you also take a look here?