Fix flaky test RemoteIndexRecoveryIT.testRerouteRecovery

[sachinpkale]

Description

• IndexRecoveryIT.testRerouteRecovery triggers shard movement and verifies few details while recovery is in progress. It also slows down recovery to make sure that shard is not moved before we assert.
• When we extended RemoteIndexRecoveryIT extends IndexRecoveryIT`, we reduced number of docs to be ingested by 5x (this is to decrease test run time as latency of ingestion is higher with remote store enabled).

• OpenSearch/server/src/internalClusterTest/java/org/opensearch/remotestore/RemoteIndexRecoveryIT.java

  Lines 67 to 70 in c06f53e

  	@Override
  	protected int numDocs() {
  	return randomIntBetween(100, 200);
  	}

• But we did not slowed down recovery by 5x. This leads to flaky behaviour in the test where recovery completes before we check recovery stats.
• To avoid the flakiness, in this PR, we have slowed down recovery by 5X.

Related Issues

• [AUTOCUT] Gradle Check Flaky Test Report for RemoteIndexRecoveryIT #14323

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

❌ Gradle check result for 3a2e2f3: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 3a2e2f3: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

✅ Gradle check result for 3a2e2f3: SUCCESS

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 72.29%. Comparing base (6fb0c1b) to head (3a2e2f3).
Report is 73 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #17228      +/-   ##
============================================
+ Coverage     72.24%   72.29%   +0.05%     
+ Complexity    65704    65660      -44     
============================================
  Files          5318     5318              
  Lines        305674   305674              
  Branches      44349    44349              
============================================
+ Hits         220834   220991     +157     
+ Misses        66769    66521     -248     
- Partials      18071    18162      +91     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.