Skip to content

Implement parallel shard refresh behind cluster settings #17782

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 4 commits into from
Apr 20, 2025
Merged

Implement parallel shard refresh behind cluster settings #17782

merged 4 commits into from
Apr 20, 2025
+342 −14

Conversation

ashking94
Copy link
Member

@ashking94 ashking94 commented Apr 3, 2025 •
edited
Loading

Description

This PR implements parallel shard-level refresh capability that can be enabled/disabled via cluster settings. Key changes include:

  • Introduces new cluster setting cluster.index.refresh.shard_level.enabled (default: false) to control refresh behavior
  • When enabled, refresh tasks run at individual shard level instead of index level
  • Maintains backward compatibility by defaulting to index-level refresh when setting is disabled
  • Supports dynamic updates to refresh intervals and refresh modes

Implementation Details

  • Added new AsyncShardRefreshTask class to handle shard-level refresh operations
  • Modified IndexService and IndexShard to manage refresh task lifecycle
  • Added synchronization mechanisms to safely handle transitions between index and shard level refresh modes
  • Implemented proper cleanup of refresh tasks during mode transitions
  • Added comprehensive test coverage for the new functionality

Benefits

  • Enables more granular refresh control at shard level
  • Allows for better resource utilization through parallel refresh operations
  • Maintains existing refresh behavior by default for backward compatibility

The feature is marked as experimental with @ExperimentalApi annotation.

Related Issues

Meta issue: #17776

Check List

  • Functionality includes testing.
  • [ ] API changes companion pull request created, if applicable.
  • Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@ashking94 ashking94 force-pushed the shard-level-refresh branch from b5886d8 to d814114 Compare April 3, 2025 19:05
@ashking94 ashking94 mentioned this pull request Apr 3, 2025
Open
2 tasks
@ashking94 ashking94 force-pushed the shard-level-refresh branch from d814114 to 74c406f Compare April 3, 2025 19:09
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Apr 3, 2025

❕ Gradle check result for 74c406f: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

@codecov Codecov
Copy link

codecov bot commented Apr 3, 2025 •
edited
Loading

Codecov Report

Attention: Patch coverage is 81.05263% with 18 lines in your changes missing coverage. Please review.

Project coverage is 72.54%. Comparing base (cbaddd3) to head (11e7831).
Report is 13 commits behind head on main.

Files with missing lines Patch % Lines
...in/java/org/opensearch/index/shard/IndexShard.java 68.96% 2 Missing and 7 partials ⚠️
...c/main/java/org/opensearch/index/IndexService.java 84.31% 1 Missing and 7 partials ⚠️
...in/java/org/opensearch/indices/IndicesService.java 92.85% 0 Missing and 1 partial ⚠️
Additional details and impacted files 
@@             Coverage Diff              @@
##               main   #17782      +/-   ##
============================================
+ Coverage     72.51%   72.54%   +0.02%     
- Complexity    67108    67128      +20     
============================================
  Files          5475     5478       +3     
  Lines        309916   310125     +209     
  Branches      45060    45080      +20     
============================================
+ Hits         224725   224966     +241     
+ Misses        66895    66820      -75     
- Partials      18296    18339      +43

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@ashking94 ashking94 marked this pull request as ready for review April 4, 2025 04:04
This was referenced Apr 14, 2025
Open
Open
@ashking94 ashking94 force-pushed the shard-level-refresh branch from d1943ab to eb1a065 Compare April 15, 2025 08:35
@github-actions GitHub Actions
Copy link
Contributor

❌ Gradle check result for eb1a065: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@github-actions GitHub Actions
Copy link
Contributor

❌ Gradle check result for 4349e25: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@github-actions GitHub Actions
Copy link
Contributor

❌ Gradle check result for 4349e25: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@github-actions GitHub Actions
Copy link
Contributor

❌ Gradle check result for 4349e25: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

This was referenced Apr 15, 2025
Open
Open
@github-actions GitHub Actions
Copy link
Contributor

❌ Gradle check result for 4349e25: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@github-actions GitHub Actions
Copy link
Contributor

❌ Gradle check result for 4349e25: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@opensearch-ci-bot opensearch-ci-bot mentioned this pull request Apr 15, 2025
Open
@github-actions GitHub Actions
Copy link
Contributor

❌ Gradle check result for 4349e25: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@github-actions GitHub Actions
Copy link
Contributor

❌ Gradle check result for 4349e25: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@ashking94 ashking94 force-pushed the shard-level-refresh branch from 4349e25 to c428af3 Compare April 16, 2025 16:42
@github-actions GitHub Actions
Copy link
Contributor

❌ Gradle check result for c428af3: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@opensearch-ci-bot opensearch-ci-bot mentioned this pull request Apr 16, 2025
Open
@ashking94 ashking94 force-pushed the shard-level-refresh branch from c428af3 to 11e7831 Compare April 19, 2025 09:00
@github-actions GitHub Actions
Copy link
Contributor

❌ Gradle check result for 11e7831: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@github-actions GitHub Actions
Copy link
Contributor

❌ Gradle check result for 11e7831: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@github-actions GitHub Actions
Copy link
Contributor

✅ Gradle check result for 11e7831: SUCCESS

@sachinpkale sachinpkale merged commit 96481cc into opensearch-project:main Apr 20, 2025
31 checks passed
Bukhtawar
Bukhtawar reviewed Apr 22, 2025
View reviewed changes
server/src/main/java/org/opensearch/index/shard/IndexShard.java
Comment on lines +5568 to +5569
refreshTask.close();
refreshTask = null;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ideally the close should de-reference the refreshTask

x-INFiN1TY-x pushed a commit to x-INFiN1TY-x/OpenSearch_Local that referenced this pull request Apr 24, 2025
@ashking94 @tqranjan
Implement parallel shard refresh behind cluster settings (opensearch-…
fd22ca2 
…project#17782)

* Implement parallel shard refresh behind cluster settings

Signed-off-by: Ashish Singh <[email protected]>

* Incorporate PR review comments

Signed-off-by: Ashish Singh <[email protected]>

* Incorporate PR review comments

Signed-off-by: Ashish Singh <[email protected]>

* Fix compilation failure

Signed-off-by: Ashish Singh <[email protected]>

---------

Signed-off-by: Ashish Singh <[email protected]>
Signed-off-by: Tanishq Ranjan <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Bukhtawar Bukhtawar Bukhtawar left review comments

@sachinpkale sachinpkale sachinpkale approved these changes

@shourya035 shourya035 shourya035 approved these changes

@anasalkouz anasalkouz Awaiting requested review from anasalkouz anasalkouz is a code owner

@andrross andrross Awaiting requested review from andrross andrross is a code owner

@CEHENKLE CEHENKLE Awaiting requested review from CEHENKLE CEHENKLE is a code owner

@cwperks cwperks Awaiting requested review from cwperks cwperks is a code owner

@dblock dblock Awaiting requested review from dblock

@dbwiddis dbwiddis Awaiting requested review from dbwiddis dbwiddis is a code owner

@gbbafna gbbafna Awaiting requested review from gbbafna gbbafna is a code owner

@jed326 jed326 Awaiting requested review from jed326 jed326 is a code owner

@kotwanikunal kotwanikunal Awaiting requested review from kotwanikunal kotwanikunal is a code owner

@mch2 mch2 Awaiting requested review from mch2 mch2 is a code owner

@msfroh msfroh Awaiting requested review from msfroh msfroh is a code owner

@nknize nknize Awaiting requested review from nknize

@owaiskazi19 owaiskazi19 Awaiting requested review from owaiskazi19 owaiskazi19 is a code owner

@reta reta Awaiting requested review from reta reta is a code owner

@Rishikesh1159 Rishikesh1159 Awaiting requested review from Rishikesh1159 Rishikesh1159 is a code owner

@saratvemulapalli saratvemulapalli Awaiting requested review from saratvemulapalli saratvemulapalli is a code owner

@shwetathareja shwetathareja Awaiting requested review from shwetathareja shwetathareja is a code owner

@sohami sohami Awaiting requested review from sohami sohami is a code owner

@VachaShah VachaShah Awaiting requested review from VachaShah VachaShah is a code owner

@bugmakerrrrrr bugmakerrrrrr Awaiting requested review from bugmakerrrrrr

@jainankitk jainankitk Awaiting requested review from jainankitk

@linuxpi linuxpi Awaiting requested review from linuxpi

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@ashking94 @sachinpkale @Bukhtawar @shourya035