Skip to content

Make HASHED_PREFIX as the default path type for remote store and snapshot #18163

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
May 1, 2025

Conversation

ashking94
Copy link
Member

@ashking94 ashking94 commented Apr 30, 2025

Description

  1. The default value for cluster.remote_store.index.path.type has been changed from FIXED to HASHED_PREFIX. This setting determines the path type used for data stored in the remote store. The HASHED_PREFIX option provides better distribution of keys and has been known to scale better with growing data size.

  2. The default value for shard_path_type has been changed from FIXED to HASHED_PREFIX. This setting determines the path type used for shard files in the blob store repository. Similar to the remote store settings, the HASHED_PREFIX option provides better distribution of shard files.

These changes are intended to improve the default performance and reliability of remote store and snapshot functionality in OpenSearch.

Let me know if you would like me to modify or expand on anything in this PR description.

Check List

  • Functionality includes testing.
  • [ ] API changes companion pull request created, if applicable.
  • [ ] Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@ashking94 ashking94 changed the title Change defaults for remote store and snapshot settings Change some defaults for remote store and snapshot settings Apr 30, 2025
@ashking94 ashking94 changed the title Change some defaults for remote store and snapshot settings Change defaults for remote store and snapshot settings Apr 30, 2025
@ashking94 ashking94 marked this pull request as ready for review April 30, 2025 09:20
@ashking94 ashking94 requested a review from a team as a code owner April 30, 2025 09:20
Copy link
Contributor

❌ Gradle check result for cd69552: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Copy link
Contributor

❌ Gradle check result for a304b4c: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Copy link
Contributor

❌ Gradle check result for ed4a9c7: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@ashking94 ashking94 changed the title Change defaults for remote store and snapshot settings Maked HASHED_PREFIX as the default path type for remote store and snapshot May 1, 2025
@ashking94 ashking94 changed the title Maked HASHED_PREFIX as the default path type for remote store and snapshot Make HASHED_PREFIX as the default path type for remote store and snapshot May 1, 2025
Copy link
Contributor

github-actions bot commented May 1, 2025

✅ Gradle check result for a2898c3: SUCCESS

Copy link

codecov bot commented May 1, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 72.55%. Comparing base (a6724d3) to head (a2898c3).
Report is 12 commits behind head on main.

Additional details and impacted files
@@             Coverage Diff              @@
##               main   #18163      +/-   ##
============================================
+ Coverage     72.53%   72.55%   +0.01%     
- Complexity    67207    67232      +25     
============================================
  Files          5476     5476              
  Lines        310436   310438       +2     
  Branches      45121    45121              
============================================
+ Hits         225179   225237      +58     
+ Misses        66894    66889       -5     
+ Partials      18363    18312      -51     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@ashking94 ashking94 merged commit 00abaac into opensearch-project:main May 1, 2025
32 checks passed
@andrross
Copy link
Member

andrross commented May 1, 2025

@ashking94 We disabled using HASHED_PREFIX in #17996 because it was causing test failures. Did we get to the bottom of that? There might still be an underlying bug there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants