Add new index and cluster level settings to limit the total primary shards per node and per index

[pandeydivyansh1803]

Description

For remote store backed cluster, Segment Replication is used as the replication strategy. With segment replication, segments are created only on primary shard and these segments are copied to the replica shards. As segment creation is CPU intensive, we have observed CPU skew between nodes of the same cluster where primary shards are not balanced.

The earlier attempts to rebalance primary shards across nodes (#6422, #12250) are definitely helping to reduce the skew but they work on the best effort basis and don’t add any constraint.

Implement new setting in OpenSearch:
index.routing.allocation.total_primary_shards_per_node: An index-level setting to limit primary shards per node for a specific index. Store this limit (indexTotalPrimaryShardsPerNodeLimit) in index metadata, similar to indexTotalShardsPerNodeLimit.
cluster.routing.allocation.total_primary_shards_per_node: A cluster-level setting to limit the total primary shards on a node.

These settings will enhance control over primary shard distribution, improving cluster balance and performance management.
The existing ShardsLimitAllocationDecider class already contains the necessary infrastructure and logic to evaluate shard allocation constraints. It has access to the current cluster state, routing information, and methods to check shard counts per node. Given this existing functionality, we propose implementing the two new primary shard limit settings within this class. This approach leverages the current decision-making framework, ensuring consistency with existing allocation rules and minimizing code duplication. By extending the ShardsLimitAllocationDecider, we can efficiently integrate the new primary shard limit checks into the existing allocation decision process.

Related Issues

Resolves #17293

Check List

• [✔️] Functionality includes testing.
• API changes companion pull request created, if applicable.
• [✔️] Public documentation issue/PR created, [https://github.com/[DOC] Add documentation for index.routing.allocation.total_primary_shards_per_node, index.routing.allocation.total_shards_per_node, cluster.routing.allocation.total_primary_shards_per_node and cluster.routing.allocation.total_shards_per_node documentation-website#9301)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

❌ Gradle check result for 721865e: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 920f71a: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

✅ Gradle check result for ebb6a2b: SUCCESS

[linuxpi]

Please resolve conflicts in CHANGELOG-3.0.md and ensure gradle check is passing

[github-actions]

✅ Gradle check result for 1eb6f20: SUCCESS

[github-actions]

❕ Gradle check result for 28def93: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[linuxpi]

@pandeydivyansh1803 Changes LGTM. Please add some tests to cover the cases where user is not able to set the new index/cluster settings for non remote store cluster step.

[github-actions]

❌ Gradle check result for 36d29c8: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

✅ Gradle check result for 36d29c8: SUCCESS

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-17295-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 bc209ee6bacbb1027dcd7ba28d56b6ceb96f4fe0
# Push it to GitHub
git push --set-upstream origin backport/backport-17295-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-17295-to-2.x.

[linuxpi]

@pandeydivyansh1803 can you manually raise the backport PR?