Update merge policy floor_setting from 2MB to 16MB

[finnroblin]

Description

Sets merge.policy.floor_segment : 16mb for knn indices to create more merges and have fewer segments after indexing. The floor_segment setting in Lucene TieredMergePolicy dictates the "size" of a segment to the merge logic. That is, the merge logic uses max(floor_segment, actualSegmentSize) when calculating merges. The OpenSearch default is 2mb. The Lucene default is 16mb since 10.2. This PR is against k-NN repo and not core because there are likely cases where a 2mb default is smarter for non-vector search 😄.

This blog post by a Lucene maintainer explains and visualizes Lucene merges. The TieredMergePolicy javadoc also describes how the merge works. The upshot is that this setting change will lead to fewer segments after indexing, which will improve search performance.

Benchmarks on the 10M 768d cohere dataset with 3x r6g.4xl instances show 5-7% improvements in query throughput and common-case latency, larger 20+% improvements in tail latency, ~4% decrease in indexing time, but ~27% increase in median merge time. The original merge floor policy results in 212 segments and this new merge floor policy results in 154 segments. The ~27% decrease in segments explains the ~27% increase in merging time.

We can see that these setting listeners worked by querying the _settings endpoint. The response includes "merge":{"policy":{"floor_segment":"16mb"}} for knn indices and policy.floor_segment":"2097152b" for non-knn indices. See the full _settings response for knn index here and the full _settings response for non-knn index here.

Check List

• New functionality includes testing.
• New functionality has been documented.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[finnroblin]

This overridden method hooks into OpenSearch here.

[finnroblin]

Currently running cohere-10M benchmark.

[finnroblin]

Here are benchmark results for 10M 768D dataset:

Setup:

instance types

• 3x r6g.4xl
• 6 primary shards
• 1 replica/shard
• deployed via opensearch-cluster-cdk
• osb client: m5.2xlarge w 8vcpu

benchmark setup

• same workload as lucene 10M but with no force merge
• tasks: delete-target-index,create-target-index,custom-vector-bulk,refresh-target-index,prod-queries
• 1st prod-queries run discarded due to lucene cold start behavior.
• The 2nd prod-queries run with the changed policy was an outlier and also discarded.

results

• Old policy: 212 segments, new policy: 154 segments
• Main takeaways: 5-7% improvements in query throughput and common-case latency, larger 20+% improvements in tail latency, ~27% increase in median merge time, ~4% decrease in indexing time (I don't think the indexing metric includes the merge time).
  

full document

lucene-floor-size-analysis.xlsx

[kotwanikunal]

nit: Use the setting key here org.opensearch.index.TieredMergePolicyProvider.INDEX_MERGE_POLICY_FLOOR_SEGMENT_SETTING

[finnroblin]

Want to call out that OS core just opened a PR for this change: opensearch-project/OpenSearch#17699.

We can merge in this listener + setting update on the k-NN side to see the performance impact on nightly benchmarks, but we should probably revert the change once it's merged into core since it's unnecessary overhead to overwrite an equivalent setting value.

[kotwanikunal]

Closing this out since core merged this in with #17687