Introducing MergedSegmentWarmerFactory to support the extension of IndexWriter.IndexReaderWarmer #17881

guojialiang92 · 2025-04-10T16:23:53Z

Description

In order to reduce the visibility delay of replicas in Segment Replication, we propose the idea of merged segment pre-copy in 17528. After discussion, we believe that it can produce good results in both local segment replication and remote store situations.

This PR is the beginning of supporting merged segments pre-copy. We introduce MergedSegmentWarmerFactory for Local/Remote merged segment pre copy.

Related Issues

Resolves #[17528]

Experiment

Test1: Replica Lag Time

We enable pre-copy in local on-disk segment replication to test the optimization effect.
We used two 16C64GB nodes, nyc_taxis workload, 2 primary shard and 1 replica shard for testing. The test command is as follows：

nohup opensearch-benchmark execute-test --target-hosts [10.204.62.132:9425](http://10.204.62.132:9425/)  --workload=nyc_taxis --pipeline=benchmark-only --workload-params='{"bulk_indexing_clients":10,"bulk_size":5000,"index_settings":{"index.number_of_shards":2,"index.number_of_replicas":1,"index.replication.type":"SEGMENT","index.routing.allocation.total_shards_per_node":2}}' --offline --results-format=csv --results-file=/home/result.csv --kill-running-processes > nohup.out 2>&1 &

We use command _cat/segment_replication to collect indicator current_lag every 5 seconds and compare the replica lag time when closing and opening pre-copy feature.

Result1: segment replication without pre-copy

Result2: segment replication with pre-copy

The X-axis represents logical time, collected every 5 seconds. The Y-axis represents the indicator current_lag collected through command _cat/segment_replication. We can see that the maximum lag time has been reduced from 66s to 2.9s, with more than 20 times optimization.

Test2: CPU util and P99 Read Latency

At the same time, we conducted a write-search test and focused on CPU util and P99 Read Latency. We can see that there is no significant change in query latency and CPU util.

Result1: segment replication without pre-copy

Result2: segment replication with pre-copy

The analysis of the reasons is as follows. CPU-intensive tasks are performed before pre-copy. There will be no significant CPU overhead due to the introduction of pre-copy. At the same time, the time cost of merging in phase SegmentMerger#merge should be much greater than the time spent in phase IndexReaderWarmer#warm. The additional synchronization time introduced by pre-copy should not account for a high proportion in the entire merge process.

Check List

Functionality includes testing.
API changes companion pull request created, if applicable.
Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: guojialiang <[email protected]>

…pre copy Signed-off-by: guojialiang <[email protected]>

github-actions · 2025-04-10T17:14:50Z

❌ Gradle check result for e012362: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

ashking94 · 2025-04-10T18:19:40Z

The replication lag improvement looks promising. There appears to be up to 5-6% higher CPU utilization with pre-copy. Is this reproducible each time, or can this be considered as general variability between multiple runs?

guojialiang92 · 2025-04-11T02:49:48Z

Hi, @ashking94

The replication lag improvement looks promising.

Yeah, the improvementt is very obvious.

There appears to be up to 5-6% higher CPU utilization with pre-copy. Is this reproducible each time, or can this be considered as general variability between multiple runs?

Some explanation is needed here. Because the current value fluctuates, we generally care about the max and avg CPU utilization.

Taking node lf-byte-es-sr-dataplatform-two-data-0 as an example, we can see that when pre-copy is enabled, the maximum CPU utilization is reduced by 4.1% (from 67.3% to 63.2%), and the average CPU utilization is slightly higher by 0.6% (from 54.5% to 55.1%).

At the same time, we repeated the test of enabling pre-copy.

It can be seen that the average CPU utilization of lf-byte-es-sr-dataplatform-two-data-0 has decreased by 2.7% (from 54.5% to 51.8%).

In summary, merged segments pre copy hardly introduces additional CPU overhead.

server/src/main/java/org/opensearch/index/engine/InternalEngine.java

Signed-off-by: guojialiang <[email protected]>

server/src/main/java/org/opensearch/index/engine/InternalEngine.java

ashking94

LGTM, do address the unresolved comments if any.

github-actions · 2025-04-14T16:37:35Z

✅ Gradle check result for a6157f0: SUCCESS

guojialiang92 · 2025-04-15T02:40:37Z

Thanks, @ashking94

LGTM, do address the unresolved comments if any.

All comments have been resolved, looking forward to merging the code.

ashking94 · 2025-04-15T04:56:34Z

@guojialiang92 can you resolve the conflicts?

…gment_warmer # Conflicts: # CHANGELOG.md # server/src/main/java/org/opensearch/common/util/FeatureFlags.java

github-actions · 2025-04-15T06:17:15Z

✅ Gradle check result for fa5481c: SUCCESS

guojialiang92 · 2025-04-15T06:26:36Z

@guojialiang92 can you resolve the conflicts?

@ashking94, It has been resolved.

…rWarmer (opensearch-project#17881) * support merged segment warmer Signed-off-by: guojialiang <[email protected]> * Introduce MergedSegmentWarmerFactory for Local/Remote merged segment pre copy Signed-off-by: guojialiang <[email protected]> * add FeatureFlags.MERGED_SEGMENT_WARMER_EXPERIMENTAL_FLAG Signed-off-by: guojialiang <[email protected]> * add an entry in CHANGELOG Signed-off-by: guojialiang <[email protected]> * add validation Signed-off-by: guojialiang <[email protected]> * modify changelog entry Signed-off-by: guojialiang <[email protected]> * add comment Signed-off-by: guojialiang <[email protected]> * add comment Signed-off-by: guojialiang <[email protected]> --------- Signed-off-by: guojialiang <[email protected]> Signed-off-by: Harsh Kothari <[email protected]>

Bukhtawar · 2025-05-22T13:46:36Z

server/src/main/java/org/opensearch/index/engine/LocalMergedSegmentWarmer.java

+ *
+ * @opensearch.internal
+ */
+public class LocalMergedSegmentWarmer implements IndexWriter.IndexReaderWarmer {


Would have preferred this in a follow-up PR :)

mch2 · 2025-05-29T21:57:54Z

server/src/main/java/org/opensearch/index/engine/LocalMergedSegmentWarmer.java

+ */
+
+/*
+ * Licensed to Elasticsearch under one or more contributor


I'm very late to this pr, but we need to remove the es part of these licenses.

guojialiang92 added 2 commits April 10, 2025 17:33

support merged segment warmer

4ab033c

Signed-off-by: guojialiang <[email protected]>

Introduce MergedSegmentWarmerFactory for Local/Remote merged segment …

e012362

…pre copy Signed-off-by: guojialiang <[email protected]>

shourya035 reviewed Apr 11, 2025

View reviewed changes

server/src/main/java/org/opensearch/index/engine/InternalEngine.java Outdated Show resolved Hide resolved

add comment

a6157f0

Signed-off-by: guojialiang <[email protected]>

guojialiang92 force-pushed the dev/support_merged_segment_warmer branch from 97fa12b to a6157f0 Compare April 14, 2025 15:46

ashking94 reviewed Apr 14, 2025

View reviewed changes

server/src/main/java/org/opensearch/index/engine/InternalEngine.java Show resolved Hide resolved

ashking94 approved these changes Apr 14, 2025

View reviewed changes

Merge remote-tracking branch 'origin/main' into dev/support_merged_se…

fa5481c

…gment_warmer # Conflicts: # CHANGELOG.md # server/src/main/java/org/opensearch/common/util/FeatureFlags.java

ashking94 approved these changes Apr 15, 2025

View reviewed changes

ashking94 merged commit 80cb033 into opensearch-project:main Apr 15, 2025
31 checks passed

opensearch-ci-bot mentioned this pull request Apr 15, 2025

[AUTOCUT] Gradle Check Flaky Test Report for MultiMatchQueryBuilderTests #17934

Closed

opensearch-ci-bot mentioned this pull request Apr 28, 2025

[AUTOCUT] Gradle Check Flaky Test Report for IndexShardTests #14321

Open

opensearch-ci-bot mentioned this pull request May 6, 2025

[AUTOCUT] Gradle Check Flaky Test Report for IndexServiceTests #14407

Open

BrewTestBot mentioned this pull request May 6, 2025

opensearch 3.0.0 Homebrew/homebrew-core#222665

Merged

opensearch-ci-bot mentioned this pull request May 8, 2025

[AUTOCUT] Gradle Check Flaky Test Report for SearchServiceTests #18238

Closed

guojialiang92 mentioned this pull request May 12, 2025

[merged segment warmer] support local merged segment warmer #18255

Merged

3 tasks

Bukhtawar reviewed May 22, 2025

View reviewed changes

mch2 reviewed May 29, 2025

View reviewed changes

guojialiang92 mentioned this pull request Jun 26, 2025

[META] Merged segment pre-copy #18625

Open

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Introducing MergedSegmentWarmerFactory to support the extension of IndexWriter.IndexReaderWarmer #17881

Introducing MergedSegmentWarmerFactory to support the extension of IndexWriter.IndexReaderWarmer #17881

Uh oh!

guojialiang92 commented Apr 10, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Apr 10, 2025

Uh oh!

ashking94 commented Apr 10, 2025

Uh oh!

guojialiang92 commented Apr 11, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

ashking94 left a comment

Uh oh!

github-actions bot commented Apr 14, 2025

Uh oh!

guojialiang92 commented Apr 15, 2025

Uh oh!

ashking94 commented Apr 15, 2025

Uh oh!

github-actions bot commented Apr 15, 2025

Uh oh!

guojialiang92 commented Apr 15, 2025

Uh oh!

Uh oh!

Bukhtawar May 22, 2025

Uh oh!

mch2 May 29, 2025

Uh oh!

Uh oh!

Introducing MergedSegmentWarmerFactory to support the extension of IndexWriter.IndexReaderWarmer #17881

Introducing MergedSegmentWarmerFactory to support the extension of IndexWriter.IndexReaderWarmer #17881

Uh oh!

Conversation

guojialiang92 commented Apr 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issues

Experiment

Test1: Replica Lag Time

Test2: CPU util and P99 Read Latency

Check List

Uh oh!

github-actions bot commented Apr 10, 2025

Uh oh!

ashking94 commented Apr 10, 2025

Uh oh!

guojialiang92 commented Apr 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ashking94 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Apr 14, 2025

Uh oh!

guojialiang92 commented Apr 15, 2025

Uh oh!

ashking94 commented Apr 15, 2025

Uh oh!

github-actions bot commented Apr 15, 2025

Uh oh!

guojialiang92 commented Apr 15, 2025

Uh oh!

Uh oh!

Bukhtawar May 22, 2025

Choose a reason for hiding this comment

Uh oh!

mch2 May 29, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

guojialiang92 commented Apr 10, 2025 •

edited

Loading

guojialiang92 commented Apr 11, 2025 •

edited

Loading