[merged segment warmer] support local merged segment warmer by guojialiang92 · Pull Request #18255 · opensearch-project/OpenSearch

guojialiang92 · 2025-05-12T05:20:51Z

Description

This PR is based on [17881]'s follow-up work. It implements the core process of local merged segment warmer, and the main modifications are as follows:

During the IndexReaderWarmer#warm, the primary shard will send send a PublishMergedSegmentRequest request containing information about the merged segment (ReplicationSegmentCheckpoint is introduced to represent the information of a merged segment) to all active state replicas.
The replica shard introduces MergedSegmentReplicationTarget to pull segment from the primary shard. It has to reuse most of the capabilities in SegmentReplicationTarget. The IndexReaderWarmer#warm process will not end until all active state replicas have completed pulling merged segment.
Use separate rate limiter and timeout controls for the merged segments pre-copy.
Added relevant UT and IT testing.

There are also some orthogonal work that needs to be supported in the future, summarized as follows:

Repilica shard cleaning up residual merged segments. For example, after the primary shard fail, the pre-copy merged segment of the previous term needs to be cleaned up, or some segments are no longer referenced after continuous merging.
Remote store support.

Related Issues

Resolves #[17528]

Check List

Functionality includes testing.
API changes companion pull request created, if applicable.
Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: guojialiang <guojialiang.2012@bytedance.com>

github-actions · 2025-05-12T05:28:29Z

❌ Gradle check result for aa13d96: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: guojialiang <guojialiang.2012@bytedance.com>

github-actions · 2025-05-12T07:33:10Z

❌ Gradle check result for 1c1f802: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-05-12T08:57:53Z

❌ Gradle check result for 78795e3: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: guojialiang <guojialiang.2012@bytedance.com>

server/src/main/java/org/opensearch/index/engine/LocalMergedSegmentWarmer.java

github-actions · 2025-05-12T10:25:39Z

✅ Gradle check result for 58528e9: SUCCESS

Signed-off-by: guojialiang <guojialiang.2012@bytedance.com>

github-actions · 2025-06-25T16:56:26Z

❌ Gradle check result for a0b4e3a: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

…ged-segment-warmer

github-actions · 2025-06-26T02:38:50Z

❌ Gradle check result for cc5ea0b: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: guojialiang <guojialiang.2012@bytedance.com>

github-actions · 2025-06-26T03:11:08Z

❌ Gradle check result for dedad5e: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: guojialiang <guojialiang.2012@bytedance.com>

github-actions · 2025-06-26T04:18:17Z

❌ Gradle check result for 3224f35: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

ashking94 · 2025-06-26T05:07:43Z

Re-triggered the PR build.

github-actions · 2025-06-26T05:48:23Z

❌ Gradle check result for 3224f35: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

guojialiang92 · 2025-06-26T06:06:32Z

@guojialiang92 Do share once you have have addressed all the comments.

Hi, @ashking94
I have addressed all the comments.
But there's a gradle test that fails frequently, and it doesn't seem to be related to this PR. I haven't reproduced it locally. I'm further troubleshooting.

guojialiang92 · 2025-06-26T06:17:41Z

I think I found the reason why the test fails. It seems that the test is not robust.
If you print the log in IngestService#executePipelinesInBatchRequests, it will throw an NPE because indexRequest is empty.
The problem can be reproduced stably by changing debug to info locally.

I copied some exception logs from gradle test, which also proves this point.

[testExecuteBulkRequestInBatchWithExceptionAndDropInCallback_requestsWithMatchingChildSlots] Test testExecuteBulkRequestInBatchWithExceptionAndDropInCallback_requestsWithMatchingChildSlots(org.opensearch.ingest.IngestServiceTests) started
[2025-06-26T12:22:53,523][INFO ][o.o.i.IngestServiceTests ] [testExecuteBulkRequestInBatchWithExceptionAndDropInCallback_requestsWithMatchingChildSlots] before test
[2025-06-26T12:22:53,533][DEBUG][o.o.i.IngestService      ] [testExecuteBulkRequestInBatchWithExceptionAndDropInCallback_requestsWithMatchingChildSlots] batchSize: 5, batches: 1
[2025-06-26T12:22:53,535][DEBUG][o.o.i.IngestService      ] [testExecuteBulkRequestInBatchWithExceptionAndDropInCallback_requestsWithMatchingChildSlots] failed to execute pipeline [_id] for documents [_index/_id1, _index/_id2, _index/_id2, _index/_id3, _index/_id3]
java.lang.NullPointerException: Cannot invoke "org.opensearch.action.index.IndexRequest.index()" because "indexRequest" is null
	at org.opensearch.ingest.IngestService.lambda$executePipelinesInBatchRequests$1(IngestService.java:1012) ~[main/:?]
	at org.apache.logging.log4j.util.LambdaUtil.get(LambdaUtil.java:76) ~[log4j-api-2.21.0.jar:?]
	at org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:1959) [log4j-api-2.21.0.jar:?]
	at org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:1804) [log4j-api-2.21.0.jar:?]
	at org.apache.logging.log4j.spi.AbstractLogger.debug(AbstractLogger.java:380) [log4j-api-2.21.0.jar:?]
	at org.opensearch.ingest.IngestService.lambda$executePipelinesInBatchRequests$0(IngestService.java:1008) ~[main/:?]
	at org.opensearch.ingest.IngestService.lambda$innerBatchExecute$0(IngestService.java:1431) ~[main/:?]
	at org.opensearch.ingest.Pipeline.lambda$batchExecute$0(Pipeline.java:277) ~[main/:?]
	at org.opensearch.ingest.CompoundProcessor.lambda$innerBatchExecute$0(CompoundProcessor.java:240) ~[main/:?]
	at

ashking94

Looks good, Thanks!

ashking94 · 2025-06-26T08:19:41Z

I have re-triggered the test. We can merge it after it becomes green.

github-actions · 2025-06-26T08:50:09Z

❌ Gradle check result for 3224f35: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

guojialiang92 · 2025-06-26T09:49:04Z

❌ Gradle check result for 3224f35: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Hi, @ashking94

It seems to be the same issue. Can we merge bugfix PR first and then run the gradle test?

[testExecuteBulkRequestInBatchWithExceptionAndDropInCallback_requestsWithMatchingChildSlots] failed to execute pipeline [_id] for documents [_index/_id1, _index/_id2, _index/_id2, _index/_id3, _index/_id3]
java.lang.NullPointerException: Cannot invoke "org.opensearch.action.index.IndexRequest.index()" because "indexRequest" is null
	at org.opensearch.ingest.IngestService.lambda$executePipelinesInBatchRequests$1(IngestService.java:1012) ~[main/:?]
	at org.apache.logging.log4j.util.LambdaUtil.get(LambdaUtil.java:76) ~[log4j-api-2.21.0.jar:?]
	at org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:1959) [log4j-api-2.21.0.jar:?]
	at org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:1804) [log4j-api-2.21.0.jar:?]
	at org.apache.logging.log4j.spi.AbstractLogger.debug(AbstractLogger.java:380) [log4j-api-2.21.0.jar:?]
	at org.opensearch.ingest.IngestService.lambda$executePipelinesInBatchRequests$0(IngestService.java:1008) ~[main/:?]
	at org.opensearch.ingest.IngestService.lambda$innerBatchExecute$0(IngestService.java:1431) ~[main/:?]
	at org.opensearch.ingest.Pipeline.lambda$batchExecute$0(Pipeline.java:277) ~[main/:?]
	at org.opensearch.ingest.CompoundProcessor.lambda$innerBatchExecute$0(CompoundProcessor.java:240) ~[main/:?]
	at org.opensearch.ingest.IngestServiceTests.lambda$testExecuteBulkRequestInBatchWithExceptionAndDropInCallback_requestsWithMatchingChildSlots$1(IngestServiceTests.java:2208) ~[test/:?]

ashking94 · 2025-06-26T09:55:27Z

@guojialiang92 can you rebase with main now?

guojialiang92 · 2025-06-26T10:00:22Z

@guojialiang92 can you rebase with main now?

Yeah

…ged-segment-warmer

github-actions · 2025-06-26T11:27:06Z

✅ Gradle check result for f5111ec: SUCCESS

guojialiang92 · 2025-06-26T11:28:30Z

@ashking94
Thanks for the patient review. All the tests have passed.

…ch-project#18255) * support local merged segment warmer Signed-off-by: guojialiang <guojialiang.2012@bytedance.com> * update log Signed-off-by: guojialiang <guojialiang.2012@bytedance.com> * fix test Signed-off-by: guojialiang <guojialiang.2012@bytedance.com> * add tests Signed-off-by: guojialiang <guojialiang.2012@bytedance.com> * IndexShard support getActiveReplicaNodes Signed-off-by: guojialiang <guojialiang.2012@bytedance.com> * add test Signed-off-by: guojialiang <guojialiang.2012@bytedance.com> * add pre-verification. Signed-off-by: guojialiang <guojialiang.2012@bytedance.com> * extend ReplicationAction instead of ActionRequest Signed-off-by: guojialiang <guojialiang.2012@bytedance.com> * reuse Store#getSegmentMetadataMap Signed-off-by: guojialiang <guojialiang.2012@bytedance.com> * refactor updateReplicationRateLimiter Signed-off-by: guojialiang <guojialiang.2012@bytedance.com> * extract an abstract base class (AbstractSegmentReplicationTarget) from SegmentReplicationTarget Signed-off-by: guojialiang <guojialiang.2012@bytedance.com> * reduce unnecessary exception judgment Signed-off-by: guojialiang <guojialiang.2012@bytedance.com> * fix UT Signed-off-by: guojialiang <guojialiang.2012@bytedance.com> * add PublishMergedSegmentActionTests Signed-off-by: guojialiang <guojialiang.2012@bytedance.com> * gradle spotlessApply Signed-off-by: guojialiang <guojialiang.2012@bytedance.com> * add test Signed-off-by: guojialiang <guojialiang.2012@bytedance.com> * rename ReplicationSegmentCheckpoint to MergeSegmentCheckpoint Signed-off-by: guojialiang <guojialiang.2012@bytedance.com> * add some description Signed-off-by: guojialiang <guojialiang.2012@bytedance.com> * refactor code Signed-off-by: guojialiang <guojialiang.2012@bytedance.com> * extract an abstract base class (AbstractPublishCheckpointAction) from PublishCheckpointAction Signed-off-by: guojialiang <guojialiang.2012@bytedance.com> * fix :server:japicmp Signed-off-by: guojialiang <guojialiang.2012@bytedance.com> * update Signed-off-by: guojialiang <guojialiang.2012@bytedance.com> * update Signed-off-by: guojialiang <guojialiang.2012@bytedance.com> --------- Signed-off-by: guojialiang <guojialiang.2012@bytedance.com>

support local merged segment warmer

aa13d96

Signed-off-by: guojialiang <guojialiang.2012@bytedance.com>

guojialiang92 requested review from a team, Bukhtawar, CEHENKLE, Rishikesh1159, VachaShah, anasalkouz, andrross, ashking94, cwperks, dbwiddis, gbbafna, jed326, kotwanikunal, mch2, msfroh, owaiskazi19, reta, sachinpkale, saratvemulapalli, shwetathareja and sohami as code owners May 12, 2025 05:20

update log

1c1f802

Signed-off-by: guojialiang <guojialiang.2012@bytedance.com>

fix test

58528e9

Signed-off-by: guojialiang <guojialiang.2012@bytedance.com>

guojialiang92 force-pushed the dev/support-local-merged-segment-warmer branch from 78795e3 to 58528e9 Compare May 12, 2025 09:15

shourya035 reviewed May 12, 2025

View reviewed changes

server/src/main/java/org/opensearch/index/engine/LocalMergedSegmentWarmer.java Outdated Show resolved Hide resolved

fix :server:japicmp

a0b4e3a

Signed-off-by: guojialiang <guojialiang.2012@bytedance.com>

guojialiang92 force-pushed the dev/support-local-merged-segment-warmer branch from d2df443 to a0b4e3a Compare June 25, 2025 16:13

Merge remote-tracking branch 'origin/main' into dev/support-local-mer…

cc5ea0b

…ged-segment-warmer

update

dedad5e

Signed-off-by: guojialiang <guojialiang.2012@bytedance.com>

update

3224f35

Signed-off-by: guojialiang <guojialiang.2012@bytedance.com>

guojialiang92 mentioned this pull request Jun 26, 2025

[BUG] Report flaky test org.opensearch.ingest.IngestServiceTests#testExecuteBulkRequestInBatchWithExceptionAndDropInCallback_requestsWithMatchingChildSlots #18622

Closed

ashking94 approved these changes Jun 26, 2025

View reviewed changes

Merge remote-tracking branch 'origin/main' into dev/support-local-mer…

f5111ec

…ged-segment-warmer

ashking94 merged commit 5ced78c into opensearch-project:main Jun 26, 2025
31 checks passed

guojialiang92 mentioned this pull request Jun 26, 2025

[META] Merged segment pre-copy #18625

Open

8 tasks

guojialiang92 mentioned this pull request Jul 10, 2025

[merged segment warmer] support cleanup redundant pending merged segments #18720

Merged

3 tasks

kh3ra mentioned this pull request Jul 21, 2025

[merged segment warmer] support remote merged segment warmer #18683

Merged

3 tasks

opensearch-ci-bot mentioned this pull request Aug 23, 2025

[AUTOCUT] Gradle Check Flaky Test Report for IngestFromKafkaIT #17215

Open

tkykenmt mentioned this pull request Jan 10, 2026

[feature] Remote Store Segment Warming tkykenmt/opensearch-feature-explorer#1120

Closed

3 tasks

Conversation

guojialiang92 commented May 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issues

Check List

Uh oh!

github-actions bot commented May 12, 2025

Uh oh!

github-actions bot commented May 12, 2025

Uh oh!

github-actions bot commented May 12, 2025

Uh oh!

Uh oh!

github-actions bot commented May 12, 2025

Uh oh!

github-actions bot commented Jun 25, 2025

Uh oh!

github-actions bot commented Jun 26, 2025

Uh oh!

github-actions bot commented Jun 26, 2025

Uh oh!

github-actions bot commented Jun 26, 2025

Uh oh!

ashking94 commented Jun 26, 2025

Uh oh!

github-actions bot commented Jun 26, 2025

Uh oh!

guojialiang92 commented Jun 26, 2025

Uh oh!

guojialiang92 commented Jun 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ashking94 left a comment

Choose a reason for hiding this comment

Uh oh!

ashking94 commented Jun 26, 2025

Uh oh!

github-actions bot commented Jun 26, 2025

Uh oh!

guojialiang92 commented Jun 26, 2025

Uh oh!

ashking94 commented Jun 26, 2025

Uh oh!

guojialiang92 commented Jun 26, 2025

Uh oh!

github-actions bot commented Jun 26, 2025

Uh oh!

guojialiang92 commented Jun 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

guojialiang92 commented May 12, 2025 •

edited

Loading

guojialiang92 commented Jun 26, 2025 •

edited

Loading

guojialiang92 commented Jun 26, 2025 •

edited

Loading