[merged segment warmer] support remote merged segment warmer#18683
[merged segment warmer] support remote merged segment warmer#18683ashking94 merged 28 commits intoopensearch-project:mainfrom
Conversation
|
❌ Gradle check result for d7f0e5b: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
d7f0e5b to
dae97b6
Compare
|
❌ Gradle check result for dae97b6: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
dae97b6 to
628d4c8
Compare
|
❌ Gradle check result for 628d4c8: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
|
❌ Gradle check result for e4aae33: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
e4aae33 to
48d3c50
Compare
|
❌ Gradle check result for 48d3c50: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
0457da0 to
21d8c13
Compare
|
❕ Gradle check result for 21d8c13: UNSTABLE Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure. |
Signed-off-by: kh3ra <33233993+kh3ra@users.noreply.github.com>
|
❌ Gradle check result for 162b3eb: null Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
|
Regarding codecov: Achieved 72.43% vs expected 72.77% The slight coverage gap is due to challenges in writing automated tests mergedSegmentWarmer, as it interacts with several final/private classes in Lucene's codebase which limits our testing capabilities at this point. |
|
@kh3ra Lets add a cluster setting to control enablement of remote merged segment warmer along with the experimental setting. If we want to disable it later, it would come in handy. |
…rch-project#18683) * Changes to support upload and download of merge segments using the IndexWriter.IndexWarmer.warm() flow Signed-off-by: kh3ra <adityakh3ra@gmail.com> * Unit tests Signed-off-by: kh3ra <adityakh3ra@gmail.com> * Fixing build issues Signed-off-by: kh3ra <adityakh3ra@gmail.com> * Fixing build issues - forbiddenAPIs/spotlessApply Signed-off-by: kh3ra <adityakh3ra@gmail.com> * Upload merge segments in low priority, minor fixes Signed-off-by: kh3ra <adityakh3ra@gmail.com> * Addressing review comments + rebase main Signed-off-by: kh3ra <adityakh3ra@gmail.com> * Test fixes + javadocs Signed-off-by: kh3ra <adityakh3ra@gmail.com> * 1. Bug fix to RemoteDirectory.DownloadRateLimiterProvider 2. Fixes to tests Signed-off-by: kh3ra <adityakh3ra@gmail.com> * Bug fix to replica updates to ActiveMergesRegistry Signed-off-by: kh3ra <adityakh3ra@gmail.com> * Addressing review comments - round 2 Signed-off-by: kh3ra <adityakh3ra@gmail.com> * new tests + test fixes + minor bug fixes Signed-off-by: kh3ra <adityakh3ra@gmail.com> * Bug fix Signed-off-by: kh3ra <adityakh3ra@gmail.com> * Fixes to RemoteStorePublishMergedSegmentActionTests Signed-off-by: kh3ra <adityakh3ra@gmail.com> * Adding integration tests Signed-off-by: kh3ra <adityakh3ra@gmail.com> * Tracking stats for merged segment warmer Signed-off-by: kh3ra <adityakh3ra@gmail.com> * Revert "Tracking stats for merged segment warmer" This reverts commit 1476e22. Signed-off-by: kh3ra <adityakh3ra@gmail.com> * Addressing review comments for tests Signed-off-by: kh3ra <adityakh3ra@gmail.com> * Addressing review comments Signed-off-by: kh3ra <adityakh3ra@gmail.com> * Rebasing Signed-off-by: kh3ra <adityakh3ra@gmail.com> * spotlessApply Signed-off-by: kh3ra <adityakh3ra@gmail.com> * test fix Signed-off-by: kh3ra <adityakh3ra@gmail.com> * Empty commit Signed-off-by: kh3ra <adityakh3ra@gmail.com> * Adding tests, enhancing logs Signed-off-by: kh3ra <adityakh3ra@gmail.com> * Adding MergedSegmentWarmerFactory tests + enhancing existing tests Signed-off-by: kh3ra <adityakh3ra@gmail.com> * Applying spotless Signed-off-by: kh3ra <adityakh3ra@gmail.com> * Added test for Timeout case for RemoteStoreReplicationSource Signed-off-by: kh3ra <adityakh3ra@gmail.com> * Restored RemoteSegmentStoreDirectory PublicAPI + added changelog Signed-off-by: kh3ra <adityakh3ra@gmail.com> --------- Signed-off-by: kh3ra <adityakh3ra@gmail.com> Signed-off-by: kh3ra <33233993+kh3ra@users.noreply.github.com>
Description
This PR is based on PR18255's follow-up work. It implements the core process of warming merged segments in remote-store-enabled domains.
During the
IndexReaderWarmer#warm,RemoteStorePublishMergedSegmentRequestrequest containing information about the newly created merged segment.IndexReaderWarmer#warmcompletes and the merge resumes.Notes -
IndexReaderWarmer#warmflow are ignored and the merged segments are then expected to get replicated in the regular replication flow.Related Issues
Resolves #[Issue number to be closed when this PR is merged]
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.