[Segment Replication] Add PIT/Scroll compatibility with Segment Replication by mch2 · Pull Request #6644 · opensearch-project/OpenSearch

mch2 · 2023-03-13T02:46:28Z

Description

This change makes updates to make PIT/Scroll queries compatible with Segment Replication. It does this by refcounting files when a new reader is created, and discarding those files after a reader is closed.

Issues Resolved

#6519

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2023-03-13T03:27:13Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/12327/
CommitID: ccd5694
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-03-13T05:32:54Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/12332/
CommitID: b32f94d

codecov-commenter · 2023-03-13T05:34:02Z

Codecov Report

❌ Patch coverage is 84.44444% with 7 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (main@ad823b6). Learn more about missing BASE report.
⚠️ Report is 4095 commits behind head on main.

Files with missing lines	Patch %	Lines
...rc/main/java/org/opensearch/index/store/Store.java	72.72%	0 Missing and 3 partials ⚠️
.../opensearch/index/engine/NRTReplicationEngine.java	80.00%	2 Missing ⚠️
...org/opensearch/index/store/ReplicaFileTracker.java	87.50%	0 Missing and 2 partials ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #6644   +/-   ##
=======================================
  Coverage        ?   70.75%           
  Complexity      ?    59246           
=======================================
  Files           ?     4810           
  Lines           ?   283497           
  Branches        ?    40873           
=======================================
  Hits            ?   200585           
  Misses          ?    66462           
  Partials        ?    16450

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

github-actions · 2023-03-13T05:40:01Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.indices.replication.SegmentReplicationIT.testPitCreatedOnReplica

URL: https://build.ci.opensearch.org/job/gradle-check/12333/
CommitID: 3742c1b
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

This change makes updates to make PIT/Scroll queries compatibile with Segment Replication. It does this by refcounting files when a new reader is created, and discarding those files after a reader is closed. Signed-off-by: Marc Handalian <handalm@amazon.com>

Signed-off-by: Marc Handalian <handalm@amazon.com>

github-actions · 2023-03-15T21:58:33Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/12500/
CommitID: eb32b8b
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-03-16T23:54:17Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/12563/
CommitID: 8822b81
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

…of current store state. Signed-off-by: Marc Handalian <handalm@amazon.com>

github-actions · 2023-03-17T00:27:27Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/12568/
CommitID: 7a60f4e

github-actions · 2023-03-17T00:39:41Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/12571/
CommitID: 939ff3a

dreamer-89 · 2023-03-20T22:12:36Z

Closing this PR in favour of #6765 (author is on leave)

CC @anasalkouz

…cation #6644 (#6765) * Segment Replication - PIT/Scroll compatibility. This change makes updates to make PIT/Scroll queries compatibile with Segment Replication. It does this by refcounting files when a new reader is created, and discarding those files after a reader is closed. Signed-off-by: Marc Handalian <handalm@amazon.com> * Fix broken test. Signed-off-by: Marc Handalian <handalm@amazon.com> * Fix test bug with PIT where snapshotted segments are queried instead of current store state. Signed-off-by: Marc Handalian <handalm@amazon.com> * Address review comments and prevent temp file deletion during reader close Signed-off-by: Suraj Singh <surajrider@gmail.com> * Fix precommit failure Signed-off-by: Suraj Singh <surajrider@gmail.com> * Use last committed segment infos reference from replication engine Signed-off-by: Suraj Singh <surajrider@gmail.com> * Clean up and prevent incref on segment info file copied from primary Signed-off-by: Suraj Singh <surajrider@gmail.com> * Fix failing test Signed-off-by: Suraj Singh <surajrider@gmail.com> --------- Signed-off-by: Marc Handalian <handalm@amazon.com> Signed-off-by: Suraj Singh <surajrider@gmail.com> Co-authored-by: Marc Handalian <handalm@amazon.com>

…cation opensearch-project#6644 (opensearch-project#6765) * Segment Replication - PIT/Scroll compatibility. This change makes updates to make PIT/Scroll queries compatibile with Segment Replication. It does this by refcounting files when a new reader is created, and discarding those files after a reader is closed. Signed-off-by: Marc Handalian <handalm@amazon.com> * Fix broken test. Signed-off-by: Marc Handalian <handalm@amazon.com> * Fix test bug with PIT where snapshotted segments are queried instead of current store state. Signed-off-by: Marc Handalian <handalm@amazon.com> * Address review comments and prevent temp file deletion during reader close Signed-off-by: Suraj Singh <surajrider@gmail.com> * Fix precommit failure Signed-off-by: Suraj Singh <surajrider@gmail.com> * Use last committed segment infos reference from replication engine Signed-off-by: Suraj Singh <surajrider@gmail.com> * Clean up and prevent incref on segment info file copied from primary Signed-off-by: Suraj Singh <surajrider@gmail.com> * Fix failing test Signed-off-by: Suraj Singh <surajrider@gmail.com> --------- Signed-off-by: Marc Handalian <handalm@amazon.com> Signed-off-by: Suraj Singh <surajrider@gmail.com> Co-authored-by: Marc Handalian <handalm@amazon.com>

…cation (#7010) * [Segment Replication] Add PIT/Scroll compatibility with Segment Replication #6644 (#6765) * Segment Replication - PIT/Scroll compatibility. This change makes updates to make PIT/Scroll queries compatibile with Segment Replication. It does this by refcounting files when a new reader is created, and discarding those files after a reader is closed. Signed-off-by: Marc Handalian <handalm@amazon.com> * Fix broken test. Signed-off-by: Marc Handalian <handalm@amazon.com> * Fix test bug with PIT where snapshotted segments are queried instead of current store state. Signed-off-by: Marc Handalian <handalm@amazon.com> * Address review comments and prevent temp file deletion during reader close Signed-off-by: Suraj Singh <surajrider@gmail.com> * Fix precommit failure Signed-off-by: Suraj Singh <surajrider@gmail.com> * Use last committed segment infos reference from replication engine Signed-off-by: Suraj Singh <surajrider@gmail.com> * Clean up and prevent incref on segment info file copied from primary Signed-off-by: Suraj Singh <surajrider@gmail.com> * Fix failing test Signed-off-by: Suraj Singh <surajrider@gmail.com> --------- Signed-off-by: Marc Handalian <handalm@amazon.com> Signed-off-by: Suraj Singh <surajrider@gmail.com> Co-authored-by: Marc Handalian <handalm@amazon.com> * Add param definition causing precommit failure Signed-off-by: Suraj Singh <surajrider@gmail.com> * Remove unnecessary override annotation Signed-off-by: Suraj Singh <surajrider@gmail.com> --------- Signed-off-by: Marc Handalian <handalm@amazon.com> Signed-off-by: Suraj Singh <surajrider@gmail.com> Co-authored-by: Marc Handalian <handalm@amazon.com>

This comment was marked as outdated.

Sign in to view

mch2 mentioned this pull request Mar 13, 2023

[Segment replication] Point in time search comaptibility #6519

Closed

mch2 force-pushed the segrep-pit branch 2 times, most recently from 26477a8 to ccd5694 Compare March 13, 2023 03:03

This comment was marked as outdated.

Sign in to view

mch2 force-pushed the segrep-pit branch from b32f94d to 3742c1b Compare March 13, 2023 05:08

mch2 added 2 commits March 15, 2023 14:30

Fix broken test.

eb32b8b

Signed-off-by: Marc Handalian <handalm@amazon.com>

mch2 force-pushed the segrep-pit branch from 3742c1b to eb32b8b Compare March 15, 2023 21:32

mch2 force-pushed the segrep-pit branch from 8822b81 to 7a60f4e Compare March 16, 2023 23:54

Fix test bug with PIT where snapshotted segments are queried instead …

939ff3a

…of current store state. Signed-off-by: Marc Handalian <handalm@amazon.com>

mch2 force-pushed the segrep-pit branch from 7a60f4e to 939ff3a Compare March 17, 2023 00:10

mch2 marked this pull request as ready for review March 17, 2023 00:17

mch2 requested review from Bukhtawar, CEHENKLE, anasalkouz, andrross, dblock, gbbafna, kartg, kotwanikunal, nknize, reta and setiah as code owners March 17, 2023 00:17

mch2 requested review from Rishikesh1159, VachaShah, dreamer-89, owaiskazi19, ryanbogan, saratvemulapalli, shwetathareja, tlfeng and xuezhou25 as code owners March 17, 2023 00:17

dreamer-89 closed this Mar 20, 2023

dreamer-89 mentioned this pull request Mar 20, 2023

[Segment Replication] Add PIT/Scroll compatibility with Segment Replication #6644 #6765

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Segment Replication] Add PIT/Scroll compatibility with Segment Replication#6644

[Segment Replication] Add PIT/Scroll compatibility with Segment Replication#6644
mch2 wants to merge 3 commits intoopensearch-project:mainfrom
mch2:segrep-pit

mch2 commented Mar 13, 2023

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

github-actions Bot commented Mar 13, 2023

Uh oh!

github-actions Bot commented Mar 13, 2023

Uh oh!

codecov-commenter commented Mar 13, 2023 •

edited by codecov Bot

Loading

Uh oh!

github-actions Bot commented Mar 13, 2023

Uh oh!

github-actions Bot commented Mar 15, 2023

Uh oh!

github-actions Bot commented Mar 16, 2023

Uh oh!

github-actions Bot commented Mar 17, 2023

Uh oh!

github-actions Bot commented Mar 17, 2023

Uh oh!

dreamer-89 commented Mar 20, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

mch2 commented Mar 13, 2023

Description

Issues Resolved

Check List

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

github-actions Bot commented Mar 13, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions Bot commented Mar 13, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

codecov-commenter commented Mar 13, 2023 • edited by codecov Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions Bot commented Mar 13, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions Bot commented Mar 15, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions Bot commented Mar 16, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions Bot commented Mar 17, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions Bot commented Mar 17, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

dreamer-89 commented Mar 20, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

codecov-commenter commented Mar 13, 2023 •

edited by codecov Bot

Loading