Optimize read write lock constructs during translog upload to remote store by ashking94 · Pull Request #9636 · opensearch-project/OpenSearch

ashking94 · 2023-08-30T10:36:48Z

Description

Optimize read write lock constructs during translog upload to remote store

Related Issues

Resolves #10013

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2023-08-30T11:19:37Z

Compatibility status:

Checks if related components are compatible with change 5595fc2

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git]

github-actions · 2023-08-30T12:10:01Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/23812/
CommitID: 8c2604a
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-08-30T12:15:15Z

Compatibility status:

Checks if related components are compatible with change 8c2604a

Incompatible components

Incompatible components: [https://github.com/opensearch-project/cross-cluster-replication.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

github-actions · 2023-08-30T12:16:26Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/23805/
CommitID: 66dbef7
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-09-12T07:49:12Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/25371/
CommitID: e48cbd7
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-09-12T10:33:16Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/25374/
CommitID: 03ffacb
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-09-12T11:25:12Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.classMethod

URL: https://build.ci.opensearch.org/job/gradle-check/25375/
CommitID: 3f784e3
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

codecov · 2023-09-12T11:26:24Z

Codecov Report

Merging #9636 (5595fc2) into main (e6dec29) will increase coverage by 0.12%.
Report is 9 commits behind head on main.
The diff coverage is 70.00%.

@@             Coverage Diff              @@
##               main    #9636      +/-   ##
============================================
+ Coverage     71.03%   71.16%   +0.12%     
- Complexity    58071    58139      +68     
============================================
  Files          4824     4824              
  Lines        273918   273928      +10     
  Branches      39918    39921       +3     
============================================
+ Hits         194591   194953     +362     
+ Misses        63017    62602     -415     
- Partials      16310    16373      +63

Files Changed	Coverage Δ
...g/transfer/TranslogCheckpointTransferSnapshot.java	`79.22% <ø> (+1.01%)`	⬆️
...rg/opensearch/index/translog/RemoteFsTranslog.java	`70.56% <68.96%> (-0.52%)`	⬇️
...dex/translog/transfer/TranslogTransferManager.java	`73.51% <100.00%> (+0.50%)`	⬆️

... and 443 files with indirect coverage changes

github-actions · 2023-09-12T15:47:57Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/25393/
CommitID: 00e20ee
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: Ashish Singh <ssashish@amazon.com>

server/src/main/java/org/opensearch/index/translog/RemoteFsTranslog.java

server/src/main/java/org/opensearch/index/translog/transfer/TranslogTransferManager.java

github-actions · 2023-09-18T05:56:50Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/25762/
CommitID: e74b1c1
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: Ashish Singh <ssashish@amazon.com>

github-actions · 2023-09-18T08:44:22Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/25769/
CommitID: 5595fc2
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-09-18T14:52:53Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.smoketest.SmokeTestMultiNodeClientYamlTestSuiteIT.test {yaml=search.aggregation/20_terms/string profiler via global ordinals}
      1 org.opensearch.smoketest.SmokeTestMultiNodeClientYamlTestSuiteIT.test {yaml=pit/10_basic/Delete all}

URL: https://build.ci.opensearch.org/job/gradle-check/25785/
CommitID: 5595fc2
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

…store (#9636) Signed-off-by: Ashish Singh <ssashish@amazon.com> (cherry picked from commit b5cc002) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…store (#9636) (#10102) (cherry picked from commit b5cc002) Signed-off-by: Ashish Singh <ssashish@amazon.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…store (#9636) (#10103) (cherry picked from commit b5cc002) Signed-off-by: Ashish Singh <ssashish@amazon.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…store (opensearch-project#9636) Signed-off-by: Ashish Singh <ssashish@amazon.com>

… remote store (opensearch-project#9636)" This reverts commit b5cc002. Signed-off-by: Ashish Singh <ssashish@amazon.com>

…ore (#10135) * Revert "Optimize read write lock constructs during translog upload to remote store (#9636)" This reverts commit b5cc002. Signed-off-by: Ashish Singh <ssashish@amazon.com> * Downgrade writelock to readlock during translog upload Signed-off-by: Ashish Singh <ssashish@amazon.com> --------- Signed-off-by: Ashish Singh <ssashish@amazon.com>

…ore (#10135) * Revert "Optimize read write lock constructs during translog upload to remote store (#9636)" This reverts commit b5cc002. Signed-off-by: Ashish Singh <ssashish@amazon.com> * Downgrade writelock to readlock during translog upload Signed-off-by: Ashish Singh <ssashish@amazon.com> --------- Signed-off-by: Ashish Singh <ssashish@amazon.com> (cherry picked from commit cbff21d) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…ore (opensearch-project#10135) * Revert "Optimize read write lock constructs during translog upload to remote store (opensearch-project#9636)" This reverts commit b5cc002. Signed-off-by: Ashish Singh <ssashish@amazon.com> * Downgrade writelock to readlock during translog upload Signed-off-by: Ashish Singh <ssashish@amazon.com> --------- Signed-off-by: Ashish Singh <ssashish@amazon.com>

…ore (#10135) (#10187) * Revert "Optimize read write lock constructs during translog upload to remote store (#9636)" This reverts commit b5cc002. * Downgrade writelock to readlock during translog upload --------- Signed-off-by: Ashish Singh <ssashish@amazon.com>

…store (opensearch-project#9636) Signed-off-by: Ashish Singh <ssashish@amazon.com> Signed-off-by: Ivan Brusic <ivan.brusic@flocksafety.com>

…ore (opensearch-project#10135) * Revert "Optimize read write lock constructs during translog upload to remote store (opensearch-project#9636)" This reverts commit b5cc002. Signed-off-by: Ashish Singh <ssashish@amazon.com> * Downgrade writelock to readlock during translog upload Signed-off-by: Ashish Singh <ssashish@amazon.com> --------- Signed-off-by: Ashish Singh <ssashish@amazon.com> Signed-off-by: Ivan Brusic <ivan.brusic@flocksafety.com>

…store (opensearch-project#9636) Signed-off-by: Ashish Singh <ssashish@amazon.com>

…ore (opensearch-project#10135) * Revert "Optimize read write lock constructs during translog upload to remote store (opensearch-project#9636)" This reverts commit b5cc002. Signed-off-by: Ashish Singh <ssashish@amazon.com> * Downgrade writelock to readlock during translog upload Signed-off-by: Ashish Singh <ssashish@amazon.com> --------- Signed-off-by: Ashish Singh <ssashish@amazon.com>

…store (opensearch-project#9636) Signed-off-by: Ashish Singh <ssashish@amazon.com> Signed-off-by: Shivansh Arora <hishiv@amazon.com>

…ore (opensearch-project#10135) * Revert "Optimize read write lock constructs during translog upload to remote store (opensearch-project#9636)" This reverts commit b5cc002. Signed-off-by: Ashish Singh <ssashish@amazon.com> * Downgrade writelock to readlock during translog upload Signed-off-by: Ashish Singh <ssashish@amazon.com> --------- Signed-off-by: Ashish Singh <ssashish@amazon.com> Signed-off-by: Shivansh Arora <hishiv@amazon.com>

ashking94 force-pushed the lockfix-readlock branch from 66dbef7 to 8c2604a Compare August 30, 2023 11:46

ashking94 added the skip-changelog label Aug 30, 2023

ashking94 force-pushed the lockfix-readlock branch from 8c2604a to e48cbd7 Compare September 12, 2023 07:28

ashking94 force-pushed the lockfix-readlock branch from e48cbd7 to 03ffacb Compare September 12, 2023 10:08

ashking94 force-pushed the lockfix-readlock branch from 03ffacb to 3f784e3 Compare September 12, 2023 10:35

ashking94 force-pushed the lockfix-readlock branch from 3f784e3 to 00e20ee Compare September 12, 2023 15:14

ashking94 force-pushed the lockfix-readlock branch from 00e20ee to 2fa2a62 Compare September 13, 2023 06:03

ashking94 changed the title ~~[Draft] Lockfix readlock~~ Optimize read write lock constructs during translog upload to remote store Sep 13, 2023

ashking94 marked this pull request as ready for review September 13, 2023 06:25

ashking94 requested review from Bukhtawar, CEHENKLE, anasalkouz, andrross, dblock, gbbafna, kartg, kotwanikunal, mch2, nknize, reta and setiah as code owners September 13, 2023 06:25

Incorporate PR review comments

e74b1c1

Signed-off-by: Ashish Singh <ssashish@amazon.com>

gbbafna reviewed Sep 18, 2023

View reviewed changes

server/src/main/java/org/opensearch/index/translog/RemoteFsTranslog.java Outdated Show resolved Hide resolved

gbbafna reviewed Sep 18, 2023

View reviewed changes

server/src/main/java/org/opensearch/index/translog/transfer/TranslogTransferManager.java Outdated Show resolved Hide resolved

Incorporate PR review comments

5595fc2

Signed-off-by: Ashish Singh <ssashish@amazon.com>

gbbafna approved these changes Sep 18, 2023

View reviewed changes

gbbafna added the backport 2.x Backport to 2.x branch label Sep 18, 2023

gbbafna merged commit b5cc002 into opensearch-project:main Sep 18, 2023

opensearch-trigger-bot bot mentioned this pull request Sep 18, 2023

[Backport 2.x] Optimize read write lock constructs during translog upload to remote store #10102

Merged

gbbafna added the backport 2.10 Backport to 2.10 branch label Sep 18, 2023

opensearch-trigger-bot bot mentioned this pull request Sep 18, 2023

[Backport 2.10] Optimize read write lock constructs during translog upload to remote store #10103

Merged

sarthakaggarwal97 pushed a commit to sarthakaggarwal97/OpenSearch that referenced this pull request Sep 20, 2023

Optimize read write lock constructs during translog upload to remote …

98e8d78

…store (opensearch-project#9636) Signed-off-by: Ashish Singh <ssashish@amazon.com>

vikasvb90 pushed a commit to vikasvb90/OpenSearch that referenced this pull request Oct 10, 2023

Optimize read write lock constructs during translog upload to remote …

c6ab52f

…store (opensearch-project#9636) Signed-off-by: Ashish Singh <ssashish@amazon.com>

Conversation

ashking94 commented Aug 30, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issues

Check List

Uh oh!

github-actions bot commented Aug 30, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Compatibility status:

Incompatible components

Skipped components

Compatible components

Uh oh!

github-actions bot commented Aug 30, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions bot commented Aug 30, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

Uh oh!

github-actions bot commented Aug 30, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions bot commented Sep 12, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions bot commented Sep 12, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions bot commented Sep 12, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

codecov bot commented Sep 12, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions bot commented Sep 12, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Sep 18, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions bot commented Sep 18, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions bot commented Sep 18, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ashking94 commented Aug 30, 2023 •

edited

Loading

github-actions bot commented Aug 30, 2023 •

edited

Loading

codecov bot commented Sep 12, 2023 •

edited

Loading