[Remote segments] Add backpressure in write path on segments lag between local and remote store by ashking94 · Pull Request #7459 · opensearch-project/OpenSearch

ashking94 · 2023-05-08T08:22:53Z

Description

Wires the code written in PR #7227 and #7400 to the write path. With this PR, we are doing the following -

Integrated tracker with RemoteStoreRefreshListener class.
Integrated RemoteRefreshSegmentPressureService with TransportShardBulkAction class. This handles the rejection if the lag is beyond certain fixed and dynamic limits.

Related Issues

Resolves #7424

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2023-05-08T08:30:47Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/14998/
CommitID: 6f1b989
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

ashking94 · 2023-05-08T08:32:18Z

Fixing the build failures.

github-actions · 2023-05-08T09:09:53Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/15000/
CommitID: 048398f

github-actions · 2023-05-08T14:43:58Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.indices.replication.SegmentReplicationIT.testScrollCreatedOnReplica

URL: https://build.ci.opensearch.org/job/gradle-check/15023/
CommitID: ce767f1
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

github-actions · 2023-05-08T15:01:34Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/15027/
CommitID: b84a82b
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-05-08T15:29:51Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/15031/
CommitID: 7335525

codecov-commenter · 2023-05-08T15:34:10Z

Codecov Report

Merging #7459 (077aa99) into main (6d9c82f) will increase coverage by 0.00%.
The diff coverage is 73.75%.

@@            Coverage Diff            @@
##               main    #7459   +/-   ##
=========================================
  Coverage     70.68%   70.68%           
+ Complexity    56024    56018    -6     
=========================================
  Files          4673     4673           
  Lines        265744   265818   +74     
  Branches      39024    39034   +10     
=========================================
+ Hits         187832   187899   +67     
+ Misses        62020    62005   -15     
- Partials      15892    15914   +22

Impacted Files	Coverage Δ
...rg/opensearch/common/settings/ClusterSettings.java	`92.85% <ø> (ø)`
...c/main/java/org/opensearch/index/IndexService.java	`74.04% <ø> (-0.46%)`	⬇️
...x/remote/RemoteRefreshSegmentPressureSettings.java	`94.44% <ø> (-5.56%)`	⬇️
...ain/java/org/opensearch/indices/IndicesModule.java	`96.46% <0.00%> (-1.74%)`	⬇️
...ensearch/action/bulk/TransportShardBulkAction.java	`76.23% <14.28%> (+0.64%)`	⬆️
...ex/remote/RemoteRefreshSegmentPressureService.java	`77.50% <25.00%> (-4.65%)`	⬇️
...ch/indices/cluster/IndicesClusterStateService.java	`68.45% <33.33%> (-5.99%)`	⬇️
...arch/index/remote/RemoteRefreshSegmentTracker.java	`88.65% <63.63%> (+0.23%)`	⬆️
...in/java/org/opensearch/index/shard/IndexShard.java	`70.10% <66.66%> (+0.03%)`	⬆️
...search/index/shard/RemoteStoreRefreshListener.java	`84.95% <83.20%> (+3.39%)`	⬆️
... and 1 more

... and 431 files with indirect coverage changes

github-actions · 2023-05-18T18:50:44Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/15705/
CommitID: 0d87c8c

github-actions · 2023-05-18T19:14:22Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/15708/
CommitID: 5ae6109
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: Ashish Singh <ssashish@amazon.com>

github-actions · 2023-05-18T20:36:22Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.action.admin.cluster.node.tasks.ResourceAwareTasksTests.testTaskResourceTrackingDuringTaskCancellation

URL: https://build.ci.opensearch.org/job/gradle-check/15716/
CommitID: dc378e8
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

gbbafna

Looks good overall . Thanks for making these changes. Few minor comments.

server/src/main/java/org/opensearch/index/shard/RemoteStoreRefreshListener.java

server/src/test/java/org/opensearch/index/shard/RemoteStoreRefreshListenerTests.java

Signed-off-by: Ashish Singh <ssashish@amazon.com>

github-actions · 2023-05-19T09:05:21Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/15741/
CommitID: 4803283
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

server/src/internalClusterTest/java/org/opensearch/remotestore/RemoteStoreBackpressureIT.java

github-actions · 2023-05-19T10:40:03Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/15746/
CommitID: 44666b7
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

ashking94 · 2023-05-19T10:41:03Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌

URL: https://build.ci.opensearch.org/job/gradle-check/15746/

CommitID: 44666b7
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Unrelated IT failure - IndexingIT.testIndexingWithSegRep

github-actions · 2023-05-19T11:19:33Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/15748/
CommitID: 98038ba
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

ashking94 · 2023-05-19T11:24:51Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌

URL: https://build.ci.opensearch.org/job/gradle-check/15748/

CommitID: 98038ba
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

some code came in unintentionally, removing it.

Signed-off-by: Ashish Singh <ssashish@amazon.com>

github-actions · 2023-05-19T12:26:32Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/15752/
CommitID: 6dad1a4
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: Ashish Singh <ssashish@amazon.com>

github-actions · 2023-05-19T17:52:23Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/15776/
CommitID: 077aa99

…een local and remote store (opensearch-project#7459) Signed-off-by: Ashish Singh <ssashish@amazon.com> Signed-off-by: Bharathwaj G <bharath78910@gmail.com>

…een local and remote store (opensearch-project#7459) Signed-off-by: Ashish Singh <ssashish@amazon.com>

…een local and remote store (#7459) (#7720) Signed-off-by: Ashish Singh <ssashish@amazon.com>

…een local and remote store (opensearch-project#7459) Signed-off-by: Ashish Singh <ssashish@amazon.com>

…een local and remote store (opensearch-project#7459) Signed-off-by: Ashish Singh <ssashish@amazon.com> Signed-off-by: Shivansh Arora <hishiv@amazon.com>

ashking94 force-pushed the 7424 branch from 6f1b989 to 048398f Compare May 8, 2023 08:35

ashking94 self-assigned this May 8, 2023

ashking94 added the skip-changelog label May 8, 2023

ashking94 force-pushed the 7424 branch from ce767f1 to b84a82b Compare May 8, 2023 14:40

ashking94 force-pushed the 7424 branch from b84a82b to 7335525 Compare May 8, 2023 14:56

ashking94 marked this pull request as ready for review May 8, 2023 15:48

ashking94 requested review from Bukhtawar, CEHENKLE, Rishikesh1159, anasalkouz, andrross, dblock, gbbafna, kartg, kotwanikunal, mch2, nknize, owaiskazi19, reta, ryanbogan, saratvemulapalli, setiah and shwetathareja as code owners May 8, 2023 15:48

Empty-Commit

dc378e8

Signed-off-by: Ashish Singh <ssashish@amazon.com>

gbbafna reviewed May 19, 2023

View reviewed changes

server/src/main/java/org/opensearch/index/shard/RemoteStoreRefreshListener.java Show resolved Hide resolved

server/src/test/java/org/opensearch/index/shard/RemoteStoreRefreshListenerTests.java Show resolved Hide resolved

Increase the default limit for variance factor

4803283

Signed-off-by: Ashish Singh <ssashish@amazon.com>

sachinpkale approved these changes May 19, 2023

View reviewed changes

server/src/internalClusterTest/java/org/opensearch/remotestore/RemoteStoreBackpressureIT.java Show resolved Hide resolved

gbbafna approved these changes May 19, 2023

View reviewed changes

Fix failing test & incorporate comments

6dad1a4

Signed-off-by: Ashish Singh <ssashish@amazon.com>

ashking94 force-pushed the 7424 branch from 98038ba to 6dad1a4 Compare May 19, 2023 11:56

gbbafna mentioned this pull request May 19, 2023

[Remote Store] Add multipart upload integration for translog and segment files #7119

Merged

6 tasks

Empty-Commit

077aa99

Signed-off-by: Ashish Singh <ssashish@amazon.com>

gbbafna merged commit 63834d9 into opensearch-project:main May 22, 2023

ashking94 added a commit to ashking94/OpenSearch that referenced this pull request May 24, 2023

[Remote segments] Add backpressure in write path on segments lag betw…

6abf472

…een local and remote store (opensearch-project#7459) Signed-off-by: Ashish Singh <ssashish@amazon.com>

ashking94 mentioned this pull request May 24, 2023

[Remote segments] Add backpressure in write path on segments lag between local and remote store (#7459) #7720

Merged

6 tasks

ashking94 added a commit to ashking94/OpenSearch that referenced this pull request May 24, 2023

[Remote segments] Add backpressure in write path on segments lag betw…

e039358

…een local and remote store (opensearch-project#7459) Signed-off-by: Ashish Singh <ssashish@amazon.com>

gbbafna pushed a commit that referenced this pull request May 24, 2023

[Remote segments] Add backpressure in write path on segments lag betw…

06fc7de

…een local and remote store (#7459) (#7720) Signed-off-by: Ashish Singh <ssashish@amazon.com>

suranjay pushed a commit to suranjay/OpenSearch that referenced this pull request May 29, 2023

[Remote segments] Add backpressure in write path on segments lag betw…

d090478

…een local and remote store (opensearch-project#7459) Signed-off-by: Ashish Singh <ssashish@amazon.com>

Bukhtawar mentioned this pull request Jun 27, 2024

Add Ashish Singh as maintainer #14567

Merged

gbbafna mentioned this pull request Aug 8, 2024

Add Varun Bansal as maintainer #15163

Merged

Conversation

ashking94 commented May 8, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issues

Check List

Uh oh!

github-actions bot commented May 8, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

ashking94 commented May 8, 2023

Uh oh!

github-actions bot commented May 8, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions bot commented May 8, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions bot commented May 8, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions bot commented May 8, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

codecov-commenter commented May 8, 2023 • edited by codecov bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions bot commented May 18, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions bot commented May 18, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions bot commented May 18, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

gbbafna left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented May 19, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

Uh oh!

github-actions bot commented May 19, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

ashking94 commented May 19, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions bot commented May 19, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

ashking94 commented May 19, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions bot commented May 19, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions bot commented May 19, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

ashking94 commented May 8, 2023 •

edited

Loading

codecov-commenter commented May 8, 2023 •

edited by codecov bot

Loading