Batch translog sync/upload per x ms for remote-backed indexes by ashking94 · Pull Request #5854 · opensearch-project/OpenSearch

ashking94 · 2023-01-13T13:11:24Z

Description

Translog sync takes care of local fsync and translog upload onto remote store. Currently, there is implicit buffering that happens as the remote store upload is a time consuming operation. However, every upload adds extra cost of network interaction along with the actual file upload. If we can buffer for a pareto-optimal duration, then we can save on the additional network interaction costs and overall achieve lower latencies and higher indexing throughput in comparison to non-buffered approach. There is also a delay optimisation in place that make sure that if the upload took considerable time, then schedule the next run with a decreased interval maintaining the overall buffer interval in check.

Credits for some of the code - Ashwin Pankaj, Laxman Muttineni

Issues Resolved

This solves #5692

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2023-01-13T13:36:25Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/9381/
CommitID: 6a2e359
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-01-13T17:31:43Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/9386/
CommitID: 3f9cc1d
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-01-13T20:22:08Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.indices.replication.SegmentReplicationIT.testCancellation

URL: https://build.ci.opensearch.org/job/gradle-check/9396/
CommitID: b0d55c0
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

github-actions · 2023-01-16T09:48:34Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.indices.replication.SegmentReplicationRelocationIT.testRelocateWhileContinuouslyIndexingAndWaitingForRefresh
      1 org.opensearch.cluster.allocation.AwarenessAllocationIT.testThreeZoneOneReplicaWithForceZoneValueAndLoadAwareness

URL: https://build.ci.opensearch.org/job/gradle-check/9424/
CommitID: 07561b9
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

github-actions · 2023-01-16T11:03:51Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/9425/
CommitID: 776d2a2

codecov-commenter · 2023-01-16T11:05:48Z

Codecov Report

❌ Patch coverage is 75.60976% with 20 lines in your changes missing coverage. Please review.
✅ Project coverage is 70.72%. Comparing base (715ff72) to head (105cbc5).
⚠️ Report is 4290 commits behind head on main.

Files with missing lines	Patch %	Lines
...mmon/util/concurrent/BufferedAsyncIOProcessor.java	68.96%	9 Missing ⚠️
...earch/common/util/concurrent/AsyncIOProcessor.java	66.66%	4 Missing ⚠️
...in/java/org/opensearch/index/shard/IndexShard.java	78.94%	4 Missing ⚠️
...org/opensearch/cluster/metadata/IndexMetadata.java	76.92%	1 Missing and 2 partials ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #5854      +/-   ##
============================================
- Coverage     70.75%   70.72%   -0.04%     
+ Complexity    58720    58704      -16     
============================================
  Files          4771     4772       +1     
  Lines        280818   280887      +69     
  Branches      40568    40572       +4     
============================================
- Hits         198704   198663      -41     
- Misses        65824    65860      +36     
- Partials      16290    16364      +74

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

github-actions · 2023-01-16T11:48:49Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/9429/
CommitID: 6833b36
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-01-18T07:28:29Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.indices.replication.SegmentReplicationIT.testDeleteOperations

URL: https://build.ci.opensearch.org/job/gradle-check/9521/
CommitID: 772addd
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

github-actions · 2023-01-19T08:27:13Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/9581/
CommitID: ad03065
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-01-19T08:33:23Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/9582/
CommitID: e1be6dd
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-01-19T10:18:35Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/9585/
CommitID: e70ce81
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-01-19T13:03:10Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/9591/
CommitID: e660f46
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-01-19T16:19:46Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/9594/
CommitID: 32a73eb
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-01-20T16:43:47Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.repositories.azure.AzureBlobStoreRepositoryTests.testWriteRead
      1 org.opensearch.repositories.azure.AzureBlobStoreRepositoryTests.testSnapshotWithLargeSegmentFiles
      1 org.opensearch.repositories.azure.AzureBlobStoreRepositoryTests.testSnapshotAndRestore
      1 org.opensearch.repositories.azure.AzureBlobStoreRepositoryTests.testReadNonExistingPath
      1 org.opensearch.repositories.azure.AzureBlobStoreRepositoryTests.testMultipleSnapshotAndRollback
      1 org.opensearch.repositories.azure.AzureBlobStoreRepositoryTests.testList
      1 org.opensearch.repositories.azure.AzureBlobStoreRepositoryTests.testIndicesDeletedFromRepository
      1 org.opensearch.repositories.azure.AzureBlobStoreRepositoryTests.testContainerCreationAndDeletion
      1 org.opensearch.indices.replication.SegmentReplicationRelocationIT.testDeleteOperations
      1 org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT.testIndexCreateBlockIsRemovedWhenAnyNodesNotExceedHighWatermark

URL: https://build.ci.opensearch.org/job/gradle-check/9646/
CommitID: 415a1cf
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

Signed-off-by: Ashish Singh <ssashish@amazon.com>

Co-authored-by: Ashwin Pankaj <appankaj@amazon.com> Co-authored-by: Laxman Muttineni <muttil@amazon.com> Signed-off-by: Ashish Singh <ssashish@amazon.com>

Signed-off-by: Ashish Singh <ssashish@amazon.com>

github-actions · 2023-01-25T08:57:25Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT.testIndexCreateBlockWithAReadOnlyBlock
      1 org.opensearch.action.admin.cluster.tasks.PendingTasksBlocksIT.testPendingTasksWithClusterNotRecoveredBlock

URL: https://build.ci.opensearch.org/job/gradle-check/9879/
CommitID: d455c21
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

andrross

Couple minor comments, otherwise looks good.

server/src/main/java/org/opensearch/index/IndexSettings.java

server/src/main/java/org/opensearch/common/util/concurrent/BufferedAsyncIOProcessor.java

Signed-off-by: Ashish Singh <ssashish@amazon.com>

github-actions · 2023-01-28T18:50:51Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/10113/
CommitID: 105cbc5

server/src/main/java/org/opensearch/common/util/concurrent/BufferedAsyncIOProcessor.java

opensearch-trigger-bot · 2023-01-29T04:36:35Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-5854-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 af566e156fefdba192343ba8f7fce84f17d2a07a
# Push it to GitHub
git push --set-upstream origin backport/backport-5854-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-5854-to-2.x.

opensearch-trigger-bot · 2023-01-30T08:10:55Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-5854-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 af566e156fefdba192343ba8f7fce84f17d2a07a
# Push it to GitHub
git push --set-upstream origin backport/backport-5854-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-5854-to-2.x.

…arch-project#5854) * Batch translog upload per x ms to allow high index throughput Signed-off-by: Ashish Singh <ssashish@amazon.com> Co-authored-by: Ashwin Pankaj <appankaj@amazon.com> Co-authored-by: Laxman Muttineni <muttil@amazon.com> Signed-off-by: Ashish Singh <ssashish@amazon.com>

…indexes (#5854) (#6066) * Batch translog sync/upload per x ms for remote-backed indexes (#5854) Signed-off-by: Ashish Singh <ssashish@amazon.com> Co-authored-by: Ashwin Pankaj <appankaj@amazon.com> Co-authored-by: Laxman Muttineni <muttil@amazon.com>

ashking94 self-assigned this Jan 13, 2023

ashking94 added Storage:Durability Issues and PRs related to the durability framework Performance This is for any performance related enhancements or bugs v2.6.0 'Issues and PRs related to version v2.6.0' distributed framework skip-changelog labels Jan 13, 2023

ashking94 force-pushed the 5692 branch from 6a2e359 to 3f9cc1d Compare January 13, 2023 17:03

ashking94 force-pushed the 5692 branch from 3f9cc1d to b0d55c0 Compare January 13, 2023 19:50

ashking94 changed the title ~~Batch translog upload per x ms to allow high index throughput~~ Batch translog upload per x ms for remote-backed indexes to allow high index throughput Jan 18, 2023

ashking94 changed the title ~~Batch translog upload per x ms for remote-backed indexes to allow high index throughput~~ Batch translog upload per x ms for remote-backed indexes Jan 18, 2023

ashking94 force-pushed the 5692 branch from 6833b36 to 772addd Compare January 18, 2023 06:52

ashking94 force-pushed the 5692 branch from ad03065 to e1be6dd Compare January 19, 2023 08:18

ashking94 force-pushed the 5692 branch from e1be6dd to e70ce81 Compare January 19, 2023 09:47

ashking94 force-pushed the 5692 branch from e70ce81 to e660f46 Compare January 19, 2023 12:34

ashking94 force-pushed the 5692 branch from e660f46 to 32a73eb Compare January 19, 2023 15:50

ashking94 force-pushed the 5692 branch from 32a73eb to 415a1cf Compare January 20, 2023 16:10

ashking94 and others added 8 commits January 25, 2023 13:54

Make buffer_interval setting Final

8a14c89

Signed-off-by: Ashish Singh <ssashish@amazon.com>

Incorporate PR review feedback

e352565

Signed-off-by: Ashish Singh <ssashish@amazon.com>

Incorporate PR review feedback

1f1ed7d

Signed-off-by: Ashish Singh <ssashish@amazon.com>

Incorporate PR review feedback

70558bd

Signed-off-by: Ashish Singh <ssashish@amazon.com>

Incorporate PR review feedback

3a5f960

Co-authored-by: Ashwin Pankaj <appankaj@amazon.com> Co-authored-by: Laxman Muttineni <muttil@amazon.com> Signed-off-by: Ashish Singh <ssashish@amazon.com>

Incorporate PR review feedback

9dd5d88

Signed-off-by: Ashish Singh <ssashish@amazon.com>

Incorporate PR review feedback

2ddda27

Signed-off-by: Ashish Singh <ssashish@amazon.com>

Handle scheduling failures

d455c21

Signed-off-by: Ashish Singh <ssashish@amazon.com>

ashking94 force-pushed the 5692 branch from 0c9105b to d455c21 Compare January 25, 2023 08:28

sachinpkale approved these changes Jan 25, 2023

View reviewed changes

andrross reviewed Jan 25, 2023

View reviewed changes

server/src/main/java/org/opensearch/index/IndexSettings.java Outdated Show resolved Hide resolved

server/src/main/java/org/opensearch/common/util/concurrent/BufferedAsyncIOProcessor.java Outdated Show resolved Hide resolved

Incorporate PR review feedback

105cbc5

Signed-off-by: Ashish Singh <ssashish@amazon.com>

gbbafna approved these changes Jan 29, 2023

View reviewed changes

server/src/main/java/org/opensearch/common/util/concurrent/BufferedAsyncIOProcessor.java Show resolved Hide resolved

gbbafna merged commit af566e1 into opensearch-project:main Jan 29, 2023

gbbafna added the backport 2.x Backport to 2.x branch label Jan 29, 2023

ashking94 mentioned this pull request Jan 30, 2023

Refactor buffer interval code for remote-backed indexes #6063

Merged

6 tasks

ashking94 added backport 2.x Backport to 2.x branch and removed backport 2.x Backport to 2.x branch labels Jan 30, 2023

ashking94 mentioned this pull request Jan 30, 2023

[Backport 2.x] Batch translog sync/upload per x ms for remote-backed indexes (#5854) #6066

Merged

6 tasks

sachinpkale mentioned this pull request Feb 4, 2023

[BUG] RemoteStoreIT.testRemoteTranslogRestore flaky test failure #6188

Closed

sachinpkale mentioned this pull request Feb 21, 2023

[Remote Translog] Add missing changelog entries #6408

Closed

6 tasks

ashking94 mentioned this pull request Mar 8, 2024

Optimised prefix pattern per shard for remote store data and metadata files for higher throughput #12567

Closed

Bukhtawar mentioned this pull request Jun 27, 2024

Add Ashish Singh as maintainer #14567

Merged

Conversation

ashking94 commented Jan 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Issues Resolved

Check List

Uh oh!

github-actions bot commented Jan 13, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions bot commented Jan 13, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions bot commented Jan 13, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions bot commented Jan 16, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions bot commented Jan 16, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

codecov-commenter commented Jan 16, 2023 • edited by codecov bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions bot commented Jan 16, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions bot commented Jan 18, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions bot commented Jan 19, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions bot commented Jan 19, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions bot commented Jan 19, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions bot commented Jan 19, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions bot commented Jan 19, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions bot commented Jan 20, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions bot commented Jan 25, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

andrross left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Jan 28, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

Uh oh!

opensearch-trigger-bot bot commented Jan 29, 2023

Uh oh!

opensearch-trigger-bot bot commented Jan 30, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

ashking94 commented Jan 13, 2023 •

edited

Loading

codecov-commenter commented Jan 16, 2023 •

edited by codecov bot

Loading