Skip to content

[improve][broker][PIP-195] Merge multiple buckets at once#19927

Merged
coderzc merged 1 commit into
apache:masterfrom
coderzc:improve/bucket_delayed_mege_num
Mar 29, 2023
Merged

[improve][broker][PIP-195] Merge multiple buckets at once#19927
coderzc merged 1 commit into
apache:masterfrom
coderzc:improve/bucket_delayed_mege_num

Conversation

@coderzc

@coderzc coderzc commented Mar 26, 2023

Copy link
Copy Markdown
Member

PIP: #16763

Motivation

Currently, we select two buckets at once to merge, which causes the merge operation to be frequently triggered. We should merge multiple buckets at once to avoid frequent merges.

Modifications

  • Use k-way merge to implement CombinedSegmentDelayedIndexQueue
  • Merge multiple buckets at once, the default select 4 buckets to merge. If the tracker can't find able merged buckets will try to reduce the number of merged buckets.
  • Improve clear code
  • Improve logs

Verifying this change

  • Make sure that the change passes the CI checks.

(Please pick either of the following options)

This change is a trivial rework / code cleanup without any test coverage.

(or)

This change is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

  • Added integration tests for end-to-end deployment with large payloads (10MB)
  • Extended integration test for recovery after broker failure

Documentation

  • doc
  • doc-required
  • doc-not-needed
  • doc-complete

Matching PR in forked repository

PR in forked repository:

@coderzc coderzc self-assigned this Mar 26, 2023
@coderzc coderzc added this to the 3.0.0 milestone Mar 26, 2023
@coderzc coderzc added the type/enhancement The enhancements for the existing features or docs. e.g. reduce memory usage of the delayed messages label Mar 26, 2023
@github-actions github-actions Bot added the doc-not-needed Your PR changes do not impact docs label Mar 26, 2023
@coderzc coderzc added area/broker ready-to-test and removed doc-not-needed Your PR changes do not impact docs labels Mar 26, 2023
@github-actions github-actions Bot added the doc-not-needed Your PR changes do not impact docs label Mar 26, 2023
@coderzc coderzc closed this Mar 26, 2023
@coderzc coderzc reopened this Mar 26, 2023
@coderzc coderzc force-pushed the improve/bucket_delayed_mege_num branch from bee2dba to 9162451 Compare March 26, 2023 06:12
@coderzc coderzc force-pushed the improve/bucket_delayed_mege_num branch from 9162451 to 4109066 Compare March 26, 2023 06:48
@codecov-commenter

codecov-commenter commented Mar 28, 2023

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 0.84746% with 117 lines in your changes missing coverage. Please review.
✅ Project coverage is 61.95%. Comparing base (329e80b) to head (4109066).
⚠️ Report is 2699 commits behind head on master.

Files with missing lines Patch % Lines
...r/delayed/bucket/BucketDelayedDeliveryTracker.java 0.00% 86 Missing ⚠️
...layed/bucket/CombinedSegmentDelayedIndexQueue.java 0.00% 31 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@             Coverage Diff              @@
##             master   #19927      +/-   ##
============================================
+ Coverage     61.50%   61.95%   +0.44%     
- Complexity    25755    26070     +315     
============================================
  Files          1859     1859              
  Lines        136792   136796       +4     
  Branches      15043    15042       -1     
============================================
+ Hits          84129    84747     +618     
+ Misses        44857    44247     -610     
+ Partials       7806     7802       -4     
Flag Coverage Δ
inttests 24.42% <0.84%> (-0.06%) ⬇️
systests 25.11% <0.00%> (+0.09%) ⬆️
unittests 59.27% <0.84%> (+0.62%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
...che/bookkeeper/mledger/impl/ManagedCursorImpl.java 78.34% <100.00%> (+0.29%) ⬆️
...layed/bucket/CombinedSegmentDelayedIndexQueue.java 0.00% <0.00%> (ø)
...r/delayed/bucket/BucketDelayedDeliveryTracker.java 0.00% <0.00%> (ø)

... and 186 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@coderzc coderzc merged commit bbf5273 into apache:master Mar 29, 2023
@coderzc coderzc deleted the improve/bucket_delayed_mege_num branch March 29, 2023 06:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/broker doc-not-needed Your PR changes do not impact docs ready-to-test type/enhancement The enhancements for the existing features or docs. e.g. reduce memory usage of the delayed messages

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants