[improve][broker][PIP-318] Support not retaining null-key message during topic compaction by coderzc · Pull Request #21578 · apache/pulsar

coderzc · 2023-11-14T08:57:52Z

PIP: #21541

Motivation

Implement #21541

Modifications

Add config topicCompactionRetainNullKey in the broker.conf, the default value is false.
If not retaining null-key message, add an empty single message placeholder to result when rebatching the message.
If the message with null-key, then add a null-key value to the result list of extractIdsAndKeysAndSize and record the delete event.

Verifying this change

Make sure that the change passes the CI checks.

(Please pick either of the following options)

This change is a trivial rework / code cleanup without any test coverage.

(or)

This change is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

Added integration tests for end-to-end deployment with large payloads (10MB)
Extended integration test for recovery after broker failure

Documentation

doc
doc-required
doc-not-needed
doc-complete

Matching PR in forked repository

PR in forked repository:

codecov-commenter · 2023-11-17T07:25:44Z

Codecov Report

Attention: Patch coverage is 87.50000% with 3 lines in your changes missing coverage. Please review.

Project coverage is 73.29%. Comparing base (44abba9) to head (1ca9ce5).
Report is 1376 commits behind head on master.

Files with missing lines	Patch %	Lines
...rg/apache/pulsar/compaction/TwoPhaseCompactor.java	85.71%	1 Missing and 1 partial ⚠️
...g/apache/pulsar/client/impl/RawBatchConverter.java	87.50%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #21578      +/-   ##
============================================
+ Coverage     73.27%   73.29%   +0.02%     
- Complexity    32676    32705      +29     
============================================
  Files          1892     1892              
  Lines        140632   140721      +89     
  Branches      15467    15489      +22     
============================================
+ Hits         103044   103146     +102     
+ Misses        29497    29470      -27     
- Partials       8091     8105      +14

Flag	Coverage Δ
inttests	`24.11% <12.50%> (-0.24%)`	⬇️
systests	`24.68% <12.50%> (-0.29%)`	⬇️
unittests	`72.57% <87.50%> (+0.02%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
...org/apache/pulsar/broker/ServiceConfiguration.java	`99.38% <100.00%> (+<0.01%)`	⬆️
...g/apache/pulsar/client/impl/RawBatchConverter.java	`93.75% <87.50%> (+0.24%)`	⬆️
...rg/apache/pulsar/compaction/TwoPhaseCompactor.java	`73.83% <85.71%> (-4.24%)`	⬇️

... and 99 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

…during topic compaction (apache#21578)

… compaction (#21578) (#21601)

… compaction (apache#21578)

… compaction (#21578) (#21662)

… compaction (apache#21578)

merlimat · 2023-11-14T16:07:08Z

 brokerServiceCompactionPhaseOneLoopTimeInSeconds=30

+# Whether retain null-key message during topic compaction
+topicCompactionRemainNullKey=false


Suggested change

topicCompactionRemainNullKey=false

topicCompactionRetainNullKey=false

@coderzc I just saw this typo in the config key :)

Sorry, I open a new PR to fix it. #21690

… compaction (#21578) (#21665) Co-authored-by: fengyubiao <9947090@qq.com>

… compaction (apache#21578) (apache#21601)

… compaction (#21578)

… compaction (apache#21578)

… compaction (apache#21578) (apache#21601) (cherry picked from commit 6ddf83d)

Support not retaining null-key message during topic compaction

cb1b764

github-actions Bot added the doc-not-needed Your PR changes do not impact docs label Nov 14, 2023

coderzc force-pushed the compaction_not_retain_nullkey branch from 7fbe7b5 to cb1b764 Compare November 14, 2023 08:58

BewareMyPower assigned coderzc Nov 14, 2023

coderzc changed the title ~~[improve][broker] Support not retaining null-key message during topic compaction~~ [improve][broker][PIP-318] Support not retaining null-key message during topic compaction Nov 15, 2023

coderzc added this to the 3.2.0 milestone Nov 15, 2023

mattisonchao reviewed Nov 15, 2023

View reviewed changes

Comment thread pulsar-broker/src/main/java/org/apache/pulsar/client/impl/RawBatchConverter.java

mattisonchao reviewed Nov 15, 2023

View reviewed changes

Comment thread pulsar-broker/src/main/java/org/apache/pulsar/client/impl/RawBatchConverter.java Outdated

Address comment

4ff06cf

coderzc force-pushed the compaction_not_retain_nullkey branch from ad93df2 to 4ff06cf Compare November 16, 2023 02:47

mattisonchao reviewed Nov 16, 2023

View reviewed changes

Comment thread pulsar-broker/src/main/java/org/apache/pulsar/client/impl/RawBatchConverter.java

Technoboy- approved these changes Nov 16, 2023

View reviewed changes

mattisonchao approved these changes Nov 16, 2023

View reviewed changes

coderzc closed this Nov 16, 2023

coderzc reopened this Nov 16, 2023

Fix test

1ca9ce5

coderzc force-pushed the compaction_not_retain_nullkey branch from a1ae5f1 to 1ca9ce5 Compare November 17, 2023 06:06

coderzc merged commit e1318b2 into apache:master Nov 20, 2023

coderzc added type/bug The PR fixed a bug or issue reported a bug area/broker release/3.1.2 release/3.0.3 release/2.11.4 release/2.10.6 labels Nov 21, 2023

coderzc added a commit to coderzc/pulsar that referenced this pull request Nov 21, 2023

[improve][broker][branch-3.1] Support not retaining null-key message …

e327219

…during topic compaction (apache#21578)

coderzc mentioned this pull request Nov 21, 2023

[improve][broker][branch-3.1] Support not retaining null-key message during topic compaction (#21578) #21601

Merged

4 tasks

coderzc added type/enhancement The enhancements for the existing features or docs. e.g. reduce memory usage of the delayed messages type/bug The PR fixed a bug or issue reported a bug and removed type/bug The PR fixed a bug or issue reported a bug labels Nov 21, 2023

Technoboy- pushed a commit that referenced this pull request Nov 22, 2023

[improve][broker] Support not retaining null-key message during topic…

6ddf83d

… compaction (#21578) (#21601)

coderzc added the cherry-picked/branch-3.1 label Nov 22, 2023

coderzc added a commit to coderzc/pulsar that referenced this pull request Dec 4, 2023

[improve][broker] Support not retaining null-key message during topic…

aa321ad

… compaction (apache#21578)

coderzc mentioned this pull request Dec 4, 2023

[improve][broker] Support not retaining null-key message during topic compaction (#21578) #21662

Merged

4 tasks

Technoboy- added the cherry-picked/branch-3.0 label Dec 4, 2023

Technoboy- pushed a commit that referenced this pull request Dec 4, 2023

[improve][broker] Support not retaining null-key message during topic…

744b7af

… compaction (#21578) (#21662)

coderzc added a commit to coderzc/pulsar that referenced this pull request Dec 4, 2023

[improve][broker] Support not retaining null-key message during topic…

1608905

… compaction (apache#21578)

coderzc mentioned this pull request Dec 4, 2023

[improve][broker] Support not retaining null-key message during topic compaction (#21578) #21665

Merged

5 tasks

coderzc added a commit to coderzc/pulsar that referenced this pull request Dec 5, 2023

[improve][broker] Support not retaining null-key message during topic…

b433200

… compaction (apache#21578)

merlimat reviewed Dec 6, 2023

View reviewed changes

coderzc mentioned this pull request Dec 7, 2023

[fix][broker] Fix typo in the config key #21690

Merged

4 tasks

Technoboy- pushed a commit that referenced this pull request Dec 14, 2023

[improve][broker] Support not retaining null-key message during topic…

cd5fa17

… compaction (#21578) (#21665) Co-authored-by: fengyubiao <9947090@qq.com>

coderzc added the cherry-picked/branch-2.11 label Dec 14, 2023

shibd added release/2.11.3 and removed release/2.11.4 labels Dec 20, 2023

nikhil-ctds pushed a commit to datastax/pulsar that referenced this pull request Dec 20, 2023

[improve][broker] Support not retaining null-key message during topic…

192fa78

… compaction (apache#21578) (apache#21601)

srinath-ctds pushed a commit to datastax/pulsar that referenced this pull request Dec 20, 2023

[improve][broker] Support not retaining null-key message during topic…

7b24dd9

… compaction (apache#21578) (apache#21601)

coderzc added a commit that referenced this pull request Jan 4, 2024

[improve][broker] Support not retaining null-key message during topic…

13b108b

… compaction (#21578)

coderzc added the cherry-picked/branch-2.10 label Jan 4, 2024

coderzc added the area/compaction label Jan 20, 2024

nodece pushed a commit to nodece/pulsar that referenced this pull request Feb 23, 2024

[improve][broker] Support not retaining null-key message during topic…

e2ffadb

… compaction (apache#21578)

mukesh-ctds pushed a commit to datastax/pulsar that referenced this pull request Mar 1, 2024

[improve][broker] Support not retaining null-key message during topic…

b8e156b

… compaction (apache#21578) (apache#21601) (cherry picked from commit 6ddf83d)

mukesh-ctds pushed a commit to datastax/pulsar that referenced this pull request Mar 6, 2024

[improve][broker] Support not retaining null-key message during topic…

509eb23

… compaction (apache#21578) (apache#21601) (cherry picked from commit 6ddf83d)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[improve][broker][PIP-318] Support not retaining null-key message during topic compaction#21578

[improve][broker][PIP-318] Support not retaining null-key message during topic compaction#21578
coderzc merged 3 commits into
apache:masterfrom
coderzc:compaction_not_retain_nullkey

coderzc commented Nov 14, 2023 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov-commenter commented Nov 17, 2023 •

edited

Loading

Uh oh!

merlimat Nov 14, 2023

Uh oh!

merlimat Dec 6, 2023

Uh oh!

coderzc Dec 7, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

	topicCompactionRemainNullKey=false
	topicCompactionRetainNullKey=false

Conversation

coderzc commented Nov 14, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modifications

Verifying this change

Documentation

Matching PR in forked repository

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov-commenter commented Nov 17, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

merlimat Nov 14, 2023

Choose a reason for hiding this comment

Uh oh!

merlimat Dec 6, 2023

Choose a reason for hiding this comment

Uh oh!

coderzc Dec 7, 2023

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

coderzc commented Nov 14, 2023 •

edited

Loading

codecov-commenter commented Nov 17, 2023 •

edited

Loading