[fix][broker] Resolve the issue of frequent updates in message expiration deletion rate #24190

zjxxzjwang · 2025-04-21T04:25:27Z

Motivation

Modifications

At present, the message expiration deletion rate is updated in the getmessageExpiryRate() method, and this rate is updated every time the topic status is obtained. This results in the statistical analysis of message expiration and deletion rates not having a fixed periodicity, making it difficult to represent the true message expiration rate.

Documentation

Only update the rate in the updateRate() method and only retrieve the value in the getmessageExpiryRate() method

doc
doc-required
doc-not-needed
doc-complete

Matching PR in forked repository

PR in forked repository:

…iration deletion rate

github-actions · 2025-04-21T04:25:57Z

@zjxxzjwang Please add the following content to your PR description and select a checkbox:

- [ ] `doc` <!-- Your PR contains doc changes -->
- [ ] `doc-required` <!-- Your PR changes impact docs and you will update later -->
- [ ] `doc-not-needed` <!-- Your PR changes do not impact docs -->
- [ ] `doc-complete` <!-- Docs have been already added -->

pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java

nodece

LGTM

jiazhai · 2025-04-28T03:11:31Z

This is similar to PR #19759, but not sure how it was reverted

jiazhai · 2025-04-28T03:22:29Z

Get the reverting information: #24073

jiazhai · 2025-04-28T03:26:57Z

@zjxxzjwang Would you please help add a unit test for this change following PR #19759. Seems this part of code is easy to be broken, it is better to protect it.

zjxxzjwang · 2025-04-30T08:23:11Z

@zjxxzjwang Would you please help add a unit test for this change following PR #19759. Seems this part of code is easy to be broken, it is better to protect it.

Good suggestion, I have also considered it before. But it seems that there are no good test examples to determine if this has not been broken.

zjxxzjwang · 2025-05-08T02:38:03Z

@lhotari Hello, can you review it

nodece · 2025-05-08T03:17:42Z

@zjxxzjwang Please rebase the master branch, and then I will merge this PR after CI passes.

HQebupt · 2025-05-22T04:21:27Z

/pulsarbot rerun-failure-checks

…iration deletion rate

…xpiration-rate' into master-Optimization-of-message-expiration-rate

zjxxzjwang · 2025-05-22T14:40:59Z

@zjxxzjwang Please rebase the master branch, and then I will merge this PR after CI passes.

hello , I have rebased the master branch and added test cases. Can you help me merge it

codecov-commenter · 2025-05-23T02:56:14Z

Codecov Report

Attention: Patch coverage is 79.88879% with 217 lines in your changes missing coverage. Please review.

Project coverage is 74.27%. Comparing base (bbc6224) to head (b94f50f).
Report is 1116 commits behind head on master.

Files with missing lines	Patch %	Lines
...rg/apache/pulsar/broker/service/HealthChecker.java	73.71%	37 Missing and 4 partials ⚠️
.../java/org/apache/pulsar/client/impl/ClientCnx.java	10.52%	22 Missing and 12 partials ⚠️
...che/bookkeeper/mledger/impl/ManagedLedgerImpl.java	51.72%	19 Missing and 9 partials ⚠️
...va/org/apache/pulsar/client/impl/ProducerImpl.java	78.82%	12 Missing and 6 partials ⚠️
...sar/broker/service/persistent/PersistentTopic.java	79.72%	12 Missing and 3 partials ⚠️
...n/java/org/apache/pulsar/broker/PulsarService.java	87.25%	8 Missing and 5 partials ⚠️
...g/apache/pulsar/PulsarClusterMetadataTeardown.java	0.00%	8 Missing ⚠️
.../service/SystemTopicBasedTopicPoliciesService.java	78.12%	6 Missing and 1 partial ⚠️
...a/org/apache/pulsar/broker/service/Dispatcher.java	0.00%	6 Missing ⚠️
.../org/apache/pulsar/broker/admin/AdminResource.java	90.38%	1 Missing and 4 partials ⚠️
... and 19 more

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #24190      +/-   ##
============================================
+ Coverage     73.57%   74.27%   +0.69%     
+ Complexity    32624    32265     -359     
============================================
  Files          1877     1866      -11     
  Lines        139502   145125    +5623     
  Branches      15299    16593    +1294     
============================================
+ Hits         102638   107788    +5150     
+ Misses        28908    28826      -82     
- Partials       7956     8511     +555

Flag	Coverage Δ
inttests	`26.96% <37.25%> (+2.37%)`	⬆️
systests	`23.30% <20.01%> (-1.02%)`	⬇️
unittests	`73.75% <79.70%> (+0.90%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
...a/org/apache/bookkeeper/mledger/ManagedLedger.java	`0.00% <ø> (ø)`
...rg/apache/bookkeeper/mledger/impl/AckSetState.java	`100.00% <ø> (ø)`
...che/bookkeeper/mledger/impl/ManagedCursorImpl.java	`78.68% <100.00%> (-0.62%)`	⬇️
...thentication/oidc/OpenIDProviderMetadataCache.java	`78.75% <ø> (+0.26%)`	⬆️
...areness/IsolatedBookieEnsemblePlacementPolicy.java	`87.21% <100.00%> (+0.19%)`	⬆️
...org/apache/pulsar/broker/ServiceConfiguration.java	`98.16% <100.00%> (-1.24%)`	⬇️
...pache/pulsar/broker/admin/impl/NamespacesBase.java	`76.09% <100.00%> (+3.01%)`	⬆️
...pulsar/broker/admin/impl/PersistentTopicsBase.java	`70.15% <100.00%> (+4.70%)`	⬆️
...g/apache/pulsar/broker/admin/impl/TenantsBase.java	`89.38% <100.00%> (-7.07%)`	⬇️
...rg/apache/pulsar/broker/admin/v3/Transactions.java	`69.17% <ø> (-6.85%)`	⬇️
... and 84 more

... and 998 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

…iration deletion rate

hanmz

LGTM

…tion deletion rate (#24190) Co-authored-by: zjxxzjwang <zjxxzjwang@tencent.com> (cherry picked from commit 81c94c8)

…tion deletion rate (apache#24190) Co-authored-by: zjxxzjwang <zjxxzjwang@tencent.com> (cherry picked from commit 81c94c8) (cherry picked from commit ef5c829)

…tion deletion rate (apache#24190) Co-authored-by: zjxxzjwang <zjxxzjwang@tencent.com> (cherry picked from commit 81c94c8) (cherry picked from commit 071dacf)

…tion deletion rate (apache#24190) Co-authored-by: zjxxzjwang <zjxxzjwang@tencent.com> (cherry picked from commit 81c94c8) (cherry picked from commit ef5c829)

…tion deletion rate (apache#24190) Co-authored-by: zjxxzjwang <zjxxzjwang@tencent.com> (cherry picked from commit 81c94c8) (cherry picked from commit 071dacf)

…tion deletion rate (apache#24190) Co-authored-by: zjxxzjwang <zjxxzjwang@tencent.com> (cherry picked from commit 81c94c8) (cherry picked from commit ef5c829)

…tion deletion rate (apache#24190) Co-authored-by: zjxxzjwang <zjxxzjwang@tencent.com> (cherry picked from commit 81c94c8)

…tion deletion rate (apache#24190) Co-authored-by: zjxxzjwang <zjxxzjwang@tencent.com>

[improve][broker]Resolve the issue of frequent updates in message exp…

0813ecc

…iration deletion rate

github-actions bot added the doc-label-missing label Apr 21, 2025

nodece reviewed Apr 21, 2025

View reviewed changes

pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java Show resolved Hide resolved

zjxxzjwang requested a review from nodece April 24, 2025 03:26

nodece approved these changes Apr 25, 2025

View reviewed changes

nodece requested review from dao-jun and lhotari April 25, 2025 05:03

dao-jun approved these changes Apr 27, 2025

View reviewed changes

jiazhai approved these changes May 7, 2025

View reviewed changes

HQebupt added the ready-to-test label May 22, 2025

github-actions bot added doc-not-needed Your PR changes do not impact docs and removed doc-label-missing labels May 22, 2025

zjxxzjwang added 3 commits May 22, 2025 22:30

[improve][broker]Resolve the issue of frequent updates in message exp…

c51db5e

…iration deletion rate

[improve][broker]Resolve the issue of frequent updates in message exp…

c7d7355

…iration deletion rate

Merge remote-tracking branch 'origin/master-Optimization-of-message-e…

b94f50f

…xpiration-rate' into master-Optimization-of-message-expiration-rate

github-actions bot added the PIP label May 22, 2025

[improve][broker]Resolve the issue of frequent updates in message exp…

aca8e60

…iration deletion rate

nodece changed the title ~~[improve][broker]Resolve the issue of frequent updates in message expiration deletion rate~~ [fix][broker] Resolve the issue of frequent updates in message expiration deletion rate May 23, 2025

nodece merged commit 81c94c8 into apache:master May 23, 2025
53 of 54 checks passed

nodece assigned zjxxzjwang May 23, 2025

nodece added release/3.0.13 release/4.0.6 release/3.3.8 labels May 23, 2025

hanmz reviewed May 26, 2025

View reviewed changes

lhotari pushed a commit that referenced this pull request Jun 2, 2025

[fix][broker] Resolve the issue of frequent updates in message expira…

071dacf

…tion deletion rate (#24190) Co-authored-by: zjxxzjwang <zjxxzjwang@tencent.com> (cherry picked from commit 81c94c8)

lhotari added the cherry-picked/branch-4.0 label Jun 2, 2025

lhotari pushed a commit that referenced this pull request Jun 2, 2025

[fix][broker] Resolve the issue of frequent updates in message expira…

ef5c829

…tion deletion rate (#24190) Co-authored-by: zjxxzjwang <zjxxzjwang@tencent.com> (cherry picked from commit 81c94c8)

lhotari added the cherry-picked/branch-3.0 label Jun 2, 2025

lhotari pushed a commit that referenced this pull request Jun 2, 2025

[fix][broker] Resolve the issue of frequent updates in message expira…

3a2d674

…tion deletion rate (#24190) Co-authored-by: zjxxzjwang <zjxxzjwang@tencent.com> (cherry picked from commit 81c94c8)

lhotari added the cherry-picked/branch-3.3 label Jun 3, 2025

KannarFr pushed a commit to CleverCloud/pulsar that referenced this pull request Sep 22, 2025

[fix][broker] Resolve the issue of frequent updates in message expira…

94aa41b

…tion deletion rate (apache#24190) Co-authored-by: zjxxzjwang <zjxxzjwang@tencent.com>

walkinggo pushed a commit to walkinggo/pulsar that referenced this pull request Oct 8, 2025

[fix][broker] Resolve the issue of frequent updates in message expira…

3b9f3fa

…tion deletion rate (apache#24190) Co-authored-by: zjxxzjwang <zjxxzjwang@tencent.com>

[fix][broker] Resolve the issue of frequent updates in message expiration deletion rate #24190

[fix][broker] Resolve the issue of frequent updates in message expiration deletion rate #24190

Conversation

zjxxzjwang commented Apr 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modifications

Documentation

Matching PR in forked repository

Uh oh!

github-actions bot commented Apr 21, 2025

Uh oh!

Uh oh!

nodece left a comment

Choose a reason for hiding this comment

Uh oh!

jiazhai commented Apr 28, 2025

Uh oh!

jiazhai commented Apr 28, 2025

Uh oh!

jiazhai commented Apr 28, 2025

Uh oh!

zjxxzjwang commented Apr 30, 2025

Uh oh!

zjxxzjwang commented May 8, 2025

Uh oh!

nodece commented May 8, 2025

Uh oh!

HQebupt commented May 22, 2025

Uh oh!

zjxxzjwang commented May 22, 2025

Uh oh!

codecov-commenter commented May 23, 2025

Codecov Report

Uh oh!

Uh oh!

hanmz left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

zjxxzjwang commented Apr 21, 2025 •

edited

Loading