Skip to content

Conversation

@zjxxzjwang
Copy link
Contributor

@zjxxzjwang zjxxzjwang commented Mar 12, 2025

Motivation

The message expiration rate is updated in both tasks, as follows:
Clipboard_Screenshot_1741772793
Clipboard_Screenshot_1741772909

the time period for calculating the rate is the interval between two tasks. If the time interval between two tasks is very small, the calculation speed will be very large.

Modifications

Delete the update method in the message expiration task, and retain the unified update method in the topic

Documentation

  • doc
  • doc-required
  • doc-not-needed
  • doc-complete

@github-actions
Copy link

@zjxxzjwang Please add the following content to your PR description and select a checkbox:

- [ ] `doc` <!-- Your PR contains doc changes -->
- [ ] `doc-required` <!-- Your PR changes impact docs and you will update later -->
- [ ] `doc-not-needed` <!-- Your PR changes do not impact docs -->
- [ ] `doc-complete` <!-- Docs have been already added -->

Copy link
Member

@lhotari lhotari left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the time period for calculating the rate is the interval between two tasks. If the time interval between two tasks is very small, the calculation speed will be very large.

Good observations, @zjxxzjwang. It seems that this PR is not only an optimization, but targets fixing invalid rate stats.

However, the current change would make "topic stats" inaccurate since the rate wouldn't get updated in that case. For consistency reasons, updating the rate should happen in the same way for all code paths.
One possible solution would be to make updateRates method private in PersistentMessageExpiryMonitor and call updateRates in the implementation of getMessageExpiryRate method, before the updated rate is returned.

@zjxxzjwang
Copy link
Contributor Author

zjxxzjwang commented Mar 14, 2025

计算速率的时间段就是两个任务之间的间隔,如果两个任务之间的时间间隔很小,计算速度就会很大。

很好的观察,@zjxxzjwang看来这个 PR 不仅仅是一个优化,还针对修复无效率统计。

但是,当前的更改会使“主题统计信息”不准确,因为在这种情况下速率不会更新。出于一致性原因,更新速率应以相同的方式在所有代码路径中进行。 一种可能的解决方案是将 PersistentMessageExpiryMonitor 中的 updateRates 方法设为私有,并在返回更新后的速率之前在 getMessageExpiryRate 方法的实现中调用 updateRates。

计算速率的时间段就是两个任务之间的间隔,如果两个任务之间的时间间隔很小,计算速度就会很大。

很好的观察,@zjxxzjwang看来这个 PR 不仅仅是一个优化,还针对修复无效率统计。

但是,当前的更改会使“主题统计信息”不准确,因为在这种情况下速率不会更新。出于一致性原因,更新速率应以相同的方式在所有代码路径中进行。 一种可能的解决方案是将 PersistentMessageExpiryMonitor 中的 updateRates 方法设为私有,并在返回更新后的速率之前在 getMessageExpiryRate 方法的实现中调用 updateRates。

Hello, the rate will continue to be updated ,because the updateRates method of the PersistentTopic class will also be executed at regular intervals, and the time period used to calculate the rate will be fixed (the timing period of the updateRates method). The calculated rate can be explained more strongly. @lhotari

@zjxxzjwang zjxxzjwang requested a review from lhotari March 14, 2025 07:07
@lhotari
Copy link
Member

lhotari commented Mar 14, 2025

However, the current change would make "topic stats" inaccurate since the rate wouldn't get updated in that case. For consistency reasons, updating the rate should happen in the same way for all code paths.
One possible solution would be to make updateRates method private in PersistentMessageExpiryMonitor and call updateRates in the implementation of getMessageExpiryRate method, before the updated rate is returned.

@zjxxzjwang Please address this review comment. The solution would be somewhat similar as there is in #24067.

@zjxxzjwang
Copy link
Contributor Author

但是,当前的更改会使“主题统计信息”不准确,因为在这种情况下速率不会更新。出于一致性原因,更新速率应以相同的方式在所有代码路径中进行。
一种可能的解决方案是将 PersistentMessageExpiryMonitor 中的 updateRates 方法设为私有,并在返回更新后的速率之前在 getMessageExpiryRate 方法的实现中调用 updateRates。

@zjxxzjwang请回复此评论。解决方案与#24067中的有些相似。

Good suggestion, I have made the modifications according to your suggestion. Please help me review it。 @lhotari

@zjxxzjwang
Copy link
Contributor Author

@lhotari Hello, please help me review it

@codecov-commenter
Copy link

codecov-commenter commented Mar 19, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 74.17%. Comparing base (bbc6224) to head (3fb529b).
Report is 978 commits behind head on master.

Additional details and impacted files

Impacted file tree graph

@@             Coverage Diff              @@
##             master   #24073      +/-   ##
============================================
+ Coverage     73.57%   74.17%   +0.59%     
+ Complexity    32624    32426     -198     
============================================
  Files          1877     1863      -14     
  Lines        139502   144311    +4809     
  Branches      15299    16457    +1158     
============================================
+ Hits         102638   107041    +4403     
+ Misses        28908    28812      -96     
- Partials       7956     8458     +502     
Flag Coverage Δ
inttests 26.67% <100.00%> (+2.08%) ⬆️
systests 23.17% <100.00%> (-1.16%) ⬇️
unittests 73.67% <100.00%> (+0.83%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
...ice/persistent/PersistentMessageExpiryMonitor.java 58.33% <100.00%> (-5.84%) ⬇️
...roker/service/persistent/PersistentReplicator.java 68.58% <ø> (-0.29%) ⬇️
...sar/broker/service/persistent/PersistentTopic.java 79.80% <ø> (+1.34%) ⬆️

... and 1062 files with indirect coverage changes

🚀 New features to boost your workflow:
  • Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@lhotari lhotari changed the title [improve][broker]Optimize message expiration rate repeated update issues [improve][broker] Optimize message expiration rate repeated update issues Mar 19, 2025
@lhotari lhotari merged commit b936f46 into apache:master Mar 19, 2025
52 checks passed
@lhotari lhotari added this to the 4.1.0 milestone Mar 19, 2025
lhotari pushed a commit that referenced this pull request Mar 20, 2025
…sues (#24073)

Co-authored-by: zjxxzjwang <zjxxzjwang@tencent.com>
(cherry picked from commit b936f46)
lhotari pushed a commit that referenced this pull request Mar 20, 2025
…sues (#24073)

Co-authored-by: zjxxzjwang <zjxxzjwang@tencent.com>
(cherry picked from commit b936f46)
lhotari pushed a commit that referenced this pull request Mar 20, 2025
…sues (#24073)

Co-authored-by: zjxxzjwang <zjxxzjwang@tencent.com>
(cherry picked from commit b936f46)
bahetimansi pushed a commit to bahetimansi/pulsar that referenced this pull request Mar 23, 2025
…sues (apache#24073)

Co-authored-by: zjxxzjwang <zjxxzjwang@tencent.com>
(cherry picked from commit b936f46)
(cherry picked from commit c51b9ca)
nodece pushed a commit to nodece/pulsar that referenced this pull request Mar 27, 2025
…sues (apache#24073)

Co-authored-by: zjxxzjwang <zjxxzjwang@tencent.com>
(cherry picked from commit b936f46)
nikhil-ctds pushed a commit to datastax/pulsar that referenced this pull request Mar 27, 2025
…sues (apache#24073)

Co-authored-by: zjxxzjwang <zjxxzjwang@tencent.com>
(cherry picked from commit b936f46)
(cherry picked from commit c51b9ca)
srinath-ctds pushed a commit to datastax/pulsar that referenced this pull request Mar 28, 2025
…sues (apache#24073)

Co-authored-by: zjxxzjwang <zjxxzjwang@tencent.com>
(cherry picked from commit b936f46)
(cherry picked from commit c51b9ca)
nikhil-ctds pushed a commit to datastax/pulsar that referenced this pull request Mar 28, 2025
…sues (apache#24073)

Co-authored-by: zjxxzjwang <zjxxzjwang@tencent.com>
(cherry picked from commit b936f46)
(cherry picked from commit f3368b4)
nikhil-ctds pushed a commit to datastax/pulsar that referenced this pull request Mar 29, 2025
…sues (apache#24073)

Co-authored-by: zjxxzjwang <zjxxzjwang@tencent.com>
(cherry picked from commit b936f46)
(cherry picked from commit f3368b4)
srinath-ctds pushed a commit to datastax/pulsar that referenced this pull request Mar 31, 2025
…sues (apache#24073)

Co-authored-by: zjxxzjwang <zjxxzjwang@tencent.com>
(cherry picked from commit b936f46)
(cherry picked from commit f3368b4)
walkinggo pushed a commit to walkinggo/pulsar that referenced this pull request Oct 8, 2025
…sues (apache#24073)

Co-authored-by: zjxxzjwang <zjxxzjwang@tencent.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants