Skip to content

Conversation

@poorbarcode
Copy link
Contributor

@poorbarcode poorbarcode commented May 12, 2025

Motivation

Issue 1: Non-global topic policies and global topic policies overwrite each other

  • Set local topic policy: dispatch rate 2000
  • Set global topic policy: dispatch rate 1000
  • Trigger a compaction for the system topic __change_events
  • Issue: get local topic policy 1000

Issue 2: Global policy can not be removed successfully

  • Set global topic policy: dispatch rate 1000
  • Delete topic, the global topic-level policies will never be deleted

Modifications

Fix two issues

Documentation

  • doc
  • doc-required
  • doc-not-needed
  • doc-complete

Matching PR in forked repository

PR in forked repository: x

@poorbarcode poorbarcode added this to the 4.1.0 milestone May 12, 2025
@poorbarcode poorbarcode self-assigned this May 12, 2025
@github-actions github-actions bot added the doc-not-needed Your PR changes do not impact docs label May 12, 2025
@poorbarcode poorbarcode changed the title [fix][broker]Non-global topic policies and global topic policies overwrites each other [fix][broker]Non-global topic policies and global topic policies overwrite each other May 12, 2025
@poorbarcode
Copy link
Contributor Author

/pulsarbot rerun-failure-checks

@poorbarcode poorbarcode requested a review from gaoran10 May 21, 2025 12:23
@codecov-commenter
Copy link

codecov-commenter commented May 21, 2025

Codecov Report

❌ Patch coverage is 73.68421% with 10 lines in your changes missing coverage. Please review.
✅ Project coverage is 74.24%. Comparing base (bbc6224) to head (e473f3d).
⚠️ Report is 1539 commits behind head on master.

Files with missing lines Patch % Lines
.../service/SystemTopicBasedTopicPoliciesService.java 72.72% 5 Missing and 1 partial ⚠️
...he/pulsar/broker/service/TopicPoliciesService.java 75.00% 4 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@             Coverage Diff              @@
##             master   #24286      +/-   ##
============================================
+ Coverage     73.57%   74.24%   +0.66%     
+ Complexity    32624    32257     -367     
============================================
  Files          1877     1866      -11     
  Lines        139502   145124    +5622     
  Branches      15299    16593    +1294     
============================================
+ Hits         102638   107746    +5108     
+ Misses        28908    28836      -72     
- Partials       7956     8542     +586     
Flag Coverage Δ
inttests 26.79% <68.42%> (+2.20%) ⬆️
systests 23.26% <65.78%> (-1.06%) ⬇️
unittests 73.73% <73.68%> (+0.88%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
...he/pulsar/broker/service/TopicPoliciesService.java 85.71% <75.00%> (+16.96%) ⬆️
.../service/SystemTopicBasedTopicPoliciesService.java 75.00% <72.72%> (+0.80%) ⬆️

... and 1082 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@gaoran10 gaoran10 merged commit 6662471 into apache:master May 21, 2025
53 checks passed
poorbarcode added a commit that referenced this pull request May 29, 2025
poorbarcode added a commit that referenced this pull request May 29, 2025
poorbarcode added a commit that referenced this pull request May 29, 2025
manas-ctds pushed a commit to datastax/pulsar that referenced this pull request Jun 4, 2025
manas-ctds pushed a commit to datastax/pulsar that referenced this pull request Jun 4, 2025
…write each other (apache#24286)

(cherry picked from commit 6662471)
(cherry picked from commit 323e2ff)
manas-ctds pushed a commit to datastax/pulsar that referenced this pull request Jun 4, 2025
…write each other (apache#24286)

(cherry picked from commit 6662471)
(cherry picked from commit 5cfd804)
manas-ctds pushed a commit to datastax/pulsar that referenced this pull request Jun 4, 2025
…write each other (apache#24286)

(cherry picked from commit 6662471)
(cherry picked from commit 5cfd804)
manas-ctds pushed a commit to datastax/pulsar that referenced this pull request Jun 5, 2025
…write each other (apache#24286)

(cherry picked from commit 6662471)
(cherry picked from commit 5cfd804)
manas-ctds pushed a commit to datastax/pulsar that referenced this pull request Jun 5, 2025
…write each other (apache#24286)

(cherry picked from commit 6662471)
(cherry picked from commit 5cfd804)
manas-ctds pushed a commit to datastax/pulsar that referenced this pull request Jun 5, 2025
…write each other (apache#24286)

(cherry picked from commit 6662471)
(cherry picked from commit 5cfd804)
srinath-ctds pushed a commit to datastax/pulsar that referenced this pull request Jun 5, 2025
…write each other (apache#24286)

(cherry picked from commit 6662471)
(cherry picked from commit 323e2ff)
srinath-ctds pushed a commit to datastax/pulsar that referenced this pull request Jun 10, 2025
…write each other (apache#24286)

(cherry picked from commit 6662471)
(cherry picked from commit 5cfd804)
nodece pushed a commit to nodece/pulsar that referenced this pull request Jun 18, 2025
@lhotari
Copy link
Member

lhotari commented Jun 19, 2025

@poorbarcode It seems that this change should be covered with a PIP. Do you agree?

@poorbarcode
Copy link
Contributor Author

@lhotari

@poorbarcode It seems that this change should be covered with a PIP. Do you agree?

Could you explain why it is needed?

@lhotari
Copy link
Member

lhotari commented Jun 19, 2025

@lhotari

@poorbarcode It seems that this change should be covered with a PIP. Do you agree?

Could you explain why it is needed?

This PR changed the way how topic policies are stored. It would be useful to have such details documented in PIPs. This change was also cherry-picked to maintenance branches and it would be useful to document the backwards compatibility in the PIP document.

Comment on lines +229 to +232
CompletableFuture<MessageId> deletePolicies = writer.deleteAsync(getEventKey(event, true), event)
.thenCompose(__ -> {
return writer.deleteAsync(getEventKey(event, false), event);
});
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@poorbarcode @lhotari It's a bit strange, why isn't CompletableFuture.allOf used here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think using CompletableFuture.allOf can benefit the performance. Thanks for mentioning this improvement

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Of course, you can submit a small PR later.

KannarFr pushed a commit to CleverCloud/pulsar that referenced this pull request Sep 22, 2025
walkinggo pushed a commit to walkinggo/pulsar that referenced this pull request Oct 8, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants