Skip to content

[fix][broker]Global topic policies do not affect after unloading topic and persistence global topic policies never affect #24279

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
poorbarcode merged 3 commits into from
May 16, 2025
Merged

[fix][broker]Global topic policies do not affect after unloading topic and persistence global topic policies never affect #24279

poorbarcode merged 3 commits into from
May 16, 2025

Conversation

@poorbarcode
Copy link
Contributor

@poorbarcode poorbarcode commented May 8, 2025

Motivation

Issue 1: Global policies do not affect after unloading the topic

  • Set non-global policy: max consumers per topic 10
  • Set global policy: max producer per topic 20
  • Both policies affect
  • After a topic unloading
  • Only non-global policy affects; the global policy does not affect.

Root cause: when calling TopicPoliciesService.getPolicies(GetType.DEFAULT), it returns non-global policies. However, the correct behavior should be a combined value of non-global policies and global policies.

Issue 2: Persistence, global policies never affect

  • Set global policy: retention size in MB 200
  • Load up the topic, the global policy does not affect

Root cause: the values of global policies never been used when building the managed ledger config.

Modifications

  • Fix two issues
  • The item TopicPoliciesService.GetType.DEFAULT is meaningless, remove it.

Documentation

  • doc
  • doc-required
  • doc-not-needed
  • doc-complete

Matching PR in forked repository

PR in forked repository: x

@poorbarcode poorbarcode added this to the 4.1.0 milestone May 8, 2025
@poorbarcode poorbarcode self-assigned this May 8, 2025
@poorbarcode poorbarcode added type/bug The PR fixed a bug or issue reported a bug release/3.0.12 release/3.3.7 release/4.0.5 labels May 8, 2025
@github-actions github-actions bot added the doc-not-needed Your PR changes do not impact docs label May 8, 2025
@poorbarcode poorbarcode force-pushed the fix/global_topic_policy_2 branch from 611a8b5 to 3baff4b Compare May 8, 2025 12:41
@poorbarcode
Copy link
Contributor Author

poorbarcode commented May 8, 2025

/pulsarbot rerun-failure-checks

@gaoran10
Copy link
Contributor

gaoran10 commented May 13, 2025

The topic has a method onUpdate(TopicPolicies policies), could we leverage this topic policies update mechanism?

@poorbarcode poorbarcode requested a review from codelipenghui May 13, 2025 15:36
@poorbarcode poorbarcode merged commit 46c2b74 into apache:master May 16, 2025
53 checks passed
@lhotari
Copy link
Member

lhotari commented May 16, 2025 •
edited
Loading

There are some merge conflicts in maintenance branches due to lack of #24118 / #24154 changes in maintenance branches (mailing list thread https://lists.apache.org/thread/vdtwlr87kyvn1s7rqozxglz381d8mhp0).

@poorbarcode Please handle cherry-picking and testing the changes in maintenance branches since things could break while cherry-picking. The behavior could be different since #24118 / #24154 changes are missing. /cc @nodece

@lhotari
Copy link
Member

lhotari commented May 16, 2025

@poorbarcode and @gaoran10 , If we are interested in making topic policy more stable, it would be worth addressing #21303. The state isn't reliable at the moment when mutations happen. That could lead into many other problems.

poorbarcode added a commit that referenced this pull request May 29, 2025
@poorbarcode
[fix][broker]Global topic policies do not affect after unloading topi…
e7b7385 
…c and persistence global topic policies never affect (#24279)

(cherry picked from commit 46c2b74)
poorbarcode added a commit that referenced this pull request May 29, 2025
@poorbarcode
[fix][broker]Global topic policies do not affect after unloading topi…
57a450e 
…c and persistence global topic policies never affect (#24279)

(cherry picked from commit 46c2b74)
poorbarcode added a commit that referenced this pull request May 29, 2025
@poorbarcode
[fix][broker]Global topic policies do not affect after unloading topi…
c0e5ce5 
…c and persistence global topic policies never affect (#24279)

(cherry picked from commit 46c2b74)
manas-ctds pushed a commit to datastax/pulsar that referenced this pull request Jun 4, 2025
@poorbarcode @ganesh-ctds
[fix][broker]Global topic policies do not affect after unloading topi…
5bd6487 
…c and persistence global topic policies never affect (apache#24279)

(cherry picked from commit 46c2b74)
(cherry picked from commit c0e5ce5)
manas-ctds pushed a commit to datastax/pulsar that referenced this pull request Jun 4, 2025
@poorbarcode @manas-ctds
[fix][broker]Global topic policies do not affect after unloading topi…
9957575 
…c and persistence global topic policies never affect (apache#24279)

(cherry picked from commit 46c2b74)
(cherry picked from commit e7b7385)
manas-ctds pushed a commit to datastax/pulsar that referenced this pull request Jun 4, 2025
@poorbarcode @ganesh-ctds
[fix][broker]Global topic policies do not affect after unloading topi…
522d1a3 
…c and persistence global topic policies never affect (apache#24279)

(cherry picked from commit 46c2b74)
(cherry picked from commit c0e5ce5)
manas-ctds pushed a commit to datastax/pulsar that referenced this pull request Jun 5, 2025
@poorbarcode @ganesh-ctds
[fix][broker]Global topic policies do not affect after unloading topi…
c43b005 
…c and persistence global topic policies never affect (apache#24279)

(cherry picked from commit 46c2b74)
(cherry picked from commit c0e5ce5)
manas-ctds pushed a commit to datastax/pulsar that referenced this pull request Jun 5, 2025
@poorbarcode @ganesh-ctds
[fix][broker]Global topic policies do not affect after unloading topi…
1f8202e 
…c and persistence global topic policies never affect (apache#24279)

(cherry picked from commit 46c2b74)
(cherry picked from commit c0e5ce5)
manas-ctds pushed a commit to datastax/pulsar that referenced this pull request Jun 5, 2025
@poorbarcode @ganesh-ctds
[fix][broker]Global topic policies do not affect after unloading topi…
77a32e9 
…c and persistence global topic policies never affect (apache#24279)

(cherry picked from commit 46c2b74)
(cherry picked from commit c0e5ce5)
manas-ctds pushed a commit to datastax/pulsar that referenced this pull request Jun 5, 2025
@poorbarcode @ganesh-ctds
[fix][broker]Global topic policies do not affect after unloading topi…
3d071b9 
…c and persistence global topic policies never affect (apache#24279)

(cherry picked from commit 46c2b74)
(cherry picked from commit c0e5ce5)
manas-ctds pushed a commit to datastax/pulsar that referenced this pull request Jun 5, 2025
@poorbarcode @ganesh-ctds
[fix][broker]Global topic policies do not affect after unloading topi…
72ab728 
…c and persistence global topic policies never affect (apache#24279)

(cherry picked from commit 46c2b74)
(cherry picked from commit c0e5ce5)
manas-ctds pushed a commit to datastax/pulsar that referenced this pull request Jun 5, 2025
@poorbarcode @ganesh-ctds
[fix][broker]Global topic policies do not affect after unloading topi…
949e2bc 
…c and persistence global topic policies never affect (apache#24279)

(cherry picked from commit 46c2b74)
(cherry picked from commit c0e5ce5)
srinath-ctds pushed a commit to datastax/pulsar that referenced this pull request Jun 5, 2025
@poorbarcode @srinath-ctds
[fix][broker]Global topic policies do not affect after unloading topi…
a550580 
…c and persistence global topic policies never affect (apache#24279)

(cherry picked from commit 46c2b74)
(cherry picked from commit e7b7385)
srinath-ctds pushed a commit to datastax/pulsar that referenced this pull request Jun 10, 2025
@poorbarcode @srinath-ctds
[fix][broker]Global topic policies do not affect after unloading topi…
5d50b2e 
…c and persistence global topic policies never affect (apache#24279)

(cherry picked from commit 46c2b74)
(cherry picked from commit c0e5ce5)
nodece pushed a commit to nodece/pulsar that referenced this pull request Jun 18, 2025
@poorbarcode @nodece
[fix][broker]Global topic policies do not affect after unloading topi…
080aebd 
…c and persistence global topic policies never affect (apache#24279)

(cherry picked from commit 46c2b74)
Signed-off-by: Zixuan Liu <nodeces@gmail.com>
@nodece
Copy link
Member

nodece commented Jun 23, 2025

@poorbarcode #24279 is causing CI failures on both the branch-3.0 and branch-3.3 branches. Could you check this? Please see https://lists.apache.org/thread/005qvhjvr5kpcf9qcgbjw82c51of28jy

@nodece nodece mentioned this pull request Jun 24, 2025
Merged
4 tasks
KannarFr pushed a commit to CleverCloud/pulsar that referenced this pull request Sep 22, 2025
@poorbarcode @KannarFr
[fix][broker]Global topic policies do not affect after unloading topi…
c7db5b3 
…c and persistence global topic policies never affect (apache#24279)
walkinggo pushed a commit to walkinggo/pulsar that referenced this pull request Oct 8, 2025
@poorbarcode @walkinggo
[fix][broker]Global topic policies do not affect after unloading topi…
22769f5 
…c and persistence global topic policies never affect (apache#24279)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@codelipenghui codelipenghui codelipenghui approved these changes

@gaoran10 gaoran10 gaoran10 approved these changes

@lhotari lhotari Awaiting requested review from lhotari

@Technoboy- Technoboy- Awaiting requested review from Technoboy-

Assignees

@poorbarcode poorbarcode

Labels

cherry-picked/branch-3.0 cherry-picked/branch-3.3 cherry-picked/branch-4.0 doc-not-needed Your PR changes do not impact docs ready-to-test release/3.0.13 release/3.3.8 release/4.0.6 type/bug The PR fixed a bug or issue reported a bug

Projects

None yet

Milestone

4.1.0

Development

Successfully merging this pull request may close these issues.

5 participants

@poorbarcode @gaoran10 @lhotari @nodece @codelipenghui