[fix][broker] Add topic consistency check #24118

nodece · 2025-03-25T04:35:38Z

Motivation

In Apache Pulsar, clients can produce or consume messages using a fully qualified partition topic name (public/default/test-partitioned-topic-partition-N). However, if the topic metadata does not exist and allowAutoTopicCreationType=non-partitioned is enabled, Pulsar automatically creates the topic as non-partitioned. This leads to inconsistencies when listing topics, as the expected partitioned topic structure is not maintained.

This issue is especially prevalent in geo-replicated environments, where topic metadata may not be fully synchronized across all regions. Currently, Pulsar lacks validation to ensure that the topic type (partitioned or non-partitioned) matches the actual partition metadata.

Modifications

Added a topic consistency check:
- For non-partitioned topic: If partition metadata exists (which shouldn't be there), the broker will reject message produce and consume operations.
- For partitioned topic: If partition metadata is missing (which should be there), the broker will reject message produce and consume operations.
Fixed affected tests to align with the new validation logic.
Added testGetPartitionedTopicMetadataByPulsarClient and testGetPartitionedTopicMedataByAdmin instead of testCompatibilityWithPartitionKeyword because the broker rejects message produce and consume operations when no metadata. Note: we still allow user lookup topic name with -partition- but no metadata: [fix][broker] Allow user lookup topic name with -partition- but no metadata #19171.

Documentation

doc
doc-required
doc-not-needed
doc-complete

Signed-off-by: Zixuan Liu <nodeces@gmail.com>

pulsar-broker/src/main/java/org/apache/pulsar/broker/namespace/NamespaceService.java

pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java

pulsar-broker/src/test/java/org/apache/pulsar/client/api/ConsumerCreationTest.java

Signed-off-by: Zixuan Liu <nodeces@gmail.com>

…umerWhenSinglePartitionIsDeleted Signed-off-by: Zixuan Liu <nodeces@gmail.com>

Signed-off-by: Zixuan Liu <nodeces@gmail.com>

codecov-commenter · 2025-03-26T02:02:23Z

Codecov Report

Attention: Patch coverage is 98.00000% with 1 line in your changes missing coverage. Please review.

Project coverage is 74.27%. Comparing base (bbc6224) to head (cb09e93).
Report is 993 commits behind head on master.

Files with missing lines	Patch %	Lines
...rg/apache/pulsar/broker/service/BrokerService.java	96.96%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #24118      +/-   ##
============================================
+ Coverage     73.57%   74.27%   +0.70%     
+ Complexity    32624    32117     -507     
============================================
  Files          1877     1864      -13     
  Lines        139502   144437    +4935     
  Branches      15299    16473    +1174     
============================================
+ Hits         102638   107286    +4648     
+ Misses        28908    28702     -206     
- Partials       7956     8449     +493

Flag	Coverage Δ
inttests	`26.93% <68.00%> (+2.34%)`	⬆️
systests	`23.16% <28.00%> (-1.17%)`	⬇️
unittests	`73.78% <98.00%> (+0.93%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
...ache/pulsar/broker/namespace/NamespaceService.java	`73.61% <100.00%> (+1.38%)`	⬆️
...pache/pulsar/client/api/PulsarClientException.java	`61.63% <100.00%> (-1.41%)`	⬇️
...rg/apache/pulsar/broker/service/BrokerService.java	`84.26% <96.96%> (+3.48%)`	⬆️

... and 1060 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

lhotari · 2025-04-23T11:15:32Z

Reverted from branch-3.0, branch-3.3 and branch-4.0 as per this dev mailing list message.

Signed-off-by: Zixuan Liu <nodeces@gmail.com> (cherry picked from commit 7f0429c) (cherry picked from commit 7671275)

This reverts commit 7671275. (cherry picked from commit baaf4a3)

Signed-off-by: Zixuan Liu <nodeces@gmail.com> (cherry picked from commit 7f0429c) (cherry picked from commit 7671275)

This reverts commit 7671275. (cherry picked from commit baaf4a3)

Signed-off-by: Zixuan Liu <nodeces@gmail.com> (cherry picked from commit 7f0429c) (cherry picked from commit 7671275)

This reverts commit 7671275. (cherry picked from commit baaf4a3)

Signed-off-by: Zixuan Liu <nodeces@gmail.com> (cherry picked from commit 7f0429c) (cherry picked from commit ed64bf9)

This reverts commit ed64bf9. (cherry picked from commit eef20ed)

Signed-off-by: Zixuan Liu <nodeces@gmail.com> (cherry picked from commit 7f0429c) (cherry picked from commit ed64bf9)

This reverts commit ed64bf9. (cherry picked from commit eef20ed)

This reverts commit ed64bf9.

Signed-off-by: Zixuan Liu <nodeces@gmail.com>

github-actions bot added the doc-not-needed Your PR changes do not impact docs label Mar 25, 2025

nodece requested review from dao-jun, lhotari, poorbarcode and shibd March 25, 2025 04:36

nodece self-assigned this Mar 25, 2025

nodece force-pushed the add-topic-consistency-check-master branch from 7426799 to 40fb779 Compare March 25, 2025 05:10

dao-jun approved these changes Mar 25, 2025

View reviewed changes

nodece closed this Mar 25, 2025

nodece force-pushed the add-topic-consistency-check-master branch from 40fb779 to d7962a1 Compare March 25, 2025 07:11

[fix][broker] Add topic consistency check

7ab11d5

Signed-off-by: Zixuan Liu <nodeces@gmail.com>

nodece reopened this Mar 25, 2025

nodece added 3 commits March 25, 2025 17:19

Remove testCompatibilityWithPartitionKeyword and add new test

b83d183

Signed-off-by: Zixuan Liu <nodeces@gmail.com>

Fix non-persistent topic existence check

bc09ab5

Signed-off-by: Zixuan Liu <nodeces@gmail.com>

Verify the topic exists, and the metadataAutoCreationEnabled is false

d5c7607

Signed-off-by: Zixuan Liu <nodeces@gmail.com>

nodece added this to the 4.1.0 milestone Mar 25, 2025

lhotari reviewed Mar 25, 2025

View reviewed changes

nodece added 9 commits March 25, 2025 20:38

Refactor validateTopicConsistency

e13e45b

Signed-off-by: Zixuan Liu <nodeces@gmail.com>

Check topic partition index in checkTopicExists

4fe15ad

Signed-off-by: Zixuan Liu <nodeces@gmail.com>

Fix testCreateNonExistentPartitions

2925547

Signed-off-by: Zixuan Liu <nodeces@gmail.com>

Fix PulsarClientBasedHandler

bc150a6

Signed-off-by: Zixuan Liu <nodeces@gmail.com>

Add testCreateProducerWhenSinglePartitionIsDeleted and testCreateCons…

d61f84a

…umerWhenSinglePartitionIsDeleted Signed-off-by: Zixuan Liu <nodeces@gmail.com>

Fix style

1e80ea7

Signed-off-by: Zixuan Liu <nodeces@gmail.com>

Fix style

c35370d

Signed-off-by: Zixuan Liu <nodeces@gmail.com>

Fix test

e7fe4af

Signed-off-by: Zixuan Liu <nodeces@gmail.com>

Fix test

4b3a6ab

Signed-off-by: Zixuan Liu <nodeces@gmail.com>

nodece force-pushed the add-topic-consistency-check-master branch from 88ab8db to 4b3a6ab Compare March 25, 2025 16:20

Fix style

cb09e93

Signed-off-by: Zixuan Liu <nodeces@gmail.com>

nodece requested a review from lhotari March 26, 2025 02:11

lhotari removed cherry-picked/branch-3.3 cherry-picked/branch-4.0 release/3.0.12 release/3.3.7 release/4.0.5 labels Apr 23, 2025

ganesh-ctds pushed a commit to datastax/pulsar that referenced this pull request Apr 24, 2025

[fix][broker] Add topic consistency check (apache#24118)

282343a

Signed-off-by: Zixuan Liu <nodeces@gmail.com> (cherry picked from commit 7f0429c) (cherry picked from commit 7671275)

ganesh-ctds pushed a commit to datastax/pulsar that referenced this pull request Apr 24, 2025

[fix][broker] Add topic consistency check (apache#24118)

5255360

Signed-off-by: Zixuan Liu <nodeces@gmail.com> (cherry picked from commit 7f0429c) (cherry picked from commit 7671275)

manas-ctds pushed a commit to datastax/pulsar that referenced this pull request Apr 25, 2025

[fix][broker] Add topic consistency check (apache#24118)

ee1d5cd

Signed-off-by: Zixuan Liu <nodeces@gmail.com> (cherry picked from commit 7f0429c) (cherry picked from commit 7671275)

manas-ctds pushed a commit to datastax/pulsar that referenced this pull request Apr 25, 2025

[fix][broker] Add topic consistency check (apache#24118)

8c79e88

Signed-off-by: Zixuan Liu <nodeces@gmail.com> (cherry picked from commit 7f0429c) (cherry picked from commit 7671275)

nodece mentioned this pull request Apr 25, 2025

[improve][pip] PIP-414: Enforce topic consistency check #24213

Merged

4 tasks

manas-ctds pushed a commit to datastax/pulsar that referenced this pull request Apr 27, 2025

[fix][broker] Add topic consistency check (apache#24118)

3f985ed

Signed-off-by: Zixuan Liu <nodeces@gmail.com> (cherry picked from commit 7f0429c) (cherry picked from commit 7671275)

manas-ctds pushed a commit to datastax/pulsar that referenced this pull request Apr 27, 2025

Revert "[fix][broker] Add topic consistency check (apache#24118)"

9c8b122

This reverts commit 7671275. (cherry picked from commit baaf4a3)

nodece mentioned this pull request Apr 28, 2025

[fix][broker] Allow recreation of partitioned topic after metadata loss #24225

Merged

4 tasks

ganesh-ctds pushed a commit to datastax/pulsar that referenced this pull request Apr 30, 2025

[fix][broker] Add topic consistency check (apache#24118)

7ccab1a

Signed-off-by: Zixuan Liu <nodeces@gmail.com> (cherry picked from commit 7f0429c) (cherry picked from commit 7671275)

ganesh-ctds pushed a commit to datastax/pulsar that referenced this pull request Apr 30, 2025

Revert "[fix][broker] Add topic consistency check (apache#24118)"

dcbb1e9

This reverts commit 7671275. (cherry picked from commit baaf4a3)

ganesh-ctds pushed a commit to datastax/pulsar that referenced this pull request May 1, 2025

Revert "[fix][broker] Add topic consistency check (apache#24118)"

da43660

This reverts commit 7671275. (cherry picked from commit baaf4a3)

ganesh-ctds pushed a commit to datastax/pulsar that referenced this pull request May 1, 2025

Revert "[fix][broker] Add topic consistency check (apache#24118)"

ad9f7ac

This reverts commit 7671275. (cherry picked from commit baaf4a3)

manas-ctds pushed a commit to datastax/pulsar that referenced this pull request May 2, 2025

[fix][broker] Add topic consistency check (apache#24118)

eea4946

Signed-off-by: Zixuan Liu <nodeces@gmail.com> (cherry picked from commit 7f0429c) (cherry picked from commit 7671275)

manas-ctds pushed a commit to datastax/pulsar that referenced this pull request May 2, 2025

Revert "[fix][broker] Add topic consistency check (apache#24118)"

1899cea

This reverts commit 7671275. (cherry picked from commit baaf4a3)

manas-ctds pushed a commit to datastax/pulsar that referenced this pull request May 2, 2025

[fix][broker] Add topic consistency check (apache#24118)

47244ce

Signed-off-by: Zixuan Liu <nodeces@gmail.com> (cherry picked from commit 7f0429c) (cherry picked from commit ed64bf9)

manas-ctds pushed a commit to datastax/pulsar that referenced this pull request May 2, 2025

Revert "[fix][broker] Add topic consistency check (apache#24118)"

0c781bb

This reverts commit ed64bf9. (cherry picked from commit eef20ed)

srinath-ctds pushed a commit to datastax/pulsar that referenced this pull request May 6, 2025

[fix][broker] Add topic consistency check (apache#24118)

38145e8

Signed-off-by: Zixuan Liu <nodeces@gmail.com> (cherry picked from commit 7f0429c) (cherry picked from commit ed64bf9)

srinath-ctds pushed a commit to datastax/pulsar that referenced this pull request May 6, 2025

Revert "[fix][broker] Add topic consistency check (apache#24118)"

33e8d3f

This reverts commit ed64bf9. (cherry picked from commit eef20ed)

lhotari mentioned this pull request May 16, 2025

[fix][broker]Global topic policies do not affect after unloading topic and persistence global topic policies never affect #24279

Merged

4 tasks

lhotari mentioned this pull request Jul 25, 2025

[fix][client] Partitioned topics are unexpectedly created by client after deletion #24554

Merged

4 tasks

nborisov pushed a commit to nborisov/pulsar that referenced this pull request Sep 12, 2025

Revert "[fix][broker] Add topic consistency check (apache#24118)"

d0d6a20

This reverts commit ed64bf9.

walkinggo pushed a commit to walkinggo/pulsar that referenced this pull request Oct 8, 2025

[fix][broker] Add topic consistency check (apache#24118)

7ed1bce

Signed-off-by: Zixuan Liu <nodeces@gmail.com>

lhotari mentioned this pull request Oct 8, 2025

[fix][broker] Allow intermittent error from topic policies service when loading topics #24829

Merged

4 tasks

TakaHiR07 mentioned this pull request Dec 12, 2025

[fix][broker] fix error process in admin.updateTopicPartition #25042

Open

14 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[fix][broker] Add topic consistency check #24118

[fix][broker] Add topic consistency check #24118

Uh oh!

nodece commented Mar 25, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov-commenter commented Mar 26, 2025 •

edited

Loading

Uh oh!

lhotari commented Apr 23, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

[fix][broker] Add topic consistency check #24118

[fix][broker] Add topic consistency check #24118

Uh oh!

Conversation

nodece commented Mar 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modifications

Documentation

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov-commenter commented Mar 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

lhotari commented Apr 23, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

nodece commented Mar 25, 2025 •

edited

Loading

codecov-commenter commented Mar 26, 2025 •

edited

Loading