Maybe running it many times can reproduce it

#21653 (comment)

Its difficult to reproduce with unit tests, manually thread.sleep() can reproduce it😂.

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 73.37%. Comparing base (495b141) to head (ca2d28c).
Report is 981 commits behind head on master.

@@              Coverage Diff              @@
##             master   #21704       +/-   ##
=============================================
+ Coverage     36.75%   73.37%   +36.62%     
- Complexity    12271    32767    +20496     
=============================================
  Files          1717     1893      +176     
  Lines        131197   140761     +9564     
  Branches      14339    15509     +1170     
=============================================
+ Hits          48220   103290    +55070     
+ Misses        76596    29360    -47236     
- Partials       6381     8111     +1730     

@crossoverJie, can we just put a time-consuming task in the thread pool to simulate the sleep behavior?
Or maybe you can just push your code with the sleep and we can find a solution together.

It's a very critical fix, if don't have a test to protect the change, things will go wrong again in the future. Here is an example: #21686

cc @mattisonchao @Technoboy-

Sorry, to clear the root cause of the issue, I left a comment:

The conditions-1 to encounter the issue:

• You enabled replication on the namespace level and did not enable replication on the topic level.
• You enabled topic level policies
• The flowing events happen at the same time
  • Modify topic-level policies, such as pulsar-admin topics set-dispatch-rate. Note: Modifying namespace-level policies will not trigger the issue.
  • The target topic is loading now.

The conditions-2 to encounter the issue:

• You enabled replication on the namespace level and did not enable replication on the topic level.
• You enabled topic-level policies
• Restart brokers
• Note:
  • this case often happens when there is only 1 broker under your cluster,
  • the probability is rare if there are more than 1 brokers under your cluster. If you restart all brokers at the same time, the probability rises as only 1 broker.

Background & Summary

1. checkCompaction will delete the topic if the attribute replicationClusters does not match the current cluster( in other words, the attribute replicationClusters does not contain broker.conf.cluster_name )
2. topic level policy modification and namespace level policy modification will trigger a task checkCompaction.
3. When a topic is loading, it is initialized by the following steps:
   a. Registered a listener to accept the topic level policy modification event
   b. Initialize the attribute replicationClusters
4. TopicPoliciesService will read all topic policies and try to update topic-level policies for every topic under this broker after the Broker restart.

After section 3-a, the task checkCompaction may be triggered by a topic-level policy modification(see section 3), but at this time the replicationClustershas not been initialized. It causes thesection 2` to delete the topic.

Detail steps of the issue: conditions-1

Detail steps of the issue: conditions-2

When there is more than one broker under your cluster, the probability of shedding topics to a new broker(when it has just started right now) is low, so the probability is high if there is only 1 broker.

Workaround-1

Before you upgrade to the new version(which contains this fix), disable the feature topicLevelPoliciesEnabled, but it also makes all topic-level policies invalidate you set before.

Workaround-2

Before you upgrade to the new version(which contains this fix), do not restart brokers or modify any topic-level policies.

@poorbarcode
Thanks for your clarification.

But based on the background of Workaround-2, if the current cluster has the conditions to trigger this issue, you can refer to the operation here to upgrade; otherwise, it may still cause the topic to be deleted during the upgrade(or restart) process.

if the current cluster has the conditions to trigger this issue, you can refer to the #21653 (comment) to upgrade; otherwise, it may still cause the topic to be deleted during the upgrade(or restart) process.

Ah, I see. Pulsar reads all topic policies and calls topic.onUpdate for every topic when it starts. @crossoverJie Thanks ❤️

@crossoverJie I updated the comment above, please review it, thanks

@poorbarcode Great job, thanks for your supplement👍.