Motivation

The topic loading timeout starts at loadOrCreatePersistentTopic:

However, there could be two asynchronous tasks before that:

1. Check if the topic exists

2. Get the topic policies (__change_events topic will be replayed)

The 1st task should be okay in most time, but the 2nd task could be time-consuming and risky. I have seen when replaying __change_events topic was stuck in production env. If this case happens, there is no metric or log and it can be only aware from client side.

P.S. as the comment shows, it actually does not make sense to execute the 2nd task here but I want to improve the topic loading process in other PRs.

The topic loading metric (pulsar_topic_load_times) starts even later at createPersistent0:

It does not even include the time when the topic loading is in BrokerService#pendingTopicLoadingQueue, which happens when the number of pending topic loading tasks exceeds the maxConcurrentTopicLoadRequest config.

Modifications

• Refactor TopicLoadingContext to add latency info and create this context from the beginning before checkNonPartitionedTopicExists, so the latency could include the time before createPersistent0
• Log latency when the topic is loaded successfully or fails to load, specially, if the topic has been added to the pending queue due to maxConcurrentTopicLoadRequest limitation, log the pending time as well

Verifying this change

• Make sure that the change passes the CI checks.

Here are example outputs from BrokerServiceTest.testConcurrentLoadTopicExceedLimitShouldNotBeAutoCreated

Created topic persistent://prop/concurrentLoad/my-topic_0 - dedup is disabled (latency: 156 ms)
Created topic persistent://prop/concurrentLoad/my-topic_1 - dedup is disabled (latency: 20 ms)
Created topic persistent://prop/concurrentLoad/my-topic_2 - dedup is disabled (latency: 20 ms)
Created topic persistent://prop/concurrentLoad/my-topic_0 - dedup is disabled (latency: 4 ms)
Created topic persistent://prop/concurrentLoad/my-topic_1 - dedup is disabled (latency: 7 ms (queued: 3))
Created topic persistent://prop/concurrentLoad/my-topic_2 - dedup is disabled (latency: 12 ms (queued: 7))

queued means how long it stays in pendingTopicLoadingQueue.

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

• Dependencies (add or upgrade a dependency)
• The public API
• The schema
• The default values of configurations
• The threading model
• The binary protocol
• The REST endpoints
• The admin CLI options
• The metrics
• Anything that affects deployment

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

Matching PR in forked repository

PR in forked repository: