-
Notifications
You must be signed in to change notification settings - Fork 3.7k
[fix][broker] Fix cannot shutdown broker gracefully by admin api #24731
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[fix][broker] Fix cannot shutdown broker gracefully by admin api #24731
Conversation
lhotari
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Great catch! just a few minor comments about parameter naming
pulsar-broker/src/main/java/org/apache/pulsar/broker/PulsarService.java
Outdated
Show resolved
Hide resolved
pulsar-broker/src/main/java/org/apache/pulsar/broker/web/WebService.java
Outdated
Show resolved
Hide resolved
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #24731 +/- ##
============================================
- Coverage 74.27% 74.18% -0.09%
- Complexity 33569 33595 +26
============================================
Files 1896 1900 +4
Lines 148111 148377 +266
Branches 17164 17204 +40
============================================
+ Hits 110009 110079 +70
- Misses 29370 29500 +130
- Partials 8732 8798 +66
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
) (cherry picked from commit 4169395)
) (cherry picked from commit 4169395)
) (cherry picked from commit 4169395)
) (cherry picked from commit 4169395)
…che#24731) (cherry picked from commit 4169395)
…che#24731) (cherry picked from commit 4169395) (cherry picked from commit 79926b9)
…che#24731) (cherry picked from commit 4169395) (cherry picked from commit 79926b9)
…che#24731) (cherry picked from commit 4169395) (cherry picked from commit 8fd9638)
…che#24731) (cherry picked from commit 4169395) (cherry picked from commit 8fd9638)
…ntUnload Revise `BrokerServiceTest.testShutDownWithMaxConcurrentUnload`, since apache#24731 fixed the issue where the broker couldn’t be shut down via Pulsar Admin. The original test contains a race condition in its assertion. ```java admin.brokers().shutDownBrokerGracefully(1, false); Awaitility.await().atLeast(bundleNum - 1, TimeUnit.SECONDS).untilAsserted(() -> { assertEquals(pulsar.getBrokerService().getTopics().size(), 0); // race condition: pulsar.getBrokerService() is possibly be null because the broker is shutting down }); ```
Fixes #23982
Motivation
The graceful broker shutdown endpoint (
/admin/v2/brokers/shutdown) was experiencing a deadlock when called via the admin API. The issue occurred because:PulsarService.closeAsync()→WebService.close()The stack trace shows the thread stuck in
CountDownLatch.await()inFutureCallback.get(), waiting indefinitely for the shutdown to complete.Click to show full stacktrace
Modifications
Modified
PulsarService.closeAsync():closeAsync(boolean waitForStop)closeAsync()now callscloseAsync(true)for backward compatibilityFixed
BrokersBase.shutDownBrokerGracefully():pulsar().closeAsync(false)instead ofpulsar().closeAsync()Verifying this change
This change added tests and can be verified as follows:
testShutdownViaAdminApi()inPulsarServiceTest.javaDoes this pull request potentially affect one of the following parts:
If the box was checked, please highlight the changes
Documentation
docdoc-requireddoc-not-neededdoc-completeMatching PR in forked repository
PR in forked repository: