KAFKA-9265: Fix kafka.log.Log instance leak on log deletion#7773
Merged
Conversation
KAFKA-8448 fixes problem with similar leak. The Log objects are being held in ScheduledExecutor PeriodicProducerExpirationCheck callback. The fix in KAFKA-8448 was to change the policy of ScheduledExecutor to remove the scheduled task when it gets canceled (by calling setRemoveOnCancelPolicy(true)). This works when a log is closed using close() method. But when a log is deleted either when the topic gets deleted or when the rebalancing operation moves the replica away from broker, the delete() operation is invoked. Log.delete() doesn't close the pending scheduled task and that leaks Log instance. Fix is to close the scheduled task in the Log.delete() method too. Tested with and without this fix and Log instances are no longer leaked and scheduled tasks are gone once log is deleted.
hachikuji
reviewed
Dec 3, 2019
This change updates delete to call close so that the functionality gets reused. Otherwise both close and delete were doing different things and missing on what one or other does. This change consolidates that logic.
hachikuji
reviewed
Dec 4, 2019
hachikuji
left a comment
Contributor
There was a problem hiding this comment.
Thanks for the updates. Left a few small suggestions.
hachikuji
approved these changes
Dec 4, 2019
hachikuji
left a comment
Contributor
There was a problem hiding this comment.
LGTM. Thanks for the fix!
Member
|
retest this please |
Member
|
1 job passed and 2 failed with known flakes:
|
ijuma
pushed a commit
that referenced
this pull request
Dec 4, 2019
KAFKA-8448 fixes problem with similar leak. The Log objects are being held in ScheduledExecutor PeriodicProducerExpirationCheck callback. The fix in KAFKA-8448 was to change the policy of ScheduledExecutor to remove the scheduled task when it gets canceled (by calling setRemoveOnCancelPolicy(true)). This works when a log is closed using close() method. But when a log is deleted either when the topic gets deleted or when the rebalancing operation moves the replica away from broker, the delete() operation is invoked. Log.delete() doesn't close the pending scheduled task and that leaks Log instance. Fix is to close the scheduled task in the Log.delete() method too. Reviewers: Jason Gustafson <jason@confluent.io>, Ismael Juma <ismael@juma.me.uk>
ijuma
pushed a commit
that referenced
this pull request
Dec 4, 2019
KAFKA-8448 fixes problem with similar leak. The Log objects are being held in ScheduledExecutor PeriodicProducerExpirationCheck callback. The fix in KAFKA-8448 was to change the policy of ScheduledExecutor to remove the scheduled task when it gets canceled (by calling setRemoveOnCancelPolicy(true)). This works when a log is closed using close() method. But when a log is deleted either when the topic gets deleted or when the rebalancing operation moves the replica away from broker, the delete() operation is invoked. Log.delete() doesn't close the pending scheduled task and that leaks Log instance. Fix is to close the scheduled task in the Log.delete() method too. Reviewers: Jason Gustafson <jason@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Member
|
Merged to trunk and cherry-picked to 2.4 and 2.3 branches. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
KAFKA-8448 fixes problem with similar leak. The Log objects are being
held in ScheduledExecutor PeriodicProducerExpirationCheck callback. The
fix in KAFKA-8448 was to change the policy of ScheduledExecutor to
remove the scheduled task when it gets canceled (by calling
setRemoveOnCancelPolicy(true)).
This works when a log is closed using close() method. But when a log is
deleted either when the topic gets deleted or when the rebalancing
operation moves the replica away from broker, the delete() operation is
invoked. Log.delete() doesn't close the pending scheduled task and that
leaks Log instance.
Fix is to close the scheduled task in the Log.delete() method too.
Tested with and without this fix and Log instances are no longer leaked
and scheduled tasks are gone once log is deleted.
More detailed description of your change,
if necessary. The PR title and PR message become
the squashed commit message, so use a separate
comment to ping reviewers.
Summary of testing strategy (including rationale)
for the feature or bug fix. Unit and/or integration
tests are expected for any behaviour change and
system tests should be considered for larger changes.
Committer Checklist (excluded from commit message)