Fix memory leak in exception handling (Issue #8882) #9799
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dd72947 to
33c7a22
Compare
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #9799 +/- ##
==========================================
- Coverage 78.62% 78.62% -0.01%
==========================================
Files 153 153
Lines 19178 19201 +23
Branches 2540 2547 +7
==========================================
+ Hits 15078 15096 +18
- Misses 3809 3812 +3
- Partials 291 293 +2
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
- Enhanced traceback cleanup in celery/app/trace.py to prevent memory leaks - Added proper cleanup of ExceptionInfo objects and traceback references - Optimized traceback_clear() function by removing redundant f_locals access - Added comprehensive memory leak test suite in t/integration/test_memory_leak_8882.py - Fixed code quality issues: removed unused imports, cleaned whitespace, added noqa comments Memory usage improvement: 92% reduction (from ~70MB to ~0.6MB for 500 failing tasks) Addresses reference cycles that prevent garbage collection of traceback frames. All pre-commit hooks passing.
a749037 to
1ca9ff4
Compare
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Contributor
Author
|
Thanks @auvipy for adding the Copilot review and providing feedback. I've addressed the comments. Let me know if you have any other suggestions. Thanks! |
…neshs21/celery into fix-memory-leak-issue-8882
Member
|
btw unit tests are separated from integration tests |
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
…neshs21/celery into fix-memory-leak-issue-8882
Contributor
Author
Got it, Separated the unit and integration tests and moved them to their respective locations. |
auvipy
requested changes
Jul 5, 2025
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
auvipy
approved these changes
Jul 6, 2025
auvipy
approved these changes
Jul 6, 2025
auvipy
requested changes
Jul 6, 2025
This was referenced Jul 6, 2025
auvipy
approved these changes
Jul 7, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This PR fixes a critical memory leak in Celery's exception handling that was causing significant memory growth when tasks raise unhandled exceptions.
Problem
Issue #8882 reported that Celery tasks raising unhandled exceptions were causing memory leaks:
Closes #8882
This PR addresses a critical memory leak in Celery's exception handling by breaking reference cycles in tracebacks, ensuring frame locals are cleared, and adding comprehensive tests to verify the fix.