Fix Django worker recursion bug + defensive checks for pool_cls.module #10048

maycuatroi1 · 2026-01-03T10:23:08Z

Description

This PR builds on top of #10045 by @bruunotrindade to add defensive checks for pool_cls.__module__ access in additional locations throughout the codebase.

Root Cause (from #10045)

DjangoWorkerFixup.__init__ was creating a WorkController instance via the worker or WorkController(app) pattern. This created a circular dependency where:

WorkController.__init__ triggers the worker_init signal
DjangoFixup.on_worker_init creates/accesses DjangoWorkerFixup
DjangoWorkerFixup.__init__ tries to create another WorkController
Infinite recursion ensues

Additionally, pool_cls can be a string (e.g., 'prefork', 'solo', 'gevent') instead of a class object, which causes AttributeError: 'str' object has no attribute '__module__' when trying to access pool_cls.__module__.

Changes in this PR

This PR includes all changes from #10045 plus:

celery/contrib/testing/worker.py: Added defensive check for pool_cls being a string in TestWorkController.__init__
celery/worker/components.py: Added defensive check for pool_cls being a string in Beat.create
New unit tests: Added tests to verify the defensive checks work correctly when pool_cls is a string

Pattern Applied

# Before (can raise AttributeError if pool_cls is a string)
if w.pool_cls.__module__.endswith(('gevent', 'eventlet')):

# After (handles both string and class pool_cls)
pool_module = w.pool_cls if isinstance(w.pool_cls, str) else w.pool_cls.__module__
if pool_module.endswith(('gevent', 'eventlet')):

Testing

All existing unit tests pass
Added 2 new tests for Beat.create with string pool_cls
Pre-commit hooks pass (flake8, mypy, isort, etc.)
Test coverage maintained

Related Issues

AttributeError: 'str' object has no attribute '__module__' in django.py fixups when using Flower #10042 - Original issue reporting RecursionError
Fix recursive WorkController instantiation in DjangoWorkerFixup + AttributeError when pool_cls is a string #10045 - Original PR by @bruunotrindade fixing the root cause
reliable prefork detection #10023 - Previous attempts to fix the issue

Acknowledgments

Thanks to @bruunotrindade for identifying and fixing the root cause in #10045. This PR extends that fix to cover additional edge cases.

- Remove worker parameter from DjangoWorkerFixup.__init__ - Set worker via on_worker_init callback instead - Add None check for worker.pool_cls to prevent AttributeError - Add regression test to prevent future recursion bugs Fixes recursive instantiation issue where DjangoWorkerFixup(app) would create WorkController(app), which in turn would create another DjangoWorkerFixup, leading to infinite recursion. The worker instance is now properly set via the on_worker_init signal callback, avoiding the circular dependency.

Extends the fix from celery#10045 by adding defensive checks for when pool_cls is a string instead of a class in two additional locations: - celery/contrib/testing/worker.py: TestWorkController.__init__ - celery/worker/components.py: Beat.create This prevents AttributeError: 'str' object has no attribute '__module__' when pool_cls is passed as a string (e.g., 'prefork', 'gevent', 'eventlet'). Also adds unit tests to verify the defensive checks work correctly.

for more information, see https://pre-commit.ci

auvipy

please check the unit test failures

codecov · 2026-01-03T14:05:21Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 88.47%. Comparing base (6a43c84) to head (34882eb).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #10048      +/-   ##
==========================================
+ Coverage   88.43%   88.47%   +0.04%     
==========================================
  Files         153      153              
  Lines       19373    19375       +2     
  Branches     2228     2228              
==========================================
+ Hits        17132    17142      +10     
+ Misses       1941     1934       -7     
+ Partials      300      299       -1

Flag	Coverage Δ
unittests	`88.44% <100.00%> (+0.04%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

auvipy · 2026-01-04T07:16:27Z

may be we can consider the remaining changes here as minor improvements

Copilot

Pull request overview

This PR extends #10045 to add defensive checks for accessing pool_cls.__module__ in two additional locations throughout the codebase, preventing AttributeError when pool_cls is a string instead of a class object.

Key Changes:

Added defensive checks in Beat.create and TestWorkController.__init__ to handle string pool_cls values
Added two unit tests for Beat.create to verify the fix works with string pool types
Applies the same pattern used in the Django fixup to prevent AttributeErrors

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File	Description
`celery/worker/components.py`	Added defensive check in `Beat.create` to handle string `pool_cls` before checking for green pools
`celery/contrib/testing/worker.py`	Added defensive check in `TestWorkController.__init__` to handle string `pool_cls` before checking for prefork pool
`t/unit/worker/test_components.py`	Added two unit tests verifying `Beat.create` raises `ImproperlyConfigured` when `pool_cls` is string 'gevent' or 'eventlet'

celery/contrib/testing/worker.py

auvipy

is it possible to improve test coverage?

maycuatroi1 · 2026-01-05T15:03:16Z

@auvipy Added unit tests for TestWorkController with string pool_cls to improve test coverage. Please review.

auvipy · 2026-01-05T15:30:00Z

can you please check why the tests are failing with new changes?

Add tests to verify defensive handling of string pool_cls in TestWorkController.__init__, addressing reviewer feedback.

maycuatroi1 · 2026-01-05T17:58:27Z

@auvipy Fixed a test pollution issue discovered during local CI testing.

Root cause: The test_init_with_string_pool_cls_prefork test was calling tblib.pickling_support.install() (via TestWorkController.__init__), which modifies global pickling behavior and caused test_retry_with_unpickleable_exception to fail when running the full test suite.

Fix: Added patch.dict('sys.modules', {'tblib': None, 'tblib.pickling_support': None}) to prevent tblib from being imported during the test.

Note: Force pushed to amend the previous commit with this fix. Apologies for not creating a separate commit.

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

bruunotrindade and others added 5 commits December 31, 2025 12:33

fix: improve prefork detection and add integration test

84c436e

fix: Copilot recommendations

d01ed94

[pre-commit.ci] auto fixes from pre-commit.com hooks

7176bdf

for more information, see https://pre-commit.ci

auvipy requested changes Jan 3, 2026

View reviewed changes

Merge branch 'main' into fix-django-worker-recursion-bug-v2

7b8a407

auvipy requested a review from Copilot January 4, 2026 07:19

Copilot started reviewing on behalf of auvipy January 4, 2026 07:20 View session

Copilot AI reviewed Jan 4, 2026

View reviewed changes

celery/contrib/testing/worker.py Show resolved Hide resolved

Nusnus added 2 commits January 4, 2026 14:18

Merge branch 'main' into fix-django-worker-recursion-bug-v2

20a1c56

Merge branch 'main' into fix-django-worker-recursion-bug-v2

237ecd7

auvipy added this to the 5.6.x milestone Jan 4, 2026

auvipy requested changes Jan 5, 2026

View reviewed changes

maycuatroi1 requested a review from auvipy January 5, 2026 15:04

Add unit tests for TestWorkController with string pool_cls

34882eb

Add tests to verify defensive handling of string pool_cls in TestWorkController.__init__, addressing reviewer feedback.

maycuatroi1 force-pushed the fix-django-worker-recursion-bug-v2 branch from a7d89b7 to 34882eb Compare January 5, 2026 17:48

auvipy requested a review from Copilot January 5, 2026 18:16

Copilot started reviewing on behalf of auvipy January 5, 2026 18:16 View session

Copilot AI reviewed Jan 5, 2026

View reviewed changes

auvipy approved these changes Jan 5, 2026

View reviewed changes

auvipy merged commit 8ea903b into celery:main Jan 5, 2026
377 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Fix Django worker recursion bug + defensive checks for pool_cls.module #10048

Fix Django worker recursion bug + defensive checks for pool_cls.module #10048

maycuatroi1 commented Jan 3, 2026

Uh oh!

auvipy left a comment

Uh oh!

codecov bot commented Jan 3, 2026 •

edited

Loading

Uh oh!

auvipy commented Jan 4, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

auvipy left a comment

Uh oh!

maycuatroi1 commented Jan 5, 2026

Uh oh!

auvipy commented Jan 5, 2026

Uh oh!

maycuatroi1 commented Jan 5, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Fix Django worker recursion bug + defensive checks for pool_cls.__module__ #10048

Fix Django worker recursion bug + defensive checks for pool_cls.__module__ #10048

Conversation

maycuatroi1 commented Jan 3, 2026

Description

Root Cause (from #10045)

Changes in this PR

Pattern Applied

Testing

Related Issues

Acknowledgments

Uh oh!

auvipy left a comment

Choose a reason for hiding this comment

Uh oh!

codecov bot commented Jan 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

auvipy commented Jan 4, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Key Changes:

Reviewed changes

Uh oh!

Uh oh!

auvipy left a comment

Choose a reason for hiding this comment

Uh oh!

maycuatroi1 commented Jan 5, 2026

Uh oh!

auvipy commented Jan 5, 2026

Uh oh!

maycuatroi1 commented Jan 5, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Fix Django worker recursion bug + defensive checks for pool_cls.module #10048

Fix Django worker recursion bug + defensive checks for pool_cls.module #10048

codecov bot commented Jan 3, 2026 •

edited

Loading