config_from_object causes infinite recursion when given a nested module that does not exist #8517
Description
Checklist
- I have verified that the issue exists against the
mainbranch of Celery. - This has already been asked to the discussions forum first.
- I have read the relevant section in the
contribution guide
on reporting bugs. - I have checked the issues list
for similar or identical bug reports. - I have checked the pull requests list
for existing proposed fixes. - I have checked the commit log
to find out if the bug was already fixed in the main branch. - I have included all related issues and possible duplicate issues
in this issue (If there are none, check this box anyway).
Mandatory Debugging Information
- I have included the output of
celery -A proj reportin the issue.
(if you are not able to do this, then at least specify the Celery
version affected). - I have verified that the issue exists against the
mainbranch of Celery. - I have included the contents of
pip freezein the issue. - I have included all the versions of all the external dependencies required
to reproduce this bug.
Optional Debugging Information
- I have tried reproducing the issue on more than one Python version
and/or implementation. - I have tried reproducing the issue on more than one message broker and/or
result backend. - I have tried reproducing the issue on more than one version of the message
broker and/or result backend. - I have tried reproducing the issue on more than one operating system.
- I have tried reproducing the issue on more than one workers pool.
- I have tried reproducing the issue with autoscaling, retries,
ETA/Countdown & rate limits disabled. - I have tried reproducing the issue after downgrading
and/or upgrading Celery and its dependencies.
Related Issues and Possible Duplicates
Related Issues
- None
Possible Duplicates
- None
Environment & Settings
Celery version: 5.3.4 (emerald-rush)
If the line causing the bug is left in, then celery -A app.tasks report fails with the same recursion error, but otherwise the result is:
celery report Output:
(celery-test) braedon ~/meltano/celery-test/recursion-bug $ celery -A app.tasks report
software -> celery:5.3.4 (emerald-rush) kombu:5.3.2 py:3.10.12
billiard:4.1.0 py-amqp:5.1.1
platform -> system:Darwin arch:64bit
kernel version:22.6.0 imp:CPython
loader -> celery.loaders.app.AppLoader
settings -> transport:amqp results:disabled
broker_url: 'amqp://guest:********@localhost:5672//'
deprecated_settings: None
Steps to Reproduce
Required Dependencies
- Minimal Python Version: Unknown
- Minimal Celery Version: Unknown
- Minimal Kombu Version: Unknown
- Minimal Broker Version: Unknown
- Minimal Result Backend Version: Unknown
- Minimal OS and/or Kernel Version: Unknown
- Minimal Broker Client Version: Unknown
- Minimal Result Backend Client Version: Unknown
Python Packages
pip freeze Output:
amqp==5.1.1
appnope==0.1.3
asttokens==2.4.0
backcall==0.2.0
billiard==4.1.0
boto3==1.28.46
botocore==1.31.46
celery==5.3.4
click==8.1.7
click-didyoumean==0.3.0
click-plugins==1.1.1
click-repl==0.3.0
comm==0.1.4
debugpy==1.8.0
decorator==5.1.1
exceptiongroup==1.1.3
executing==1.2.0
ipykernel==6.25.2
ipython==8.15.0
jedi==0.19.0
jmespath==1.0.1
jupyter_client==8.3.1
jupyter_core==5.3.1
kombu==5.3.2
matplotlib-inline==0.1.6
nest-asyncio==1.5.7
packaging==23.1
parso==0.8.3
pexpect==4.8.0
pickleshare==0.7.5
platformdirs==3.10.0
prompt-toolkit==3.0.39
psutil==5.9.5
psycopg2==2.9.7
ptyprocess==0.7.0
pure-eval==0.2.2
pycurl==7.45.2
Pygments==2.16.1
python-dateutil==2.8.2
pyzmq==25.1.1
s3transfer==0.6.2
six==1.16.0
SQLAlchemy==2.0.20
stack-data==0.6.2
tornado==6.3.3
traitlets==5.9.0
typing_extensions==4.7.1
tzdata==2023.3
urllib3==1.26.16
vine==5.0.0
wcwidth==0.2.6
Other Dependencies
Details
N/A
Minimally Reproducible Test Case
Details
tree:
app
├── __init__.py
└── tasks.pytasks.py:
from celery import Celery
app = Celery("tasks", broker="amqp://guest@localhost//")
@app.task
def add(x, y):
return x + y
app.config_from_object("app.fake_module")init.py is empty.
Expected Behavior
I expect a ModuleNotFoundError, such as what happens when the top-level module does not exist (e.g. app.config_from_object("fake_module"))
Actual Behavior
Instead, there's an infinite recursion issue:
celery -A app.tasks worker --loglevel=INFO
Traceback (most recent call last):
File "/Users/braedon/.pyenv/versions/celery-test/bin/celery", line 8, in <module>
sys.exit(main())
File "/Users/braedon/.pyenv/versions/3.10.12/envs/celery-test/lib/python3.10/site-packages/celery/__main__.py", line 15, in main
sys.exit(_main())
File "/Users/braedon/.pyenv/versions/3.10.12/envs/celery-test/lib/python3.10/site-packages/celery/bin/celery.py", line 236, in main
return celery(auto_envvar_prefix="CELERY")
File "/Users/braedon/.pyenv/versions/3.10.12/envs/celery-test/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
return self.main(*args, **kwargs)
File "/Users/braedon/.pyenv/versions/3.10.12/envs/celery-test/lib/python3.10/site-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
File "/Users/braedon/.pyenv/versions/3.10.12/envs/celery-test/lib/python3.10/site-packages/click/core.py", line 1686, in invoke
sub_ctx = cmd.make_context(cmd_name, args, parent=ctx)
File "/Users/braedon/.pyenv/versions/3.10.12/envs/celery-test/lib/python3.10/site-packages/click/core.py", line 943, in make_context
self.parse_args(ctx, args)
File "/Users/braedon/.pyenv/versions/3.10.12/envs/celery-test/lib/python3.10/site-packages/click/core.py", line 1408, in parse_args
value, args = param.handle_parse_result(ctx, opts, args)
File "/Users/braedon/.pyenv/versions/3.10.12/envs/celery-test/lib/python3.10/site-packages/click/core.py", line 2400, in handle_parse_result
value = self.process_value(ctx, value)
File "/Users/braedon/.pyenv/versions/3.10.12/envs/celery-test/lib/python3.10/site-packages/click/core.py", line 2362, in process_value
value = self.callback(ctx, self, value)
File "/Users/braedon/.pyenv/versions/3.10.12/envs/celery-test/lib/python3.10/site-packages/celery/bin/worker.py", line 158, in <lambda>
value: value or ctx.obj.app.conf.worker_state_db,
File "/Users/braedon/.pyenv/versions/3.10.12/envs/celery-test/lib/python3.10/site-packages/celery/utils/collections.py", line 111, in __getattr__
return self[k]
File "/Users/braedon/.pyenv/versions/3.10.12/envs/celery-test/lib/python3.10/site-packages/celery/utils/collections.py", line 391, in __getitem__
return getitem(k)
File "/Users/braedon/.pyenv/versions/3.10.12/envs/celery-test/lib/python3.10/site-packages/celery/utils/collections.py", line 249, in __getitem__
return mapping[_key]
File "/Users/braedon/.pyenv/versions/3.10.12/lib/python3.10/collections/__init__.py", line 1102, in __getitem__
if key in self.data:
File "/Users/braedon/.pyenv/versions/3.10.12/envs/celery-test/lib/python3.10/site-packages/celery/utils/collections.py", line 111, in __getattr__
return self[k]
File "/Users/braedon/.pyenv/versions/3.10.12/lib/python3.10/collections/__init__.py", line 1102, in __getitem__
if key in self.data:
File "/Users/braedon/.pyenv/versions/3.10.12/envs/celery-test/lib/python3.10/site-packages/celery/utils/collections.py", line 111, in __getattr__
return self[k]
...
...
...
File "/Users/braedon/.pyenv/versions/3.10.12/envs/celery-test/lib/python3.10/site-packages/celery/utils/collections.py", line 111, in __getattr__
return self[k]
File "/Users/braedon/.pyenv/versions/3.10.12/lib/python3.10/collections/__init__.py", line 1102, in __getitem__
if key in self.data:
File "/Users/braedon/.pyenv/versions/3.10.12/envs/celery-test/lib/python3.10/site-packages/kombu/utils/objects.py", line 31, in __get__
return super().__get__(instance, owner)
File "/Users/braedon/.pyenv/versions/3.10.12/lib/python3.10/functools.py", line 981, in __get__
val = self.func(instance)
File "/Users/braedon/.pyenv/versions/3.10.12/envs/celery-test/lib/python3.10/site-packages/celery/app/base.py", line 134, in data
return self.callback()
File "/Users/braedon/.pyenv/versions/3.10.12/envs/celery-test/lib/python3.10/site-packages/celery/app/base.py", line 977, in _finalize_pending_conf
conf = self._conf = self._load_config()
File "/Users/braedon/.pyenv/versions/3.10.12/envs/celery-test/lib/python3.10/site-packages/celery/app/base.py", line 987, in _load_config
self.loader.config_from_object(self._config_source)
File "/Users/braedon/.pyenv/versions/3.10.12/envs/celery-test/lib/python3.10/site-packages/celery/loaders/base.py", line 122, in config_from_object
obj = self._smart_import(obj, imp=self.import_from_cwd)
File "/Users/braedon/.pyenv/versions/3.10.12/envs/celery-test/lib/python3.10/site-packages/celery/loaders/base.py", line 142, in _smart_import
return imp(path)
File "/Users/braedon/.pyenv/versions/3.10.12/envs/celery-test/lib/python3.10/site-packages/celery/loaders/base.py", line 91, in import_from_cwd
return import_from_cwd(
File "/Users/braedon/.pyenv/versions/3.10.12/envs/celery-test/lib/python3.10/site-packages/celery/utils/imports.py", line 104, in import_from_cwd
return imp(module, package=package)
File "/Users/braedon/.pyenv/versions/3.10.12/envs/celery-test/lib/python3.10/site-packages/celery/loaders/base.py", line 88, in import_module
return importlib.import_module(module, package=package)
File "/Users/braedon/.pyenv/versions/3.10.12/lib/python3.10/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
File "<frozen importlib._bootstrap>", line 1002, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 945, in _find_spec
File "<frozen importlib._bootstrap_external>", line 1439, in find_spec
File "<frozen importlib._bootstrap_external>", line 1411, in _get_spec
File "<frozen importlib._bootstrap_external>", line 1572, in find_spec
File "<frozen importlib._bootstrap_external>", line 128, in _path_join
File "<frozen importlib._bootstrap_external>", line 128, in <listcomp>
RecursionError: maximum recursion depth exceeded while calling a Python object