Skip to content

Worker will not re-connect anymore due to Celery 6.0 and above issue #7521

@samip5

Description

@samip5

Describe the bug
Worker might not re-connect if it fails enough of times anymore.

To Reproduce
Steps to reproduce the behavior:

  1. Deploy Authentik
  2. Yeet Redis deployment for a moment
  3. See error
  4. De-deploy Redis
  5. See issue with Worker not re-connecting

Expected behavior
I would expect it to re-connect.

Logs

{"event": "/ak-root/venv/lib/python3.11/site-packages/celery/worker/consumer/consumer.py:391: CPendingDeprecationWarning: \nIn Celery 5.1 we introduced an optional breaking change which\non connection loss cancels all currently executed tasks with late acknowledgement enabled.\nThese tasks cannot be acknowledged as the connection is gone, and the tasks are automatically redelivered\nback to the queue. You can enable this behavior using the worker_cancel_long_running_tasks_on_connection_loss\nsetting. In Celery 5.1 it is set to False by default. The setting will be set to True by default in Celery 6.0.\n\n  warnings.warn(CANCEL_TASKS_BY_DEFAULT, CPendingDeprecationWarning)\n", "level": "warning", "logger": "py.warnings", "timestamp": 1699070849.2876282}
{"event": "/ak-root/venv/lib/python3.11/site-packages/celery/worker/consumer/consumer.py:507: CPendingDeprecationWarning: The broker_connection_retry configuration setting will no longer determine\nwhether broker connection retries are made during startup in Celery 6.0 and above.\nIf you wish to retain the existing behavior for retrying connections on startup,\nyou should set broker_connection_retry_on_startup to True.\n  warnings.warn(\n", "level": "warning", "logger": "py.warnings", "timestamp": 1699070849.4042337}
{"event": "consumer: Cannot connect to redis://redis.databases.svc.cluster.local:6379/0: Error 111 connecting to redis.databases.svc.cluster.local:6379. Connection refused..\nTrying again in 2.00 seconds... (1/100)\n", "level": "error", "logger": "celery.worker.consumer.consumer", "timestamp": 1699070849.4176264}
{"event": "consumer: Cannot connect to redis://redis.databases.svc.cluster.local:6379/0: Error 1 connecting to redis.databases.svc.cluster.local:6379. Operation not permitted..\nTrying again in 4.00 seconds... (2/100)\n", "level": "error", "logger": "celery.worker.consumer.consumer", "timestamp": 1699070851.7179375}
{"event": "consumer: Cannot connect to redis://redis.databases.svc.cluster.local:6379/0: Error 1 connecting to redis.databases.svc.cluster.local:6379. Operation not permitted..\nTrying again in 6.00 seconds... (3/100)\n", "level": "error", "logger": "celery.worker.consumer.consumer", "timestamp": 1699070855.7518742}
{"event": "/ak-root/venv/lib/python3.11/site-packages/celery/worker/consumer/consumer.py:507: CPendingDeprecationWarning: The broker_connection_retry configuration setting will no longer determine\nwhether broker connection retries are made during startup in Celery 6.0 and above.\nIf you wish to retain the existing behavior for retrying connections on startup,\nyou should set broker_connection_retry_on_startup to True.\n  warnings.warn(\n", "level": "warning", "logger": "py.warnings", "timestamp": 1699070862.060859}

Version and Deployment (please complete the following information):

  • authentik version: 2023.10.2
  • Deployment: Helm

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions