Skip to content

multiple workers: repeated tasks are scheduled by each worker instead of just once #6815

@a-gerhard

Description

@a-gerhard

Describe the bug
We've got 10 workers and 1 server in our setup. ldap_sync_all is scheduled 10 times in each 2 hour window (to be more accurate, 10 times within 1 hour after each full even hour). This looks like each worker is creating its own schedule (crontab(minute=fqdn_rand("sources_ldap_sync"), hour="*/2") according to `sources/ldap/settings.py).

This may affect other schedules as well

To Reproduce
Steps to reproduce the behavior:

  1. create a setup with multiple workers (e.g., using docker-compose replication)
  2. use celery flower to observe tasks
  3. wait until you see the schedules

Expected behavior
The task is scheduled according to its schedule, and only once per time interval. Not once per worker, but once in total.

Screenshots
Some tasks failed because redis ran out of memory due to the amount of data that was synchronized, which also caused some other issues.
image

Version and Deployment (please complete the following information):

  • authentik version: 2023.8.2
  • Deployment: docker-compose

Additional context
This has a severe impact on the issues I described in #6810

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingbug/confirmedConfirmed bugs

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions