Skip to content

*: replace Celery with Dramatiq#13492

Merged
rissson merged 309 commits intomainfrom
celery-2-dramatiq
Jul 28, 2025
Merged

*: replace Celery with Dramatiq#13492
rissson merged 309 commits intomainfrom
celery-2-dramatiq

Conversation

@rissson
Copy link
Member

@rissson rissson commented Mar 12, 2025

Details

Old TODO:

  • fix the authentik_events.view_systemtask permission check for sync_status endpoints, and associated code
  • modify schedules reconciliation to atomically create/delete/update schedules on startup, instead of hoping outdated schedules get deleted
  • ak worker embedded in ak server (?) -> no, need to update doc to add ak worker in commands to run
  • UI
  • bootstrap tasks
  • convert sources/ldap tasks
  • convert lib/sync/outgoing tasks
  • tasks fixes, see todos

TODO when extracting the code to its package:

TODO that modify a bit the implementation:

  • messages:
    • Needs events for errors
    • needs translations -> follow-up PR probably
  • updating crontab doesn't update next_run
  • read only schedules -> wontdo
  • add utility to dispatch on startup, probably run_on_startup as an argument to a middleware dispatching the tasks on worker boot
  • schedules fixes, grep TODO
  • custom currenttask middleware for tests only with list instead of single task

TODO things to figure out:

  • outpost service connection schedule broken because on top class instead of subclass. Actually figure out if ScheduledModel should be on "lowest" class or if intermediary classes are fine, like for OutgoingSyncProvider
  • figure out what is going on in outpost_post_save and things, and how to handle rel_obj in there
  • re-introduce "meta" tasks dispatching tasks for each object, for performance reasons
    • ssf
    • sync outgoing
    • events notifications send
  • workers API
  • metrics
  • task cancellation? -> wontdo
  • retry failed tasks
  • handle results
  • container healthcheck

TODO cleanup:

  • timeout for all tasks (+ tests)
  • description for all tasks (+ tests)
  • check that timelimit is miliseconds everywhere
  • check that tasksmodels (i.e. what is put in rel_obj) inherit from TaskModel for task deletion after model deletion
  • reset migrations
  • find out why tests don't run in CI
  • cleanup testing things
  • move metricsmiddleware to package

TODO frontend:

  • frontend hide successful tasks (do we hide warning? probably not)
  • frontend updates to show status and things
  • task retry
  • detail pages to show tasks and schedules
  • sync status dashbooard card

TODO docs:

  • find out what docs to change
  • dev docs
  • config
  • worker and system tasks page
  • release notes
    • authentik_admin_workers metric -> authentik_tasks_workers

TODO deploy:

  • update Helm chart prom rules
  • update Grafana dashboard

closes #15652
closes #14824
closes #14818
closes #9252
closes #13147
closes #6815
closes #9976
closes #6839


Checklist

  • Local tests pass (ak test authentik/)
  • The code has been formatted (make lint-fix)

If an API change has been made

  • The API schema has been updated (make gen-build)

If changes to the frontend have been made

  • The code has been formatted (make web)

If applicable

  • The documentation has been updated
  • The documentation has been formatted (make website)

@rissson rissson self-assigned this Mar 12, 2025
@rissson rissson requested a review from a team as a code owner March 12, 2025 17:08
@netlify
Copy link

netlify bot commented Mar 12, 2025

Deploy Preview for authentik-storybook canceled.

Name Link
🔨 Latest commit 93eeda6
🔍 Latest deploy log https://app.netlify.com/projects/authentik-storybook/deploys/6887874f90a63f00086b28a4

@netlify
Copy link

netlify bot commented Mar 12, 2025

Deploy Preview for authentik-docs canceled.

Name Link
🔨 Latest commit 93eeda6
🔍 Latest deploy log https://app.netlify.com/projects/authentik-docs/deploys/6887874f6f97e10008550073

@rissson
Copy link
Member Author

rissson commented Mar 12, 2025

Superseeds #13436 #9607 #6845

@rissson rissson marked this pull request as draft March 12, 2025 17:10
@codecov
Copy link

codecov bot commented Mar 24, 2025

Codecov Report

❌ Patch coverage is 80.86459% with 301 lines in your changes missing coverage. Please review.
✅ Project coverage is 92.70%. Comparing base (3ae2831) to head (93eeda6).
⚠️ Report is 3 commits behind head on main.
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
authentik/tasks/middleware.py 71.55% 33 Missing ⚠️
authentik/lib/sync/outgoing/tasks.py 69.07% 30 Missing ⚠️
authentik/tasks/forks.py 25.00% 21 Missing ⚠️
authentik/lib/sync/outgoing/api.py 28.57% 20 Missing ⚠️
authentik/sources/kerberos/api/source.py 27.27% 16 Missing ⚠️
authentik/tasks/api/tasks.py 75.38% 16 Missing ⚠️
authentik/tasks/schedules/api.py 73.77% 16 Missing ⚠️
authentik/sources/ldap/api.py 25.00% 15 Missing ⚠️
authentik/outposts/signals.py 80.43% 9 Missing ⚠️
authentik/tasks/schedules/scheduler.py 47.05% 9 Missing ⚠️
... and 36 more
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #13492      +/-   ##
==========================================
- Coverage   93.17%   92.70%   -0.48%     
==========================================
  Files         831      831              
  Lines       44251    44469     +218     
==========================================
- Hits        41230    41224       -6     
- Misses       3021     3245     +224     
Flag Coverage Δ
e2e 46.79% <53.44%> (+0.22%) ⬆️
integration 23.61% <40.54%> (+0.16%) ⬆️
unit 90.82% <80.48%> (-0.45%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

rissson added 12 commits April 2, 2025 11:47
Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>
Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>
Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>
Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>
Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>
Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>
Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>
Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>
Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>
Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>
@rissson rissson mentioned this pull request Jun 3, 2025
6 tasks
rissson added 3 commits June 3, 2025 18:25
Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>
Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>
Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>
Copy link
Contributor

@tanberry tanberry left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for answering my questions from my first review; I'd still like to add an image of the retry icon but not really required. I re-edited a few areas based on your answers, please confirm that they look right, and there is one question, but yeah, LGTM!!

rissson and others added 14 commits July 24, 2025 13:17
Co-authored-by: Tana M Berry <tanamarieberry@yahoo.com>
Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>
Co-authored-by: Tana M Berry <tanamarieberry@yahoo.com>
Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>
Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>
Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>
Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>
This reverts commit 70d9d53.
This reverts commit 52bbc82.
Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>
Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>
Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>
Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>
Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>
Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>
Copy link
Member

@BeryJu BeryJu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

need a way to run the bootstrap blueprint when AUTHENTIK_BOOTSTRAP_* is set, and ideally a way to run the worker with ak server in development, otherwise LGTM

rissson added 5 commits July 28, 2025 15:10
… defined

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>
Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>
Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>
Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>
Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>
@rissson rissson enabled auto-merge (squash) July 28, 2025 14:54
@rissson rissson merged commit 29f20a4 into main Jul 28, 2025
107 of 109 checks passed
@rissson rissson deleted the celery-2-dramatiq branch July 28, 2025 15:00
kensternberg-authentik added a commit that referenced this pull request Aug 6, 2025
* main: (91 commits)
  *: replace Celery with Dramatiq (#13492)
  website/docs: stages/mtls: Clean up stage configuration section (#15753)
  translate: Updates for file locale/en/LC_MESSAGES/django.po in es (#15818)
  website: bump the eslint group in /website with 2 updates (#15805)
  web: bump typedoc from 0.28.7 to 0.28.8 in /packages/esbuild-plugin-live-reload (#15809)
  core: bump axllent/mailpit from v1.27.1 to v1.27.2 in /tests/e2e (#15813)
  web: bump the rollup group across 1 directory with 4 updates (#15806)
  web: bump the eslint group across 3 directories with 2 updates (#15808)
  lifecycle/aws: bump cross-env from 7.0.3 to 10.0.0 in /lifecycle/aws (#15807)
  web: bump ts-pattern from 5.7.1 to 5.8.0 in /web (#15810)
  web: bump @sentry/browser from 9.41.0 to 9.42.0 in /web in the sentry group across 1 directory (#15811)
  web: Add support for placeholder in <ak-text-input> (#15795)
  providers/rac: fix incorrect caching (#15779)
  root: support for custom postgresql connection options (#15577)
  website: bump the build group in /website with 3 updates (#15784)
  web: bump @sentry/browser from 9.40.0 to 9.41.0 in /web in the sentry group across 1 directory (#15785)
  core: bump astral-sh/uv from 0.8.2 to 0.8.3 (#15786)
  lifecycle/aws: bump aws-cdk from 2.1021.0 to 2.1022.0 in /lifecycle/aws (#15787)
  core: bump twilio from 9.6.5 to 9.7.0 (#15788)
  revert: web: Font fixes (#15581) (#15789)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

deploy_me Deploy the PR changes on a test environment

Projects

None yet

4 participants