Sentry is a great tool for monitoring celery tasks, and alerting when they fail or don’t run on time. But it requires a bit of work to setup properly. Below is some sample code for setting up sentry monitoring of periodic tasks, followed by an explanation.
import math
import sentry_sdk
from celery import signals
from sentry_sdk import monitor
from sentry_sdk.integrations.celery import CeleryIntegration
@signals.beat_init.connect # if you use beats
@signals.celeryd_init.connect
def init_sentry(**kwargs):
sentry_sdk.init(
dsn=...,
integrations=[
CeleryIntegration(monitor_beat_tasks=False)
]
)
@signals.worker_shutdown.connect
@signals.task_postrun.connect
def flush_sentry(**kwargs):
sentry_sdk.flush(timeout=5)
def add_periodic_task(celery, schedule, task):
max_runtime = math.ceil(schedule * 4 / 60)
monitor_config = {
"recovery_threshold": 1,
"failure_issue_threshold": 10,
"checkin_margin": max_runtime,
"max_runtime": max_runtime,
"schedule": {
"type": "interval",
"value": math.ceil(schedule / 60.0)
"unit": "minute"
}
}
name = task.__name__
task = monitor(monitor_slug=name, monitor_config=monitor_config)(task)
celery.add_periodic_task(schedule, celery.task(task).s(), name=name)Initialize Sentry
The init_sentry function must be called before any tasks start executing. The sentry docs for celery recommend using the celeryd_init signal. And if you use celery beats for periodic task execution, then you also need to initialize on the beat_init signal.
Monitoring Beats Tasks
In this example, I’m setting monitor_beat_tasks=False to show how you can do manual monitoring. monitor_beat_tasks=True is much simpler, and doesn’t require any code like in add_periodic_task. But in my experience, it’s not reliable when using async celery functions. The automatic beats monitoring uses some celery signals that likely don’t get executed correctly under async conditions. But manual monitoring isn’t that hard with a function wrapper, as shown above.
Adding a Periodic Task
The add_periodic_task function takes a Celery instance, a periodic interval in seconds, and a function to execute. This function can be normal or async. It then does the following:
- Calculates a
max_runtimein minutes, so that sentry knows when a task has gone over time. This is also used forcheckin_margin, giving the task plenty of buffer time before an issue is created. You should adjust these according to your needs. - Creates a
monitor_configfor sentry, specifying the following:- schedule in minutes (rounded up, because sentry doesn’t handle schedules in seconds)
- the number of failures allowed before creating an issue (I put 10, but you should adjust as needed)
- how many successful checkins are required before the issue is marked as resolved (1 is the default, but adjust as needed)
- Wraps the function in the sentry monitor decorator, using the function’s name as the
monitor_slug. With default beats monitoring, the slug is set to the fullpackage.module.functionpath, which can be quite long and becomes hard to scan when you have many tasks. - Schedules the task in celery.
Sentry Flush
While this may not be strictly necessary, calling sentry_sdk.flush on the worker_shutdown and task_postrun signals ensures that events are sent to sentry when a celery task completes.
Monitoring your crons
Once this is all setup and running, you should be able to go to Insights > Crons in your sentry web UI, and see all your celery tasks. Double check your monitor settings to make sure they’re correct, then sit back and relax, while sentry keeps track of how your tasks are running.
