Skip to content

Conversation

@tw4l
Copy link
Member

@tw4l tw4l commented Dec 2, 2025

Fixes #3044

  • Use asyncio.create_task to send background job failure emails (tracking tasks in set in ops class to avoid garbage collection)
  • Update job in database before sending emails
  • Add try/except around email sending so failures are logged instead of throwing uncaught exception
  • Rename handle_replica_job_finished to handle_replica_job_succeeded to better reflect intent
  • Fix failed background job email template and ensure that job sent to it is JSON-serializable with `model_dump(mode="json")

Note: The latest commit should be dropped before merging but allows for easy testing of a failed replica job that will trigger an email.

Further job-related improvements will be handled in a future release. Some work toward that has been removed from this PR and saved as main...job-failure-improvements-extended

@tw4l tw4l force-pushed the job-failure-email-fix branch 5 times, most recently from 247d3ce to 365e1b7 Compare December 3, 2025 17:50
@tw4l tw4l marked this pull request as ready for review December 3, 2025 17:53
@tw4l tw4l requested a review from ikreymer December 3, 2025 17:56
{"$set": {"success": success, "finished": finished}},
)

if not success:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, yeah, good to do it in this order, first update db, then send email.

Copy link
Member

@ikreymer ikreymer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a minor logging fix, tested on dev and emails sending as expected now!

@ikreymer ikreymer merged commit 6f1c342 into main Dec 4, 2025
25 checks passed
@ikreymer ikreymer deleted the job-failure-email-fix branch December 4, 2025 01:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Background job failure emails are failing to send

3 participants