Skip to content

Conversation

@whtsky
Copy link
Contributor

@whtsky whtsky commented Jun 1, 2020

Note: Before submitting this pull request, please review our contributing
guidelines
.

Description

Fixes #4116

#4292 makes Celery respect Django's CONN_MAX_AGE settings, but was reverted in #5515.

This PR tries to respect Django's CONN_MAX_AGE while force close db connections after fork.

@lgtm-com
Copy link

lgtm-com bot commented Jun 1, 2020

This pull request introduces 5 alerts when merging 2db18a8 into 2479f95 - view on LGTM.com

new alerts:

  • 4 for Module is imported with 'import' and 'import from'
  • 1 for Non-exception in 'except' clause

@auvipy auvipy added this to the 4.4.x milestone Jun 1, 2020
@auvipy auvipy merged commit f0c9b40 into celery:master Jun 1, 2020
@whtsky whtsky deleted the django_CONN_MAX_AGE branch June 16, 2020 08:44
@auvipy
Copy link
Member

auvipy commented Jul 8, 2020

can you comment on #6216

danlamanna added a commit to danlamanna/celery that referenced this pull request Jul 15, 2025
This reverts commit f0c9b40.

This reverts PR celery#6134 and stops using the close_if_unusable_or_obsolete API since there are edge cases where it's
unable to detect if a connection if actually unusable. This is most obvious when Celery interrupts a query in
progress via a time limit handler. Django has marked this issue as
wontfix (https://code.djangoproject.com/ticket/30646). Since this is effectively an optimization for Celery that
can't be reliably used, Celery ought to close the connection after each task instead of trying to manage connections
in a way similar to how the Django application does.
danlamanna added a commit to danlamanna/celery that referenced this pull request Jul 21, 2025
This reverts commit f0c9b40.

This reverts PR celery#6134 and stops using the close_if_unusable_or_obsolete API since there are edge cases where it's
unable to detect if a connection if actually unusable. This is most obvious when Celery interrupts a query in
progress via a time limit handler. Django has marked this issue as
wontfix (https://code.djangoproject.com/ticket/30646). Since this is effectively an optimization for Celery that
can't be reliably used, Celery ought to close the connection after each task instead of trying to manage connections
in a way similar to how the Django application does.
danlamanna added a commit to danlamanna/celery that referenced this pull request Jul 21, 2025
This reverts commit f0c9b40.

This reverts PR celery#6134 and stops using the close_if_unusable_or_obsolete API since there are edge cases where it's
unable to detect if a connection if actually unusable. This is most obvious when Celery interrupts a query in
progress via a time limit handler. Django has marked this issue as
wontfix (https://code.djangoproject.com/ticket/30646). Since this is effectively an optimization for Celery that
can't be reliably used, Celery ought to close the connection after each task instead of trying to manage connections
in a way similar to how the Django application does.
auvipy pushed a commit to danlamanna/celery that referenced this pull request Aug 7, 2025
This reverts commit f0c9b40.

This reverts PR celery#6134 and stops using the close_if_unusable_or_obsolete API since there are edge cases where it's
unable to detect if a connection if actually unusable. This is most obvious when Celery interrupts a query in
progress via a time limit handler. Django has marked this issue as
wontfix (https://code.djangoproject.com/ticket/30646). Since this is effectively an optimization for Celery that
can't be reliably used, Celery ought to close the connection after each task instead of trying to manage connections
in a way similar to how the Django application does.
danlamanna added a commit to danlamanna/celery that referenced this pull request Aug 8, 2025
This reverts commit f0c9b40.

This reverts PR celery#6134 and stops using the close_if_unusable_or_obsolete API since there are edge cases where it's
unable to detect if a connection if actually unusable. This is most obvious when Celery interrupts a query in
progress via a time limit handler. Django has marked this issue as
wontfix (https://code.djangoproject.com/ticket/30646). Since this is effectively an optimization for Celery that
can't be reliably used, Celery ought to close the connection after each task instead of trying to manage connections
in a way similar to how the Django application does.
auvipy pushed a commit to danlamanna/celery that referenced this pull request Aug 9, 2025
This reverts commit f0c9b40.

This reverts PR celery#6134 and stops using the close_if_unusable_or_obsolete API since there are edge cases where it's
unable to detect if a connection if actually unusable. This is most obvious when Celery interrupts a query in
progress via a time limit handler. Django has marked this issue as
wontfix (https://code.djangoproject.com/ticket/30646). Since this is effectively an optimization for Celery that
can't be reliably used, Celery ought to close the connection after each task instead of trying to manage connections
in a way similar to how the Django application does.
auvipy pushed a commit that referenced this pull request Aug 10, 2025
* Revert "Use Django DB max age connection setting"

This reverts commit f0c9b40.

This reverts PR #6134 and stops using the close_if_unusable_or_obsolete API since there are edge cases where it's
unable to detect if a connection if actually unusable. This is most obvious when Celery interrupts a query in
progress via a time limit handler. Django has marked this issue as
wontfix (https://code.djangoproject.com/ticket/30646). Since this is effectively an optimization for Celery that
can't be reliably used, Celery ought to close the connection after each task instead of trying to manage connections
in a way similar to how the Django application does.

* Ensure django fixup never calls close_if_unusable_or_obsolete

This API can fail to close unusable connections in certain scenarios,
namely database failovers and ungraceful terminations (e.g. signal
handler for time limit exceeded tasks). This makes
close_if_unusable_or_obsolete adequate for HTTP request lifecycle
management but inappropriate for use within celery workers.

See also:
https://code.djangoproject.com/ticket/30646
https://forum.djangoproject.com/t/close-if-unusable-or-obsolete-fails-to-close-unusable-connections/41900

* Add test for close_cache
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Django celery fixup doesn't respect Django settings for PostgreSQL connections

2 participants