-
-
Notifications
You must be signed in to change notification settings - Fork 4.9k
Choose queue type and exchange type when creating missing queues (fix #9671) #9815
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #9815 +/- ##
=======================================
Coverage 78.63% 78.63%
=======================================
Files 153 153
Lines 19199 19222 +23
Branches 2546 2555 +9
=======================================
+ Hits 15097 15116 +19
- Misses 3807 3810 +3
- Partials 295 296 +1
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lets also fix the kombu front as well as described here (#9759 (comment))
|
Hi @auvipy I just open the PR on Kombu I added to the control.py file on the celery part the two parameter control_queue_durable and control_queue_exclusive, after kombu is fixed i will make the commit with these modifications. |
|
can you please also look at the approach in this PR? celery/kombu#2313 and #9750 ? are they related to what you are trying to do? |
|
I checked both of those drafts. They flip every queue to My patch keeps the current behaviour unless the user asks for something different. It adds two flags: you can set the pidbox queue to exclusive (still transient, but now accepted by RabbitMQ 4.x) or, if you really need persistence, you can set it to durable. If you do nothing, it stays exactly as it is today. If the project would rather move straight to the “everything durable” solution, I can drop this and follow that path o set is as default as Please let me know if I’ve misunderstood anything. |
|
your approach seems better so im ok with it. |
|
Hi @auvipy I’ve added support for configuring durable and exclusive options on both event queues and control queue:
There’s also validation to prevent both being set to True, plus I added docs and tests. This is meant to fix #9759 on the celery part. That said, I can’t fully test the changes because they depend on feature that we introduced in Kombu PR 2338 Just wondering what’s the best way to proceed here, i need some guidance on how i can test it or if is necessary to test with the right version of kombu. Thanks! |
|
can you check why the unit tests are failing, please? |
|
My bad, I’ve just pushed a follow‑up commit that adds the missing guard clause to the failing test I introduced. Also figured out how to pin my local env to the unreleased kombu main, so I run the full test‑suite and the CI should be green. |
|
I think the tests are now failing because they are not pointing to the main of Kombu, I think i need some guidance here to understand how to fix them. |
|
yes kombu new version is required. |
|
kombu has been updated, so lets see |
|
the redis failures are not related to this rabbitmq bug fix pr |
|
@ghirailghiro please address the review comments... |
(`task_create_missing_queues=True`): *`task_create_missing_queue_type` *`task_create_missing_queue_exchange_type` Backwards compatibility: default behaviour (classic queue + direct exchange) is unchanged. Closes celery#9671
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
…t queues - Added `event_queue_durable` and `event_queue_exclusive` settings. - Added `control_exchange_durable` and `control_exchange_exclusive` settings. - Updated `EventReceiver` and `Control` to support these options. - Prevented invalid config: both options cannot be True at the same time. - Added related tests and updated documentation accordingly. This commit fix Issue (celery#9759) on the celery part
…e and control_queue_exclusive are True Prevent misconfiguration by raising ImproperlyConfigured in Control when both control_queue_durable and control_queue_exclusive options are enabled.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice work!
Very clean PR.
One change and we can merge it, please fix the missing .. versionadded:: in the docs.
|
unit tests seems to be failing after the last commit. can you please cross check and fix them? |
|
I checked the celery/app/amqp.py on main and i think the right way is to explicit set the routing_key. I think we should roll back the last commit for keeping compatibility. |
|
🚀🚀 |
|
Thank you guys for the support and sorry for the delay on the response! |
Description
This PR adds two new optional settings that control how Celery auto-creates
queues when
task_create_missing_queues = True:task_create_missing_queue_type"classic""classic"or choose the type you need, e.g."quorum").task_create_missing_queue_exchange_typeNone"topic","fanout", etc.Changes(Fixes #9671)
Add two new settings that apply when Celery autogenerates a queue
(
task_create_missing_queues=True):task_create_missing_queue_typetask_create_missing_queue_exchange_typeBackwards compatibility: default behaviour (classic queue + direct exchange) is unchanged.