Skip to content

Conversation

@ghirailghiro
Copy link
Contributor

Description

This PR adds two new optional settings that control how Celery auto-creates
queues when task_create_missing_queues = True:

Setting Default What it does
task_create_missing_queue_type "classic" Select the AMQP queue kind: keep the default "classic" or choose the type you need, e.g. "quorum").
task_create_missing_queue_exchange_type None Override the exchange type used for the auto-exchange: set to "topic", "fanout", etc.

Changes(Fixes #9671)

Add two new settings that apply when Celery autogenerates a queue
(task_create_missing_queues=True):

  • task_create_missing_queue_type

  • task_create_missing_queue_exchange_type

Backwards compatibility: default behaviour (classic queue + direct exchange) is unchanged.

@codecov
Copy link

codecov bot commented Jul 16, 2025

Codecov Report

❌ Patch coverage is 85.18519% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 78.63%. Comparing base (31d05ed) to head (213104e).
⚠️ Report is 70 commits behind head on main.

Files with missing lines Patch % Lines
celery/app/amqp.py 88.23% 1 Missing and 1 partial ⚠️
celery/events/receiver.py 71.42% 0 Missing and 2 partials ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##             main    #9815   +/-   ##
=======================================
  Coverage   78.63%   78.63%           
=======================================
  Files         153      153           
  Lines       19199    19222   +23     
  Branches     2546     2555    +9     
=======================================
+ Hits        15097    15116   +19     
- Misses       3807     3810    +3     
- Partials      295      296    +1     
Flag Coverage Δ
unittests 78.61% <85.18%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@auvipy auvipy requested review from auvipy and Copilot July 16, 2025 03:05

This comment was marked as resolved.

@auvipy auvipy added this to the 5.6.0 milestone Jul 16, 2025
Copy link
Member

@auvipy auvipy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lets also fix the kombu front as well as described here (#9759 (comment))

@ghirailghiro
Copy link
Contributor Author

Hi @auvipy I just open the PR on Kombu

I added to the control.py file on the celery part the two parameter control_queue_durable and control_queue_exclusive, after kombu is fixed i will make the commit with these modifications.

@auvipy
Copy link
Member

auvipy commented Jul 22, 2025

can you please also look at the approach in this PR? celery/kombu#2313 and #9750 ? are they related to what you are trying to do?

@ghirailghiro
Copy link
Contributor Author

I checked both of those drafts. They flip every queue to durable=True, so the RabbitMQ warning disappears. For what I undestand pidbox queues are meant for short-lived messages that you don't need to keep after the broker is restarted.

My patch keeps the current behaviour unless the user asks for something different. It adds two flags: you can set the pidbox queue to exclusive (still transient, but now accepted by RabbitMQ 4.x) or, if you really need persistence, you can set it to durable. If you do nothing, it stays exactly as it is today.

If the project would rather move straight to the “everything durable” solution, I can drop this and follow that path o set is as default as durable=True

Please let me know if I’ve misunderstood anything.

@auvipy
Copy link
Member

auvipy commented Jul 23, 2025

your approach seems better so im ok with it.

@ghirailghiro
Copy link
Contributor Author

Hi @auvipy

I’ve added support for configuring durable and exclusive options on both event queues and control queue:

  • event_queue_durable, event_queue_exclusive
  • control_queue_durable, control_queue_exclusive

There’s also validation to prevent both being set to True, plus I added docs and tests. This is meant to fix #9759 on the celery part.

That said, I can’t fully test the changes because they depend on feature that we introduced in Kombu PR 2338

Just wondering what’s the best way to proceed here, i need some guidance on how i can test it or if is necessary to test with the right version of kombu.

Thanks!

@auvipy auvipy requested a review from Nusnus July 26, 2025 05:58
@auvipy
Copy link
Member

auvipy commented Jul 26, 2025

can you check why the unit tests are failing, please?

@ghirailghiro
Copy link
Contributor Author

ghirailghiro commented Jul 26, 2025

My bad, I’ve just pushed a follow‑up commit that adds the missing guard clause to the failing test I introduced.

Also figured out how to pin my local env to the unreleased kombu main, so I run the full test‑suite and the CI should be green.

@ghirailghiro
Copy link
Contributor Author

I think the tests are now failing because they are not pointing to the main of Kombu, I think i need some guidance here to understand how to fix them.

@auvipy
Copy link
Member

auvipy commented Jul 27, 2025

yes kombu new version is required.

@auvipy
Copy link
Member

auvipy commented Aug 7, 2025

kombu has been updated, so lets see

@auvipy auvipy requested a review from Copilot August 9, 2025 06:56

This comment was marked as outdated.

@auvipy
Copy link
Member

auvipy commented Aug 9, 2025

the redis failures are not related to this rabbitmq bug fix pr

@auvipy auvipy requested a review from Copilot August 10, 2025 11:47

This comment was marked as outdated.

@auvipy auvipy requested a review from Copilot August 10, 2025 11:51

This comment was marked as resolved.

@auvipy
Copy link
Member

auvipy commented Aug 27, 2025

@ghirailghiro please address the review comments...

ghirailghiro and others added 10 commits August 28, 2025 16:05
(`task_create_missing_queues=True`):

*`task_create_missing_queue_type`

*`task_create_missing_queue_exchange_type`

Backwards compatibility: default behaviour (classic queue + direct
exchange) is unchanged.

Closes celery#9671
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
…t queues

- Added `event_queue_durable` and `event_queue_exclusive` settings.
- Added `control_exchange_durable` and `control_exchange_exclusive` settings.
- Updated `EventReceiver` and `Control` to support these options.
- Prevented invalid config: both options cannot be True at the same time.
- Added related tests and updated documentation accordingly.

This commit fix Issue (celery#9759) on the celery part
…e and control_queue_exclusive are True

Prevent misconfiguration by raising ImproperlyConfigured in Control when both control_queue_durable and control_queue_exclusive options are enabled.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Nusnus
Nusnus previously requested changes Aug 28, 2025
Copy link
Member

@Nusnus Nusnus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work!
Very clean PR.

One change and we can merge it, please fix the missing .. versionadded:: in the docs.

@auvipy
Copy link
Member

auvipy commented Aug 30, 2025

unit tests seems to be failing after the last commit. can you please cross check and fix them?

@ghirailghiro
Copy link
Contributor Author

I checked the celery/app/amqp.py on main and i think the right way is to explicit set the routing_key. I think we should roll back the last commit for keeping compatibility.

@auvipy auvipy merged commit 166f705 into celery:main Aug 31, 2025
380 of 393 checks passed
@Nusnus
Copy link
Member

Nusnus commented Aug 31, 2025

🚀🚀

@ghirailghiro
Copy link
Contributor Author

Thank you guys for the support and sorry for the delay on the response!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Adding task_create_missing_queues_type in order to autocreate missing queues with quorum type

3 participants