Add xfail test for RabbitMQ quorum queue global QoS race condition #9770

maccinza · 2025-06-18T22:06:07Z

Description:

This PR adds an integration test that simulates a RabbitMQ quorum queue cluster visibility race during Celery worker startup.

The test forces a quorum detection failure on the first worker, introduces a delay in queue declaration, and starts multiple workers concurrently. It asserts that at least one worker hits an AMQP-level error (e.g., 540 NOT_IMPLEMENTED or connection loss) caused by attempting to apply global QoS on a quorum queue during the race.

The test is marked xfail since this is a known RabbitMQ limitation.

This behavior has been seen in a production environment with celery 5.5.3, kombu 5.5.2, and rabbitmq 3.13.7-management, where the broker is clustered.

❌ Observed Behavior

When multiple workers start concurrently:

The first worker fails quorum detection (simulated fault).
Subsequent workers proceed with quorum detection and start normally.
At least one worker triggers a RabbitMQ 540 NOT_IMPLEMENTED error or similar AMQP-level failure due to global QoS being applied to a quorum queue.

✅ Expected Behavior

At least one worker should experience a broker-side failure (540 NOT_IMPLEMENTED or equivalent AMQP error) as a result of this race condition.

Test Status

The test is marked xfail to acknowledge the known RabbitMQ limitation regarding global QoS on quorum queues.
Test intentionally expects failure (AMQP error presence) to validate that the race condition surfaces as designed.

This test simulates a quorum queue cluster propagation race where the first worker fails quorum detection and others succeed. It starts multiple workers concurrently and expects at least one AMQP error (e.g., 540 NOT_IMPLEMENTED) caused by applying global QoS to a quorum queue. The test is marked xfail since this behavior is a known RabbitMQ limitation.

for more information, see https://pre-commit.ci

codecov · 2025-06-18T22:12:23Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 78.52%. Comparing base (de104a2) to head (9178e43).
⚠️ Report is 45 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #9770   +/-   ##
=======================================
  Coverage   78.51%   78.52%           
=======================================
  Files         153      153           
  Lines       19130    19130           
  Branches     2533     2533           
=======================================
+ Hits        15020    15021    +1     
+ Misses       3821     3819    -2     
- Partials      289      290    +1

Flag	Coverage Δ
unittests	`78.49% <ø> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

…th CI

…error.

for more information, see https://pre-commit.ci

auvipy · 2025-06-28T03:21:17Z

tests and CI looks much better now

t/integration/conftest.py

for more information, see https://pre-commit.ci

Copilot

Pull Request Overview

This PR adds an integration test to simulate a RabbitMQ quorum queue global QoS race condition during Celery worker startup and refines timing and error-handling patterns in existing integration tests.

Refactors task tests to consistently use billiard’s multiprocessing and functions from the time module.
Improves the security test by using apply_async and introducing exception handling with clearer failure messages.
Introduces a new integration test file simulating the quorum queue QoS race condition.

Reviewed Changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File	Description
t/integration/test_tasks.py	Refactored multiprocessing and timing function usage for clarity and consistency.
t/integration/test_security.py	Enhanced error handling in a security test to better capture timeout issues.
t/integration/test_quorum_queue_qos_cluster_simulation.py	Added tests to simulate a known RabbitMQ quorum queue QoS race condition.
t/integration/conftest.py	Updated configuration file handling and added logging to support smoother test execution.

t/integration/conftest.py

t/integration/test_security.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

…ition (celery#9770)" This reverts commit 6d8bfd1.

…dition (celery#9770)" This reverts commit c889df4.

…dition (celery#9770)" This reverts commit b10a55c.

* Disabled tests test_multiprocess_producer and test_multithread_producer * Refactor integration tests CI * Disable test_quorum_queue_qos_cluster_simulation.py * Reduce integration tests timeout from 30m -> 20m and increase attempts from 2 -> 3 (fail/retry faster) * Increase max attempts from 3 -> 5 with 1m break between each retry * TMP Dont wait for unit tests * Changed retry settings * Revert "Add xfail test for RabbitMQ quorum queue global QoS race condition (#9770)" This reverts commit 6d8bfd1. * Remove test_quorum_queue_qos_cluster_simulation.py from CI * Prevent the billiard QueueListener from deadlocking during worker shutdown * Revert "TMP Dont wait for unit tests" This reverts commit 3da612d. * Run smoke if integration passed * Changed retry settings * Disable test_groupresult_serialization * timeout 5, attempts 10, instead of 30m x 2 attempts * Split integration test jobs to be test per module * Disabled dep with unit tests (for faster testing) * Run all integration jobs together * Split smoke test jobs * Simplifed python-package.yml * Cleanup * max-parallel: 4 * fixed smoked tests ci * Removed Python 3.10-3.12 from the integration and smoke tests CI * Integration & Smoke run only if unit tests pass * Reduced more python versions for now (integration min/max, smoke-max) * Revert "Prevent the billiard QueueListener from deadlocking during worker shutdown" This reverts commit cebcb2b. * Reapply "Add xfail test for RabbitMQ quorum queue global QoS race condition (#9770)" This reverts commit b10a55c. * Added back `test_quorum_queue_qos_cluster_simulation` to the integration tests * max-parallel: 5 * Revert "Disable test_groupresult_serialization" This reverts commit ade8cbd.

maccinza and others added 3 commits June 18, 2025 18:59

Merge branch 'main' into failing_to_disable_qos_for_quorum_queues

bd7f4a7

[pre-commit.ci] auto fixes from pre-commit.com hooks

798da3d

for more information, see https://pre-commit.ci

auvipy requested review from Nusnus, auvipy and Copilot June 19, 2025 05:11

This comment was marked as outdated.

Sign in to view

auvipy and others added 2 commits June 19, 2025 05:15

Update t/integration/test_quorum_queue_qos_cluster_simulation.py

49c73ba

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update t/integration/test_quorum_queue_qos_cluster_simulation.py

3cfe661

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

auvipy added this to the 5.5.4 milestone Jun 19, 2025

maccinza and others added 19 commits June 20, 2025 13:15

Final version with comments

bd67cab

Uses processes instead of threads for better isolation

f3ae9ce

Fixes for CI

e30f70f

Tries adding extra safeguard to stop test

27e3a4c

Tries to make connection parameters to rabbit and redis compatible wi…

c9d2505

…th CI

Tries to fix test on CI

93a023c

Fixes linting

84ebdbf

Tries to force execution with no cache

2774fd0

Replaces cache sanitization steps. Tries fixing actions indentation.

eec0c64

Tries improving termination

34282a0

Fixes pytest cache clearing. Tries to terminate process when catches …

704a3c5

…error.

Fixes linting

80d9f31

Removes cache cleaning. Total refactory to try to avoid ci hanging

755336a

Adds missing xfail

42283e3

[pre-commit.ci] auto fixes from pre-commit.com hooks

ca40d1b

for more information, see https://pre-commit.ci

Fixes linting

6ec15ca

[pre-commit.ci] auto fixes from pre-commit.com hooks

5d733ab

for more information, see https://pre-commit.ci

Fixes linting

888eee4

Yet another try to avoid hanging on CI

1a03b40

[pre-commit.ci] auto fixes from pre-commit.com hooks

d8100d6

for more information, see https://pre-commit.ci

maccinza requested a review from auvipy June 27, 2025 17:22

maccinza marked this pull request as ready for review June 27, 2025 17:22

maccinza added 2 commits June 27, 2025 14:24

Removes debug step from integration tests flow

27cc269

Reverts changes on python-package.yml

ba3fd43

auvipy reviewed Jun 28, 2025

View reviewed changes

t/integration/conftest.py Show resolved Hide resolved

t/integration/conftest.py Show resolved Hide resolved

maccinza and others added 2 commits June 30, 2025 10:54

Reverts comments

5607d4e

[pre-commit.ci] auto fixes from pre-commit.com hooks

694f3d5

for more information, see https://pre-commit.ci

maccinza requested a review from auvipy June 30, 2025 13:56

auvipy requested a review from Copilot June 30, 2025 15:59

Copilot AI reviewed Jun 30, 2025

View reviewed changes

t/integration/conftest.py Outdated Show resolved Hide resolved

t/integration/test_security.py Outdated Show resolved Hide resolved

auvipy and others added 2 commits June 30, 2025 22:03

Update t/integration/conftest.py

5ed0d20

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update t/integration/test_security.py

9178e43

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

auvipy approved these changes Jun 30, 2025

View reviewed changes

auvipy added the Component: RabbitMQ Broker label Jul 1, 2025

auvipy merged commit 6d8bfd1 into celery:main Jul 1, 2025
123 of 124 checks passed

auvipy added the PR Type: Test Case label Jul 1, 2025

auvipy modified the milestones: 5.5.4, 5.6.0 Jul 13, 2025

Nusnus mentioned this pull request Jul 23, 2025

Revert "Add xfail test for RabbitMQ quorum queue global QoS race condition" #9829

Closed

Nusnus added a commit to Nusnus/celery that referenced this pull request Aug 12, 2025

Revert "Add xfail test for RabbitMQ quorum queue global QoS race cond…

ae1ba4c

…ition (celery#9770)" This reverts commit 6d8bfd1.

Nusnus added a commit to Nusnus/celery that referenced this pull request Aug 12, 2025

Revert "Add xfail test for RabbitMQ quorum queue global QoS race cond…

b10a55c

…ition (celery#9770)" This reverts commit 6d8bfd1.

Nusnus added a commit to Nusnus/celery that referenced this pull request Aug 12, 2025

Revert "Add xfail test for RabbitMQ quorum queue global QoS race cond…

0448696

…ition (celery#9770)" This reverts commit 6d8bfd1.

Nusnus added a commit to Nusnus/celery that referenced this pull request Aug 12, 2025

Revert "Add xfail test for RabbitMQ quorum queue global QoS race cond…

c889df4

…ition (celery#9770)" This reverts commit 6d8bfd1.

Nusnus added a commit to Nusnus/celery that referenced this pull request Aug 13, 2025

Reapply "Add xfail test for RabbitMQ quorum queue global QoS race con…

dbb1592

…dition (celery#9770)" This reverts commit c889df4.

Nusnus added a commit to Nusnus/celery that referenced this pull request Aug 25, 2025

Reapply "Add xfail test for RabbitMQ quorum queue global QoS race con…

41b02a5

…dition (celery#9770)" This reverts commit b10a55c.

Nusnus added a commit to Nusnus/celery that referenced this pull request Aug 25, 2025

Reapply "Add xfail test for RabbitMQ quorum queue global QoS race con…

906e803

…dition (celery#9770)" This reverts commit b10a55c.

Nusnus added a commit to Nusnus/celery that referenced this pull request Aug 25, 2025

Reapply "Add xfail test for RabbitMQ quorum queue global QoS race con…

9c83bec

…dition (celery#9770)" This reverts commit b10a55c.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Add xfail test for RabbitMQ quorum queue global QoS race condition #9770

Add xfail test for RabbitMQ quorum queue global QoS race condition #9770

maccinza commented Jun 18, 2025

Uh oh!

codecov bot commented Jun 18, 2025 •

edited

Loading

Uh oh!

This comment was marked as outdated.

Uh oh!

auvipy commented Jun 28, 2025

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Add xfail test for RabbitMQ quorum queue global QoS race condition #9770

Add xfail test for RabbitMQ quorum queue global QoS race condition #9770

Conversation

maccinza commented Jun 18, 2025

Description:

❌ Observed Behavior

✅ Expected Behavior

Test Status

Uh oh!

codecov bot commented Jun 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

This comment was marked as outdated.

Uh oh!

auvipy commented Jun 28, 2025

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

codecov bot commented Jun 18, 2025 •

edited

Loading