Adds integration test for chord_unlock bug when routed to quorum/topic queue #9766
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #9766 +/- ##
=======================================
Coverage 78.52% 78.52%
=======================================
Files 153 153
Lines 19130 19130
Branches 2533 2533
=======================================
Hits 15021 15021
Misses 3821 3821
Partials 288 288
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
|
interestingly, the tests are passing! |
I guess it is actually |
There was a problem hiding this comment.
Pull Request Overview
This PR adds an integration test that reproduces the chord_unlock routing issue when using a quorum queue with a topic exchange. Key changes include:
- A new test file with fixtures to set up a Celery app with specific queue and routing configurations.
- A stress test that submits multiple chords before starting a worker to capture the routing deficiencies.
- An update to the pytest markers in pyproject.toml to include the "amqp" marker.
Reviewed Changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| t/integration/test_rabbitmq_chord_unlock_routing.py | Introduces an integration test to replicate chord_unlock routing issues. |
| pyproject.toml | Updates marker list by adding the "amqp" marker. |
auvipy
left a comment
There was a problem hiding this comment.
what do you think about using logging instead of print?
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
auvipy
left a comment
There was a problem hiding this comment.
the tests looks good. need to work on the fix next
Description
This PR introduces an integration test that reproduces the issue described in Discussion #9742, where:
celery.chord_unlock tasks are routed to a declared quorum queue (bound to a topic exchange), but are either never consumed or endlessly retried, with rare occasions when it is successfully run.
✅ Summary of the test:
Chords are submitted before any worker is running.
chord_unlock is explicitly routed to a quorum queue via a topic exchange.
A worker is later started that consumes from both the header and callback queues.
The test verifies that all chord results complete successfully.
❌ Observed behavior:
A significant number of chord results remain pending or fail due to the chord_unlock task being ignored or retried indefinitely.
✅ Expected behavior:
The chord_unlock task should be routed correctly and consumed by the worker since the queue is explicitly declared and consumed.