fix: restrict disable-prefetch feature to Redis brokers only #9919
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The disable-prefetch feature was originally implemented for all brokers, but users reported compatibility issues with RabbitMQ. This change restricts the feature to Redis brokers only, which is where most testing was conducted. Changes: - Add broker transport detection in Tasks.start() - Only apply disable-prefetch logic for Redis brokers - Log warning for non-Redis brokers when setting is enabled - Update CLI help text to mention Redis-only support - Update documentation to reflect Redis-only limitation - Add tests for non-Redis broker behavior Fixes compatibility issues with RabbitMQ and other non-Redis brokers while maintaining the feature for Redis users.
- Break long line in worker.py help text - Remove trailing whitespace in test file
19 tasks
…test The test_log_when_qos_is_false test was failing because our new warning for non-Redis brokers was being logged, causing the test to expect 2 log records instead of 1. Fixed by explicitly setting worker_disable_prefetch to False in this test.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #9919 +/- ##
==========================================
- Coverage 78.67% 78.67% -0.01%
==========================================
Files 153 153
Lines 19295 19299 +4
Branches 2569 2570 +1
==========================================
+ Hits 15181 15184 +3
Misses 3816 3816
- Partials 298 299 +1
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
auvipy
approved these changes
Sep 30, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This pull request restricts the
--disable-prefetchfeature to Redis brokers only to resolve compatibility issues with RabbitMQ and other brokers.Background:
The
--disable-prefetchfeature was introduced in PR #9863 to allow workers to only fetch tasks when process slots are available, preventing task hoarding and improving fair distribution. However, as reported in issue #9913, this feature causes crashes when used with RabbitMQ brokers due to differences in channel implementation - RabbitMQ channels don't have theqosattribute that the feature relies on.Root Cause:
The original implementation assumed all broker channels would have the same interface as Redis channels, but RabbitMQ's Kombu channel implementation doesn't extend
virtual.Channeland lacks theqosattribute, causingAttributeError: 'Channel' object has no attribute 'qos'.Solution:
This PR makes the feature Redis-specific by:
driver_type == 'redis') before applying disable-prefetch logicChanges Made:
celery/worker/consumer/tasks.pyto check broker type and only apply prefetch logic for Rediscelery/bin/worker.pyhelp text to mention Redis-only supportTesting:
This is a surgical fix that maintains backward compatibility while preventing crashes on unsupported brokers. Redis users continue to benefit from the feature, while RabbitMQ and other broker users get clear messaging about the limitation instead of cryptic errors.
Fixes #9913