Feature/disable prefetch fixes #9863

Dkhodos · 2025-08-14T14:35:31Z

Note: Before submitting this pull request, please review our contributing guidelines.

Description

Introduce an option to completely disable broker prefetching without requiring acks_late. When enabled, the worker only fetches a new task when an execution slot is available. This prevents head‑of‑line blocking and starvation caused by long‑running tasks.

Real‑world need: we run long‑lasting tasks (hours) across multiple workers/pods. Prefetch caused busy workers to hold tasks in reserve while idle workers sat empty, leading to severe starvation. Disabling prefetch resolves this by fetching only when a slot is free.
This work revives and supersedes the stalled discussion and implementation in PR #9818. All reviewer suggestions from that PR have been applied here.

What’s included

New config: worker_disable_prefetch (default: False)
New CLI flag: --disable-prefetch
Core logic: when enabled, wrap channel.qos.can_consume to check reserved_requests against effective concurrency:
- use autoscaler max_concurrency if present, otherwise pool size
Documentation updates:
- user guide (optimizing, configuration) and FAQ explain usage and impact
Tests:
- unit tests for consumer/autoscale behavior (including autoscaling edge cases)
- pytest‑click based CLI test covering --disable-prefetch
Minor nit applied: compute limit via getattr(c.controller, "max_concurrency", None) or c.pool.num_processes

Backward compatibility

Off by default; existing behavior unchanged.
When enabled, only reservation timing changes (fetch on free slot); no protocol or API changes.

Implementation notes

This is a pragmatic Celery‑side QoS guard. Longer‑term, the ideal place for this behavior would be in Kombu’s QoS, with Celery surfacing the option.

Testing

Unit suites for consumer/autoscale/CLI pass locally.
Docs build and configuration references updated.

References

Prior discussion/implementation: PR #9818

for more information, see https://pre-commit.ci

…pool size; add CLI --disable-prefetch test; ensure consumer tests pass

codecov · 2025-08-14T15:46:41Z

Codecov Report

❌ Patch coverage is 92.85714% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 78.70%. Comparing base (6804ea8) to head (d8c7d2b).
⚠️ Report is 65 commits behind head on main.

Files with missing lines	Patch %	Lines
celery/bin/worker.py	66.66%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #9863      +/-   ##
==========================================
+ Coverage   78.69%   78.70%   +0.01%     
==========================================
  Files         153      153              
  Lines       19243    19257      +14     
  Branches     2557     2561       +4     
==========================================
+ Hits        15143    15156      +13     
  Misses       3803     3803              
- Partials      297      298       +1

Flag	Coverage Δ
unittests	`78.68% <92.85%> (+0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Add dedicated tests for disable_prefetch flag handling in worker.py to improve test coverage. This addresses the coverage issue identified by Codecov in PR #9863.

Dkhodos · 2025-08-16T18:10:39Z

Codecov Report

❌ Patch coverage is 92.85714% with 1 line in your changes missing coverage. Please review. ✅ Project coverage is 78.64%. Comparing base (33eb148) to head (48b254b).

Files with missing lines Patch % Lines
celery/bin/worker.py 66.66% 0 Missing and 1 partial ⚠️
Additional details and impacted files
☔ View full report in Codecov by Sentry. 📢 Have feedback on the report? Share it here.

should be covered well now

Add comprehensive tests for the worker's disable_prefetch flag handling to improve test coverage. Refactored tests to be more concise, focused, and pass all lint checks. This addresses the coverage issue identified by Codecov in PR #9863.

Nusnus

Please note the integration tests are currently broken in main, preventing merges of non-trivial changes.

A fix is in progress 🙏

bakwc · 2025-08-25T10:08:36Z

Thanks a lot for working on this! Any chances to get it merged in the nearest future?

Dkhodos · 2025-08-25T10:55:30Z

Thanks a lot for working on this! Any chances to get it merged in the nearest future?

@Nusnus is CI fixed?

Nusnus · 2025-08-25T12:52:55Z

Thanks a lot for working on this! Any chances to get it merged in the nearest future?

@Nusnus is CI fixed?

Hopefully by (my) EOD today.
I'm going to try a different approach, let's see if it works 🙏

auvipy

also, do we have relevant issues open in kombu for the following note:

This is a pragmatic Celery‑side QoS guard. Longer‑term, the ideal place for this behavior would be in Kombu’s QoS, with Celery surfacing the option.

docs/userguide/configuration.rst

… setting

nasis · 2025-08-31T08:09:48Z

Needed
Looking for it to be merged.

docs/userguide/configuration.rst

Dkhodos · 2025-08-31T09:19:13Z

also, do we have relevant issues open in kombu for the following note:

This is a pragmatic Celery‑side QoS guard. Longer‑term, the ideal place for this behavior would be in Kombu’s QoS, with Celery surfacing the option.

Added
celery/kombu#2353

Might even have a PR for this later today :)

Side question:
Wen can we expect a package release with this fix? what number would it be 4.6.?

auvipy · 2025-08-31T09:44:57Z

v5.6

auvipy · 2025-09-28T04:18:16Z

we got a regression report of this. can you please check it? we might need to revert it for a while if needed #9913

This reverts commit 246bca1.

bakwc · 2025-09-28T19:00:56Z

May be you could leave it at least for redis?

auvipy · 2025-09-29T06:46:33Z

yes, that is possible

rbehal and others added 6 commits July 17, 2025 12:40

Add option to disable prefetch

0e05197

[pre-commit.ci] auto fixes from pre-commit.com hooks

2f0f1af

for more information, see https://pre-commit.ci

Fix pre-commit hooks failing

3ebcd40

Merge branch 'main' into add_disable_prefetch

7ddf7a5

worker: disable-prefetch QoS guard uses autoscale max_concurrency or …

2bc5419

…pool size; add CLI --disable-prefetch test; ensure consumer tests pass

Merge branch 'main' into feature/disable-prefetch-fixes

48b254b

auvipy mentioned this pull request Aug 14, 2025

Add option to completely disable prefetching of tasks without using acks_late #9818

Closed

Fix test coverage for worker disable_prefetch handling

9c47520

Add dedicated tests for disable_prefetch flag handling in worker.py to improve test coverage. This addresses the coverage issue identified by Codecov in PR #9863.

Nusnus reviewed Aug 16, 2025

View reviewed changes

avigaffa approved these changes Aug 17, 2025

View reviewed changes

sashu2310 approved these changes Aug 17, 2025

View reviewed changes

auvipy added this to the 5.7.0 milestone Aug 21, 2025

Merge branch 'main' into feature/disable-prefetch-fixes

ba76a91

auvipy modified the milestones: 5.7.0, 5.6.0 Aug 26, 2025

Dkhodos and others added 2 commits August 26, 2025 10:18

Merge branch 'main' into feature/disable-prefetch-fixes

e553b18

Merge branch 'main' into feature/disable-prefetch-fixes

58c57ef

auvipy self-requested a review August 28, 2025 13:12

auvipy requested changes Aug 31, 2025

View reviewed changes

docs/userguide/configuration.rst Show resolved Hide resolved

Dkhodos added 2 commits August 31, 2025 10:51

Merge branch 'main' into feature/disable-prefetch-fixes

d66870b

docs: add missing versionadded annotation for worker_disable_prefetch…

fa28fd0

… setting

amirgur approved these changes Aug 31, 2025

View reviewed changes

nasis approved these changes Aug 31, 2025

View reviewed changes

auvipy reviewed Aug 31, 2025

View reviewed changes

docs/userguide/configuration.rst Outdated Show resolved Hide resolved

Update docs/userguide/configuration.rst

d8c7d2b

auvipy approved these changes Aug 31, 2025

View reviewed changes

docs/userguide/configuration.rst Show resolved Hide resolved

auvipy merged commit 246bca1 into celery:main Aug 31, 2025
105 of 108 checks passed

Dkhodos mentioned this pull request Aug 31, 2025

Move QoS Guard Functionality from Celery to Kombu celery/kombu#2353

Open

5 tasks

tdzienniak mentioned this pull request Sep 26, 2025

New --disable-prefetch worker option doesn't work with RabbitMQ broker #9913

Closed

19 tasks

auvipy added a commit that referenced this pull request Sep 28, 2025

Revert "Feature/disable prefetch fixes (#9863)"

8a83d2b

This reverts commit 246bca1.

auvipy mentioned this pull request Sep 28, 2025

Revert "Feature/disable prefetch fixes" #9915

Closed

Dkhodos mentioned this pull request Sep 29, 2025

fix: restrict disable-prefetch feature to Redis brokers only #9919

Merged

Uh oh!

Feature/disable prefetch fixes #9863

Feature/disable prefetch fixes #9863

Uh oh!

Conversation

Dkhodos commented Aug 14, 2025

Description

What’s included

Backward compatibility

Implementation notes

Testing

References

Uh oh!

codecov bot commented Aug 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Dkhodos commented Aug 16, 2025

Codecov Report

Uh oh!

Nusnus left a comment

Choose a reason for hiding this comment

Uh oh!

bakwc commented Aug 25, 2025

Uh oh!

Dkhodos commented Aug 25, 2025

Uh oh!

Nusnus commented Aug 25, 2025

Uh oh!

auvipy left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

nasis commented Aug 31, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Dkhodos commented Aug 31, 2025

Uh oh!

auvipy commented Aug 31, 2025

Uh oh!

auvipy commented Sep 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bakwc commented Sep 28, 2025

Uh oh!

auvipy commented Sep 29, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

codecov bot commented Aug 14, 2025 •

edited

Loading

auvipy commented Sep 28, 2025 •

edited

Loading