Move broadcaster backpressure from task-level to stream-level#25325
Merged
nickvikeras merged 2 commits intomainfrom Feb 7, 2026
Merged
Move broadcaster backpressure from task-level to stream-level#25325nickvikeras merged 2 commits intomainfrom
nickvikeras merged 2 commits intomainfrom
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
2 Skipped Deployments
|
062f477 to
4b0d9e2
Compare
Previously, each spawned fetch task independently waited on ingest_hi inside itself, meaning tasks were already spawned and consuming resources while blocked on backpressure. Now checkpoints are gated at the stream level via backpressured_checkpoint_stream(), so tasks are only spawned when the backpressure window allows.
4b0d9e2 to
3d5dbd1
Compare
amnn
approved these changes
Feb 7, 2026
Contributor
amnn
left a comment
There was a problem hiding this comment.
Nice change -- this does really motivate updating the backpressure system to be the same for concurrent and sequential pipelines.
| stream::iter(start..end) | ||
| // Backpressure is enforced at the stream level: checkpoints are only yielded when | ||
| // ingest_hi allows, preventing spawned tasks from piling up while blocked. | ||
| let checkpoints = backpressured_checkpoint_stream(start, end, ingest_hi_rx); |
wlmyng
approved these changes
Feb 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Previously, each spawned fetch task independently waited on ingest_hi inside itself, meaning tasks were already spawned and consuming resources while blocked on backpressure. Now checkpoints are gated at the stream level via backpressured_checkpoint_stream(), so tasks are only spawned when the backpressure window allows.
Without this change, it's impossible to let adaptive concurrency limiters find the upper bound for object store ingestion, because we will just keep spinning up futures (that are blocked on backpressure) until we OOM.
Test plan
I have tested it during a testnet backfill as part of a larger effort to enable adaptive rate limiting.
Release notes