Adaptive concurrency controller for ingestion and processing.#25641
Merged
nickvikeras merged 4 commits intomainfrom Mar 4, 2026
Merged
Adaptive concurrency controller for ingestion and processing.#25641nickvikeras merged 4 commits intomainfrom
nickvikeras merged 4 commits intomainfrom
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
2 Skipped Deployments
|
5284359 to
c683fc5
Compare
9b76c0a to
8be186f
Compare
8be186f to
1cbdb51
Compare
1cbdb51 to
7eb3079
Compare
da42a1f to
277aa9a
Compare
277aa9a to
2a5cff9
Compare
2a5cff9 to
e16f228
Compare
5fbf6a0 to
4c560b9
Compare
wlmyng
approved these changes
Mar 4, 2026
Contributor
wlmyng
left a comment
There was a problem hiding this comment.
LGTM, thanks for the detailed commentary in stream.rs, and the thorough testing, prevents_stampede and triggers_decrease were helpful to review to understand the mechanisms
do we ever want to expose the formula constants (dead band, log10)?
1 task
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adaptive concurrency controller for ingestion and processing.
The algorithm is a result of weeks of trial and error guided by lots of Claude Research queries. I tried to document where each of the ideas came from in the comments in the code.
It's working very well now.
Test plan
I've been testing by backfilling test bigtable clusters from genesis in mainnet and testnet.
Release Notes
Processor concurrency (
fanout) and ingestion concurrency (ingest_concurrency) now use adaptive concurrency control by default. Instead of a fixed number of workers, concurrency starts at 1 and scales automatically based on downstream channel backpressure — up tonum_cpusfor processors and up to 500 for ingestion.Breaking changes:
Processor::FANOUTtrait constant has been removed. Processor concurrency is now configured via thefanoutfield onConcurrentConfig/SequentialConfig, which accepts aConcurrencyConfigenum instead ofusize.ingest_concurrencyinIngestionConfigchanged fromusizetoConcurrencyConfig.processor_channel_sizefield controls the channel between the processor and downstream stage (defaults tonum_cpus / 2). This channel previously sized itself fromFANOUT + PIPELINE_BUFFER.Migration: To preserve previous fixed-concurrency behavior, set
fanout: Some(ConcurrencyConfig::Fixed { value: N })oringest_concurrency: ConcurrencyConfig::Fixed { value: N }. Otherwise, no changes are needed — the adaptive defaults should work well for most workloads.