Skip to content

Attempt to Fix RST_STREAM by draining http2 channels#8019

Merged
generall merged 2 commits intodevfrom
drain-http2-channel
Feb 5, 2026
Merged

Attempt to Fix RST_STREAM by draining http2 channels#8019
generall merged 2 commits intodevfrom
drain-http2-channel

Conversation

@agourlay
Copy link
Member

@agourlay agourlay commented Jan 30, 2026

Trying to fix

WARN collection::shards::replica_set::update: Failed to update shard , error: Rate limiting exceeded: status: ResourceExhausted, message: "h2 protocol error: http2 error: connection error detected: detected excessive load generating behavior (b\"too_many_internal_resets\")", details: [], metadata: MetadataMap { headers: {}

Chasing HTTP2 frames with Wireshark while running search on a 3 nodes cluster.

bfb -b100 -p10 -n100000 --indexing-threshold 1000000 --shards 9

bfb -b100 -p500 -n100000 --indexing-threshold 1000000 --skip-create --skip-upload --skip-wait-index --search --search-limit 5000 --timeout 2

Rational

I think our manual channel pool mechanism is dropping connection which are currently under use.

This PR introduces a draining mechanism to gracefully shutdown the channel.

Before

before

After

after

@agourlay agourlay force-pushed the drain-http2-channel branch from facc2e3 to 2653ce2 Compare January 30, 2026 14:15
@generall
Copy link
Member

generall commented Feb 2, 2026

currently, this PR doesn't fix too_many_internal_resets problem

@generall
Copy link
Member

generall commented Feb 3, 2026

So far, I am only able to repro it on v1.16.3+

@generall generall changed the title (WIP) Fix RST_STREAM by draining http2 channels Attempt to Fix RST_STREAM by draining http2 channels Feb 5, 2026
@generall generall marked this pull request as ready for review February 5, 2026 11:31
@generall generall requested a review from timvisee February 5, 2026 11:32
@generall generall merged commit c7438cc into dev Feb 5, 2026
15 checks passed
@generall generall deleted the drain-http2-channel branch February 5, 2026 13:07
@qdrant qdrant deleted a comment from coderabbitai bot Feb 5, 2026
generall added a commit that referenced this pull request Feb 9, 2026
* Fix RST_STREAM by draining http2 channels

* remove controversial duration time for now

---------

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
@timvisee timvisee mentioned this pull request Feb 17, 2026
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants