Add --remote_max_concurrency_per_connection
#27466
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
RELNOTES: The new `--remote_max_concurrency_per_connection` can be used to specify the maximum number of concurrent gRPC requests Bazel will issue on a single connection to the server. The default value of 100 matches the previous behavior.
Collaborator
Author
|
@bazel-io flag 8.5.0 |
Collaborator
Author
Good point, done! Let me know if you would prefer more details. |
tjgq
approved these changes
Nov 4, 2025
Collaborator
Author
|
@bazel-io fork 9.0.0 |
bazel-io
pushed a commit
to bazel-io/bazel
that referenced
this pull request
Nov 5, 2025
The stack traces obtained bazelbuild#25232 (comment) indicate that all gRPC threads are waiting on events when the hang reported in bazelbuild#25232 occurs, with no other threads being active except for the virtual threads blocked on upload futures. This situation is reminiscent of grpc/grpc-java#8334 (comment) and further experimentation showed that reducing the maximum number of concurrent requests per gRPC connection down to 20 (from 100) resolved the hangs. Reducing the number to 50 made them less likely. Since it is not clear that there is a single number that avoids hangs for all backends while not sacrificing performance with some, this change makes the limit configurable for further experimentation. RELNOTES: The new `--remote_max_concurrency_per_connection` can be used to specify the maximum number of concurrent gRPC requests Bazel will issue on a single connection to the server. The default value of 100 matches the previous behavior. Work towards bazelbuild#25232 Closes bazelbuild#27466. PiperOrigin-RevId: 828555281 Change-Id: I901cfb13be7f4f0a4ef1d406845d96e88cecd02f
github-merge-queue bot
pushed a commit
that referenced
this pull request
Nov 6, 2025
The stack traces obtained #25232 (comment) indicate that all gRPC threads are waiting on events when the hang reported in #25232 occurs, with no other threads being active except for the virtual threads blocked on upload futures. This situation is reminiscent of grpc/grpc-java#8334 (comment) and further experimentation showed that reducing the maximum number of concurrent requests per gRPC connection down to 20 (from 100) resolved the hangs. Reducing the number to 50 made them less likely. Since it is not clear that there is a single number that avoids hangs for all backends while not sacrificing performance with some, this change makes the limit configurable for further experimentation. RELNOTES: The new `--remote_max_concurrency_per_connection` can be used to specify the maximum number of concurrent gRPC requests Bazel will issue on a single connection to the server. The default value of 100 matches the previous behavior. Work towards #25232 Closes #27466. PiperOrigin-RevId: 828555281 Change-Id: I901cfb13be7f4f0a4ef1d406845d96e88cecd02f Commit 51a3f0e Co-authored-by: Fabian Meumertzheim <fabian@meumertzhe.im>
Collaborator
Author
|
@bazel-io fork 8.5.0 |
bazel-io
pushed a commit
to bazel-io/bazel
that referenced
this pull request
Nov 6, 2025
The stack traces obtained bazelbuild#25232 (comment) indicate that all gRPC threads are waiting on events when the hang reported in bazelbuild#25232 occurs, with no other threads being active except for the virtual threads blocked on upload futures. This situation is reminiscent of grpc/grpc-java#8334 (comment) and further experimentation showed that reducing the maximum number of concurrent requests per gRPC connection down to 20 (from 100) resolved the hangs. Reducing the number to 50 made them less likely. Since it is not clear that there is a single number that avoids hangs for all backends while not sacrificing performance with some, this change makes the limit configurable for further experimentation. RELNOTES: The new `--remote_max_concurrency_per_connection` can be used to specify the maximum number of concurrent gRPC requests Bazel will issue on a single connection to the server. The default value of 100 matches the previous behavior. Work towards bazelbuild#25232 Closes bazelbuild#27466. PiperOrigin-RevId: 828555281 Change-Id: I901cfb13be7f4f0a4ef1d406845d96e88cecd02f
github-merge-queue bot
pushed a commit
that referenced
this pull request
Nov 6, 2025
The stack traces obtained #25232 (comment) indicate that all gRPC threads are waiting on events when the hang reported in #25232 occurs, with no other threads being active except for the virtual threads blocked on upload futures. This situation is reminiscent of grpc/grpc-java#8334 (comment) and further experimentation showed that reducing the maximum number of concurrent requests per gRPC connection down to 20 (from 100) resolved the hangs. Reducing the number to 50 made them less likely. Since it is not clear that there is a single number that avoids hangs for all backends while not sacrificing performance with some, this change makes the limit configurable for further experimentation. RELNOTES: The new `--remote_max_concurrency_per_connection` can be used to specify the maximum number of concurrent gRPC requests Bazel will issue on a single connection to the server. The default value of 100 matches the previous behavior. Work towards #25232 Closes #27466. PiperOrigin-RevId: 828555281 Change-Id: I901cfb13be7f4f0a4ef1d406845d96e88cecd02f Commit 51a3f0e Co-authored-by: Fabian Meumertzheim <fabian@meumertzhe.im>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The stack traces obtained #25232 (comment) indicate that all gRPC threads are waiting on events when the hang reported in #25232 occurs, with no other threads being active except for the virtual threads blocked on upload futures.
This situation is reminiscent of grpc/grpc-java#8334 (comment) and further experimentation showed that reducing the maximum number of concurrent requests per gRPC connection down to 20 (from 100) resolved the hangs. Reducing the number to 50 made them less likely. Since it is not clear that there is a single number that avoids hangs for all backends while not sacrificing performance with some, this change makes the limit configurable for further experimentation.
RELNOTES: The new
--remote_max_concurrency_per_connectioncan be used to specify the maximum number of concurrent gRPC requests Bazel will issue on a single connection to the server. The default value of 100 matches the previous behavior.Work towards #25232