[Fix][Python] Increase timeout for linux distribtests jobs for release and master branch#41000
Closed
sreenithi wants to merge 3 commits intogrpc:masterfrom
Closed
[Fix][Python] Increase timeout for linux distribtests jobs for release and master branch#41000sreenithi wants to merge 3 commits intogrpc:masterfrom
sreenithi wants to merge 3 commits intogrpc:masterfrom
Conversation
Contributor
Author
|
This took about 6 hours |
This reverts commit 4d27ea3.
sergiitk
approved these changes
Nov 3, 2025
Member
sergiitk
left a comment
There was a problem hiding this comment.
Accepted as a temporary solution. 6 hours is way too much, we should figure out how to decrease it.
sreenithi
added a commit
to sreenithi/grpc
that referenced
this pull request
Nov 26, 2025
…e and master branch (grpc#41000) The newly migrated pyproject.toml build system (in grpc#40833) seems to have much slower build times compared to the previous setup.py builds. This difference has also been noted by few users ([Ref1](pypa/pip#7294), [Ref2](https://zameermanji.com/blog/2021/6/14/building-wheels-with-pip-is-getting-slower/)) most likely due to the overhead of creating isolated build environments for each build. Some new runtimes for each variant target noted from different recent runs below: ``` 2025-11-01 10:19:02,937 PASSED: build_artifact.python_musllinux_1_2_x86_cp312-cp312 [time=4267.1sec, retries=0:0] 2025-11-01 10:19:05,501 PASSED: build_artifact.python_musllinux_1_2_x86_cp311-cp311 [time=4269.7sec, retries=0:0] 2025-11-01 10:19:07,152 PASSED: build_artifact.python_musllinux_1_2_x86_cp310-cp310 [time=4287.0sec, retries=0:0] 2025-11-01 11:43:22,269 PASSED: build_artifact.python_manylinux2014_x86_cp312-cp312 [time=4644.4sec, retries=0:0] 2025-11-01 11:43:37,257 PASSED: build_artifact.python_manylinux2014_x86_cp314-cp314 [time=4659.4sec, retries=0:0] 2025-11-01 11:43:37,525 PASSED: build_artifact.python_manylinux2014_x86_cp311-cp311 [time=4659.7sec, retries=0:0] 2025-11-01 11:43:40,671 PASSED: build_artifact.python_manylinux2014_x86_cp310-cp310 [time=4662.8sec, retries=0:0] 2025-11-01 11:43:51,663 PASSED: build_artifact.python_manylinux2014_x86_cp39-cp39 [time=4673.8sec, retries=0:0] 2025-11-01 11:43:51,708 PASSED: build_artifact.python_manylinux2014_x86_cp313-cp313 [time=4673.8sec, retries=0:0] 2025-11-01 04:59:11,117 PASSED: build_artifact.python_linux_extra_armv7_cp311-cp311 [time=5172.2sec, retries=0:0] 2025-11-01 04:59:31,002 PASSED: build_artifact.python_linux_extra_armv7_cp314-cp314 [time=5157.0sec, retries=0:0] 2025-11-01 04:59:53,485 PASSED: build_artifact.python_linux_extra_armv7_cp39-cp39 [time=5221.7sec, retries=0:0] ``` The builds take anywhere between 70-90 minutes for each target. We have a total of 30 targets currently and 12 targets are built parallelly in a batch. So for about 3 batches of 12 targets each and a worst case time of 90 minutes per batch, the `build_artifact` phase itself should take about 4.5 hours to complete, followed by the remaining `package` and `distribtest` phases which should be comparatively shorter. So, I believe about 7 hours should be enough, but just keeping an extra buffer and increasing the timeout to 8 hours. Closes grpc#41000 COPYBARA_INTEGRATE_REVIEW=grpc#41000 from sreenithi:increase_distribtests_timeout 44c94b0 PiperOrigin-RevId: 827672050
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The newly migrated pyproject.toml build system (in #40833) seems to have much slower build times compared to the previous setup.py builds. This difference has also been noted by few users (Ref1, Ref2) most likely due to the overhead of creating isolated build environments for each build.
Some new runtimes for each variant target noted from different recent runs below:
The builds take anywhere between 70-90 minutes for each target. We have a total of 30 targets currently and 12 targets are built parallelly in a batch. So for about 3 batches of 12 targets each and a worst case time of 90 minutes per batch, the
build_artifactphase itself should take about 4.5 hours to complete, followed by the remainingpackageanddistribtestphases which should be comparatively shorter.So, I believe about 7 hours should be enough, but just keeping an extra buffer and increasing the timeout to 8 hours.