ci: Move multigpu to periodic#79894
Closed
seemethere wants to merge 1 commit intogh/seemethere/253/basefrom
Closed
ci: Move multigpu to periodic#79894seemethere wants to merge 1 commit intogh/seemethere/253/basefrom
seemethere wants to merge 1 commit intogh/seemethere/253/basefrom
Conversation
We have hard limitations on the number of linux.16xlarge.nvidia.gpu machines we can spin up. Considering that the TTS for this specific job has increased 2x over the past 7 days. Signed-off-by: Eli Uriegas <eliuriegas@fb.com> [ghstack-poisoned]
malfet
approved these changes
Jun 20, 2022
Contributor
malfet
left a comment
There was a problem hiding this comment.
Do we have any signal on how frequently we had to revert due to that?
janeyx99
approved these changes
Jun 20, 2022
Contributor
🔗 Helpful links
❌ 4 New Failures, 1 Base Failures, 2 PendingAs of commit 5302e30 (more details on the Dr. CI page): Expand to see more
🕵️ 4 new failures recognized by patternsThe following CI failures do not appear to be due to upstream breakages
|
Contributor
|
@rohan-varma The code that increased the TTS is #77947. FYI this change will move all multigpu tests to periodic (every 4 hours) instead of every commit. If this sounds like something the distributed team would prefer not to do, the other alternative is removing tests from the multigpu config (like reverting the fsdp change). |
Member
Author
|
@rohan-varma I remember you said you had an opinion on moving this? |
Contributor
|
Spoke with @janeyx99 offline, I think right now, we have to go ahead with the move to periodic, until we can decrease TTS to < 75min, after which it can be moved to master-only. |
Member
Author
|
@pytorchbot merge -f |
Collaborator
|
@pytorchbot successfully started a merge job. Check the current status here |
Collaborator
|
@seemethere your PR has been successfully merged. |
Contributor
|
Hey @seemethere. |
facebook-github-bot
pushed a commit
that referenced
this pull request
Jun 22, 2022
Summary: We have hard limitations on the number of linux.16xlarge.nvidia.gpu machines we can spin up. Considering that the TTS for this specific job has increased 2x over the past 7 days. Signed-off-by: Eli Uriegas <eliuriegasfb.com> Pull Request resolved: #79894 Approved by: https://github.com/malfet, https://github.com/janeyx99 Test Plan: contbuild & OSS CI, see https://hud.pytorch.org/commit/pytorch/pytorch/02d01707a6f40b7a189c3353010798766609df71 Reviewed By: atalman Differential Revision: D37327474 Pulled By: seemethere fbshipit-source-id: 525846ea9da645075f7778e7186753005f637f09
miladm
pushed a commit
to miladm/pytorch
that referenced
this pull request
Jun 27, 2022
We have hard limitations on the number of linux.16xlarge.nvidia.gpu machines we can spin up. Considering that the TTS for this specific job has increased 2x over the past 7 days. Signed-off-by: Eli Uriegas <eliuriegasfb.com> Pull Request resolved: pytorch#79894 Approved by: https://github.com/malfet, https://github.com/janeyx99
laurentdupin
pushed a commit
to laurentdupin/pytorch
that referenced
this pull request
Apr 25, 2026
We have hard limitations on the number of linux.16xlarge.nvidia.gpu machines we can spin up. Considering that the TTS for this specific job has increased 2x over the past 7 days. Signed-off-by: Eli Uriegas <eliuriegasfb.com> Pull Request resolved: pytorch#79894 Approved by: https://github.com/malfet, https://github.com/janeyx99
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack from ghstack:
We have hard limitations on the number of linux.16xlarge.nvidia.gpu
machines we can spin up. Considering that the TTS for this specific job
has increased 2x over the past 7 days.
Signed-off-by: Eli Uriegas eliuriegas@fb.com