Skip to content

Update _linux-test to support B200 runner#157341

Closed
huydhn wants to merge 16 commits intomainfrom
support-b200-runners
Closed

Update _linux-test to support B200 runner#157341
huydhn wants to merge 16 commits intomainfrom
support-b200-runners

Conversation

@huydhn
Copy link
Contributor

@huydhn huydhn commented Jul 1, 2025 •
edited
Loading

This unblocks pytorch/test-infra#6869. The key changes to call out:

  • B200 needs OIDC to access ECR and upload stats to S3, so we need to set id-token: write in _linux-test. All workflows calling _linux-test also need to be updated accordingly
  • Connecting sccache to S3 on B200 doesn't seem to work, so I disable it. It still works locally though.

Testing

https://github.com/pytorch/pytorch/actions/runs/16055549292/job/45312298376

@huydhn
Update _linux-test to support B200 runner
8f29154 
Signed-off-by: Huy Do <huydhn@gmail.com>
@pytorch-bot
Copy link

pytorch-bot bot commented Jul 1, 2025 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/157341

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 45d6a75 with merge base 6dc2b22 (image):
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@pytorch-bot pytorch-bot bot added the topic: not user facing topic category label Jul 1, 2025
@huydhn huydhn mentioned this pull request Jul 1, 2025
Closed
huydhn added 2 commits July 1, 2025 14:55
@huydhn
Need to set aws-role-to-assume
9cae9a4 
Signed-off-by: Huy Do <huydhn@gmail.com>
@huydhn
Need id-token write
b117273 
Signed-off-by: Huy Do <huydhn@gmail.com>
@nWEIdia
Copy link
Collaborator

nWEIdia commented Jul 2, 2025 •
edited
Loading

time to open up the docker permissions on the runner? To fix this ?

@huydhn
Copy link
Contributor Author

huydhn commented Jul 2, 2025 •
edited
Loading

time to open up the docker permissions on the runner? To fix this ?

Yes, we need to make sure that docker daemon is running on the runner, and whatever the user that runs GitHub daemon is in docker group so that it has the necessary permission. Could you help set that up plz?

From what I see in https://github.com/pytorch/pytorch/actions/runs/16011916708/job/45173325662?pr=157341#step:9:67, at least the logging in to ECR is ok now and docker can see the image https://github.com/pytorch/pytorch/actions/runs/16011916708/job/45173325662?pr=157341#step:9:67

@huydhn
Copy link
Contributor Author

huydhn commented Jul 2, 2025 •
edited
Loading

@nWEIdia If possible, could you also help add awscli to the runner to address this issue https://github.com/pytorch/pytorch/actions/runs/16011916708/job/45173325662?pr=157341#step:9:52. Usually, a pip install awscli command will do

Another question: Does the GitHub daemon user have sudo?

@nWEIdia
Copy link
Collaborator

nWEIdia commented Jul 2, 2025

Both of the above steps are done:

  1. docker command is runnable without encountering the daemon issue
  2. aws command could be found.
    I am restarting your job.

huydhn added 10 commits July 1, 2025 22:05
@huydhn
Skip setup-nvidia
7a318ca 
Signed-off-by: Huy Do <huydhn@gmail.com>
@huydhn
Setup python on B200
7a56b91 
Signed-off-by: Huy Do <huydhn@gmail.com>
@huydhn
Wrong commit
b9d5fbf 
Signed-off-by: Huy Do <huydhn@gmail.com>
@huydhn
Store GitHub env variables
d0cd65d 
Signed-off-by: Huy Do <huydhn@gmail.com>
@huydhn
Attempt to not use remote S3 bucket for sccache
c7316d2 
Signed-off-by: Huy Do <huydhn@gmail.com>
@huydhn
Try to use sccache locally
170ea95 
Signed-off-by: Huy Do <huydhn@gmail.com>
@huydhn
Does this work?
db0af80 
Signed-off-by: Huy Do <huydhn@gmail.com>
@huydhn
Update permissions on other workflows
6615dc2 
Signed-off-by: Huy Do <huydhn@gmail.com>
@huydhn
Need to load common-build
209632b 
Signed-off-by: Huy Do <huydhn@gmail.com>
@huydhn
Ready for review
6c17fd5 
Signed-off-by: Huy Do <huydhn@gmail.com>
@huydhn huydhn requested a review from ZainRizvi July 3, 2025 18:10
@huydhn huydhn marked this pull request as ready for review July 3, 2025 18:10
@huydhn huydhn requested a review from a team as a code owner July 3, 2025 18:10
@huydhn huydhn requested a review from nWEIdia July 3, 2025 18:11
nWEIdia
nWEIdia approved these changes Jul 3, 2025
View reviewed changes
Copy link
Collaborator

@nWEIdia nWEIdia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the successful B200 job! https://github.com/pytorch/pytorch/actions/runs/16055549292/job/45312298376

@huydhn huydhn requested a review from seemethere July 4, 2025 00:46
@huydhn
Copy link
Contributor Author

huydhn commented Jul 4, 2025

@pytorchbot merge

@pytorch-bot pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Jul 4, 2025
@pytorchmergebot
Copy link
Collaborator

pytorchmergebot commented Jul 4, 2025

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

@pytorchmergebot
Copy link
Collaborator

pytorchmergebot commented Jul 4, 2025

Merge failed

Reason: 1 jobs have failed, first few of them are: trunk / macos-py3-arm64 / build

Details for Dev Infra team Raised by workflow job

huydhn added 3 commits July 4, 2025 12:26
@huydhn
Fix bash syntax
a23bede 
Signed-off-by: Huy Do <huydhn@gmail.com>
@huydhn
Weird bug in macOS build workflow
3df91fb 
Signed-off-by: Huy Do <huydhn@gmail.com>
@huydhn
I was silly
45d6a75 
Signed-off-by: Huy Do <huydhn@gmail.com>
@huydhn
Copy link
Contributor Author

huydhn commented Jul 4, 2025

@pytorchbot merge

@pytorchmergebot
Copy link
Collaborator

pytorchmergebot commented Jul 4, 2025

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

@chuanqi129 chuanqi129 mentioned this pull request Jul 7, 2025
Closed
@huydhn huydhn mentioned this pull request Jul 7, 2025
Closed
pytorchmergebot pushed a commit that referenced this pull request Jul 7, 2025
@chuanqi129 @pytorchmergebot
[CI] Fix xpu ci test sccache issue (#157693)
8a47f9d 
With PR #157341 land, it broken the PXU CI test on sccache which has been disabled by #143851. Re-disable it
Pull Request resolved: #157693
Approved by: https://github.com/atalman, https://github.com/huydhn
@d4l3k d4l3k mentioned this pull request Jul 8, 2025
Closed
pytorchmergebot pushed a commit that referenced this pull request Jul 9, 2025
@d4l3k @pytorchmergebot
[ci] fix h100-distributed (#157826)
3eb7084 
This was broken by #157341

This should resolve the permission issue
Pull Request resolved: #157826
Approved by: https://github.com/fduwjj, https://github.com/Skylion007, https://github.com/huydhn
@github-actions github-actions bot deleted the support-b200-runners branch August 4, 2025 02:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@malfet malfet malfet approved these changes

@atalman atalman atalman approved these changes

@nWEIdia nWEIdia nWEIdia approved these changes

@ZainRizvi ZainRizvi Awaiting requested review from ZainRizvi

@seemethere seemethere Awaiting requested review from seemethere

Assignees

No one assigned

Labels

ciflow/trunk Trigger trunk jobs on your pull request Merged test-config/default topic: not user facing topic category

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@huydhn @nWEIdia @pytorchmergebot @malfet @atalman