Skip to content

Fix the skip condition for test_c10d tests#119938

Closed
tinglvv wants to merge 1 commit intopytorch:mainfrom
tinglvv:tingl-c10d-fix
Closed

Fix the skip condition for test_c10d tests#119938
tinglvv wants to merge 1 commit intopytorch:mainfrom
tinglvv:tingl-c10d-fix

Conversation

@tinglvv
Copy link
Copy Markdown
Collaborator

@tinglvv tinglvv commented Feb 14, 2024 •
edited by pytorch-bot bot
Loading

Seeing the error for c10d tests when running on 1GPU. Adding the skip when there is insufficient GPU.

RuntimeError: CUDA error: invalid device ordinal
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

referring to #84980

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu @penguinwu @fegin @XilunWu @wanchaol @fduwjj @wz337 @tianyu-l @wconstab @yf225 @eqy @ptrblck

@pytorch-bot pytorch-bot bot added the topic: not user facing topic category label Feb 14, 2024
@pytorch-bot
Copy link
Copy Markdown

pytorch-bot bot commented Feb 14, 2024 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/119938

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (1 Unrelated Failure)

As of commit c2b81b7 with merge base 8f27fde (image):

FLAKY - The following job failed but was likely due to flakiness present on trunk:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@github-actions github-actions bot added the oncall: distributed Add this issue/PR to distributed oncall triage queue label Feb 14, 2024
fegin
fegin approved these changes Feb 15, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@fegin fegin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fix!

@tinglvv
Copy link
Copy Markdown
Collaborator Author

tinglvv commented Feb 15, 2024

Thanks for the stamp! @fegin

@fegin fegin added the ciflow/trunk Trigger trunk jobs on your pull request label Feb 15, 2024
@fegin
Copy link
Copy Markdown
Contributor

fegin commented Feb 15, 2024

@pytorchbot merge

@pytorchmergebot
Copy link
Copy Markdown
Collaborator

pytorchmergebot commented Feb 15, 2024

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

@tinglvv tinglvv deleted the tingl-c10d-fix branch February 15, 2024 22:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@eqy eqy eqy approved these changes

@fegin fegin fegin approved these changes

Assignees

No one assigned

Labels

ciflow/trunk Trigger trunk jobs on your pull request Merged oncall: distributed Add this issue/PR to distributed oncall triage queue open source topic: not user facing topic category

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@tinglvv @fegin @pytorchmergebot @eqy @pytorchbot