fix by wz337 · Pull Request #136366 · pytorch/pytorch

wz337 · 2024-09-20T17:35:58Z

Stack from ghstack (oldest at bottom):

cc @XilunWu @H-Huang @awgu @kwen2501 @wanchaol @fegin @fduwjj @wconstab @d4l3k @c-p-i-o @LucasLLC @MeetVadakkanchery @mhorowitz @pradeepfn

[ghstack-poisoned]

pytorch-bot · 2024-09-20T17:36:01Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/136366

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 56 Cancelled Jobs

As of commit 2249c86 with merge base d3647d1 ():

NEW FAILURE - The following job has failed:

Check Labels / Check labels (gh)
# This PR needs a release notes: label

CANCELLED JOBS - The following jobs were cancelled. Please retry:

Lint / lintrunner-clang / linux-job (gh)
##[error]The operation was canceled.
Lint / lintrunner-noclang / linux-job (gh)
##[error]The operation was canceled.
pull / cuda12.1-py3.10-gcc9-sm75 (gh)
pull / cuda12.1-py3.10-gcc9-sm75 / build (gh)
##[error]The operation was canceled.
pull / linux-docs (gh)
pull / linux-focal-cpu-py3.10-gcc9-bazel-test / build-and-test (default, 1, 1, lf.linux.4xlarge) (gh)
##[error]The operation was canceled.
pull / linux-focal-cuda11.8-py3.10-gcc9 (gh)
pull / linux-focal-cuda11.8-py3.10-gcc9 / build (gh)
##[error]The operation was canceled.
pull / linux-focal-cuda12.1-py3.10-gcc9 / build (gh)
##[error]The operation was canceled.
pull / linux-focal-cuda12.1-py3.10-gcc9-bazel-test / build-and-test (default, 1, 1, lf.linux.4xlarge.nvidia.gpu) (gh)
##[error]The operation was canceled.
pull / linux-focal-cuda12.1-py3.10-gcc9-sm86 (gh)
pull / linux-focal-cuda12.1-py3.10-gcc9-sm86 / build (gh)
##[error]The operation was canceled.
pull / linux-focal-cuda12.4-py3.10-gcc9-bazel-test / build-and-test (default, 1, 1, lf.linux.4xlarge.nvidia.gpu) (gh)
##[error]The operation was canceled.
pull / linux-focal-py3_9-clang9-xla (gh)
pull / linux-focal-py3_9-clang9-xla / build (gh)
##[error]The operation was canceled.
pull / linux-focal-py3-clang9-android-ndk-r21e-gradle-custom-build-single / build-and-test (default, 1, 1, linux.2xlarge) (gh)
##[error]The operation was canceled.
pull / linux-focal-py3-clang9-android-ndk-r21e-gradle-custom-build-single-full-jit / build-and-test (default, 1, 1, linux.2xlarge) (gh)
##[error]The operation was canceled.
pull / linux-focal-py3.11-clang10 (gh)
pull / linux-focal-py3.11-clang10 / build (gh)
pull / linux-focal-py3.12-clang10 / test (default, 1, 4, lf.linux.2xlarge) (gh)
##[error]The operation was canceled.
pull / linux-focal-py3.12-clang10 / test (default, 2, 4, lf.linux.2xlarge) (gh)
##[error]The operation was canceled.
pull / linux-focal-py3.12-clang10 / test (default, 3, 4, lf.linux.2xlarge) (gh)
##[error]The operation was canceled.
pull / linux-focal-py3.12-clang10 / test (default, 4, 4, lf.linux.2xlarge) (gh)
##[error]The operation was canceled.
pull / linux-focal-py3.12-clang10 / test (dynamo, 1, 3, lf.linux.2xlarge) (gh)
##[error]The operation was canceled.
pull / linux-focal-py3.12-clang10 / test (dynamo, 2, 3, lf.linux.2xlarge) (gh)
##[error]The operation was canceled.
pull / linux-focal-py3.12-clang10 / test (dynamo, 3, 3, lf.linux.2xlarge) (gh)
##[error]The operation was canceled.
pull / linux-focal-py3.12-clang10-experimental-split-build / test (default, 1, 3, linux.2xlarge) (gh)
##[error]The operation was canceled.
pull / linux-focal-py3.12-clang10-experimental-split-build / test (default, 2, 3, linux.2xlarge) (gh)
##[error]The operation was canceled.
pull / linux-focal-py3.12-clang10-experimental-split-build / test (default, 3, 3, linux.2xlarge) (gh)
##[error]The operation was canceled.
pull / linux-focal-py3.12-clang10-experimental-split-build / test (dynamo, 1, 3, linux.2xlarge) (gh)
##[error]The operation was canceled.
pull / linux-focal-py3.12-clang10-experimental-split-build / test (dynamo, 2, 3, linux.2xlarge) (gh)
##[error]The operation was canceled.
pull / linux-focal-py3.12-clang10-experimental-split-build / test (dynamo, 3, 3, linux.2xlarge) (gh)
##[error]The operation was canceled.
pull / linux-focal-py3.9-clang10 / test (crossref, 1, 2, lf.linux.2xlarge) (gh)
##[error]The operation was canceled.
pull / linux-focal-py3.9-clang10 / test (crossref, 2, 2, lf.linux.2xlarge) (gh)
##[error]The operation was canceled.
pull / linux-focal-py3.9-clang10 / test (default, 1, 4, lf.linux.2xlarge) (gh)
pull / linux-focal-py3.9-clang10 / test (default, 2, 4, lf.linux.2xlarge) (gh)
##[error]The operation was canceled.
pull / linux-focal-py3.9-clang10 / test (default, 3, 4, lf.linux.2xlarge) (gh)
##[error]The operation was canceled.
pull / linux-focal-py3.9-clang10 / test (default, 4, 4, lf.linux.2xlarge) (gh)
##[error]The operation was canceled.
pull / linux-focal-py3.9-clang10 / test (dynamo, 1, 3, lf.linux.2xlarge) (gh)
##[error]The operation was canceled.
pull / linux-focal-py3.9-clang10 / test (dynamo, 2, 3, lf.linux.2xlarge) (gh)
##[error]The operation was canceled.
pull / linux-focal-py3.9-clang10 / test (dynamo, 3, 3, lf.linux.2xlarge) (gh)
##[error]The operation was canceled.
pull / linux-focal-py3.9-clang10-onnx / test (default, 1, 2, lf.linux.2xlarge) (gh)
##[error]The operation was canceled.
pull / linux-focal-py3.9-clang10-onnx / test (default, 2, 2, lf.linux.2xlarge) (gh)
##[error]The operation was canceled.
pull / linux-focal-rocm6.1-py3.8 / build (gh)
##[error]The operation was canceled.
pull / linux-jammy-cuda11.8-cudnn9-py3.9-clang12 / build (gh)
##[error]The operation was canceled.
pull / linux-jammy-py3-clang12-executorch / test (executorch, 1, 1, lf.linux.2xlarge) (gh)
##[error]The operation was canceled.
pull / linux-jammy-py3.10-clang15-asan / test (default, 1, 6, lf.linux.4xlarge) (gh)
##[error]The operation was canceled.
pull / linux-jammy-py3.10-clang15-asan / test (default, 2, 6, lf.linux.4xlarge) (gh)
##[error]The operation was canceled.
pull / linux-jammy-py3.10-clang15-asan / test (default, 3, 6, lf.linux.4xlarge) (gh)
##[error]The operation was canceled.
pull / linux-jammy-py3.10-clang15-asan / test (default, 4, 6, lf.linux.4xlarge) (gh)
##[error]The operation was canceled.
pull / linux-jammy-py3.10-clang15-asan / test (default, 5, 6, lf.linux.4xlarge) (gh)
##[error]The operation was canceled.
pull / linux-jammy-py3.10-clang15-asan / test (default, 6, 6, lf.linux.4xlarge) (gh)
##[error]The operation was canceled.
pull / linux-jammy-py3.9-gcc11 (gh)
pull / linux-jammy-py3.9-gcc11 / build (gh)
##[error]The operation was canceled.
pull / linux-jammy-py3.9-gcc11-pch / build (gh)
##[error]The operation was canceled.
pull / win-vs2019-cpu-py3 / build (gh)
##[error]The operation was canceled.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

ghstack-source-id: 02e042a Pull Request resolved: #136366

[ghstack-poisoned]

ghstack-source-id: bf2f3b9 Pull Request resolved: #136366

ghstack-source-id: e6b781e Pull Request resolved: #136365 fix ghstack-source-id: e6b781e Pull Request resolved: #136366

Update

8f26973

[ghstack-poisoned]

wz337 mentioned this pull request Sep 20, 2024

[DSD] Fix loading uneven full tensor into sharded state dict #136365

Closed

pytorch-bot bot added module: distributed_checkpoint oncall: distributed Add this issue/PR to distributed oncall triage queue labels Sep 20, 2024

wz337 added a commit that referenced this pull request Sep 20, 2024

fix

84168bd

ghstack-source-id: 02e042a Pull Request resolved: #136366

Update

2249c86

[ghstack-poisoned]

wz337 added a commit that referenced this pull request Sep 20, 2024

fix

b89c0c7

ghstack-source-id: bf2f3b9 Pull Request resolved: #136366

wz337 added a commit that referenced this pull request Sep 20, 2024

add shape and stride from local state

ba586ef

ghstack-source-id: e6b781e Pull Request resolved: #136365 fix ghstack-source-id: e6b781e Pull Request resolved: #136366

wz337 closed this Sep 20, 2024

github-actions bot deleted the gh/wz337/33/head branch October 21, 2024 02:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix#136366

fix#136366
wz337 wants to merge 2 commits intogh/wz337/33/basefrom
gh/wz337/33/head

wz337 commented Sep 20, 2024 •

edited by pytorch-bot bot

Loading

Uh oh!

pytorch-bot bot commented Sep 20, 2024 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

wz337 commented Sep 20, 2024 • edited by pytorch-bot bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Sep 20, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/136366

❌ 1 New Failure, 56 Cancelled Jobs

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

wz337 commented Sep 20, 2024 •

edited by pytorch-bot bot

Loading

pytorch-bot bot commented Sep 20, 2024 •

edited

Loading