[DCP] Fixes the stateless optimizer issue of distributed state_dict by fegin · Pull Request #135535 · pytorch/pytorch

fegin · 2024-09-09T21:13:45Z

Stack from ghstack (oldest at bottom):

-> [DCP] Fixes the stateless optimizer issue of distributed state_dict #135535

Some optimizers don't have states that can cause get_state_dict/set_state_dict behave incorrectly. This PR fixes the issues.

fixes: #133415

cc @XilunWu @H-Huang @awgu @kwen2501 @wanchaol @fduwjj @wz337 @wconstab @d4l3k @c-p-i-o @LucasLLC @MeetVadakkanchery @mhorowitz @pradeepfn

[ghstack-poisoned]

pytorch-bot · 2024-09-09T21:13:48Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/135535

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 5 Unrelated Failures

As of commit 87dc47f with merge base 9b76449 ():

NEW FAILURE - The following job has failed:

trunk / linux-focal-cuda12.4-py3.10-gcc9-sm86 / test (default, 5, 5, linux.g5.4xlarge.nvidia.gpu) (gh)
'test/inductor/test_cudacodecache.py::TestCUDACodeCache::test_async_compile'

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

pull / linux-docs / build-docs-python-false (gh) (trunk failure)
pull / linux-focal-py3.11-clang10 / test (dynamo, 3, 3, linux.2xlarge) (gh) (trunk failure)
test_nestedtensor.py::TestNestedTensorSubclassCPU::test_to_padded_tensor_nt_dim_2_requires_grad_False_cpu_float16
pull / linux-focal-py3.12-clang10 / test (dynamo, 3, 3, linux.2xlarge) (gh) (trunk failure)
test_nestedtensor.py::TestNestedTensorSubclassCPU::test_to_padded_tensor_nt_dim_2_requires_grad_False_cpu_float16
pull / linux-focal-py3.12-clang10-experimental-split-build / test (dynamo, 3, 3, linux.2xlarge) (gh) (trunk failure)
test_nestedtensor.py::TestNestedTensorSubclassCPU::test_to_padded_tensor_nt_dim_2_requires_grad_False_cpu_float16
pull / linux-focal-py3.9-clang10 / test (dynamo, 3, 3, linux.2xlarge) (gh) (trunk failure)
test_nestedtensor.py::TestNestedTensorSubclassCPU::test_to_padded_tensor_nt_dim_2_requires_grad_False_cpu_float16

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[ghstack-poisoned]

wz337

LGTM. stamp to unblock

wz337 · 2024-09-09T22:14:14Z

test/distributed/checkpoint/test_state_dict.py

            {
                "use_composable": [True, False],
-                "optimizer_class": [torch.optim.Adam, torch.optim.AdamW],
+                "optimizer_class": [torch.optim.SGD],


Do we want to add torch.optim.SGD to the original list instead?

[ghstack-poisoned]

Some optimizers don't have states that can cause get_state_dict/set_state_dict behave incorrectly. This PR fixes the issues. fixes: #133415 ghstack-source-id: 7dd1c09 Pull Request resolved: #135535

fegin · 2024-09-10T01:44:33Z

@pytorchbot mege

pytorch-bot · 2024-09-10T01:44:35Z

❌ 🤖 pytorchbot command failed:

@pytorchbot: error: argument command: invalid choice: 'mege' (choose from 'merge', 'revert', 'rebase', 'label', 'drci', 'cherry-pick', 'close')

usage: @pytorchbot [-h] {merge,revert,rebase,label,drci,cherry-pick,close} ...

Try @pytorchbot --help for more info.

fegin · 2024-09-10T01:45:36Z

@pytorchbot merge

pytorchmergebot · 2024-09-10T01:47:14Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2024-09-10T01:47:26Z

Merge failed

Reason: 1 jobs have failed, first few of them are: trunk / linux-focal-cuda12.4-py3.10-gcc9-sm86 / test (default, 5, 5, linux.g5.4xlarge.nvidia.gpu)

Details for Dev Infra team

Raised by workflow job

fegin · 2024-09-10T03:08:11Z

@pytorchbot merge -f "The failing test is not related."

pytorchmergebot · 2024-09-10T03:09:47Z

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

…ytorch#135535) Some optimizers don't have states that can cause get_state_dict/set_state_dict behave incorrectly. This PR fixes the issues. fixes: pytorch#133415 Pull Request resolved: pytorch#135535 Approved by: https://github.com/wz337

#136000) [DCP] Fixes the stateless optimizer issue of distributed state_dict (#135535) Some optimizers don't have states that can cause get_state_dict/set_state_dict behave incorrectly. This PR fixes the issues. fixes: #133415 Pull Request resolved: #135535 Approved by: https://github.com/wz337 Co-authored-by: Chien-Chin Huang <chienchin@fb.com>

Update

4cb7e4c

[ghstack-poisoned]

pytorch-bot bot added module: distributed_checkpoint oncall: distributed Add this issue/PR to distributed oncall triage queue labels Sep 9, 2024

fegin requested a review from wz337 September 9, 2024 21:13

fegin added the ciflow/trunk Trigger trunk jobs on your pull request label Sep 9, 2024

Update

73b14d4

[ghstack-poisoned]

Update

a2c792b

[ghstack-poisoned]

fegin added module: distributed_checkpoint and removed module: distributed_checkpoint labels Sep 9, 2024

Update

5c4acc0

[ghstack-poisoned]

fegin added the release notes: distributed (checkpoint) label Sep 9, 2024

wz337 approved these changes Sep 9, 2024

View reviewed changes

Update

87dc47f

[ghstack-poisoned]

pytorchmergebot added the merging label Sep 10, 2024

pytorchmergebot removed the merging label Sep 10, 2024

pytorchmergebot added the merging label Sep 10, 2024

pytorchmergebot added the Merged label Sep 10, 2024

pytorchmergebot closed this in 1d9feff Sep 10, 2024

pytorchmergebot removed the merging label Sep 10, 2024

This was referenced Sep 13, 2024

[DCP] Fixes the stateless optimizer issue of distributed state_dict (… #136000

Merged

[v.2.5.0] Release Tracker #135522

Closed

github-actions bot deleted the gh/fegin/288/head branch October 12, 2024 02:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[DCP] Fixes the stateless optimizer issue of distributed state_dict#135535

[DCP] Fixes the stateless optimizer issue of distributed state_dict#135535
fegin wants to merge 5 commits intogh/fegin/288/basefrom
gh/fegin/288/head

fegin commented Sep 9, 2024 •

edited by pytorch-bot bot

Loading

Uh oh!

pytorch-bot bot commented Sep 9, 2024 •

edited

Loading

Uh oh!

wz337 left a comment •

edited

Loading

Uh oh!

wz337 Sep 9, 2024

Uh oh!

fegin Sep 9, 2024

Uh oh!

fegin commented Sep 10, 2024

Uh oh!

pytorch-bot bot commented Sep 10, 2024

Uh oh!

fegin commented Sep 10, 2024

Uh oh!

pytorchmergebot commented Sep 10, 2024

Uh oh!

pytorchmergebot commented Sep 10, 2024

Uh oh!

fegin commented Sep 10, 2024

Uh oh!

pytorchmergebot commented Sep 10, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

fegin commented Sep 9, 2024 • edited by pytorch-bot bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Sep 9, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/135535

❌ 1 New Failure, 5 Unrelated Failures

Uh oh!

wz337 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wz337 Sep 9, 2024

Choose a reason for hiding this comment

Uh oh!

fegin Sep 9, 2024

Choose a reason for hiding this comment

Uh oh!

fegin commented Sep 10, 2024

Uh oh!

pytorch-bot bot commented Sep 10, 2024

Uh oh!

fegin commented Sep 10, 2024

Uh oh!

pytorchmergebot commented Sep 10, 2024

Merge started

Uh oh!

pytorchmergebot commented Sep 10, 2024

Merge failed

Uh oh!

fegin commented Sep 10, 2024

Uh oh!

pytorchmergebot commented Sep 10, 2024

Merge started

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

fegin commented Sep 9, 2024 •

edited by pytorch-bot bot

Loading

pytorch-bot bot commented Sep 9, 2024 •

edited

Loading

wz337 left a comment •

edited

Loading