Skip to content

[SPMD] Add debug of SPMD for single/multi host#5742

Merged
ManfeiBai merged 53 commits intomasterfrom
newmastercheckcommitoct19
Dec 5, 2023
Merged

[SPMD] Add debug of SPMD for single/multi host#5742
ManfeiBai merged 53 commits intomasterfrom
newmastercheckcommitoct19

Conversation

@ManfeiBai
Copy link
Copy Markdown
Collaborator

@ManfeiBai ManfeiBai commented Oct 26, 2023 •
edited
Loading

Add debug of SPMD for single/multi device for visualize sharded tensors

@ManfeiBai ManfeiBai changed the title [debug] add debug for FSDP for single device [debug] add debug for SPMD for single device Oct 26, 2023
@yeounoh yeounoh self-requested a review October 26, 2023 17:17
@yeounoh yeounoh added the distributed SPMD and other distributed things. label Oct 26, 2023
@ManfeiBai ManfeiBai force-pushed the newmastercheckcommitoct19 branch from ba40a9b to ef482ea Compare October 26, 2023 17:45
@yeounoh
Copy link
Copy Markdown
Contributor

yeounoh commented Oct 27, 2023

Hi @ManfeiBai could you also move the file to torch_xla/experimental/spmd/ you can create a new dir, I will refactor in the upcoming release.

@ManfeiBai
Copy link
Copy Markdown
Collaborator Author

ManfeiBai commented Nov 2, 2023

Hi @ManfeiBai could you also move the file to torch_xla/experimental/spmd/ you can create a new dir, I will refactor in the upcoming release.

Thanks, updated to torch_xla/experimental/spmd/ and created torch_xla/experimental/spmd/ too

@ManfeiBai ManfeiBai marked this pull request as ready for review November 8, 2023 19:38
@ManfeiBai ManfeiBai changed the title [debug] add debug for SPMD for single device Add debug of SPMD for single device Nov 8, 2023
@ManfeiBai ManfeiBai marked this pull request as draft November 8, 2023 19:41
ManfeiBai
ManfeiBai commented Nov 23, 2023
View reviewed changes
Comment thread torch_xla/distributed/spmd/test_debugging.py Outdated
@ManfeiBai ManfeiBai marked this pull request as ready for review November 26, 2023 22:49
ManfeiBai
ManfeiBai commented Nov 27, 2023
View reviewed changes
Comment thread torch_xla/distributed/spmd/__init__.py Outdated
yeounoh
yeounoh reviewed Nov 28, 2023
View reviewed changes
Comment thread test/spmd/test_spmd_debugging.py Outdated
yeounoh
yeounoh reviewed Nov 28, 2023
View reviewed changes
Comment thread test/spmd/test_spmd_debugging.py Outdated
yeounoh
yeounoh reviewed Nov 28, 2023
View reviewed changes
Comment thread test/spmd/test_spmd_debugging.py Outdated
yeounoh
yeounoh reviewed Nov 28, 2023
View reviewed changes
Comment thread test/spmd/test_spmd_debugging.py Outdated
yeounoh
yeounoh reviewed Nov 28, 2023
View reviewed changes
Comment thread test/spmd/test_spmd_debugging.py Outdated
yeounoh
yeounoh reviewed Nov 28, 2023
View reviewed changes
Comment thread test/spmd/test_spmd_debugging.py Outdated
yeounoh
yeounoh reviewed Nov 28, 2023
View reviewed changes
Comment thread test/spmd/test_spmd_debugging.py Outdated
yeounoh
yeounoh reviewed Nov 28, 2023
View reviewed changes
Comment thread torch_xla/distributed/spmd/debugging.py Outdated
@ManfeiBai ManfeiBai changed the title Add debug of SPMD for single device Add debug of SPMD for single host Nov 28, 2023
yeounoh
yeounoh reviewed Nov 28, 2023
View reviewed changes
Comment thread torch_xla/distributed/spmd/debugging.py Outdated
yeounoh
yeounoh reviewed Nov 28, 2023
View reviewed changes
Comment thread torch_xla/distributed/spmd/debugging.py Outdated
yeounoh
yeounoh reviewed Nov 28, 2023
View reviewed changes
Comment thread torch_xla/distributed/spmd/debugging.py Outdated
yeounoh
yeounoh suggested changes Nov 28, 2023
View reviewed changes
Copy link
Copy Markdown
Contributor

@yeounoh yeounoh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left comments, thanks @ManfeiBai

@ManfeiBai ManfeiBai force-pushed the newmastercheckcommitoct19 branch 2 times, most recently from c83a2a5 to 281685b Compare November 29, 2023 00:25
@ManfeiBai ManfeiBai requested a review from yeounoh December 4, 2023 02:18
@ManfeiBai ManfeiBai mentioned this pull request Dec 4, 2023
Closed
yeounoh
yeounoh reviewed Dec 4, 2023
View reviewed changes
Comment thread torch_xla/distributed/spmd/__init__.py Outdated
yeounoh
yeounoh approved these changes Dec 4, 2023
View reviewed changes
Copy link
Copy Markdown
Contributor

@yeounoh yeounoh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, left one more minor comment https://github.com/pytorch/xla/pull/5742/files#r1414546255

@ManfeiBai ManfeiBai merged commit f94b53c into master Dec 5, 2023
chunnienc pushed a commit to chunnienc/xla that referenced this pull request Dec 14, 2023
@ManfeiBai @chunnienc
[SPMD] Add debug of SPMD for single/multi host (pytorch#5742)
5175072
golechwierowicz pushed a commit that referenced this pull request Jan 12, 2024
@ManfeiBai @golechwierowicz
[SPMD] Add debug of SPMD for single/multi host (#5742)
4d8c0c8
bhavya01 pushed a commit that referenced this pull request Apr 22, 2024
@ManfeiBai @bhavya01
[SPMD] Add debug of SPMD for single/multi host (#5742)
071a0c4
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@JackCaoG JackCaoG Awaiting requested review from JackCaoG

1 more reviewer

@yeounoh yeounoh yeounoh approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@ManfeiBai ManfeiBai

Labels

backport_2.2 distributed SPMD and other distributed things.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@ManfeiBai @yeounoh @JackCaoG