Enable capturing of comm collective parameters (#98) by louisfeng · Pull Request #85368 · pytorch/pytorch

louisfeng · 2022-09-20T19:50:23Z

Summary:
X-link: facebookresearch/torch_ucc#98

Add tensor input, output, and other metadata for PyTorch comms.

Test Plan: P517138779

Reviewed By: Pavani-Panakanti

Differential Revision: D38357077

pytorch-bot · 2022-09-20T19:50:26Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/85368

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures, 1 Pending

As of commit 7dcc84c:
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

facebook-github-bot · 2022-09-20T19:50:59Z

This pull request was exported from Phabricator. Differential Revision: D38357077

facebook-github-bot · 2022-09-20T21:01:57Z

This pull request was exported from Phabricator. Differential Revision: D38357077

facebook-github-bot · 2022-09-20T22:19:22Z

This pull request was exported from Phabricator. Differential Revision: D38357077

Summary: X-link: pytorch/pytorch#85368 Pull Request resolved: facebookresearch#98 Add tensor input, output, and other metadata for PyTorch comms. Reviewed By: Pavani-Panakanti Differential Revision: D38357077 fbshipit-source-id: 144852fffaba47434d15f121970121b2419026a1

facebook-github-bot · 2022-09-20T23:08:14Z

This pull request was exported from Phabricator. Differential Revision: D38357077

Summary: X-link: pytorch/pytorch#85368 Pull Request resolved: facebookresearch#98 Add tensor input, output, and other metadata for PyTorch comms. Reviewed By: Pavani-Panakanti Differential Revision: D38357077 fbshipit-source-id: 6945ed29905be2466fe1056f5ad77b332da70005

facebook-github-bot · 2022-09-21T02:50:07Z

This pull request was exported from Phabricator. Differential Revision: D38357077

H-Huang

Hi, thanks for the PR. I am actually not familiar with the original use case for RECORD_PARAM_COMMS. Where does it get logged and is there a way to see the before / after of this change? Thanks!

H-Huang · 2022-09-26T17:03:44Z

torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp

Reason for adding this? (since initialization is not a collective)

The network.ai team requested this feature. We can collect when a network collective is created during initialization. cc: Pavani.

H-Huang · 2022-09-26T17:05:52Z

torch/csrc/distributed/c10d/ParamCommsUtils.hpp

Since this is a general comm util. It probably shouldn't be named WorkNCCL (a backend specific object) and make instead named to just Work instead? I have another comment to clarify what is actually getting logged here.

Yes, makes sense, will fix.

This supposed to be the ProcessGroup object, with the seq number can uniquely identify each collective operation.

H-Huang · 2022-09-26T17:07:01Z

torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp

Confused about recording this. Isn't this a pointer to ProcessGroupNCCL rather than a work obj? Same applies to the other collectives

Modified the comment, the code is correct, but the comment was wrong. Should be the process group object.

Summary: X-link: pytorch/pytorch#85368 Pull Request resolved: facebookresearch#98 Add tensor input, output, and other metadata for PyTorch comms. Reviewed By: Pavani-Panakanti Differential Revision: D38357077 fbshipit-source-id: 51f4c616816e8fb5bc72e4d713df24a7fecdcd41

H-Huang

LGTM, thanks for updating

facebook-github-bot · 2022-10-04T00:17:53Z

/easycla

As part of the transition to the PyTorch Foundation, this project now requires contributions be covered under the new CLA. See #85559 for additional details.

This comment will trigger a new check of this PR. If you are already covered, you will simply see a new "EasyCLA" check that passes. If you are not covered, a bot will leave a new comment with a link to sign.

linux-foundation-easycla · 2022-10-04T00:17:56Z

The committers listed above are authorized under a signed CLA.

✅ login: louisfeng / name: Louis Feng (294a0ed)

Summary: Pull Request resolved: pytorch#85368 X-link: facebookresearch/torch_ucc#98 Add tensor input, output, and other metadata for PyTorch comms. Test Plan: P517138779 Reviewed By: Pavani-Panakanti Differential Revision: D38357077 fbshipit-source-id: 71c256c6355abe8c384508d450c33fe8f8a51da4

facebook-github-bot · 2022-10-04T16:40:31Z

This pull request was exported from Phabricator. Differential Revision: D38357077

facebook-github-bot · 2022-10-06T16:20:29Z

This pull request was exported from Phabricator. Differential Revision: D38357077

linux-foundation-easycla · 2022-10-06T16:20:36Z

The committers listed above are authorized under a signed CLA.

✅ login: louisfeng / name: Louis Feng (7dcc84c)

Summary: Pull Request resolved: pytorch#85368 X-link: facebookresearch/torch_ucc#98 Add tensor input, output, and other metadata for PyTorch comms. Test Plan: P517138779 Reviewed By: Pavani-Panakanti Differential Revision: D38357077 fbshipit-source-id: 379a6b951d2051967cb4c90391241e42456c6497

facebook-github-bot · 2022-10-07T22:16:21Z

This pull request was exported from Phabricator. Differential Revision: D38357077

Summary: Pull Request resolved: pytorch#85368 X-link: facebookresearch/torch_ucc#98 Add tensor input, output, and other metadata for PyTorch comms. Test Plan: ``` buck build mode/opt-split-dwarf -c fbcode.nvcc_arch=a100 //hpc/models/ads:ads_10x_launcher buck-out/gen/hpc/models/ads/ads_10x_launcher.par -- +launcher=local launcher.num_trainers=2 +data_loader=random ``` P517138779 Reviewed By: Pavani-Panakanti Differential Revision: D38357077 fbshipit-source-id: f339b00ab09d5b7a21d2607ef54d21ff078af501

Summary: Pull Request resolved: pytorch#85368 X-link: facebookresearch/torch_ucc#98 Add tensor input, output, and other metadata for PyTorch comms. Test Plan: ``` buck build mode/opt-split-dwarf -c fbcode.nvcc_arch=a100 //hpc/models/ads:ads_10x_launcher buck-out/gen/hpc/models/ads/ads_10x_launcher.par -- +launcher=local launcher.num_trainers=2 +data_loader=random ``` P517138779 Reviewed By: Pavani-Panakanti Differential Revision: D38357077 fbshipit-source-id: c5394ff04ebe6af223c0e0a69a9cb558cc4c3a3f

facebook-github-bot · 2022-10-10T21:14:47Z

This pull request was exported from Phabricator. Differential Revision: D38357077

Summary: X-link: pytorch/pytorch#85368 Pull Request resolved: #98 Add tensor input, output, and other metadata for PyTorch comms. Reviewed By: Pavani-Panakanti Differential Revision: D38357077 fbshipit-source-id: e9545eeea3311b68762781a5f6aa585971aa08fe

facebook-github-bot · 2022-10-11T04:33:40Z

@pytorchbot merge

(Initiating merge automatically since Phabricator Diff has merged)

pytorchmergebot · 2022-10-11T04:38:22Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

github-actions · 2022-10-11T04:39:12Z

Hey @louisfeng.
You've committed this PR, but it does not have both a 'release notes: ...' and 'topics: ...' label. Please add one of each to the PR. The 'release notes: ...' label should represent the part of PyTorch that this PR changes (fx, autograd, distributed, etc) and the 'topics: ...' label should represent the kind of PR it is (not user facing, new feature, bug fix, perf improvement, etc). The list of valid labels can be found here for the 'release notes: ...' and here for the 'topics: ...'.
For changes that are 'topic: not user facing' there is no need for a release notes label.

Summary: Pull Request resolved: #85368 X-link: facebookresearch/torch_ucc#98 Add tensor input, output, and other metadata for PyTorch comms. Test Plan: ``` buck build mode/opt-split-dwarf -c fbcode.nvcc_arch=a100 //hpc/models/ads:ads_10x_launcher buck-out/gen/hpc/models/ads/ads_10x_launcher.par -- +launcher=local launcher.num_trainers=2 +data_loader=random ``` P517138779 Reviewed By: Pavani-Panakanti Differential Revision: D38357077 fbshipit-source-id: e9545eeea3311b68762781a5f6aa585971aa08fe

louisfeng requested review from H-Huang, awgu, mingzhe09088, mrshenli, pritamdamania87, rohan-varma and zhaojuanmao as code owners September 20, 2022 19:50

pytorch-bot bot added the release notes: distributed (c10d) release notes category label Sep 20, 2022

facebook-github-bot added cla signed fb-exported labels Sep 20, 2022

facebook-github-bot added the oncall: distributed Add this issue/PR to distributed oncall triage queue label Sep 20, 2022

louisfeng force-pushed the export-D38357077 branch from de77af2 to 67d6c15 Compare September 20, 2022 21:01

louisfeng force-pushed the export-D38357077 branch from 67d6c15 to 58d1398 Compare September 20, 2022 22:19

louisfeng force-pushed the export-D38357077 branch from 58d1398 to 37f48cd Compare September 20, 2022 23:08

louisfeng force-pushed the export-D38357077 branch from 37f48cd to fcd9e3c Compare September 21, 2022 02:50

H-Huang requested a review from wz337 September 22, 2022 23:39

H-Huang reviewed Sep 26, 2022

View reviewed changes

louisfeng requested review from H-Huang and removed request for awgu, mingzhe09088, wz337 and zhaojuanmao October 3, 2022 20:18

H-Huang approved these changes Oct 3, 2022

View reviewed changes

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Oct 3, 2022

louisfeng force-pushed the export-D38357077 branch from 294a0ed to 7bec7ad Compare October 4, 2022 16:40

louisfeng force-pushed the export-D38357077 branch from 7bec7ad to 86ca64f Compare October 6, 2022 16:20

louisfeng force-pushed the export-D38357077 branch from 86ca64f to 32171c3 Compare October 7, 2022 22:16

louisfeng force-pushed the export-D38357077 branch from 32171c3 to 7dcc84c Compare October 10, 2022 21:14

pytorchmergebot added the Merged label Oct 11, 2022

pytorchmergebot closed this in 55479fe Oct 11, 2022

Conversation

louisfeng commented Sep 20, 2022

Uh oh!

pytorch-bot bot commented Sep 20, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/85368

✅ No Failures, 1 Pending

Uh oh!

facebook-github-bot commented Sep 20, 2022

Uh oh!

facebook-github-bot commented Sep 20, 2022

Uh oh!

facebook-github-bot commented Sep 20, 2022

Uh oh!

facebook-github-bot commented Sep 20, 2022

Uh oh!

facebook-github-bot commented Sep 21, 2022

Uh oh!

H-Huang left a comment

Choose a reason for hiding this comment

Uh oh!

H-Huang Sep 26, 2022

Choose a reason for hiding this comment

Uh oh!

louisfeng Sep 28, 2022

Choose a reason for hiding this comment

Uh oh!

H-Huang Sep 26, 2022

Choose a reason for hiding this comment

Uh oh!

louisfeng Sep 26, 2022

Choose a reason for hiding this comment

Uh oh!

louisfeng Sep 28, 2022

Choose a reason for hiding this comment

Uh oh!

H-Huang Sep 26, 2022

Choose a reason for hiding this comment

Uh oh!

louisfeng Sep 28, 2022

Choose a reason for hiding this comment

Uh oh!

H-Huang left a comment

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot commented Oct 4, 2022

Uh oh!

linux-foundation-easycla bot commented Oct 4, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

facebook-github-bot commented Oct 4, 2022

Uh oh!

facebook-github-bot commented Oct 6, 2022

Uh oh!

linux-foundation-easycla bot commented Oct 6, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

facebook-github-bot commented Oct 7, 2022

Uh oh!

facebook-github-bot commented Oct 10, 2022

Uh oh!

facebook-github-bot commented Oct 11, 2022

Uh oh!

pytorchmergebot commented Oct 11, 2022

Merge started

Uh oh!

github-actions bot commented Oct 11, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

pytorch-bot bot commented Sep 20, 2022 •

edited

Loading

linux-foundation-easycla bot commented Oct 4, 2022 •

edited

Loading

linux-foundation-easycla bot commented Oct 6, 2022 •

edited

Loading