Skip to content

Add torch.nn.init.normal_ and torch.nn.init.kaiming_uniform_ ops to ShardedTensor#67057

Closed
bowangbj wants to merge 5 commits into
gh/bowangbj/12/basefrom
gh/bowangbj/12/head
Closed

Add torch.nn.init.normal_ and torch.nn.init.kaiming_uniform_ ops to ShardedTensor#67057
bowangbj wants to merge 5 commits into
gh/bowangbj/12/basefrom
gh/bowangbj/12/head

Conversation

@bowangbj

@bowangbj bowangbj commented Oct 21, 2021 •
edited
Loading

Copy link
Copy Markdown
Contributor

Stack from ghstack:

Summary:
Extend ShardedTensor with torch.nn.init.[normal_, and kaiming_uniform_] ops
Follow up from #63997

Test Plan:
a) Unit Test
(pytorch) ... $ python test/distributed/_sharded_tensor/ops/test_init.py TestShardedTensorNNInit --v

or b) Manual run: Instruction here: https://docs.google.com/document/d/1_m1Hdo5w51-hhPlZ_F8Y6PIWrN7UgJZqiSpARYvhsaE/edit#
s/uniform_/normal_ or kaiming_uniform_

Reviewers:

Subscribers:

Tasks:

Tags:

Differential Revision: D31845654

@bowangbj
Add torch.nn.init.normal_ and torch.nn.init.kaiming_uniform_ ops to S…
649f2d0 
…hardedTensor

Summary:
Extend ShardedTensor with torch.nn.init.[normal_, and kaiming_uniform_] ops
Follow up from #63997

Test Plan:
a) Unit Test
(pytorch) ... $ python test/distributed/_sharded_tensor/ops/test_init.py TestShardedTensorNNInit --v

or b) Manual run: Instruction here: https://docs.google.com/document/d/1_m1Hdo5w51-hhPlZ_F8Y6PIWrN7UgJZqiSpARYvhsaE/edit#
s/uniform_/normal_ or kaiming_uniform_

Reviewers:

Subscribers:

Tasks:

Tags:

[ghstack-poisoned]
@pytorch-probot

pytorch-probot Bot commented Oct 21, 2021 •
edited
Loading

Copy link
Copy Markdown
CI Flow Status

⚛️ CI Flow

Ruleset - Version: v1
Ruleset - File: https://github.com/pytorch/pytorch/blob/2f1458fee735861373f6b1c9fb5c2e78986e1b18/.github/generated-ciflow-ruleset.json
PR ciflow labels: ciflow/default

Workflows Labels (bold enabled) Status
Triggered Workflows
linux-bionic-py3.6-clang9 ciflow/all, ciflow/cpu, ciflow/default, ciflow/linux, ciflow/noarch, ciflow/xla ✅ triggered
linux-vulkan-bionic-py3.6-clang9 ciflow/all, ciflow/cpu, ciflow/default, ciflow/linux, ciflow/vulkan ✅ triggered
linux-xenial-cuda11.3-py3.6-gcc7 ciflow/all, ciflow/cuda, ciflow/default, ciflow/linux ✅ triggered
linux-xenial-py3-clang5-mobile-build ciflow/all, ciflow/default, ciflow/linux, ciflow/mobile ✅ triggered
linux-xenial-py3-clang5-mobile-custom-build-dynamic ciflow/all, ciflow/default, ciflow/linux, ciflow/mobile ✅ triggered
linux-xenial-py3-clang5-mobile-custom-build-static ciflow/all, ciflow/default, ciflow/linux, ciflow/mobile ✅ triggered
linux-xenial-py3.6-clang7-asan ciflow/all, ciflow/cpu, ciflow/default, ciflow/linux, ciflow/sanitizers ✅ triggered
linux-xenial-py3.6-clang7-onnx ciflow/all, ciflow/cpu, ciflow/default, ciflow/linux, ciflow/onnx ✅ triggered
linux-xenial-py3.6-gcc5.4 ciflow/all, ciflow/cpu, ciflow/default, ciflow/linux ✅ triggered
linux-xenial-py3.6-gcc7-bazel-test ciflow/all, ciflow/bazel, ciflow/cpu, ciflow/default, ciflow/linux ✅ triggered
win-vs2019-cpu-py3 ciflow/all, ciflow/cpu, ciflow/default, ciflow/win ✅ triggered
win-vs2019-cuda11.3-py3 ciflow/all, ciflow/cuda, ciflow/default, ciflow/win ✅ triggered
Skipped Workflows
libtorch-linux-xenial-cuda10.2-py3.6-gcc7 ciflow/all, ciflow/cuda, ciflow/libtorch, ciflow/linux 🚫 skipped
libtorch-linux-xenial-cuda11.3-py3.6-gcc7 ciflow/all, ciflow/cuda, ciflow/libtorch, ciflow/linux 🚫 skipped
linux-bionic-cuda10.2-py3.9-gcc7 ciflow/all, ciflow/cuda, ciflow/linux, ciflow/slow 🚫 skipped
linux-xenial-cuda10.2-py3.6-gcc7 ciflow/all, ciflow/cuda, ciflow/linux, ciflow/slow 🚫 skipped
linux-xenial-py3-clang5-mobile-code-analysis ciflow/all, ciflow/linux, ciflow/mobile 🚫 skipped
parallelnative-linux-xenial-py3.6-gcc5.4 ciflow/all, ciflow/cpu, ciflow/linux 🚫 skipped
periodic-libtorch-linux-xenial-cuda11.1-py3.6-gcc7 ciflow/all, ciflow/cuda, ciflow/libtorch, ciflow/linux, ciflow/scheduled 🚫 skipped
periodic-linux-xenial-cuda10.2-py3-gcc7-slow-gradcheck ciflow/all, ciflow/cuda, ciflow/linux, ciflow/scheduled, ciflow/slow, ciflow/slow-gradcheck 🚫 skipped
periodic-linux-xenial-cuda11.1-py3.6-gcc7 ciflow/all, ciflow/cuda, ciflow/linux, ciflow/scheduled 🚫 skipped
periodic-win-vs2019-cuda11.1-py3 ciflow/all, ciflow/cuda, ciflow/scheduled, ciflow/win 🚫 skipped
puretorch-linux-xenial-py3.6-gcc5.4 ciflow/all, ciflow/cpu, ciflow/linux 🚫 skipped

You can add a comment to the PR and tag @pytorchbot with the following commands: 
# ciflow rerun, "ciflow/default" will always be added automatically
@pytorchbot ciflow rerun

# ciflow rerun with additional labels "-l <ciflow/label_name>", which is equivalent to adding these labels manually and trigger the rerun
@pytorchbot ciflow rerun -l ciflow/scheduled -l ciflow/slow

For more information, please take a look at the CI Flow Wiki.

@facebook-github-bot

facebook-github-bot commented Oct 21, 2021 •
edited
Loading

Copy link
Copy Markdown
Contributor

🔗 Helpful links

💊 CI failures summary and remediations

As of commit 2f1458f (more details on the Dr. CI page):

💚 💚 Looks good so far! There are no failures yet. 💚 💚

This comment was automatically generated by Dr. CI (expand for details).Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions to the (internal) Dr. CI Users group.

Click here to manually regenerate this comment.

@facebook-github-bot facebook-github-bot added the oncall: distributed Add this issue/PR to distributed oncall triage queue label Oct 21, 2021
@bowangbj

bowangbj commented Oct 21, 2021

Copy link
Copy Markdown
Contributor Author

@bowangbj has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

@bowangbj
lint on "Add torch.nn.init.normal_ and torch.nn.init.kaiming_uniform_…
d926195 
… ops to ShardedTensor"

Summary:
Extend ShardedTensor with torch.nn.init.[normal_, and kaiming_uniform_] ops
Follow up from #63997

Test Plan:
a) Unit Test
(pytorch) ... $ python test/distributed/_sharded_tensor/ops/test_init.py TestShardedTensorNNInit --v

or b) Manual run: Instruction here: https://docs.google.com/document/d/1_m1Hdo5w51-hhPlZ_F8Y6PIWrN7UgJZqiSpARYvhsaE/edit#
s/uniform_/normal_ or kaiming_uniform_

Reviewers:

Subscribers:

Tasks:

Tags:

Differential Revision: [D31845654](https://our.internmc.facebook.com/intern/diff/D31845654)

[ghstack-poisoned]
@bowangbj
rebase on "Add torch.nn.init.normal_ and torch.nn.init.kaiming_unifor…
a7d74b4 
…m_ ops to ShardedTensor"

Summary:
Extend ShardedTensor with torch.nn.init.[normal_, and kaiming_uniform_] ops
Follow up from #63997

Test Plan:
a) Unit Test
(pytorch) ... $ python test/distributed/_sharded_tensor/ops/test_init.py TestShardedTensorNNInit --v

or b) Manual run: Instruction here: https://docs.google.com/document/d/1_m1Hdo5w51-hhPlZ_F8Y6PIWrN7UgJZqiSpARYvhsaE/edit#
s/uniform_/normal_ or kaiming_uniform_

Reviewers:

Subscribers:

Tasks:

Tags:

Differential Revision: [D31845654](https://our.internmc.facebook.com/intern/diff/D31845654)

[ghstack-poisoned]
bowangbj added a commit that referenced this pull request Oct 21, 2021
@bowangbj
Add torch.nn.init.normal_ and torch.nn.init.kaiming_uniform_ ops to S…
1157707 
…hardedTensor

Summary:
Extend ShardedTensor with torch.nn.init.[normal_, and kaiming_uniform_] ops
Follow up from #63997

Test Plan:
a) Unit Test
(pytorch) ... $ python test/distributed/_sharded_tensor/ops/test_init.py TestShardedTensorNNInit --v

or b) Manual run: Instruction here: https://docs.google.com/document/d/1_m1Hdo5w51-hhPlZ_F8Y6PIWrN7UgJZqiSpARYvhsaE/edit#
s/uniform_/normal_ or kaiming_uniform_

Reviewers:

Subscribers:

Tasks:

Tags:

ghstack-source-id: 012f671
Pull Request resolved: #67057
@bowangbj

bowangbj commented Oct 21, 2021

Copy link
Copy Markdown
Contributor Author

@bowangbj has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

albanD
albanD reviewed Oct 22, 2021
View reviewed changes
Comment thread torch/nn/init.py Outdated
Comment thread torch/nn/init.py Outdated
@bowangbj
resolve albanD cmt on "Add torch.nn.init.normal_ and torch.nn.init.ka…
d913ea0 
…iming_uniform_ ops to ShardedTensor"

Summary:
Extend ShardedTensor with torch.nn.init.[normal_, and kaiming_uniform_] ops
Follow up from #63997

Test Plan:
a) Unit Test
(pytorch) ... $ python test/distributed/_sharded_tensor/ops/test_init.py TestShardedTensorNNInit --v

or b) Manual run: Instruction here: https://docs.google.com/document/d/1_m1Hdo5w51-hhPlZ_F8Y6PIWrN7UgJZqiSpARYvhsaE/edit#
s/uniform_/normal_ or kaiming_uniform_

Reviewers:

Subscribers:

Tasks:

Tags:

Differential Revision: [D31845654](https://our.internmc.facebook.com/intern/diff/D31845654)

[ghstack-poisoned]
bowangbj added a commit that referenced this pull request Oct 22, 2021
@bowangbj
Add torch.nn.init.normal_ and torch.nn.init.kaiming_uniform_ ops to S…
73ec3e4 
…hardedTensor

Summary:
Extend ShardedTensor with torch.nn.init.[normal_, and kaiming_uniform_] ops
Follow up from #63997

Test Plan:
a) Unit Test
(pytorch) ... $ python test/distributed/_sharded_tensor/ops/test_init.py TestShardedTensorNNInit --v

or b) Manual run: Instruction here: https://docs.google.com/document/d/1_m1Hdo5w51-hhPlZ_F8Y6PIWrN7UgJZqiSpARYvhsaE/edit#
s/uniform_/normal_ or kaiming_uniform_

Reviewers:

Subscribers:

Tasks:

Tags:

ghstack-source-id: 54ce4ba
Pull Request resolved: #67057
@bowangbj

bowangbj commented Oct 22, 2021

Copy link
Copy Markdown
Contributor Author

@bowangbj has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

@bowangbj bowangbj requested a review from albanD October 22, 2021 17:55
@bowangbj

bowangbj commented Oct 22, 2021

Copy link
Copy Markdown
Contributor Author

Thanks Alban for your super fast review, resolved all cmts, PTAL

albanD
albanD reviewed Oct 22, 2021
View reviewed changes
Comment thread torch/nn/init.py
@bowangbj
albanD cmt keep tensor, on "Add torch.nn.init.normal_ and torch.nn.in…
2f1458f 
…it.kaiming_uniform_ ops to ShardedTensor"

Summary:
Extend ShardedTensor with torch.nn.init.[normal_, and kaiming_uniform_] ops
Follow up from #63997

Test Plan:
a) Unit Test
(pytorch) ... $ python test/distributed/_sharded_tensor/ops/test_init.py TestShardedTensorNNInit --v

or b) Manual run: Instruction here: https://docs.google.com/document/d/1_m1Hdo5w51-hhPlZ_F8Y6PIWrN7UgJZqiSpARYvhsaE/edit#
s/uniform_/normal_ or kaiming_uniform_

Reviewers:

Subscribers:

Tasks:

Tags:

Differential Revision: [D31845654](https://our.internmc.facebook.com/intern/diff/D31845654)

[ghstack-poisoned]
@bowangbj bowangbj mentioned this pull request Oct 22, 2021
Closed
@bowangbj

bowangbj commented Oct 22, 2021

Copy link
Copy Markdown
Contributor Author

@bowangbj has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

@bowangbj bowangbj requested a review from albanD October 22, 2021 19:18
@bowangbj

bowangbj commented Oct 22, 2021

Copy link
Copy Markdown
Contributor Author

Thanks Alban for the super quick review, resolved ur cmts, PTAL

albanD
albanD reviewed Oct 22, 2021
View reviewed changes

@albanD albanD left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The nn part looks good.
I'll let the distributed people comment on the rest.

pritamdamania87
pritamdamania87 approved these changes Oct 26, 2021
View reviewed changes

@pritamdamania87 pritamdamania87 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great!

facebook-github-bot pushed a commit that referenced this pull request Oct 26, 2021
@bowangbj @facebook-github-bot
Add torch.nn.init.normal_ and torch.nn.init.kaiming_uniform_ ops to S…
3596e13 
…hardedTensor (#67057)

Summary:
Pull Request resolved: #67057

Extend ShardedTensor with torch.nn.init.[normal_, and kaiming_uniform_] ops
Follow up from #63997

Test Plan:
a) Unit Test
(pytorch) ... $ python test/distributed/_sharded_tensor/ops/test_init.py TestShardedTensorNNInit --v

or b) Manual run: Instruction here: https://docs.google.com/document/d/1_m1Hdo5w51-hhPlZ_F8Y6PIWrN7UgJZqiSpARYvhsaE/edit#
s/uniform_/normal_ or kaiming_uniform_

Imported from OSS

Reviewed By: pritamdamania87

Differential Revision: D31845654

fbshipit-source-id: e7aedc0972539da59f7b84bbbf617caf6b206d52
@facebook-github-bot facebook-github-bot deleted the gh/bowangbj/12/head branch November 25, 2021 15:17
laurentdupin pushed a commit to laurentdupin/pytorch that referenced this pull request Apr 25, 2026
@bowangbj @facebook-github-bot
Add torch.nn.init.normal_ and torch.nn.init.kaiming_uniform_ ops to S…
6783339 
…hardedTensor (pytorch#67057)

Summary:
Pull Request resolved: pytorch#67057

Extend ShardedTensor with torch.nn.init.[normal_, and kaiming_uniform_] ops
Follow up from pytorch#63997

Test Plan:
a) Unit Test
(pytorch) ... $ python test/distributed/_sharded_tensor/ops/test_init.py TestShardedTensorNNInit --v

or b) Manual run: Instruction here: https://docs.google.com/document/d/1_m1Hdo5w51-hhPlZ_F8Y6PIWrN7UgJZqiSpARYvhsaE/edit#
s/uniform_/normal_ or kaiming_uniform_

Imported from OSS

Reviewed By: pritamdamania87

Differential Revision: D31845654

fbshipit-source-id: e7aedc0972539da59f7b84bbbf617caf6b206d52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@albanD albanD albanD left review comments

@wanchaol wanchaol Awaiting requested review from wanchaol

@fduwjj fduwjj Awaiting requested review from fduwjj

+1 more reviewer

@pritamdamania87 pritamdamania87 pritamdamania87 approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

cla signed oncall: distributed Add this issue/PR to distributed oncall triage queue

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@bowangbj @facebook-github-bot @albanD @pritamdamania87