[WIP] Dispatch Friendly collectives#93379

Closed

wconstab wants to merge 10 commits intopytorch:masterfrom

kumpera:land_will_pr

Contributor

wconstab commented Jan 31, 2023 •

edited by kumpera

Loading

cc @mlazos @soumith @voznesenskym @yanboliang @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @chunyuan-w @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @desertfire

Rodrigo Kumpera added 6 commits

January 27, 2023 00:58


          trace trace

05519b2


          sketch of alternative API, ignore the inductor bits, they broken

b5ad672


          update OP to use stride.

839bde5


          cleanup

15f8058

wip

aa1ee37


          fixes plus plug DTensor.

71714d7

wconstab requested review from a team, H-Huang, albanD, awgu, kwen2501, mrshenli, rohan-varma, soulitzer, wanchaol and zhaojuanmao as code owners

January 31, 2023 17:53

pytorch-bot bot commented Jan 31, 2023 •

edited

Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/93379

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 6 Failures

As of commit d008410:

NEW FAILURES - The following jobs have failed:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

pytorch-bot bot added the release notes: releng label

github-actions bot added ciflow/inductor module: inductor labels

wconstab commented

View reviewed changes

torch/distributed/traceable_collectives.py Show resolved Hide resolved

wconstab commented

View reviewed changes

torch/distributed/traceable_collectives.py Outdated Show resolved Hide resolved

wconstab commented

View reviewed changes

torch/_meta_registrations.py Outdated Show resolved Hide resolved

kumpera changed the title ~~Land will pr~~ [WIP] Dispatch Friendly collectives

wconstab commented

View reviewed changes

tools/autograd/derivatives.yaml

                 self: grad.reshape_symint(self.sym_sizes())
                 result: auto_linear
+              - name: all_reduce(Tensor self, str reduceOp, str tag, int[] ranks, int stride) -> Tensor

Contributor Author

wconstab Jan 31, 2023

have you thought about how to handle str reduceOp?

i think it is OK to leave it as str instead of a new C++ type.

however, we need a way to translate between str/enum, and I didn't think about where to put that translator yet.

currently, i think we only need this in python (since inductor also calls into py api), so maybe we can keep it together with the pg mapping utils.

Contributor

kumpera Jan 31, 2023

I think it should be in the same place as the op implementation.

albanD removed their request for review

January 31, 2023 18:28

Rodrigo Kumpera added 2 commits

January 31, 2023 21:12


          linting

54ef69e


          lint it all

4ce5fd9

wconstab commented

View reviewed changes

torch/distributed/traceable_collectives.py Show resolved Hide resolved


          add some tests.

4dfe1ea

wconstab added a commit that referenced this pull request


          Update on "[mockup of Rodrigo's PR] Traceable AllReduce Support"

7f0ee2b

NOTE: this is a hacked squashed version of Rodrigo's PR (#93379) designed to unblock inductor impl work.  Wait for real PR to land then rebase the inductor stuff on top of that.

[ghstack-poisoned]

wconstab added a commit that referenced this pull request


          Update base for Update on "[mockup of Rodrigo's PR] Traceable AllRedu…

6d4b18f

…ce Support"

NOTE: this is a hacked squashed version of Rodrigo's PR (#93379) designed to unblock inductor impl work.  Wait for real PR to land then rebase the inductor stuff on top of that.

[ghstack-poisoned]

wconstab added a commit that referenced this pull request


          Update on "[mockup of Rodrigo's PR] Traceable AllReduce Support"

10c489a

NOTE: this is a hacked squashed version of Rodrigo's PR (#93379) designed to unblock inductor impl work.  Wait for real PR to land then rebase the inductor stuff on top of that.

[ghstack-poisoned]

wconstab added a commit that referenced this pull request


          Update base for Update on "[mockup of Rodrigo's PR] Traceable AllRedu…

84cfda9

…ce Support"

NOTE: this is a hacked squashed version of Rodrigo's PR (#93379) designed to unblock inductor impl work.  Wait for real PR to land then rebase the inductor stuff on top of that.

[ghstack-poisoned]

wconstab added a commit that referenced this pull request


          Update on "[mockup of Rodrigo's PR] Traceable AllReduce Support"

3775b5e

NOTE: this is a hacked squashed version of Rodrigo's PR (#93379) designed to unblock inductor impl work.  Wait for real PR to land then rebase the inductor stuff on top of that.

[ghstack-poisoned]

wconstab added a commit that referenced this pull request


          Update base for Update on "[mockup of Rodrigo's PR] Traceable AllRedu…

2959d8f

…ce Support"

NOTE: this is a hacked squashed version of Rodrigo's PR (#93379) designed to unblock inductor impl work.  Wait for real PR to land then rebase the inductor stuff on top of that.

[ghstack-poisoned]

wconstab added a commit that referenced this pull request


          Update on "[mockup of Rodrigo's PR] Traceable AllReduce Support"

030d6cd

NOTE: this is a hacked squashed version of Rodrigo's PR (#93379) designed to unblock inductor impl work.  Wait for real PR to land then rebase the inductor stuff on top of that.

[ghstack-poisoned]

wconstab added a commit that referenced this pull request


          Update base for Update on "[mockup of Rodrigo's PR] Traceable AllRedu…

7b3cea8

…ce Support"

NOTE: this is a hacked squashed version of Rodrigo's PR (#93379) designed to unblock inductor impl work.  Wait for real PR to land then rebase the inductor stuff on top of that.

[ghstack-poisoned]

wconstab added a commit that referenced this pull request


          Update on "[mockup of Rodrigo's PR] Traceable AllReduce Support"

f2f96c3

NOTE: this is a hacked squashed version of Rodrigo's PR (#93379) designed to unblock inductor impl work.  Wait for real PR to land then rebase the inductor stuff on top of that.

[ghstack-poisoned]

wconstab added a commit that referenced this pull request


          Update base for Update on "[mockup of Rodrigo's PR] Traceable AllRedu…

90a1f3f

…ce Support"

NOTE: this is a hacked squashed version of Rodrigo's PR (#93379) designed to unblock inductor impl work.  Wait for real PR to land then rebase the inductor stuff on top of that.

[ghstack-poisoned]

wconstab added a commit that referenced this pull request


          Update on "[mockup of Rodrigo's PR] Traceable AllReduce Support"

a1b8972

NOTE: this is a hacked squashed version of Rodrigo's PR (#93379) designed to unblock inductor impl work.  Wait for real PR to land then rebase the inductor stuff on top of that.

[ghstack-poisoned]

wconstab added a commit that referenced this pull request


          Update base for Update on "[mockup of Rodrigo's PR] Traceable AllRedu…

26be98e

…ce Support"

NOTE: this is a hacked squashed version of Rodrigo's PR (#93379) designed to unblock inductor impl work.  Wait for real PR to land then rebase the inductor stuff on top of that.

[ghstack-poisoned]

wconstab added a commit that referenced this pull request


          Update on "[mockup of Rodrigo's PR] Traceable AllReduce Support"

c27bfbc

NOTE: this is a hacked squashed version of Rodrigo's PR (#93379) designed to unblock inductor impl work.  Wait for real PR to land then rebase the inductor stuff on top of that.

[ghstack-poisoned]

wconstab added a commit that referenced this pull request


          Update base for Update on "[mockup of Rodrigo's PR] Traceable AllRedu…

c1d4b2a

…ce Support"

NOTE: this is a hacked squashed version of Rodrigo's PR (#93379) designed to unblock inductor impl work.  Wait for real PR to land then rebase the inductor stuff on top of that.

[ghstack-poisoned]

wconstab added a commit that referenced this pull request


          Update on "[mockup of Rodrigo's PR] Traceable AllReduce Support"

a7a5d0e

NOTE: this is a hacked squashed version of Rodrigo's PR (#93379) designed to unblock inductor impl work.  Wait for real PR to land then rebase the inductor stuff on top of that.

[ghstack-poisoned]

wconstab added a commit that referenced this pull request


          Update base for Update on "[mockup of Rodrigo's PR] Traceable AllRedu…

f6ce906

…ce Support"

NOTE: this is a hacked squashed version of Rodrigo's PR (#93379) designed to unblock inductor impl work.  Wait for real PR to land then rebase the inductor stuff on top of that.

[ghstack-poisoned]

wconstab added a commit that referenced this pull request


          Update on "[mockup of Rodrigo's PR] Traceable AllReduce Support"

b048bea

NOTE: this is a hacked squashed version of Rodrigo's PR (#93379) designed to unblock inductor impl work.  Wait for real PR to land then rebase the inductor stuff on top of that.

[ghstack-poisoned]

wconstab added a commit that referenced this pull request


          Update base for Update on "[mockup of Rodrigo's PR] Traceable AllRedu…

7c8a432

…ce Support"

NOTE: this is a hacked squashed version of Rodrigo's PR (#93379) designed to unblock inductor impl work.  Wait for real PR to land then rebase the inductor stuff on top of that.

[ghstack-poisoned]

wconstab added a commit that referenced this pull request


          Update on "[mockup of Rodrigo's PR] Traceable AllReduce Support"

8a996db

NOTE: this is a hacked squashed version of Rodrigo's PR (#93379) designed to unblock inductor impl work.  Wait for real PR to land then rebase the inductor stuff on top of that.

[ghstack-poisoned]

wconstab added a commit that referenced this pull request


          Update base for Update on "[mockup of Rodrigo's PR] Traceable AllRedu…

db1e512

…ce Support"

NOTE: this is a hacked squashed version of Rodrigo's PR (#93379) designed to unblock inductor impl work.  Wait for real PR to land then rebase the inductor stuff on top of that.

[ghstack-poisoned]

wconstab added a commit that referenced this pull request


          Update on "[mockup of Rodrigo's PR] Traceable AllReduce Support"

17aafe8

NOTE: this is a hacked squashed version of Rodrigo's PR (#93379) designed to unblock inductor impl work.  Wait for real PR to land then rebase the inductor stuff on top of that.

[ghstack-poisoned]

kumpera closed this

kumpera deleted the land_will_pr branch

February 10, 2023 18:38

wconstab added a commit that referenced this pull request


          Update base for Update on "[mockup of Rodrigo's PR] Traceable AllRedu…

c5becc5

…ce Support"

NOTE: this is a hacked squashed version of Rodrigo's PR (#93379) designed to unblock inductor impl work.  Wait for real PR to land then rebase the inductor stuff on top of that.

[ghstack-poisoned]

wconstab added a commit that referenced this pull request


          Update on "[mockup of Rodrigo's PR] Traceable AllReduce Support"

edbe38e

NOTE: this is a hacked squashed version of Rodrigo's PR (#93379) designed to unblock inductor impl work.  Wait for real PR to land then rebase the inductor stuff on top of that.

[ghstack-poisoned]

wconstab added a commit that referenced this pull request


          Update base for Update on "[mockup of Rodrigo's PR] Traceable AllRedu…

bc9ce75

…ce Support"

NOTE: this is a hacked squashed version of Rodrigo's PR (#93379) designed to unblock inductor impl work.  Wait for real PR to land then rebase the inductor stuff on top of that.

[ghstack-poisoned]

wconstab added a commit that referenced this pull request


          Update on "[mockup of Rodrigo's PR] Traceable AllReduce Support"

9b4b9b0

NOTE: this is a hacked squashed version of Rodrigo's PR (#93379) designed to unblock inductor impl work.  Wait for real PR to land then rebase the inductor stuff on top of that.

[ghstack-poisoned]

wconstab added a commit that referenced this pull request


          [Mockup of Rodrigo's PR] Traceable AllReduce Support

41615dd

(updated 2/10)

NOTE: this is a hacked squashed version of Rodrigo's PR (#93379) designed to unblock inductor impl work.  Wait for real PR to land then rebase the inductor stuff on top of that.

ghstack-source-id: 2030643
Pull Request resolved: #94059

wconstab added a commit that referenced this pull request


          Update base for Update on "[mockup of Rodrigo's PR] Traceable AllRedu…

51b5764

…ce Support"

NOTE: this is a hacked squashed version of Rodrigo's PR (#93379) designed to unblock inductor impl work.  Wait for real PR to land then rebase the inductor stuff on top of that.

[ghstack-poisoned]

wconstab added a commit that referenced this pull request


          Update on "[mockup of Rodrigo's PR] Traceable AllReduce Support"

NOTE: this is a hacked squashed version of Rodrigo's PR (#93379) designed to unblock inductor impl work.  Wait for real PR to land then rebase the inductor stuff on top of that.

[ghstack-poisoned]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

mrshenli Awaiting requested review from mrshenli

zhaojuanmao Awaiting requested review from zhaojuanmao

rohan-varma Awaiting requested review from rohan-varma

H-Huang Awaiting requested review from H-Huang

awgu Awaiting requested review from awgu

kwen2501 Awaiting requested review from kwen2501

wanchaol Awaiting requested review from wanchaol

soulitzer Awaiting requested review from soulitzer

1 more reviewer

kumpera kumpera left review comments

Labels

ciflow/inductor module: inductor release notes: releng