[Distributed] Add more runtime distributed communication functions #314
Merged
yaoyaoding merged 15 commits intohidet-org:mainfrom Jul 14, 2023
Merged
[Distributed] Add more runtime distributed communication functions #314yaoyaoding merged 15 commits intohidet-org:mainfrom
yaoyaoding merged 15 commits intohidet-org:mainfrom
Conversation
Collaborator
Author
|
@yaoyaoding Hi, could you please take a look at this pr? It's basically the correspondence of |
Member
|
The PR looks good to me, thanks @soodoshll ! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
We add the following communication functions related to distributed functionalities:
Runtime API
We mimic the APIs of PyTorch's distributed package
Hidet OPs
We distinguish these two different usages of distributed primitives because many communication primitives cannot be directly viewed as nodes in computational graphs.
Still working on documentation and comments