DeepSpeed Comm. Backend v1 #1985

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merged

jeffra merged 24 commits into master from staging-comms-next

Jun 10, 2022

Contributor

awan-10 commented May 27, 2022 •

edited by Quentin-Anthony

Loading

This PR introduces the DeepSpeed Comm. Backend (v1).

Current advanced communication schemes rely on mixing python-level communication packages (e.g. torch.distributed, mpi4py for 1-bit Adam). In order to simplify comms prototypes, we're looking to add support for custom communication backends within DeepSpeed built directly on top of their respective libraries (e.g. NCCL, MPI, etc).

This PR completes the first phase towards this goal by introducing:

The new comms interface deepspeed.comms
A complete wrapper around torch.distributed called TorchBackend for backwards-compatibility
A rough skeleton for custom backends that we can use for phase 2

Co-authored-by: Quentin Anthony qganthony@yahoo.com
Co-authored-by: Ammar Ahmad Awan ammar.awan@microsoft.com
Co-authored-by: Jeff Rasley jerasley@microsoft.com


          Staging comms v1 (#301)

867a853

Co-authored-by: Quentin Anthony <qganthony@yahoo.com>
Co-authored-by: Ammar Ahmad Awan <ammar.awan@microsoft.com>
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

awan-10 requested review from RezaYazdaniAminabadi, ShadenSmith, cli99, conglongli, eltonzheng, jeffra, minjiaz, samyam and tjruwase as code owners

May 27, 2022 22:29

awan-10 commented

View reviewed changes

deepspeed/__init__.py Outdated Show resolved Hide resolved

awan-10 commented

View reviewed changes

deepspeed/comm/backend.py Outdated Show resolved Hide resolved

awan-10 commented

View reviewed changes

deepspeed/moe/sharded_moe.py Outdated Show resolved Hide resolved

awan-10 commented

View reviewed changes

deepspeed/runtime/comm/coalesced_collectives.py Outdated Show resolved Hide resolved

awan-10 commented

View reviewed changes

deepspeed/runtime/comm/coalesced_collectives.py Outdated Show resolved Hide resolved

awan-10 commented

View reviewed changes

deepspeed/runtime/engine.py Outdated Show resolved Hide resolved

awan-10 commented

View reviewed changes

deepspeed/runtime/engine.py Show resolved Hide resolved

awan-10 commented

View reviewed changes

deepspeed/runtime/pipe/module.py Outdated Show resolved Hide resolved

awan-10 commented

View reviewed changes

deepspeed/runtime/zero/partition_parameters.py Outdated Show resolved Hide resolved

awan-10 added 2 commits

May 27, 2022 15:45


          Delete stage1.py

c93fcfe


          Delete distributed.py

7f8ca01

awan-10 commented

View reviewed changes

op_builder/comm.py Outdated Show resolved Hide resolved

awan-10 commented

View reviewed changes

tests/comm/test.py Outdated Show resolved Hide resolved

awan-10 commented

View reviewed changes

Contributor Author

awan-10 left a comment

I reviewed this PR with Quentin. He will take care of minor comments. @jeffra, please review this one.

Quentin-Anthony added 5 commits

May 28, 2022 05:48


          revert deepspeed/__init__.py logging calls

977ee32


          Delete test.py

68eb9f4


          Update comments and move custom comm ops to internal functions

54796bb


          Merge branch 'staging-comms-next' of https://github.com/microsoft/Dee…

c06c72d

…pSpeed into staging-comms-next


          Remove unnecessary print and update backend description

f070a0c

Collaborator

jeffra commented May 31, 2022

@SeanNaren can you take a look at the lightning test failure? We're seeing this on other PRs and in master right now as well. It appears to be a protobuf issue, have you seen this on your side before?

Quentin-Anthony added 2 commits

June 1, 2022 14:51


          Delete runall.py

73afaa8


          Delete all_reduce.py

e51021f

Contributor

Quentin-Anthony commented Jun 7, 2022

@jeffra and @tjruwase -- Any further comments?


          Merge branch 'master' into staging-comms-next

jeffra approved these changes

View reviewed changes

Quentin-Anthony added 3 commits

June 8, 2022 01:08


          Add isend/irecv comm ops

604eae7


          Add pre-commit check and clean up references to torch.dist in comments

2dd23f3


          Updated pre-commit error

b902744

tjruwase reviewed

View reviewed changes

deepspeed/comm/comm.py Outdated Show resolved Hide resolved

tjruwase reviewed

View reviewed changes

deepspeed/comm/comm.py Outdated Show resolved Hide resolved

tjruwase reviewed

View reviewed changes

deepspeed/comm/comm.py Outdated Show resolved Hide resolved

tjruwase reviewed

View reviewed changes

deepspeed/comm/comm.py Show resolved Hide resolved

tjruwase reviewed

View reviewed changes

deepspeed/comm/comm.py Outdated Show resolved Hide resolved

tjruwase reviewed

View reviewed changes

deepspeed/comm/comm.py Outdated Show resolved Hide resolved

tjruwase reviewed

View reviewed changes

deepspeed/runtime/zero/stage_1_and_2.py Outdated Show resolved Hide resolved

tjruwase reviewed

View reviewed changes

deepspeed/comm/comm.py Show resolved Hide resolved


          Clean up comm.py logic

3c5e065

deepspeedai deleted a comment from rocm-mici


          Added comm constants file to avoid raw string assignment

971c349

Quentin-Anthony mentioned this pull request

Relax DeepSpeed MoE ZeRO-1 Assertion #2007

Merged


          Revert "Relax assertion to allow Megatron-DeepSpeed MoE to use ZeRO 1"

5e0c8c1

This reverts commit 9976681.

tjruwase approved these changes

View reviewed changes

deepspeed/comm/comm.py Outdated Show resolved Hide resolved

jeffra merged commit 36ad311 into master

jeffra deleted the staging-comms-next branch

June 10, 2022 23:47

jeffra restored the staging-comms-next branch

June 10, 2022 23:47

This was referenced Jun 13, 2022

Add support for DS comms deepspeedai/Megatron-DeepSpeed#50

Open

DeepSpeed Communication Profiling and Logging #2012

Merged

kisseternity mentioned this pull request

[BUG]DeepSpeed Comm. Backend not compatible with outside torch.distributed module #2063

Open

jeffra mentioned this pull request

Fix regression w. dist_init_required #2225

Merged

mrwyattii deleted the staging-comms-next branch

July 7, 2023 02:39

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

jeffra jeffra approved these changes

tjruwase tjruwase approved these changes

samyam Awaiting requested review from samyam

ShadenSmith Awaiting requested review from ShadenSmith

conglongli Awaiting requested review from conglongli

cli99 Awaiting requested review from cli99

eltonzheng Awaiting requested review from eltonzheng

minjiaz Awaiting requested review from minjiaz

RezaYazdaniAminabadi Awaiting requested review from RezaYazdaniAminabadi

+1 more reviewer

Quentin-Anthony Quentin-Anthony left review comments

Labels

None yet