Skip to content

Trajepl/nebula ckpt engine #2085

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
tjruwase merged 42 commits into from
Jul 28, 2022
Merged

Trajepl/nebula ckpt engine #2085

tjruwase merged 42 commits into from
Jul 28, 2022

Conversation

@trajepl
Copy link
Contributor

@trajepl trajepl commented Jul 11, 2022

No description provided.

@trajepl
Copy link
Contributor Author

trajepl commented Jul 11, 2022 •
edited
Loading

It feels like to me that the NebulaCheckpointEngine should be part of the Nebula library itself. Is it possible to make Nebula APIs align more with torch save/load APIs? Then on the DeepSpeed or any other Nebular user will require less code change to support this feature.

Currently, it is hard to make nebula save/load api totally same with torch save/load, as nebula relys on the tag to build up the concept of checkpoint to manager the saved files.

Instead of modifying the core deepspeed.intiialize() API can we instead add Nebula config options into the deepspeed config json? We try very hard not to add additional params to our initialize API. In the near future we are going to simplify these args even further.

Updated in latest commit.

This warning will be printed for all ranks for anyone who doesn't have torch_nebula installed (e.g., 256 gpu -> 256 prints). Can we ignore this import warning and raise an explicit error if and only if a user attempts to enable nebula?

Updated in latest commit.

trajepl and others added 22 commits July 21, 2022 17:08
@trajepl
make checkpoint_engine as required args
4b42bc2
@trajepl
Merge branch 'trajepl/nebula_ckpt_engine' of github.com:trajepl/DeepS…
22f8c2a 
…peed into trajepl/nebula_ckpt_engine
@trajepl
Merge pull request #2 from microsoft/master
bbd2bde 
Merge from master
@trajepl
Merge branch 'master' of github.com:trajepl/DeepSpeed into trajepl/ne…
d9298cf 
…bula_ckpt_engine
@trajepl
fix args
1906398
@trajepl
extract parameters out to config
432e7c6
@trajepl
fix: load state dict
7dbb6d8
@trajepl
separate load engine
e912e31
@trajepl
linter fix
7fc279b
@tjruwase
Merge branch 'master' into trajepl/nebula_ckpt_engine
5ebacc6
@trajepl
extract checkpoint engine to abstract calss
c70c818
@trajepl
linter fix
e6dd794
@trajepl
Merge branch 'trajepl/nebula_ckpt_engine' of github.com:trajepl/DeepS…
3788ada 
…peed into trajepl/nebula_ckpt_engine
@trajepl
construct function args fix
1efd2ce
@trajepl
add docs for dev/customers
dce0fb5
@trajepl
linter fix
bb5bb7c
@tjruwase
Merge branch 'master' into trajepl/nebula_ckpt_engine
0c21dc2
@trajepl
remove load engine
3e8c238
@trajepl
print->log_dist
a5c8897
@trajepl
linter fix
44d687b
@trajepl
add tag flag to distinguish the loading order
82ad297
@tjruwase
Merge branch 'master' into trajepl/nebula_ckpt_engine
cf12a8d
@tjruwase tjruwase merged commit e669aaf into deepspeedai:master Jul 28, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jeffra jeffra jeffra approved these changes

@tjruwase tjruwase tjruwase approved these changes

@samyam samyam Awaiting requested review from samyam

@ShadenSmith ShadenSmith Awaiting requested review from ShadenSmith

@conglongli conglongli Awaiting requested review from conglongli

@awan-10 awan-10 Awaiting requested review from awan-10

@cli99 cli99 Awaiting requested review from cli99

@eltonzheng eltonzheng Awaiting requested review from eltonzheng

@minjiaz minjiaz Awaiting requested review from minjiaz

@RezaYazdaniAminabadi RezaYazdaniAminabadi Awaiting requested review from RezaYazdaniAminabadi

@duli2012 duli2012 Awaiting requested review from duli2012

@mrwyattii mrwyattii Awaiting requested review from mrwyattii

@yaozhewei yaozhewei Awaiting requested review from yaozhewei

@arashb arashb Awaiting requested review from arashb

@xiaoxiawu-microsoft xiaoxiawu-microsoft Awaiting requested review from xiaoxiawu-microsoft

@samadejacobs samadejacobs Awaiting requested review from samadejacobs

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@trajepl @tjruwase @jeffra @Anbang-Hu @mrwyattii