Co-authored-by: Ammar Ahmad Awan <ammar.awan@microsoft.com>
Co-authored-by: yaozhewei <zheweiy@berkeley.edu>
Co-authored-by: Samyam Rajbhandari <samyamr@microsoft.com>

Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com>
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

Bumps [tzinfo](https://github.com/tzinfo/tzinfo) from 1.2.9 to 1.2.10.
- [Release notes](https://github.com/tzinfo/tzinfo/releases)
- [Changelog](https://github.com/tzinfo/tzinfo/blob/master/CHANGES.md)
- [Commits](tzinfo/tzinfo@v1.2.9...v1.2.10)

---
updated-dependencies:
- dependency-name: tzinfo
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

Co-authored-by: Michael Wyatt <michaelwyatt@microsoft.com>

Co-authored-by: Ammar Ahmad Awan <ammar.awan@microsoft.com>
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

* fix conflit between tutel and top-2 gate
Co-authored-by: Ammar Ahmad Awan <ammar.awan@microsoft.com>
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

* first pass at pydanticifying Zero Configs

* added pydantic to reqs

* fixed bug with deprecated values not being type-checked

* fixing zero config bugs from unit tests

* fixed access of Config values

* removing zero constants

* formatting/fix broken import

* fixed bad merge

* fixed issue with missing aliased field

* fix for failing tests

* fix how deprecated fields are processed

* only process dep params when they are set

* fix mistyped field name

* fixes, docs, removed more constants

* fix merge

* more fixes after merge w master

* added unit tests

* formatting

* added fix for transformers unit tests

* separated offload config from zero config

* fixed bad import

* formatting and flake fixes

* implement suggestion from review

Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com>
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com>
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

* enable checkpoint engine

* seprated nebula config

* add __init__.py for nebula importing

* linter fix

* fix: ds_config is None

* fix: ds config

* fix: get sd loader fix

* align the API with torch raw code

* linter fix

* remove duplicate tag params

* make checkpoint_engine as required args

* fix args

* extract parameters out to config

* fix: load state dict

* separate load engine

* linter fix

* extract checkpoint engine to abstract calss

* linter fix

* construct function args fix

* add docs for dev/customers

* linter fix

* remove load engine

* print->log_dist

* linter fix

* add tag flag to distinguish the loading order

Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com>
Co-authored-by: Michael Wyatt <michaelwyatt@microsoft.com>
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

* Use Popen.terminate() to stop the child processes gracefully; Kill them if terminate doesn't work

* The Popen.kill() command cause the training processes to end abruptly. This may cause the child processes to become zombies without communicating properly to the parent process about the kill signal. So the ssh session continue to wait for signals from the child processes, causing it to not return back to the pdsh command

Fixes microsoft#1995

Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com>

…m tensor (#2152)

Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com>

…1149)

Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

Co-authored-by: Arpan Jain <t-arpanjain@microsoft.com>
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

* Split parameter offload from z3

* Format fixes

* Bug fixes

* Cleanup

* Remove dead code

* Release swap buffers for persisted params

* Format fixes

* Format fixes

* Pass args correctly

* Use pinned memory for nvme offload

* Merge with masster

* Fix missing import

* model pesistence params

* Fix merge issues

* Handle none device

* Usse log_dist

* tensor parallelism for mixture of experts
Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com>
Co-authored-by: Ammar Ahmad Awan <ammar.awan@microsoft.com>