ZeRO tutorials #384
Merged
ZeRO tutorials #384
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
arashashari
approved these changes
Sep 10, 2020
tjruwase
added a commit
that referenced
this pull request
Apr 12, 2025
* Integrate NVIDIA GPUDirect Storage into nvme library * 1) Remove debug prints 2) Create write file with random data 3) Delete target file before new writes * Workaround gds perf issue by leaking buffers * DGX2 mount/unmount utililties * Formatting * Add torch save/load * Add torch save/load * Remove gds * Add torch legacy save * Update to new cli * Add function signatures Add file_offset arg to read/write apis * Remove redundant asserts * Add DeepSpeedFileWriter * Add mock and python file writers * Format fixes * More perf counters * Fix pinned_offset bug; Show as not real python file object * Buffer copy speed * Add torch_fastio option * Format fixes * Measure torch_fastio perf * Force flush * Formatting * Renamings * Fix device bug * Disable torch.distributed requirement * Renaming * Integrate fast model checkpointing * Double I/O buffer optimization * Support larger sizes * Refactoring; save_storage api * Cast to byte tensor * Handle storage object saves * Remove mysterious import * Api to save storage object list; refactor stats * add pytorch optimization * fixed some syntax errors * comment out save_storage for mock * uncomment save storage for mock * fixed indentation * Yangli2/fastio double buffer pytorch optimized (#291) * Double I/O buffer optimization * add pytorch optimization * fixed some syntax errors * comment out save_storage for mock * uncomment save storage for mock * fixed indentation Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com> Co-authored-by: Yang Li <yangli2@microsoft.com> * making deepspeed/runtime/fp16/loss_scaler/dynamiclossscale serializable * Dump fast_writer stats only on rank 0 * Configuration option for fused fp16 optimizer * Update to new API * Format fixes * Update to master (#340) * Integrate NVIDIA GPUDirect Storage into nvme library * 1) Remove debug prints 2) Create write file with random data 3) Delete target file before new writes * Workaround gds perf issue by leaking buffers * DGX2 mount/unmount utililties * Formatting * Add torch save/load * Add torch save/load * Remove gds * Add torch legacy save * Update to new cli * Add function signatures Add file_offset arg to read/write apis * Remove redundant asserts * Add DeepSpeedFileWriter * Add mock and python file writers * Format fixes * More perf counters * Fix pinned_offset bug; Show as not real python file object * Buffer copy speed * Add torch_fastio option * Format fixes * Measure torch_fastio perf * Force flush * Formatting * Renamings * Fix device bug * Disable torch.distributed requirement * Renaming * Integrate fast model checkpointing * Double I/O buffer optimization * Support larger sizes * Refactoring; save_storage api * Cast to byte tensor * Handle storage object saves * Remove mysterious import * Api to save storage object list; refactor stats * add pytorch optimization * fixed some syntax errors * comment out save_storage for mock * uncomment save storage for mock * fixed indentation * Yangli2/fastio double buffer pytorch optimized (#291) * Double I/O buffer optimization * add pytorch optimization * fixed some syntax errors * comment out save_storage for mock * uncomment save storage for mock * fixed indentation Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com> Co-authored-by: Yang Li <yangli2@microsoft.com> * making deepspeed/runtime/fp16/loss_scaler/dynamiclossscale serializable * Dump fast_writer stats only on rank 0 * Configuration option for fused fp16 optimizer * Update to new API * Format fixes Co-authored-by: jerryyangli <jerryyangli@gmail.com> Co-authored-by: Yang Li <yangli2@microsoft.com> * Support torch* optimization for version 1.12 * Formatting * Versioned torch* optimization * Versioned torch* optimizations (#341) * Integrate NVIDIA GPUDirect Storage into nvme library * 1) Remove debug prints 2) Create write file with random data 3) Delete target file before new writes * Workaround gds perf issue by leaking buffers * DGX2 mount/unmount utililties * Formatting * Add torch save/load * Add torch save/load * Remove gds * Add torch legacy save * Update to new cli * Add function signatures Add file_offset arg to read/write apis * Remove redundant asserts * Add DeepSpeedFileWriter * Add mock and python file writers * Format fixes * More perf counters * Fix pinned_offset bug; Show as not real python file object * Buffer copy speed * Add torch_fastio option * Format fixes * Measure torch_fastio perf * Force flush * Formatting * Renamings * Fix device bug * Disable torch.distributed requirement * Renaming * Integrate fast model checkpointing * Double I/O buffer optimization * Support larger sizes * Refactoring; save_storage api * Cast to byte tensor * Handle storage object saves * Remove mysterious import * Api to save storage object list; refactor stats * add pytorch optimization * fixed some syntax errors * comment out save_storage for mock * uncomment save storage for mock * fixed indentation * Yangli2/fastio double buffer pytorch optimized (#291) * Double I/O buffer optimization * add pytorch optimization * fixed some syntax errors * comment out save_storage for mock * uncomment save storage for mock * fixed indentation Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com> Co-authored-by: Yang Li <yangli2@microsoft.com> * making deepspeed/runtime/fp16/loss_scaler/dynamiclossscale serializable * Dump fast_writer stats only on rank 0 * Configuration option for fused fp16 optimizer * Update to new API * Format fixes * Support torch* optimization for version 1.12 * Formatting * Versioned torch* optimization Co-authored-by: jerryyangli <jerryyangli@gmail.com> Co-authored-by: Yang Li <yangli2@microsoft.com> * fp16 fused mode * fp16 fused mode (#342) * Integrate NVIDIA GPUDirect Storage into nvme library * 1) Remove debug prints 2) Create write file with random data 3) Delete target file before new writes * Workaround gds perf issue by leaking buffers * DGX2 mount/unmount utililties * Formatting * Add torch save/load * Add torch save/load * Remove gds * Add torch legacy save * Update to new cli * Add function signatures Add file_offset arg to read/write apis * Remove redundant asserts * Add DeepSpeedFileWriter * Add mock and python file writers * Format fixes * More perf counters * Fix pinned_offset bug; Show as not real python file object * Buffer copy speed * Add torch_fastio option * Format fixes * Measure torch_fastio perf * Force flush * Formatting * Renamings * Fix device bug * Disable torch.distributed requirement * Renaming * Integrate fast model checkpointing * Double I/O buffer optimization * Support larger sizes * Refactoring; save_storage api * Cast to byte tensor * Handle storage object saves * Remove mysterious import * Api to save storage object list; refactor stats * add pytorch optimization * fixed some syntax errors * comment out save_storage for mock * uncomment save storage for mock * fixed indentation * Yangli2/fastio double buffer pytorch optimized (#291) * Double I/O buffer optimization * add pytorch optimization * fixed some syntax errors * comment out save_storage for mock * uncomment save storage for mock * fixed indentation Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com> Co-authored-by: Yang Li <yangli2@microsoft.com> * making deepspeed/runtime/fp16/loss_scaler/dynamiclossscale serializable * Dump fast_writer stats only on rank 0 * Configuration option for fused fp16 optimizer * Update to new API * Format fixes * Support torch* optimization for version 1.12 * Formatting * Versioned torch* optimization * fp16 fused mode Co-authored-by: jerryyangli <jerryyangli@gmail.com> Co-authored-by: Yang Li <yangli2@microsoft.com> * Support serialization versions * Support serialization of different torch versions (#343) * Integrate NVIDIA GPUDirect Storage into nvme library * 1) Remove debug prints 2) Create write file with random data 3) Delete target file before new writes * Workaround gds perf issue by leaking buffers * DGX2 mount/unmount utililties * Formatting * Add torch save/load * Add torch save/load * Remove gds * Add torch legacy save * Update to new cli * Add function signatures Add file_offset arg to read/write apis * Remove redundant asserts * Add DeepSpeedFileWriter * Add mock and python file writers * Format fixes * More perf counters * Fix pinned_offset bug; Show as not real python file object * Buffer copy speed * Add torch_fastio option * Format fixes * Measure torch_fastio perf * Force flush * Formatting * Renamings * Fix device bug * Disable torch.distributed requirement * Renaming * Integrate fast model checkpointing * Double I/O buffer optimization * Support larger sizes * Refactoring; save_storage api * Cast to byte tensor * Handle storage object saves * Remove mysterious import * Api to save storage object list; refactor stats * add pytorch optimization * fixed some syntax errors * comment out save_storage for mock * uncomment save storage for mock * fixed indentation * Yangli2/fastio double buffer pytorch optimized (#291) * Double I/O buffer optimization * add pytorch optimization * fixed some syntax errors * comment out save_storage for mock * uncomment save storage for mock * fixed indentation Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com> Co-authored-by: Yang Li <yangli2@microsoft.com> * making deepspeed/runtime/fp16/loss_scaler/dynamiclossscale serializable * Dump fast_writer stats only on rank 0 * Configuration option for fused fp16 optimizer * Update to new API * Format fixes * Support torch* optimization for version 1.12 * Formatting * Versioned torch* optimization * fp16 fused mode * Support serialization versions Co-authored-by: jerryyangli <jerryyangli@gmail.com> Co-authored-by: Yang Li <yangli2@microsoft.com> * distributed ckpt draft (#349) * inject parallel write * Support serialization of different torch versions (#343) (#345) * Integrate NVIDIA GPUDirect Storage into nvme library * 1) Remove debug prints 2) Create write file with random data 3) Delete target file before new writes * Workaround gds perf issue by leaking buffers * DGX2 mount/unmount utililties * Formatting * Add torch save/load * Add torch save/load * Remove gds * Add torch legacy save * Update to new cli * Add function signatures Add file_offset arg to read/write apis * Remove redundant asserts * Add DeepSpeedFileWriter * Add mock and python file writers * Format fixes * More perf counters * Fix pinned_offset bug; Show as not real python file object * Buffer copy speed * Add torch_fastio option * Format fixes * Measure torch_fastio perf * Force flush * Formatting * Renamings * Fix device bug * Disable torch.distributed requirement * Renaming * Integrate fast model checkpointing * Double I/O buffer optimization * Support larger sizes * Refactoring; save_storage api * Cast to byte tensor * Handle storage object saves * Remove mysterious import * Api to save storage object list; refactor stats * add pytorch optimization * fixed some syntax errors * comment out save_storage for mock * uncomment save storage for mock * fixed indentation * Yangli2/fastio double buffer pytorch optimized (#291) * Double I/O buffer optimization * add pytorch optimization * fixed some syntax errors * comment out save_storage for mock * uncomment save storage for mock * fixed indentation Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com> Co-authored-by: Yang Li <yangli2@microsoft.com> * making deepspeed/runtime/fp16/loss_scaler/dynamiclossscale serializable * Dump fast_writer stats only on rank 0 * Configuration option for fused fp16 optimizer * Update to new API * Format fixes * Support torch* optimization for version 1.12 * Formatting * Versioned torch* optimization * fp16 fused mode * Support serialization versions Co-authored-by: jerryyangli <jerryyangli@gmail.com> Co-authored-by: Yang Li <yangli2@microsoft.com> Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com> Co-authored-by: jerryyangli <jerryyangli@gmail.com> Co-authored-by: Yang Li <yangli2@microsoft.com> * finish split distributed write * split based-on num_bytes * resolving single node python test * remove irrelavent prints * format Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com> Co-authored-by: jerryyangli <jerryyangli@gmail.com> Co-authored-by: Yang Li <yangli2@microsoft.com> * torch serialization options * Configurable torch serialization (#350) * Integrate NVIDIA GPUDirect Storage into nvme library * 1) Remove debug prints 2) Create write file with random data 3) Delete target file before new writes * Workaround gds perf issue by leaking buffers * DGX2 mount/unmount utililties * Formatting * Add torch save/load * Add torch save/load * Remove gds * Add torch legacy save * Update to new cli * Add function signatures Add file_offset arg to read/write apis * Remove redundant asserts * Add DeepSpeedFileWriter * Add mock and python file writers * Format fixes * More perf counters * Fix pinned_offset bug; Show as not real python file object * Buffer copy speed * Add torch_fastio option * Format fixes * Measure torch_fastio perf * Force flush * Formatting * Renamings * Fix device bug * Disable torch.distributed requirement * Renaming * Integrate fast model checkpointing * Double I/O buffer optimization * Support larger sizes * Refactoring; save_storage api * Cast to byte tensor * Handle storage object saves * Remove mysterious import * Api to save storage object list; refactor stats * add pytorch optimization * fixed some syntax errors * comment out save_storage for mock * uncomment save storage for mock * fixed indentation * Yangli2/fastio double buffer pytorch optimized (#291) * Double I/O buffer optimization * add pytorch optimization * fixed some syntax errors * comment out save_storage for mock * uncomment save storage for mock * fixed indentation Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com> Co-authored-by: Yang Li <yangli2@microsoft.com> * making deepspeed/runtime/fp16/loss_scaler/dynamiclossscale serializable * Dump fast_writer stats only on rank 0 * Configuration option for fused fp16 optimizer * Update to new API * Format fixes * Support torch* optimization for version 1.12 * Formatting * Versioned torch* optimization * fp16 fused mode * Support serialization versions * torch serialization options Co-authored-by: jerryyangli <jerryyangli@gmail.com> Co-authored-by: Yang Li <yangli2@microsoft.com> * Distributed writer slicing on byte boundary * Fix typo * FastFileWriter Config; Parallel writer nodes * Minor fix * remove warning from fast-io-ckpt (#354) * Relocate debug print * Parallel writing through byte boundary slicing (#351) * Integrate NVIDIA GPUDirect Storage into nvme library * 1) Remove debug prints 2) Create write file with random data 3) Delete target file before new writes * Workaround gds perf issue by leaking buffers * DGX2 mount/unmount utililties * Formatting * Add torch save/load * Add torch save/load * Remove gds * Add torch legacy save * Update to new cli * Add function signatures Add file_offset arg to read/write apis * Remove redundant asserts * Add DeepSpeedFileWriter * Add mock and python file writers * Format fixes * More perf counters * Fix pinned_offset bug; Show as not real python file object * Buffer copy speed * Add torch_fastio option * Format fixes * Measure torch_fastio perf * Force flush * Formatting * Renamings * Fix device bug * Disable torch.distributed requirement * Renaming * Integrate fast model checkpointing * Double I/O buffer optimization * Support larger sizes * Refactoring; save_storage api * Cast to byte tensor * Handle storage object saves * Remove mysterious import * Api to save storage object list; refactor stats * add pytorch optimization * fixed some syntax errors * comment out save_storage for mock * uncomment save storage for mock * fixed indentation * Yangli2/fastio double buffer pytorch optimized (#291) * Double I/O buffer optimization * add pytorch optimization * fixed some syntax errors * comment out save_storage for mock * uncomment save storage for mock * fixed indentation Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com> Co-authored-by: Yang Li <yangli2@microsoft.com> * making deepspeed/runtime/fp16/loss_scaler/dynamiclossscale serializable * Dump fast_writer stats only on rank 0 * Configuration option for fused fp16 optimizer * Update to new API * Format fixes * Support torch* optimization for version 1.12 * Formatting * Versioned torch* optimization * fp16 fused mode * Support serialization versions * torch serialization options * Distributed writer slicing on byte boundary * Fix typo * FastFileWriter Config; Parallel writer nodes * Minor fix * remove warning from fast-io-ckpt (#354) * Relocate debug print Co-authored-by: jerryyangli <jerryyangli@gmail.com> Co-authored-by: Yang Li <yangli2@microsoft.com> Co-authored-by: Guanhua Wang <alexwgh333@gmail.com> * fix broken mock_file_writer (#357) * Report write speed * DP writing * DP MoE checkpoints Generalize DP dense checkpoints for socket/machine options * Various improvements (#376) * Integrate NVIDIA GPUDirect Storage into nvme library * 1) Remove debug prints 2) Create write file with random data 3) Delete target file before new writes * Workaround gds perf issue by leaking buffers * DGX2 mount/unmount utililties * Formatting * Add torch save/load * Add torch save/load * Remove gds * Add torch legacy save * Update to new cli * Add function signatures Add file_offset arg to read/write apis * Remove redundant asserts * Add DeepSpeedFileWriter * Add mock and python file writers * Format fixes * More perf counters * Fix pinned_offset bug; Show as not real python file object * Buffer copy speed * Add torch_fastio option * Format fixes * Measure torch_fastio perf * Force flush * Formatting * Renamings * Fix device bug * Disable torch.distributed requirement * Renaming * Integrate fast model checkpointing * Double I/O buffer optimization * Support larger sizes * Refactoring; save_storage api * Cast to byte tensor * Handle storage object saves * Remove mysterious import * Api to save storage object list; refactor stats * add pytorch optimization * fixed some syntax errors * comment out save_storage for mock * uncomment save storage for mock * fixed indentation * Yangli2/fastio double buffer pytorch optimized (#291) * Double I/O buffer optimization * add pytorch optimization * fixed some syntax errors * comment out save_storage for mock * uncomment save storage for mock * fixed indentation Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com> Co-authored-by: Yang Li <yangli2@microsoft.com> * making deepspeed/runtime/fp16/loss_scaler/dynamiclossscale serializable * Dump fast_writer stats only on rank 0 * Configuration option for fused fp16 optimizer * Update to new API * Format fixes * Support torch* optimization for version 1.12 * Formatting * Versioned torch* optimization * fp16 fused mode * Support serialization versions * torch serialization options * Distributed writer slicing on byte boundary * Fix typo * FastFileWriter Config; Parallel writer nodes * Minor fix * remove warning from fast-io-ckpt (#354) * Relocate debug print * Report write speed * DP writing * DP MoE checkpoints Generalize DP dense checkpoints for socket/machine options Co-authored-by: jerryyangli <jerryyangli@gmail.com> Co-authored-by: Yang Li <yangli2@microsoft.com> Co-authored-by: Guanhua Wang <alexwgh333@gmail.com> * Decoupled checkpointing * New MP slicing algorithm * Format fixes * Decoupled checkpointing support (#384) * Integrate NVIDIA GPUDirect Storage into nvme library * 1) Remove debug prints 2) Create write file with random data 3) Delete target file before new writes * Workaround gds perf issue by leaking buffers * DGX2 mount/unmount utililties * Formatting * Add torch save/load * Add torch save/load * Remove gds * Add torch legacy save * Update to new cli * Add function signatures Add file_offset arg to read/write apis * Remove redundant asserts * Add DeepSpeedFileWriter * Add mock and python file writers * Format fixes * More perf counters * Fix pinned_offset bug; Show as not real python file object * Buffer copy speed * Add torch_fastio option * Format fixes * Measure torch_fastio perf * Force flush * Formatting * Renamings * Fix device bug * Disable torch.distributed requirement * Renaming * Integrate fast model checkpointing * Double I/O buffer optimization * Support larger sizes * Refactoring; save_storage api * Cast to byte tensor * Handle storage object saves * Remove mysterious import * Api to save storage object list; refactor stats * add pytorch optimization * fixed some syntax errors * comment out save_storage for mock * uncomment save storage for mock * fixed indentation * Yangli2/fastio double buffer pytorch optimized (#291) * Double I/O buffer optimization * add pytorch optimization * fixed some syntax errors * comment out save_storage for mock * uncomment save storage for mock * fixed indentation Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com> Co-authored-by: Yang Li <yangli2@microsoft.com> * making deepspeed/runtime/fp16/loss_scaler/dynamiclossscale serializable * Dump fast_writer stats only on rank 0 * Configuration option for fused fp16 optimizer * Update to new API * Format fixes * Support torch* optimization for version 1.12 * Formatting * Versioned torch* optimization * fp16 fused mode * Support serialization versions * torch serialization options * Distributed writer slicing on byte boundary * Fix typo * FastFileWriter Config; Parallel writer nodes * Minor fix * remove warning from fast-io-ckpt (#354) * Relocate debug print * Report write speed * DP writing * DP MoE checkpoints Generalize DP dense checkpoints for socket/machine options * Decoupled checkpointing * New MP slicing algorithm * Format fixes Co-authored-by: jerryyangli <jerryyangli@gmail.com> Co-authored-by: Yang Li <yangli2@microsoft.com> Co-authored-by: Guanhua Wang <alexwgh333@gmail.com> * add io multiplier for larger scale simulation (#411) * add io multiplier config for simulation * remove prints and test correctness * format * Merge with master * Format fixes * Guanhua/fast io clean v5 (#435) * Add environment variable to make nvcc compilation more verbose (#2759) * Bing/formatting correction (#2764) * modify engine.py for formatting * commit formatting changes on engine.py * Add links to new azureML examples (#2756) Co-authored-by: Jeff Rasley <jerasley@microsoft.com> * Fix hardcoded instances to fp16 in optimizer creation log messages to the correct dtype. (#2743) * Remove hardcoded instances to fp16 in log messages. * Add model_dtype to print the correct format * Respond to PR feedback --------- Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com> * Refactor/Pydantify monitoring config (#2640) * pydantify monitoring configs --------- Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com> * Pin minimum `packaging` requirement (#2771) Co-authored-by: Jeff Rasley <jerasley@microsoft.com> * Fix for diffusers v0.12.0 (#2753) Co-authored-by: Jeff Rasley <jerasley@microsoft.com> * update copy right in aio * type fix in ds_py_aio_handle * update year in aio/py_test * fix description in util pybind * update and remove prints in fast_file_writer * remove del print * remove dist barrier in engine.py * update year in runtime/model_ckpt * add todo in runtime/model_ckpt/util.py * update year * reverse pip3 * update opbuilder * format * modify print for python * fix print capability * fix print * some fix in flops_profiler (#2068) * bugs in profiler: 1. Tensor.bmm missed in _patch_tensor_methods function 2. missed funtions in _reload_functionals and _reload_tensor_methods functions 3. torch.mm and torch.Tensor.mm will have same __name__ in wrapFunc, my suggustion is use __str__ instead. * formatting --------- Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com> Co-authored-by: Cheng Li <pistasable@gmail.com> * fix upsample flops compute by skipping unused kargs (#2773) * fix upsample flops compute by skipping unused kargs * fix format * format * Fix broken kernel inject bug (#2776) * format * remove zero change * fix engine issue --------- Co-authored-by: Connor Holmes <connorholmes@microsoft.com> Co-authored-by: Bing Xie <67908712+xiexbing@users.noreply.github.com> Co-authored-by: cassieesvelt <73311224+cassieesvelt@users.noreply.github.com> Co-authored-by: Jeff Rasley <jerasley@microsoft.com> Co-authored-by: Logan Adams <114770087+loadams@users.noreply.github.com> Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com> Co-authored-by: Michael Wyatt <michaelwyatt@microsoft.com> Co-authored-by: Carlos Mocholí <carlossmocholi@gmail.com> Co-authored-by: swli <47371259+lucasleesw@users.noreply.github.com> Co-authored-by: Cheng Li <pistasable@gmail.com> Co-authored-by: Molly Smith <112220543+molly-smith@users.noreply.github.com> * Formatting * Formatting * Debug file delete slowdown * Investigate write perf * Investigate write perf * Fix mising args * Fix microbenchmark and unit tests (#450) * Debug file delete slowdown * Investigate write perf * Investigate write perf * Fix mising args * Formatting * Rebase attempts * updates for running with newest dependencies * Pydantic fixes * Rebase fixes * Fix rebase bugs * Add DS utils for tensor casting * Fomat fixes * Fix GDS * Update with io_engine API * Continued rebase * Integrate GDS into writer factory * Add --venv_script option * Formatting fix Signed-off-by: Olatunji Ruwase <olruwase@microsoft.com> --------- Signed-off-by: Olatunji Ruwase <olruwase@microsoft.com> Co-authored-by: jerryyangli <jerryyangli@gmail.com> Co-authored-by: Yang Li <yangli2@microsoft.com> Co-authored-by: Guanhua Wang <alexwgh333@gmail.com> Co-authored-by: Connor Holmes <connorholmes@microsoft.com> Co-authored-by: Bing Xie <67908712+xiexbing@users.noreply.github.com> Co-authored-by: cassieesvelt <73311224+cassieesvelt@users.noreply.github.com> Co-authored-by: Jeff Rasley <jerasley@microsoft.com> Co-authored-by: Logan Adams <114770087+loadams@users.noreply.github.com> Co-authored-by: Michael Wyatt <michaelwyatt@microsoft.com> Co-authored-by: Carlos Mocholí <carlossmocholi@gmail.com> Co-authored-by: swli <47371259+lucasleesw@users.noreply.github.com> Co-authored-by: Cheng Li <pistasable@gmail.com> Co-authored-by: Molly Smith <112220543+molly-smith@users.noreply.github.com> Co-authored-by: Ubuntu <jomayeri@microsoft.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add ZeRO tutorial