DeepSpeed Communication Profiling and Logging #2012

Quentin-Anthony · 2022-06-13T22:11:09Z

This PR implements logging for all DeepSpeed communication calls

This PR introduces the DeepSpeed Communication Logger

After this PR, all communication calls from #1985 are automatically detected and logged (depending on config options). A final summary is then printed. For example:

Comm. Op            Message Size        Count               Total Latency(ms)   Avg Latency(ms)     tput_avg (Gbps)     busbw_avg (Gbps)    
broadcast
                    0B                  2                   0.19                0.10                0.00                0.00                
                    2.0 KB              146                 11.12               0.08                0.43                0.41                
                    6.0 KB              24                  1.84                0.08                1.29                1.21                
                    6.31 KB             1                   0.12                0.12                0.87                0.82                
                    8.0 KB              24                  1.85                0.08                1.73                1.62                
                    2.0 MB              24                  2.05                0.08                397.06              372.24              
                    4.0 MB              1                   0.15                0.15                434.01              406.89              
                    6.0 MB              24                  2.78                0.12                872.00              817.50              
                    8.0 MB              48                  6.36                0.13                1020.36             956.59              
                    98.25 MB            1                   8317.12             8317.12             0.20                0.19                
barrier
                    0B                  3                   237.96              79.32               0.00                0.00                
all_gather
                    128.0 B             146                 70.81               0.38                0.01                0.01                
                    384.0 B             24                  11.45               0.37                0.02                0.02                
                    512.0 B             24                  10.72               0.38                0.02                0.02                
                    128.0 KB            24                  12.44               0.52                4.20                3.94                
                    256.0 KB            1                   0.45                0.45                9.29                8.71                
                    384.0 KB            24                  18.01               0.57                11.50               10.78               
                    512.0 KB            48                  43.42               0.65                13.22               12.40               
                    6.14 MB             2                   17.76               8.88                40.96               38.40               
all_gather_base
                    128.0 B             1460                87.36               0.06                0.03                0.03                
                    256.0 B             147                 3.18                0.02                0.67                0.63                
                    384.0 B             240                 14.28               0.06                0.10                0.10                
                    512.0 B             240                 14.43               0.06                0.14                0.13                
                    128.0 KB            48                  3.38                0.07                29.79               27.93               
                    128.12 KB           24                  0.24                0.00                471.78              442.30              
                    256.0 KB            2                   0.18                0.09                45.52               42.67               
                    384.38 KB           72                  3.69                0.05                353.74              331.63              
                    512.0 KB            96                  5.22                0.06                412.72              386.92              
                    512.12 KB           24                  1.16                0.05                275.55              258.33              
                    512.5 KB            24                  1.70                0.07                120.35              112.83              
                    6.0 MB              219                 16.63               0.07                1401.24             1313.66             
                    6.0 MB              6                   0.51                0.08                1234.17             1157.03             
                    6.14 MB             11                  1.00                0.08                1308.19             1226.43             
                    6.25 MB             9                   0.31                0.03                15263.03            14309.09            
reduce_scatter_base
                    678.86 MB           40                  602.29              9.69                1468.06             1376.31             
all_reduce
                    1.0 B               20                  5572.57             6.37                0.00                0.00                
                    8.0 B               40                  100.00              0.58                0.00                0.00                
log_summary_barrier
                    0B                  1                   0.11                0.11                0.00                0.00

This PR contributes the following features:

Automatic detection and logging of all comms calls with custom log names
Final comms summary (manual prints with log_summary method)
Config support
verbosity levels (for automatic grouping within DeepSpeed, e.g. all_gather_zero3)
An associated tutorial/documentation

Co-authored-by: Quentin Anthony qganthony@yahoo.com
Co-authored-by: Ammar Ahmad Awan ammar.awan@microsoft.com
Co-authored-by: Jeff Rasley jerasley@microsoft.com

Co-authored-by: Quentin Anthony <qganthony@yahoo.com> Co-authored-by: Ammar Ahmad Awan <ammar.awan@microsoft.com> Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

…pSpeed into staging-comms-next

deepspeed/comm/comm.py

deepspeed/comm/torch.py

deepspeed/runtime/pipe/module.py

deepspeed/utils/comms_logging.py

deepspeed/utils/logging.py

deepspeed/utils/timer.py

Quentin-Anthony

This has been reviewed by Ammar

deepspeed/comm/comm.py

jeffra

LGTM

…staging-comms-logging-v1

Quentin-Anthony · 2022-07-22T21:04:37Z

@jeffra and @awan-10 -- All comments have been resolved and we're ready to merge from my side.

Quentin-Anthony and others added 19 commits May 27, 2022 15:26

Staging comms v1 (#301)

867a853

Co-authored-by: Quentin Anthony <qganthony@yahoo.com> Co-authored-by: Ammar Ahmad Awan <ammar.awan@microsoft.com> Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

Delete stage1.py

c93fcfe

Delete distributed.py

7f8ca01

revert deepspeed/__init__.py logging calls

977ee32

Delete test.py

68eb9f4

Update comments and move custom comm ops to internal functions

54796bb

Merge branch 'staging-comms-next' of https://github.com/microsoft/Dee…

c06c72d

…pSpeed into staging-comms-next

Remove unnecessary print and update backend description

f070a0c

Relax assertion to allow Megatron-DeepSpeed MoE to use ZeRO 1

9976681

Simplify ZeRO stage 1 check for previous commit

09063a3

Remove misleading world_size prints

656b415

Add commslogger class, and introduce rough prototype comms logging

2e7129c

Clean up logger

0023b3e

Add more robust arg checks

e55c8e9

Add labels to common collective calls for logger

31c7dcf

Add more annotations

8e23f50

Fix up log_summary_new and fix logging bug for barrier

7998350

Clean up arg sweep logic and add isend/irecv

227874e

Merge branch 'master' into staging-comms-logging-v1

27c38f9

Quentin-Anthony requested review from RezaYazdaniAminabadi, ShadenSmith, awan-10, cli99, conglongli, eltonzheng, jeffra, minjiaz, samyam and tjruwase as code owners June 13, 2022 22:11

Clean up logging branch

26e15ae

Quentin-Anthony and others added 4 commits June 28, 2022 03:54

Updated fn signatures for torch1.2

298349d

Remove lingering prof arg

102ae1d

Merge branch 'master' into staging-comms-logging-v1

2185f16

Update logging tutorial

4faf3b9