I'm experiencing almost the same issue in Windows 10

CUDA 10.1
Pytorch 16 in conda environment
gcc
g++
Visual Studio 2019 plus C++ development tools

(pytorch) C:\Users\rgval>nvidia-smi
Thu Sep 24 11:31:36 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 431.40 Driver Version: 431.40 CUDA Version: 10.1 |
|-------------------------------+----------------------+----------------------+
| GPU Name TCC/WDDM | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 166... WDDM | 00000000:01:00.0 Off | N/A |
| N/A 50C P8 3W / N/A | 153MiB / 6144MiB | 0% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+

(pytorch) C:\Users\rgval>python -c "import torch; print(torch.version.cuda)"
10.1

(pytorch) C:\Users\rgval>gcc --version
gcc (x86_64-posix-seh-rev0, Built by MinGW-W64 project) 8.1.0
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

(pytorch) C:\Users\rgval>g++ --version
g++ (x86_64-posix-seh-rev0, Built by MinGW-W64 project) 8.1.0
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

(base) C:\Users\rgval>nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Sun_Jul_28_19:12:52_Pacific_Daylight_Time_2019
Cuda compilation tools, release 10.1, V10.1.243

bash install.sh
......

C:/Users/rgval/miniconda3/envs/pytorch/lib/site-packages/torch/include\c10/util/ThreadLocalDebugInfo.h(12): warning: modifier is ignored on an enum specifier

csrc/lamb/fused_lamb_cuda_kernel.cu(284): error: expected a ")"
detected during instantiation of "void lamb_cuda_kernel_part3(T *, GRAD_T *, T *, T *, const GRAD_T *, float, float, float, float, float, float, float, size_t, adamMode_t, float, T *, T *, T *) [with T=at::acc_type<double, true>, GRAD_T=double]"
(370): here

csrc/lamb/fused_lamb_cuda_kernel.cu(290): error: expected a ")"
detected during instantiation of "void lamb_cuda_kernel_part3(T *, GRAD_T *, T *, T *, const GRAD_T *, float, float, float, float, float, float, float, size_t, adamMode_t, float, T *, T *, T *) [with T=at::acc_type<double, true>, GRAD_T=double]"
(370): here

csrc/lamb/fused_lamb_cuda_kernel.cu(284): error: expected a ")"
detected during instantiation of "void lamb_cuda_kernel_part3(T *, GRAD_T *, T *, T *, const GRAD_T *, float, float, float, float, float, float, float, size_t, adamMode_t, float, T *, T *, T *) [with T=at::acc_type<float, true>, GRAD_T=float]"
(370): here

csrc/lamb/fused_lamb_cuda_kernel.cu(290): error: expected a ")"
detected during instantiation of "void lamb_cuda_kernel_part3(T *, GRAD_T *, T *, T *, const GRAD_T *, float, float, float, float, float, float, float, size_t, adamMode_t, float, T *, T *, T *) [with T=at::acc_type<float, true>, GRAD_T=float]"
(370): here

csrc/lamb/fused_lamb_cuda_kernel.cu(284): error: expected a ")"
detected during instantiation of "void lamb_cuda_kernel_part3(T *, GRAD_T *, T *, T *, const GRAD_T *, float, float, float, float, float, float, float, size_t, adamMode_t, float, T *, T *, T *) [with T=at::acc_type<c10::Half, true>, GRAD_T=c10::Half]"
(370): here

csrc/lamb/fused_lamb_cuda_kernel.cu(290): error: expected a ")"
detected during instantiation of "void lamb_cuda_kernel_part3(T *, GRAD_T *, T *, T *, const GRAD_T *, float, float, float, float, float, float, float, size_t, adamMode_t, float, T *, T *, T *) [with T=at::acc_type<c10::Half, true>, GRAD_T=c10::Half]"
(370): here

6 errors detected in the compilation of "C:/Users/rgval/AppData/Local/Temp/tmpxft_00002c9c_00000000-10_fused_lamb_cuda_kernel.cpp1.ii".
error: command 'C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\bin\nvcc.exe' failed with exit status 1
Error on line 195
Fail to install deepspeed

Cheers!

Originally posted by @rvalenciaaz in #397 (comment)