Skip to content

CUDA debug build failed on Windows #38024

@peterjc123

Description

@peterjc123

🐛 Bug

Error text:

[2034/2674] Building NVCC (Device) object caffe2/CMakeFiles/torch_cuda.dir/utils/torch_cuda_generated_math_gpu.cu.obj
FAILED: caffe2/CMakeFiles/torch_cuda.dir/utils/torch_cuda_generated_math_gpu.cu.obj 
cmd.exe /C "cd /D C:\w\b\windows\pytorch\build\build\caffe2\CMakeFiles\torch_cuda.dir\utils && C:\w\b\windows\conda\envs\py37\Library\bin\cmake.exe -E make_directory C:/w/b/windows/pytorch/build/build/caffe2/CMakeFiles/torch_cuda.dir/utils/. && C:\w\b\windows\conda\envs\py37\Library\bin\cmake.exe -D verbose:BOOL=OFF -D build_configuration:STRING=Debug -D generated_file:STRING=C:/w/b/windows/pytorch/build/build/caffe2/CMakeFiles/torch_cuda.dir/utils/./torch_cuda_generated_math_gpu.cu.obj -D generated_cubin_file:STRING=C:/w/b/windows/pytorch/build/build/caffe2/CMakeFiles/torch_cuda.dir/utils/./torch_cuda_generated_math_gpu.cu.obj.cubin.txt -P C:/w/b/windows/pytorch/build/build/caffe2/CMakeFiles/torch_cuda.dir/utils/torch_cuda_generated_math_gpu.cu.obj.Debug.cmake"
math_gpu.cu
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.2\include\thrust/detail/contiguous_storage.inl(282): error: calling a __host__ function("std::_Iterator_base12::_Iterator_base12") from a __host__ __device__ function("thrust::detail::contiguous_storage<const     ::__half *,  ::thrust::device_allocator<const     ::__half *> > ::uninitialized_copy<    ::std::_Vector_const_iterator<    ::std::_Vector_val<    ::std::_Simple_types<const     ::__half *> > > > ") is not allowed

C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.2\include\thrust/detail/contiguous_storage.inl(282): error: calling a __host__ function("std::_Iterator_base12::_Iterator_base12") from a __host__ __device__ function("thrust::detail::contiguous_storage<const     ::__half *,  ::thrust::device_allocator<const     ::__half *> > ::uninitialized_copy<    ::std::_Vector_const_iterator<    ::std::_Vector_val<    ::std::_Simple_types<const     ::__half *> > > > ") is not allowed

C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.2\include\thrust/detail/contiguous_storage.inl(282): error: calling a __host__ function("std::_Iterator_base12::~_Iterator_base12") from a __host__ __device__ function("thrust::detail::contiguous_storage<const     ::__half *,  ::thrust::device_allocator<const     ::__half *> > ::uninitialized_copy<    ::std::_Vector_const_iterator<    ::std::_Vector_val<    ::std::_Simple_types<const     ::__half *> > > > ") is not allowed

C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.2\include\thrust/detail/contiguous_storage.inl(282): error: calling a __host__ function("std::_Iterator_base12::~_Iterator_base12") from a __host__ __device__ function("thrust::detail::contiguous_storage<const     ::__half *,  ::thrust::device_allocator<const     ::__half *> > ::uninitialized_copy<    ::std::_Vector_const_iterator<    ::std::_Vector_val<    ::std::_Simple_types<const     ::__half *> > > > ") is not allowed

C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.2\include\thrust/detail/contiguous_storage.inl(282): error: calling a __host__ function("std::_Iterator_base12::_Iterator_base12") from a __host__ __device__ function("thrust::detail::contiguous_storage<    ::__half *,  ::thrust::device_allocator<    ::__half *> > ::uninitialized_copy<    ::std::_Vector_const_iterator<    ::std::_Vector_val<    ::std::_Simple_types<    ::__half *> > > > ") is not allowed

C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.2\include\thrust/detail/contiguous_storage.inl(282): error: calling a __host__ function("std::_Iterator_base12::_Iterator_base12") from a __host__ __device__ function("thrust::detail::contiguous_storage<    ::__half *,  ::thrust::device_allocator<    ::__half *> > ::uninitialized_copy<    ::std::_Vector_const_iterator<    ::std::_Vector_val<    ::std::_Simple_types<    ::__half *> > > > ") is not allowed

C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.2\include\thrust/detail/contiguous_storage.inl(282): error: calling a __host__ function("std::_Iterator_base12::~_Iterator_base12") from a __host__ __device__ function("thrust::detail::contiguous_storage<    ::__half *,  ::thrust::device_allocator<    ::__half *> > ::uninitialized_copy<    ::std::_Vector_const_iterator<    ::std::_Vector_val<    ::std::_Simple_types<    ::__half *> > > > ") is not allowed

C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.2\include\thrust/detail/contiguous_storage.inl(282): error: calling a __host__ function("std::_Iterator_base12::~_Iterator_base12") from a __host__ __device__ function("thrust::detail::contiguous_storage<    ::__half *,  ::thrust::device_allocator<    ::__half *> > ::uninitialized_copy<    ::std::_Vector_const_iterator<    ::std::_Vector_val<    ::std::_Simple_types<    ::__half *> > > > ") is not allowed

C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.2\include\thrust/detail/allocator/allocator_traits.inl(359): error: calling a __host__ function("std::_Iterator_base12::_Iterator_base12") from a __host__ __device__ function("thrust::detail::allocator_traits_detail::copy_construct_range< ::thrust::system::cpp::detail::tag,  ::thrust::device_allocator<const     ::__half *> ,     ::std::_Vector_const_iterator<    ::std::_Vector_val<    ::std::_Simple_types<const     ::__half *> > > ,  ::thrust::device_ptr<const     ::__half *> > ") is not allowed

C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.2\include\thrust/detail/allocator/allocator_traits.inl(359): error: calling a __host__ function("std::_Iterator_base12::_Iterator_base12") from a __host__ __device__ function("thrust::detail::allocator_traits_detail::copy_construct_range< ::thrust::system::cpp::detail::tag,  ::thrust::device_allocator<const     ::__half *> ,     ::std::_Vector_const_iterator<    ::std::_Vector_val<    ::std::_Simple_types<const     ::__half *> > > ,  ::thrust::device_ptr<const     ::__half *> > ") is not allowed

C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.2\include\thrust/detail/allocator/copy_construct_range.inl(218): error: calling a __host__ function("std::_Iterator_base12::~_Iterator_base12") from a __host__ __device__ function("thrust::detail::allocator_traits_detail::copy_construct_range< ::thrust::system::cpp::detail::tag,  ::thrust::device_allocator<const     ::__half *> ,     ::std::_Vector_const_iterator<    ::std::_Vector_val<    ::std::_Simple_types<const     ::__half *> > > ,  ::thrust::device_ptr<const     ::__half *> > ") is not allowed

C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.2\include\thrust/detail/allocator/copy_construct_range.inl(218): error: calling a __host__ function("std::_Iterator_base12::~_Iterator_base12") from a __host__ __device__ function("thrust::detail::allocator_traits_detail::copy_construct_range< ::thrust::system::cpp::detail::tag,  ::thrust::device_allocator<const     ::__half *> ,     ::std::_Vector_const_iterator<    ::std::_Vector_val<    ::std::_Simple_types<const     ::__half *> > > ,  ::thrust::device_ptr<const     ::__half *> > ") is not allowed

C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.2\include\thrust/detail/allocator/allocator_traits.inl(359): error: calling a __host__ function("std::_Iterator_base12::_Iterator_base12") from a __host__ __device__ function("thrust::detail::allocator_traits_detail::copy_construct_range< ::thrust::system::cpp::detail::tag,  ::thrust::device_allocator<    ::__half *> ,     ::std::_Vector_const_iterator<    ::std::_Vector_val<    ::std::_Simple_types<    ::__half *> > > ,  ::thrust::device_ptr<    ::__half *> > ") is not allowed

C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.2\include\thrust/detail/allocator/allocator_traits.inl(359): error: calling a __host__ function("std::_Iterator_base12::_Iterator_base12") from a __host__ __device__ function("thrust::detail::allocator_traits_detail::copy_construct_range< ::thrust::system::cpp::detail::tag,  ::thrust::device_allocator<    ::__half *> ,     ::std::_Vector_const_iterator<    ::std::_Vector_val<    ::std::_Simple_types<    ::__half *> > > ,  ::thrust::device_ptr<    ::__half *> > ") is not allowed

C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.2\include\thrust/detail/allocator/copy_construct_range.inl(218): error: calling a __host__ function("std::_Iterator_base12::~_Iterator_base12") from a __host__ __device__ function("thrust::detail::allocator_traits_detail::copy_construct_range< ::thrust::system::cpp::detail::tag,  ::thrust::device_allocator<    ::__half *> ,     ::std::_Vector_const_iterator<    ::std::_Vector_val<    ::std::_Simple_types<    ::__half *> > > ,  ::thrust::device_ptr<    ::__half *> > ") is not allowed

C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.2\include\thrust/detail/allocator/copy_construct_range.inl(218): error: calling a __host__ function("std::_Iterator_base12::~_Iterator_base12") from a __host__ __device__ function("thrust::detail::allocator_traits_detail::copy_construct_range< ::thrust::system::cpp::detail::tag,  ::thrust::device_allocator<    ::__half *> ,     ::std::_Vector_const_iterator<    ::std::_Vector_val<    ::std::_Simple_types<    ::__half *> > > ,  ::thrust::device_ptr<    ::__half *> > ") is not allowed

C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.2\include\thrust/detail/allocator/copy_construct_range.inl(291): error: calling a __host__ function("std::_Iterator_base12::_Iterator_base12") from a __host__ __device__ function("thrust::detail::copy_construct_range< ::thrust::system::cpp::detail::tag,  ::thrust::device_allocator<const     ::__half *> ,     ::std::_Vector_const_iterator<    ::std::_Vector_val<    ::std::_Simple_types<const     ::__half *> > > ,  ::thrust::device_ptr<const     ::__half *> > ") is not allowed

C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.2\include\thrust/detail/allocator/copy_construct_range.inl(291): error: calling a __host__ function("std::_Iterator_base12::_Iterator_base12") from a __host__ __device__ function("thrust::detail::copy_construct_range< ::thrust::system::cpp::detail::tag,  ::thrust::device_allocator<const     ::__half *> ,     ::std::_Vector_const_iterator<    ::std::_Vector_val<    ::std::_Simple_types<const     ::__half *> > > ,  ::thrust::device_ptr<const     ::__half *> > ") is not allowed

C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.2\include\thrust/detail/allocator/copy_construct_range.inl(291): error: calling a __host__ function("std::_Iterator_base12::~_Iterator_base12") from a __host__ __device__ function("thrust::detail::copy_construct_range< ::thrust::system::cpp::detail::tag,  ::thrust::device_allocator<const     ::__half *> ,     ::std::_Vector_const_iterator<    ::std::_Vector_val<    ::std::_Simple_types<const     ::__half *> > > ,  ::thrust::device_ptr<const     ::__half *> > ") is not allowed

C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.2\include\thrust/detail/allocator/copy_construct_range.inl(291): error: calling a __host__ function("std::_Iterator_base12::~_Iterator_base12") from a __host__ __device__ function("thrust::detail::copy_construct_range< ::thrust::system::cpp::detail::tag,  ::thrust::device_allocator<const     ::__half *> ,     ::std::_Vector_const_iterator<    ::std::_Vector_val<    ::std::_Simple_types<const     ::__half *> > > ,  ::thrust::device_ptr<const     ::__half *> > ") is not allowed

C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.2\include\thrust/detail/allocator/copy_construct_range.inl(291): error: calling a __host__ function("std::_Iterator_base12::_Iterator_base12") from a __host__ __device__ function("thrust::detail::copy_construct_range< ::thrust::system::cpp::detail::tag,  ::thrust::device_allocator<    ::__half *> ,     ::std::_Vector_const_iterator<    ::std::_Vector_val<    ::std::_Simple_types<    ::__half *> > > ,  ::thrust::device_ptr<    ::__half *> > ") is not allowed

C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.2\include\thrust/detail/allocator/copy_construct_range.inl(291): error: calling a __host__ function("std::_Iterator_base12::_Iterator_base12") from a __host__ __device__ function("thrust::detail::copy_construct_range< ::thrust::system::cpp::detail::tag,  ::thrust::device_allocator<    ::__half *> ,     ::std::_Vector_const_iterator<    ::std::_Vector_val<    ::std::_Simple_types<    ::__half *> > > ,  ::thrust::device_ptr<    ::__half *> > ") is not allowed

C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.2\include\thrust/detail/allocator/copy_construct_range.inl(291): error: calling a __host__ function("std::_Iterator_base12::~_Iterator_base12") from a __host__ __device__ function("thrust::detail::copy_construct_range< ::thrust::system::cpp::detail::tag,  ::thrust::device_allocator<    ::__half *> ,     ::std::_Vector_const_iterator<    ::std::_Vector_val<    ::std::_Simple_types<    ::__half *> > > ,  ::thrust::device_ptr<    ::__half *> > ") is not allowed

C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.2\include\thrust/detail/allocator/copy_construct_range.inl(291): error: calling a __host__ function("std::_Iterator_base12::~_Iterator_base12") from a __host__ __device__ function("thrust::detail::copy_construct_range< ::thrust::system::cpp::detail::tag,  ::thrust::device_allocator<    ::__half *> ,     ::std::_Vector_const_iterator<    ::std::_Vector_val<    ::std::_Simple_types<    ::__half *> > > ,  ::thrust::device_ptr<    ::__half *> > ") is not allowed

24 errors detected in the compilation of "C:/Users/circleci/AppData/Local/Temp/tmpxft_0000018c_00000000-12_math_gpu.compute_75.cpp1.ii".
math_gpu.cu
CMake Error at torch_cuda_generated_math_gpu.cu.obj.Debug.cmake:281 (message):
  Error generating file
  C:/w/b/windows/pytorch/build/build/caffe2/CMakeFiles/torch_cuda.dir/utils/./torch_cuda_generated_math_gpu.cu.obj

See https://app.circleci.com/pipelines/github/pytorch/pytorch/165899/workflows/977c46e5-2bfd-4a99-aa57-e6ce50d20cfc/jobs/5383261.

To Reproduce

Steps to reproduce the behavior:

  1. set DEBUG=1
  2. python tools/build_libtorch.py

Expected behavior

Build pass.

Environment

Please copy and paste the output from our
environment collection script
(or fill out the checklist below manually).

You can get the script and run it with:

wget https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py
# For security purposes, please check the contents of collect_env.py before running it.
python collect_env.py
  • PyTorch Version (e.g., 1.0): master
  • OS (e.g., Linux): Windows
  • How you installed PyTorch (conda, pip, source): source
  • Build command you used (if compiling from source): python tools/build_libtorch.py
  • Python version: 3.7
  • CUDA/cuDNN version: CUDA 9.2/10.1/10.2
  • GPU models and configuration: Tesla T4
  • Any other relevant information:

Additional context

cc @ezyang @gchanan @zou3519 @malfet @peterjc123 @nbcsm @guyang3532

Metadata

Metadata

Assignees

No one assigned

    Labels

    high prioritymodule: buildBuild system issuesmodule: windowsWindows support for PyTorchtriagedThis issue has been looked at a team member, and triaged and prioritized into an appropriate module

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions