I am installing on the brand-new google compute engine with Ubuntu 2004 and and Tesla V100. All my previous installation steps are successfully (MKL, Cuda - 11.2, Cudnn 8.1, nccl-2.8.4) and vcpkg. I actually have successfully finished the step:

$ ./vcpkg install flashlight-cuda

But when I ran the next step

$ ./vcpkg install flashlight-cuda[asr]

The 'vcpkg' always prompts me that I need to pass the '--recursive' flag to have the [core,lib,asr] all built, and when I actually did that, the build failed with an error as following:

[186/193] : && /usr/bin/c++ -fPIC -g -rdynamic CMakeFiles/fl_asr_tutorial_inference_ctc.dir/flashlight/app/asr/tutorial/InferenceCTC.cpp.o -o bin/asr/fl_asr_tutorial_inference_ctc -L/usr/local/cuda-11.2/targets/x86_64-linux/lib/stubs -L/usr/local/cuda-11.2/targets/x86_64-linux/lib -Wl,-rpath,/home/xiaopanzhang/vcpkg/packages/arrayfire_x64-linux/lib:/home/xiaopanzhang/vcpkg/installed/x64-linux/debug/lib:/home/xiaopanzhang/vcpkg/installed/x64-linux/lib libflashlight-app-asr.a -ldl /home/xiaopanzhang/vcpkg/installed/x64-linux/debug/lib/libglog.a /home/xiaopanzhang/vcpkg/installed/x64-linux/lib/libgflags.a libflashlight.a libfl-libraries.a /opt/intel/oneapi/mkl/latest/lib/intel64/libmkl_intel_lp64.so /opt/intel/oneapi/mkl/latest/lib/intel64/libmkl_gnu_thread.so /opt/intel/oneapi/mkl/latest/lib/intel64/libmkl_core.so -fopenmp /usr/lib/x86_64-linux-gnu/libpthread.so -lm /usr/lib/x86_64-linux-gnu/libdl.so /home/xiaopanzhang/vcpkg/installed/x64-linux/debug/lib/libfftw3.a /home/xiaopanzhang/vcpkg/installed/x64-linux/debug/lib/libkenlm.a /home/xiaopanzhang/vcpkg/installed/x64-linux/debug/lib/libkenlm_util.a /home/xiaopanzhang/vcpkg/installed/x64-linux/debug/lib/liblzmad.a /home/xiaopanzhang/vcpkg/installed/x64-linux/debug/lib/libbz2d.a /home/xiaopanzhang/vcpkg/installed/x64-linux/debug/lib/libz.a /home/xiaopanzhang/vcpkg/packages/arrayfire_x64-linux/lib/libafcuda.so.3.7.3 -pthread /home/xiaopanzhang/vcpkg/installed/x64-linux/lib/libcudnn.so /home/xiaopanzhang/vcpkg/installed/x64-linux/debug/lib/libmpi.so /home/xiaopanzhang/vcpkg/installed/x64-linux/lib/libnccl.so /usr/local/cuda-11.2/lib64/libcudart_static.a -ldl -lpthread /usr/lib/x86_64-linux-gnu/librt.so /home/xiaopanzhang/vcpkg/installed/x64-linux/debug/lib/libsndfile.a /home/xiaopanzhang/vcpkg/installed/x64-linux/debug/lib/libvorbisenc.a /home/xiaopanzhang/vcpkg/installed/x64-linux/debug/lib/libvorbis.a /home/xiaopanzhang/vcpkg/installed/x64-linux/debug/lib/libFLAC.a /home/xiaopanzhang/vcpkg/installed/x64-linux/debug/lib/libogg.a /home/xiaopanzhang/vcpkg/installed/x64-linux/debug/lib/libopus.a -lm -lcudadevrt -lcudart_static -lrt -lpthread -ldl && :
FAILED: bin/asr/fl_asr_tutorial_inference_ctc
: && /usr/bin/c++ -fPIC -g -rdynamic CMakeFiles/fl_asr_tutorial_inference_ctc.dir/flashlight/app/asr/tutorial/InferenceCTC.cpp.o -o bin/asr/fl_asr_tutorial_inference_ctc -L/usr/local/cuda-11.2/targets/x86_64-linux/lib/stubs -L/usr/local/cuda-11.2/targets/x86_64-linux/lib -Wl,-rpath,/home/xiaopanzhang/vcpkg/packages/arrayfire_x64-linux/lib:/home/xiaopanzhang/vcpkg/installed/x64-linux/debug/lib:/home/xiaopanzhang/vcpkg/installed/x64-linux/lib libflashlight-app-asr.a -ldl /home/xiaopanzhang/vcpkg/installed/x64-linux/debug/lib/libglog.a /home/xiaopanzhang/vcpkg/installed/x64-linux/lib/libgflags.a libflashlight.a libfl-libraries.a /opt/intel/oneapi/mkl/latest/lib/intel64/libmkl_intel_lp64.so /opt/intel/oneapi/mkl/latest/lib/intel64/libmkl_gnu_thread.so /opt/intel/oneapi/mkl/latest/lib/intel64/libmkl_core.so -fopenmp /usr/lib/x86_64-linux-gnu/libpthread.so -lm /usr/lib/x86_64-linux-gnu/libdl.so /home/xiaopanzhang/vcpkg/installed/x64-linux/debug/lib/libfftw3.a /home/xiaopanzhang/vcpkg/installed/x64-linux/debug/lib/libkenlm.a /home/xiaopanzhang/vcpkg/installed/x64-linux/debug/lib/libkenlm_util.a /home/xiaopanzhang/vcpkg/installed/x64-linux/debug/lib/liblzmad.a /home/xiaopanzhang/vcpkg/installed/x64-linux/debug/lib/libbz2d.a /home/xiaopanzhang/vcpkg/installed/x64-linux/debug/lib/libz.a /home/xiaopanzhang/vcpkg/packages/arrayfire_x64-linux/lib/libafcuda.so.3.7.3 -pthread /home/xiaopanzhang/vcpkg/installed/x64-linux/lib/libcudnn.so /home/xiaopanzhang/vcpkg/installed/x64-linux/debug/lib/libmpi.so /home/xiaopanzhang/vcpkg/installed/x64-linux/lib/libnccl.so /usr/local/cuda-11.2/lib64/libcudart_static.a -ldl -lpthread /usr/lib/x86_64-linux-gnu/librt.so /home/xiaopanzhang/vcpkg/installed/x64-linux/debug/lib/libsndfile.a /home/xiaopanzhang/vcpkg/installed/x64-linux/debug/lib/libvorbisenc.a /home/xiaopanzhang/vcpkg/installed/x64-linux/debug/lib/libvorbis.a /home/xiaopanzhang/vcpkg/installed/x64-linux/debug/lib/libFLAC.a /home/xiaopanzhang/vcpkg/installed/x64-linux/debug/lib/libogg.a /home/xiaopanzhang/vcpkg/installed/x64-linux/debug/lib/libopus.a -lm -lcudadevrt -lcudart_static -lrt -lpthread -ldl && :
/usr/bin/ld: warning: libcuda.so.1, needed by /home/xiaopanzhang/vcpkg/packages/arrayfire_x64-linux/lib/libafcuda.so.3.7.3, not found (try using -rpath or -rpath-link)
/usr/bin/ld: warning: libcudart.so.10.1, needed by /home/xiaopanzhang/vcpkg/installed/x64-linux/lib/libnccl.so, not found (try using -rpath or -rpath-link)
/usr/bin/ld: /home/xiaopanzhang/vcpkg/installed/x64-linux/lib/libnccl.so: undefined reference to cudaGetDeviceCount@libcudart.so.10.1' /usr/bin/ld: /home/xiaopanzhang/vcpkg/installed/x64-linux/lib/libnccl.so: undefined reference to __cudaRegisterFunction@libcudart.so.10.1'
/usr/bin/ld: /home/xiaopanzhang/vcpkg/installed/x64-linux/lib/libnccl.so: undefined reference to `cudaIpcCloseMemHandle@libcudart.so.10.1'

As far as I can see, the link failed because it looks like the linker is looking for dynamic libraries which is different from the version I installed, but I do have LD_LIBRARY_PATH and PATH point to the cuda installation location (/usr/local/cuda-11.2/... here), I also check the library list of 'vcpkg' and it says:

xiaopanzhang@wav2letter:~/vcpkg$ ./vcpkg list | grep cuda
arrayfire[cuda]:x64-linux ArrayFire CUDA backend
cuda:x64-linux 10.1#5 A parallel computing platform and programming model

xiaopanzhang@wav2letter:~/vcpkg$ ./vcpkg list | grep nccl
nccl:x64-linux 2.4.6 Optimized primitives for collective multi-GPU co..

It looks like the vcpkg also has its own old version of cuda, cudnn and nccl, could anybody tell me what might go wrong here?