This improves the compile time, but it generates the following warnings:

nvcc warning : '--device-debug (-G)' overrides '--generate-line-info (-lineinfo)'
ptxas warning : Conflicting options --device-debug and --generate-line-info specified, ignoring --generate-line-info option

Not sure what are the implications of overriding the -lineinfo option

-lineinfo essentially attaches a copy of the CUDA source code and allows you to map the binary code to specific source code lines when profiling the code with NVIDIA NSight compute. --device-debug additionally disables optimizations but this makes the information from profiling inaccurate. So I would interpret the warning in that context, that the two options do similar things but are intended for different purposes. For debug builds I think disabling optimizations is the correct things to do; when I profile the code I never use LLAMA_DEBUG anyways but just manually add -lineinfo to the NVCC flags.

This needs to be optional. The only case this helps is when looking for bugs on the kernels that are slow to compile, otherwise these files do not need to be recompiled when making changes to other parts of the code. However, this makes the CUDA backend so slow that it will make debug builds unusable for quick iteration in other areas.

I thought I had measured a much smaller performance regression when I tested it but I must have done something wrong because upon renewed testing I'm getting very similar results. I added a new flag LLAMA_CUDA_DEBUG instead. There is still a warning if one compiles both with LLAMA_DEBUG and LLAMA_CUDA_DEBUG but I think it's inconsequential.