Sorry, I don't understand this PR. Why can the linked loops be unrolled if and only if the strides are unsigned 64 bit integers? And how is the preference for signed loop variables relevant? In any case, there is a template ggml_cuda_unroll in common.cuh that can be used to manually unroll loops if the compiler can't do it due to e.g. continue. I think in cases such as this it's preferable to use that instead.

Sorry, I don't understand this PR. Why can the linked loops be unrolled if and only if the strides are unsigned 64 bit integers? And how is the preference for signed loop variables relevant?

Let me rephrase: The compiler does unroll on all CCs, but due to changes introduced for >= BW in how signed int overflow is handled for 32-bit pointer arithmetic it will no longer do constant folding and group all data-accessing LDG/STG instructions in the aforementioned loop, sequentially processing them instead:

As a consequence, we pay the latency cost of data-access more often than we have to. By explicitly performing pointer-arithmetic in 64-bit the compiler will group all data-accessing LDG/STG instructions, reducing the # of times we have to wait for data:

.

In any case, there is a template ggml_cuda_unroll in common.cuh that can be used to manually unroll loops if the compiler can't do it due to e.g. continue. I think in cases such as this it's preferable to use that instead.

Consequentially, applying ggml_cuda_unroll while doing 32-bit pointer arithmetic will not do complete constant-folding/grouping

, whereas combining it with explicitly declared 64-bit pointer arithmetic will:

1. My current understanding is that this is a permanent change that will apply to future HW gens also. Hence, moving forward, performance guidance will likely be to do pointer-arithmetic in 64-bit per default. This is guaranteed to give best performance on all CCs and is in line with the prevalence of 64-bit applications. While 32-bit pointer arithmetic will most often be optimized to the same performance degree (we use it extensively throughout ggml without issues for BW GPUs), it may fail in edge cases such as the one observed here & require compiler massaging to work (I did confirm that this specific instance is not a compiler bug).
2. Since ggml_cuda_unroll + 64-bit pointer arithmetic does not give better perf over doing 64-bit pointer arithmetic alone I did not push related changes I show screenshots of.

Hope that clarifies the PR.

Sorry, I am not at all familiar with SASS. Do I understand you correctly that Is this not a difference in terms of which PTX instructions the CUDA compiler emits but rather that the same PTX instructions get optimized differently? I am asking because instructions like st support up to 16 bytes per thread and my impression was that that would be the bandwidth-optimal way to copy data.

Generally speaking, I'm not sure my use of signed 32 bit integers for most things is optimal. It's how I started the code when I had no CUDA experience at all and only later on did the code become so optimized that the specific data types would make a meaningful difference. For the vast majority of kernels the used integers are non-negative and fit within the 32 bit range. Does the compiler optimize the code correctly if uint32_t is used instead of size_t? Because then we can simply start using uint32_t as the default integer data type without having to worry about register pressure or (presumably) slower 64 bit integer arithmetic.

One more question: is the recommendation still to use signed integers for loops that cannot be unrolled at compile time?

Sorry, I am not at all familiar with SASS. Do I understand you correctly that Is this not a difference in terms of which PTX instructions the CUDA compiler emits but rather that the same PTX instructions get optimized differently? I am asking because instructions like st support up to 16 bytes per thread and my impression was that that would be the bandwidth-optimal way to copy data.

Explicitly doing pointer arithmetic in 64 bit will generate you different ptx than doing it in 32 bit and letting the compiler resolve the type-mismatch when addressing your actual 64-bit pointer (which is always loaded as a 64-bit int). This is fine, as you are asking for two different things already in CUDA, not in PTX. Consequentially, two explicit cvt instructions are present in the 64-bit function's ptx code that upcast tid.x and tid.y to 64 bit + different mix of ptx instructions are used for the subsequent pointer arithmetic in the godbolt example here.

The issue is also not in the width of the individual data-accessing instructions, but in the potential inter-dependency of array between the individual loop iterations. While array is marked as __restrict__, the 32-bit pointer arithmetic used to subscript it could overflow. Consequentially, the pointer at iteration n+1 could point to the same value as iteration n, and the loop can no longer be unrolled as the data for n+1 would then depend on n. While the issue I highlighted is also there for pre-BW GPUs, the undefinedness of 32-bit signed int overflow was exploited more aggressively on previous HW generations: a >8GB array (2**31= 2 Gigaelements * 4byte/element) is something feasible on current HW, and the above optimization may lead to issues/faulty behavior here should the pointer arithmetic overflow. While 64-bit pointer arithmetic may overflow also, having 2**63= 9 ExaElements is infeasible on current HW and thus overflow is currently a non-issue.

Side note: For bandwidth-optimal data copies, one should make sure to read/write whole sectors (or better yet cache lines) in a coalesced manner across all threads in a CTA; Basically bandwidth-optimality requires CTA-level optimization, similar to how compute-optimality for GEMM requires tensor-core use that are also shared within a CTA. Since CTA-level-optimal code is at times convoluted to write in a SIMT language such as CUDA, there exists meta-compiler such as CuTILE and TRITON that work on CTA-level. While using wider loads are a good way to read/write sectors and cache lines in a coalesced way, they don't strictly have to be used unless the MMU queue gets full (LG/MIO throttle would be the key words to look out for in Nsight Compute).

Generally speaking, I'm not sure my use of signed 32 bit integers for most things is optimal. It's how I started the code when I had no CUDA experience at all and only later on did the code become so optimized that the specific data types would make a meaningful difference. For the vast majority of kernels the used integers are non-negative and fit within the 32 bit range. Does the compiler optimize the code correctly if uint32_t is used instead of size_t? Because then we can simply start using uint32_t as the default integer data type without having to worry about register pressure or (presumably) slower 64 bit integer arithmetic.

1. uint32_t is worse than int, since unsigned int overflow is defined in C and nvcc complies with it; loop unrolling will not happen, even on older HW (double_loop_32u vs double_loop_32s in the godbolt example). Hence, recommendation is still to use signed int for loop counters when working in 32 bit precision.
2. NVGPUs are optimized for 64-bit integer arithmetic, and slow-downs are not expected to happen based on instruction throughput.
3. Register pressure may be an issue, but should be tackled once its observed.

In summary: From a performance perspective, pointer arithmetic should default to 64-bit (ideally using size_t since that's the way to represent object/pointer sizes in c++), as the compiler will always fully optimize this. The main use-case for 32-bit pointer arithmetic in a 64-bit application is register pressure. Here, one should use signed int where possible for some degree of compiler optimization (most often reaching the same optimization-level as 64-bit pointer arithmetic), and be ready for compiler massaging on BW in the cases where the compiler fails to yield the same level of optimization (i.e. do manual loop hoisting etc.). As the kernel targeted in this PR does not suffer from register pressure, one can simply switch to 64-bit pointer arithmetic on all platforms for full compiler optimization.

@JohannesGaessler lmk should there be a need for additional clarifications

Thank you, the situation is clear to me now. I checked the performance impact for the GPUs I have:

My RTX 4090 seems to also be benefiting slightly in some scenarios, for the RTX 5090 the benefits are larger. I am not seeing differences for any other GPUs that are larger than run-to-run variance so probably it should be fine to use size_t universally for strides (instead of e.g. having to define a type per GPU vendor or generation).