cc @drisspg , thoughts on how far away we are from also supporting the nuz float8 flavors here?

does this match how other matmul codegen cases in inductor are tested?

rtol=5e-1, atol=5e-1 is used in e.g. test_valid_cast(). In the Triton matmul tutorial, for fp8 it tested atol=0.125, rtol=0 for M, N, K = 512 as a reference. Let me know if you have concerns!

the float8 test in the triton tutorial is comparing PyTorch bf16 matmul to triton float8 matmul. The comparison here is between eager mode float8 matmul and compiled float8 matmul - should that be tested with a tighter tolerance?

You're right, let me check the actual eager vs compiled difference for some shapes to see what's a good strict tolerance.

@vkuzo updated the tolerance for each after checking the actual differences for the shapes tested. I initially read 5e-1 as 1e-5 in my head which was not helpful...

Could you help confirm how will this be used? Also, what's the alignment requirement for fp8 gemm from Triton and Cublas? Could you add some test cases for e.g. odd k, k divisible by 2 / 4 / 8 / 16? If it's not supported we should throw an error somewhere.

EVEN_K is used in the kernel template to load matrix blocks without masks when K is even. This is used in the mm's lowering as well.

[EDITED] Alignment requirement:

Tensorwise Cublas:
- K divisible by 16
- N divisible by 8

Rowwise Cutlass (same):
- K divisible by 16
- N divisible by 8

Triton (tensorwise and rowwise):
- K >= 16
- (no restrictions on N and M)

Perf for Triton for odd K, K divisible by 16 / 8 / 4 / 2: https://docs.google.com/spreadsheets/d/1KnBFXH-4aUbUXWWVyEKNEIToz5q5cjZ6bipTs7ObUXs/edit?gid=2086012542#gid=2086012542 (internal):

• Odd K compared to K divisible by 16: 6x slower
• Perf worse to best: odd, divisible by 4, divisible by 2 / 8 (about the same), divisible by 16.

For the compiled case, requirement for K and N is checked in _meta_registrations already. I'll add a test case to test the error message. Added test cases for rowwise/tensorwise for when M is (1, 3, 33, 257, 1024), and K, N are 16 or a larger multiple of 32.

"Tensorwise cublas": is the alignment requirement on N real?

"Rowwise Cutlass: no requirement": I'm surprised about this. I don't really believe that the CUTLASS fp8 gemm kernel doesn't have requirements on K. Are we sure about this?

Sorry I meant no additional requirements from Cutlass. Edited my comment above with more info (commenting out current 16-divisibility checks).

the tolerance is higher than I would have expected. If we compare, for example, the cuBLAS kernel (via torch._scaled_mm) versus the triton fp8 tutorial's matmul, is the tolerance similar?

I see one issue is that I include input quantization inside the compiled function, making the difference greater than if only the matmul is compiled. Updated the UTs to only include matmul in the compiled function, and all rtol to 0.01.

I also realized that I made the mistake of using rand() (Uniform[0, 1)) instead of randn() (Normal(0, 1)) to generate input matrices. Using randn(), some result elements are close to 0, so the relative difference can be quite large (for M, K, N = 2, 1024, 2048, one element is -0.0432 in the compiled case and -0.0234 via the cuBLAS kernel, 80% relative difference). So I've increased the atol to 0.05 or 0.07 to allow for this (see below on justification).

If we compare, for example, the cuBLAS kernel (via torch._scaled_mm) versus the triton fp8 tutorial's matmul, is the tolerance similar?

The Triton template used for the compiled case follows that of the mm kernel (with scaling and bias added), which is expressed slightly differently than the tutorial's matmul but the approach is exactly the same. When I compared the cuBLAS kernel, tutorial's Triton and the templated Triton kernels, I did see larger relative difference (for element close to 0) between the templated Triton and cuBLAS for some shapes e.g. (3, 1024, 2048), (2, 1024, 2048. This turned out to be because certain configs (block M size etc) led to different results (expected as float addition is not associative - see my post in the internal Triton group), and once we use the same config for the tutorial's Triton kernel, then both Triton kernels had the same relative difference to cuBLAS. The difference is very small compared to how imprecise they are from the bfloat16 result. Repro script and output: P1496984171.