This change breaks Q4_0. Repro:

make -j && ./bin/test-backend-ops -o MUL_MAT -b CPU

On M1 Pro it crashes like this:

...
  MUL_MAT(type_a=q4_0,type_b=f32,m=16,n=1,k=256,bs=[10,10],nr=[1,2],per=[0,1,2,3]): OK
  MUL_MAT(type_a=q4_0,type_b=f32,m=16,n=1,k=256,bs=[10,10],nr=[2,2],per=[0,1,2,3]): OK
  MUL_MAT(type_a=q4_0,type_b=f32,m=16,n=16,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3]): Illegal instruction: 4

On M2 Ultra, the tests fail:

  MUL_MAT(type_a=q4_0,type_b=f32,m=16,n=1,k=256,bs=[10,10],nr=[1,2],per=[0,1,2,3]): OK
  MUL_MAT(type_a=q4_0,type_b=f32,m=16,n=1,k=256,bs=[10,10],nr=[2,2],per=[0,1,2,3]): OK
  MUL_MAT(type_a=q4_0,type_b=f32,m=16,n=16,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3]): sentinel mismatch: sent_3 FAIL
  MUL_MAT(type_a=q4_0,type_b=f32,m=16,n=16,k=256,bs=[10,1],nr=[1,1],per=[0,1,2,3]): [MUL_MAT] NMSE = 0.059779312 > 0.000500000 FAIL
  MUL_MAT(type_a=q4_0,type_b=f32,m=16,n=16,k=256,bs=[10,1],nr=[2,1],per=[0,1,2,3]): OK

The problem is even though the sources compile successfully, they do not run:

g++ -march=armv8.2a+i8mm test.cpp && ./a.out
Illegal instruction: 4

I was able to reproduce the failing cases on an M3. The issue appears to be related to the test test-backend-ops, which is a special case that doesn't require model loading. As a result, the Q4_0 optimized kernel was not activated despite the hardware supporting i8mm. Meanwhile the macro GGML_USE_LLAMAFILE was undefined as result of __ARM_FEATURE_MATMUL_INT8 definition. This mismatch caused the kernel selection logic to skip both the optimized path for Q4_0 and GGML_USE_LLAMAFILE path.

I applied a temporary patch to prevent GGML_USE_LLAMAFILE from being undefined and make -j && ./bin/test-backend-ops -o MUL_MAT -b CPU worked as expected without errors.

diff --git a/ggml/src/ggml-cpu/ggml-cpu.c b/ggml/src/ggml-cpu/ggml-cpu.c
index 4b58254e..e1224161 100644
--- a/ggml/src/ggml-cpu/ggml-cpu.c
+++ b/ggml/src/ggml-cpu/ggml-cpu.c
@@ -40,7 +40,7 @@
 #endif

 #if defined(__ARM_FEATURE_SVE) || defined(__ARM_FEATURE_MATMUL_INT8)
-#undef GGML_USE_LLAMAFILE
+//#undef GGML_USE_LLAMAFILE
 #endif

The test-backend-ops should work without GGML_USE_LLAMAFILE. I haven't had time yet to look in details, but what @slaren discovered earlier today in #10543, it seems that the I8MM kernels have out-of-bounds writes for certain sizes?

The test-backend-ops didn't seem use I8MM optimized kernel in my test on a M3 even though __ARM_FEATURE_MATMUL_INT8 was enabled.

test-backend-ops does not use the repack buffers (that's something we need to add to be able to test the repacking functionality), but the standard Q4_0 implementation in ggml_vec_dot_q4_0_q8_0 also supports __ARM_FEATURE_MATMUL_INT8, and that is what is failing here.