Hm, it looks like the "CPU reference" mechanism that we implemented in #19209 does not actually work as I thought it would. I was thinking that we can run:

test-backend-ops -b CPU

And it would compare the reference vs non-reference CPU implementation. But this is not the case, because the ggml_backend_cpu_set_use_ref() applies to both backend1 and backend2 - they are the same backends.

How do you test the non-reference implementation against the reference?

That's what I use and it correctly fails if there is some bug. For example on master if I do something like this

diff --git a/ggml/src/ggml-cpu/ops.cpp b/ggml/src/ggml-cpu/ops.cpp
index ed4535020..6223b202a 100644
--- a/ggml/src/ggml-cpu/ops.cpp
+++ b/ggml/src/ggml-cpu/ops.cpp
@@ -8247,7 +8247,7 @@ static void ggml_compute_forward_flash_attn_ext_f16_one_chunk(
 
             if (s > M) {
                 ms = expf(M - s);
-                M = s;
+                //M = s;
                 ggml_vec_scale_f32(DV, VKQ32, ms);
             } else {
                 vs = expf(s - M);

I get failures of the form

[FLASH_ATTN_EXT] ERR = 0.135610349 > 0.000500000 FLASH_ATTN_EXT(hsk=40,hsv=40,nh=4,nr23=[4,1],kv=512,nb=1,mask=0,sinks=1,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_KV=f16,permute=[0,1,2,3]): FAIL

when running test-backend-ops -b CPU

When I print pointers in make_test_cases_eval, I get different pointers for backend and backend_cpu

Ah correct - this works as expected.

The problem is that the tiled version is never exercised atm. use_tiled is always false because in all tests the batch size is less than Q_TILE_SZ.

Ah I see, you're right. So I can change in test-backend-ops:

-                                            for (int nb : { 1, 3, 32, 35, }) {
+                                            for (int nb : { 1, 3, 32, 65, }) {

To exercise this path, though would have to be careful when tuning Q_TILE_SZ

I did some bench with zen5x16 (a IA-MAX+ 395)

• the AOCL use aocl AMD hight optimized gemm full BF16.
• FA-0_BF16 use std tinyblas with KV as BF16 no FA
• FA-0_FP16 use std tinyblas with KV as FP16 no FA
  les nexts use FA with FP16 KV: I test some TILE size :
  GGML_FA_TILE_Q/GGML_FA_TILE_KV GEMM_RMxGEMM_RN

For now on this my zen5 with llama 8B the faster I have is:

GGML_FA_TILE_Q 300
GGML_FA_TILE_KV 64
GEMM_RM 6
GEMM_RN 4

If I can find some time I like to "hack" your FA/gemm for BF16 support....

Do you think (or @ggerganov ) it is possible to store the KV-cache packed? so we don"t need to do full repack in FA ?

@Djip007 The primary limitation for FA modifications is the code to remain easy to understand and test.

Do you think (or @ggerganov ) it is possible to store the KV-cache packed? so we don"t need to do full repack in FA ?

Not sure, haven't thought about this. My feeling is even if it is possible, it wouldn't be worth the extra complexity.

My feeling is even if it is possible, it wouldn't be worth the extra complexity.

I completely agree with that, and I saw your reservations about the testing part.

I haven't thought about it very long, so it might not be possible:

We could extend the extra buffers to handle KV in addition to weight cases.
This would allow us to keep the simple case in the basic CPU backend and add optimized versions of the FA, as was done for GEMM.
Well, we might also need to extend the tests to work in these cases.

It might not be a good idea to discuss on this pull request. Should I start a new discussion on this topic?

And it may not be possible or very complicated 😎