Great work!

Why not make an IQ4_XXS by using IQ3_S for the attn_k and attn_q ?
At 4.1bpw, that fits the bill!

This is basically the same as IQ4_NL, but in super-blocks of 256 with 6-bit scales for the blocks of 32 weights. It looks pretty good on the quantization error vs quantized model size curve:

It is possible to move the point closer to the IQ2_XXS...IQ3_M fit line by using IQ3_S for the attn_k and attn_q tensors. This reduces the quantized model size to about 4.1 bpw at the expense of a ~0.3% increase in PPL. But given that currently CPU performance for IQ3_S is pretty bad, I decided against this. Speaking of performance, it is excellent on all platforms where I can test except Metal (as usual):

• 133.7 t/s on CUDA (RTX-4080) vs 128.8 t/s for Q4_0
• 15.8 t/s on AVX2 (Ryzen-7950X) vs 14.5 t/s for Q4_0
• 28.8 t/s on ARM_NEON (M2 Max CPU) vs 28.2 t/s for Q4_0
• 53.9 t/s on Metal (30-core M2 Max GPU) vs 63.1 t/s for Q4_0
  @ikawrakow
  I think i found a bug

get_k_quant_type : tensor cols 13696 x 5120 are not divisible by 256, required for iq4_xsllama_model_quantize: failed to quantize:
Unsupported tensor size encountered
It should be fallback to IQ4NL？

Why not make an IQ4_XXS by using IQ3_S for the attn_k and attn_q ? At 4.1bpw, that fits the bill!

Because I need to fix IQ3_S performance on the CPU first. With attn_q and attn_k quantized with IQ3_S (these two tensors contain about 16% of the model weights for a 7B LLaMA model), performance on my M2 Max CPU drops from 28.8 t/s to 21 t/s. On a Ryzen-7950X CPU performance goes down from 15.8 t/s (achieved with 4 threads) to 12.5 t/s (4 threads) or 14.5 (8 threads). I think IQ4_XS is a really nice alternative to Q4_0 with 6% smaller model size combined with better inference performance (except on Metal), so I don't want to destroy the performance benefit. Let me look more into what is the best way to get to 4 bpw quants.

@sorasoras Thanks! I keep forgetting this check. It should be fixed now.

@sorasoras Thanks! I keep forgetting this check. It should be fixed now.

It's working.

Q4KM   4.6321                                       8.79 GB
Q3KXS  4.6299 +/- 0.04409                    6.12 GB   
IQ4NL  4.6048 +/- 0.04419                     7.61 GB
IQ4XS  4.5885 +/- 0.04395                     7.30 GB
Q6K    4.5787 +/- 0.04407                      11.4 GB
Q5_KS  4.5761 +/- 0.04412                     9.33 GB

That's great!

| qwen 13B IQ4_NL - 4.5 bpw      |   7.61 GiB |    14.17 B | ROCm       |  99 | pp 512     |  1488.78 ± 11.45 |
| qwen 13B IQ4_NL - 4.5 bpw      |   7.61 GiB |    14.17 B | ROCm       |  99 | tg 128     |     73.13 ± 0.18 |
| qwen 13B IQ4_XS - 4.25 bpw     |   7.30 GiB |    14.17 B | ROCm       |  99 | pp 512     |   1547.23 ± 9.30 |
| qwen 13B IQ4_XS - 4.25 bpw     |   7.30 GiB |    14.17 B | ROCm       |  99 | tg 128     |     76.88 ± 0.78 |

@ikawrakow Thanks a lot for your hard work! It is very much appreciated. Do you think that we can fix the slower Metal speeds with better kernels or does it require a whole new quantisation type? I am wondering why there is such a difference. Is it because of the additional overhead/calculations that are required for the new IQ quant methods?

Very nice, seems to be a solid replacement for Q4KS, which was my default recommendation.

Do you think that we can fix the slower Metal speeds with better kernels or does it require a whole new quantisation type?

The quantization in this PR is non-linear, hence it requires a table lookup. If you compare to Q4_0, there are two quants packed in one uint8_t, so getting these is just a matter of q & 0xf and q >> 4. Here we need lookup_table[q & 0xf] and lookup_table[q >> 4]. On the other platforms this makes zero difference. At least on my GPU the calculation is almost always memory bound, so this one additional lookup doesn't matter. On the CPU there are vector shuffle instructions that are very fast, so the cost is negligible too. But for some reason the Apple GPU very much dislikes this additional memory load. I'm already putting the lookup table in shared memory (and that gave a ~30% boost in performance compared to having the lookup table in constant memory), so not sure what else can be done. But I would very much appreciate if someone more knowledgeable than me in Apple GPU matters would find a better approach compared to my implementation.

@ikawrakow
IQs don't seems support Forced dmmv.
Forced dmmv is about 7-8 percent faster for Q5KS. Do you any plan to implements that in the future to further improve performance of iq quants？