Replies: 64 comments 184 replies
-
|
It is also something other vendors out there are championing such as nvidia (KTVC): Article: https://venturebeat.com/orchestration/nvidia-shrinks-llm-memory-20x-without-changing-model-weights More links within that reference. It would be great to hear from the developers what is ahead regarding such features! |
Beta Was this translation helpful? Give feedback.
-
|
EDIT (April 1): The data below has been corrected. Original claims about a 92.5% prompt collapse and memory paradox were wrong (RSS measurement, silent request failures). See my correction reply below for accurate numbers. Corrected DGX Spark GB10 baseline data (llama.cpp build 8399, Nemotron-3-Nano-30B-A3B Q4_K_XL, 128K context, measured via nvidia-smi + llama.cpp internal KV buffer reporting): Memory: Cache KV Buffer Total GPU Savings Context f16 q8_0 q4_0 Context f16 q8_0 q4_0 q4_0 delta Still planning to build from @TheTom's fork and benchmark turbo3/turbo4 on this hardware. GB10 (sm_121) would be the first Blackwell-class validation of the CUDA path. Full data + methodology comparison: https://github.com/Memoriant/dgx-spark-kv-cache-benchmark |
Beta Was this translation helpful? Give feedback.
-
Beta Was this translation helpful? Give feedback.
-
|
@dentity007 The 92.5% prompt processing collapse at 64K is a clear demonstration of why TurboQuant matters. The unified memory result (q4_0 costing more than fp16) is also valuable -- scale/zero-point metadata overhead exceeding compression on shared-memory architectures is worth documenting. One note on build source: @TheTom's repo (TheTom/llama-cpp-turboquant, branch Your SM 121 would be the first Blackwell-class validation of the block_size=128 CUDA path. @AmesianX has Blackwell results on a separate implementation but not on this codebase. Would be interesting to see how the WHT rotation performs on that hardware. |
Beta Was this translation helpful? Give feedback.
-
|
Small correction on attribution. The CUDA work in my repo is from @signalnine (CUDA port merged as PR #3, plus InnerQ per-channel equalization). spiritbuun has their own separate CUDA fork with different optimizations (which is what i believedentity007 is using?). I've been collaborating with spiritbuun and he's been doing amazing work on the CUDA side. Both are great contributions, just want to make sure credit goes to the right people. Also a few clarifications:
Appreciate the detailed summary though. and thanks again for the block_size=128 CUDA fix (PR #32), that was a fast catch. |
Beta Was this translation helpful? Give feedback.
-
|
Correction to my post above: the benchmark data I shared was flawed. u/audioen on r/LocalLLaMA caught the methodology error and they were right. What was wrong: "92.5% prompt throughput collapse at 64K" -- Wrong. I measured throughput from requests that failed silently. Prompt throughput is identical across all cache types at all context lengths. Memory: f16: 768 MiB KV buffer, 23,092 MiB total GPU Generation throughput (tok/s): degrades at long context ~6K: f16 44.7 / q4_0 45.0 (+0.7%) @HyperionMS2040 apologies for the misleading data in my earlier post. The "unified memory paradox" was a measurement artifact. Still planning to build from TheTom's fork and benchmark turbo3/turbo4 on GB10 (sm_121). Will report corrected numbers. Full corrected data + methodology comparison: https://github.com/Memoriant/dgx-spark-kv-cache-benchmark |
Beta Was this translation helpful? Give feedback.
-
|
I've got something going here: unixsysdev/llama-turboquant@16e93d5 PS: Closer to optimal. |
Beta Was this translation helpful? Give feedback.
-
|
Be sure to read the CONTRIBUTING.md Pull requests section, esp. the part about adding new data types. |
Beta Was this translation helpful? Give feedback.
-
|
Working TurboQuant Implementation Available Memory layout: |
Beta Was this translation helpful? Give feedback.
-
|
I have a working implementation of TurboQuant as native KV cache types in llama.cpp with Metal GPU support. Repo: https://github.com/TheTom/turboquant_plus What's working:
Benchmarks (M5 Max 128GB):
Compression target is met. Speed gap is from the unoptimized WHT rotation (O(d^2) per block). Working on Hadamard rotation (O(d log d)) and fused flash attention dequant next. Gotcha for anyone else implementing this: Metal JIT silently falls back to CPU if you Happy to collaborate with anyone else working on this. |
Beta Was this translation helpful? Give feedback.
-
|
Here is llama-bench I had run a few weeks ago: |
Beta Was this translation helpful? Give feedback.
-
|
UPDATE: Context scaling regression FIXED. Turns out the root cause wasn't the graph-side rotation matmul. The custom WHT op (O(d log d) butterfly) gave identical performance to the dense matmul, meaning the matmul was never the bottleneck. The actual bottleneck: the Metal dequant was re-reading shared Results after fix (M5 Max, Qwen3.5-35B-A3B):
Flat at 98.7-99.5% through 32K. No degradation trend. PPL +1.1% (5.471 vs 5.414). The fix is on Full investigation documenting every failed approach (custom WHT op, group-32 rotation) before finding the real fix: @tarruda would love to see your 397B numbers with this fix. The dequant optimization should help significantly on your M1 Ultra since the per-position cost was the main issue, and you have 100+ layers compounding it. @Aaryan-Kapoor the dequant unroll pattern might help your CPU implementation too. Same principle: batch the byte reads instead of extracting per element. |
Beta Was this translation helpful? Give feedback.
-
|
I just built commit @9cd043108d61edcd34a8dddf3378d8de6856dff6 and didn't see any difference from the previous build. That is: speed still degrades very severely with increased context. I tried to run |
Beta Was this translation helpful? Give feedback.
-
@tarruda a few things since your last test:
your 397B with 100+ layers will compound differently but these should close the gap significantly vs what you saw before. would love to see your numbers on the latest build. if you can run this it would help me diagnose any remaining issues on M1 Ultra: |
Beta Was this translation helpful? Give feedback.
-
|
Hi @TheTom I will test it later today. Thanks! |
Beta Was this translation helpful? Give feedback.
-
Couldn't wait, so I spun something up; hopefully, it helps the final implementation. Feel free to cherry-pick :)
Working TurboQuant TQ3_0 implementation (CPU, both K+V cache) Branch: https://github.com/Aaryan-Kapoor/llama.cpp/tree/turboquant-tq3_0 Implements Algorithm 1 (TurboQuant_mse) from the paper as
Benchmarks (Qwen3.5-35B-A3B Q4_K_M, CPU, 4 threads):
Output is identical to f16 baseline on the 35B model at temperature 0. Quality degrades on very small models (0.6B) as expected - the paper's claims hold for reasonably-sized models. Usage:
Known limitations:
|
Beta Was this translation helpful? Give feedback.
-
|
great work Aaryan. a few things I noticed that line up with my implementation (github.com/TheTom/turboquant_plus):
one thing to watch: I hit a nasty bug where the graph-side WHT rotation gave PPL 23.5 instead of 6.2. turned out to be ggml column-major storage silently transposing the rotation matrix. documented the full investigation at github.com/TheTom/turboquant_plus/blob/main/docs/pre-rotate-queries-investigation.md in case you go down the GPU path. happy to compare notes. nice to see multiple implementations converging on the same findings. |
Beta Was this translation helpful? Give feedback.
-
At what context length is the output identical? That is an important question as degradation from KV cache quanting can often only be seen as minor calculation errors accumulate over the context length. So I would suggest trying some complex tasks near the max context of the model and then look if the output is still identical. Also, is the output still the same as bf16 with a q8_0 and q4_0 kv cache? If not, then that would mean the accuracy is indeed higher even at long context tasks than the current kv cache quants. |
Beta Was this translation helpful? Give feedback.
-
|
Sonnet 4.6 thinks the qjl part is important: "For attention scores specifically, unbiasedness matters because errors accumulate across all keys in softmax. A small systematic bias in every dot product can skew the softmax distribution noticeably, whereas zero-mean noise tends to cancel." |
Beta Was this translation helpful? Give feedback.
-
Same results on me. I found all bits on MSE (Lloyd-max) is always better in top-1 and top-5 token consistency rate, KL-divergence, etc. https://github.com/arclabs001/YATQ . QJL eliminates bias but increases variance, and:
|
Beta Was this translation helpful? Give feedback.
-
|
Got CUDA + Flash Attention turbo3 working on RTX 5090. Ported @TheTom's Metal turbo3 kernels to CUDA with full Flash Attention support for both K and V. Hardware: RTX 5090 32GB, CUDA 12.8, sm_120, WSL2 Ubuntu 24.04 NIAH: 6/6 exact retrieval Qwen3.5-27B is a hybrid architecture — only 16 of 64 layers have KV cache (the GatedAttention layers). 16 layers × 4 KV heads × 256 head_dim. What's implemented (15 files, 4 new + 11 modified): All dispatch paths: convert, set-rows, get-rows, cpy, MUL_MAT routing (turbo3 excluded from mmvq/mmq, routed through dequant-then-cublas for MUL_MAT) Build: Known limitations: |
Beta Was this translation helpful? Give feedback.
-
|
This is awesome, thanks for porting and testing! Great to see turbo3 running on 5090 with full FA support. The dequant-then-cublas routing for MUL_MAT is a smart approach. @spiritbuun has a separate CUDA fork (RTX 3090) that just hit 98.8% of q8_0 prefill speed using a similar dequant-then-MMA path — bulk dequant to fp16 temp buffer during prefill, vec_dot for decode. Their fork also has a norm correction that makes turbo3 PPL beat q8_0 on CUDA. Worth comparing notes: https://github.com/spiritbuun/llama-cpp-turboquant-cuda/tree/feature/turboquant-kv-cache A few things on the roadmap that might interest you:
Good catch on the CUDA 13.1 MMQ segfault and the hybrid architecture detail (16/64 layers with KV cache). Will add both to our docs. The 700K context result on 32GB is great data. Would you be willing to run our diagnostic script? It standardizes benchmarking across hardware for me to diagnose issues: https://github.com/TheTom/turboquant_plus/blob/main/scripts/README.md |
Beta Was this translation helpful? Give feedback.
-
|
The Metal implementation was clean to port from, really solid work! I will check out @spiritbuun's fork, especially the norm correction — That's a nice optimization. |
Beta Was this translation helpful? Give feedback.
-
|
It's impossible for me to compile your branch, for some reason it's reporting me error C2065: 'M_PI' is undeclared, I am following your compilation guide. I have no issues to compile llama.cpp on my Blackwell card, but yours for some reason is bringing me this issue. |
Beta Was this translation helpful? Give feedback.
-
Fixed in the latest commit (1545941). The issue was a missing #define _USE_MATH_DEFINES before the includes in Pull the latest from main and rebuild: git pull Let me know if you hit anything else. |
Beta Was this translation helpful? Give feedback.
-
|
Yeah, it's giving another set of errors now: error C2375: 'quantize_row_turbo3_0_ref': |
Beta Was this translation helpful? Give feedback.
-
|
anyone working on Vulkan backend? |
Beta Was this translation helpful? Give feedback.
-
|
@jesusmb1995 That fails for me with Confirmed it runs without the cache key params |
Beta Was this translation helpful? Give feedback.
-
@StrangeBytesDev Recently fixed a crash a AMD integrated GPU. So far for two GPU's I tested Nvidia 1080Ti and RADV GDX1150. Can you re-build with latest commit and see if it also did fix it for you? |
Beta Was this translation helpful? Give feedback.
-
|
@jesusmb1995 Can confirm it now passes with flying colors, and now GPU is the bottleneck as expected. I'll go play around some more, thanks for the update! |
Beta Was this translation helpful? Give feedback.
-
|
Great, added support for mixed K-V types (only a subset of them is supported, tbq pq q8 f16) with flash attention kernel on few recent commits. The original case that was failing for you ( (Note: theses results below are from a different laptop, worse speed in general, disclaimer rather a short run. On this machine pq3/pq4 for both had the best speed) |
Beta Was this translation helpful? Give feedback.
-
from your EDIT, and non-mixed results |
Beta Was this translation helpful? Give feedback.
-
|
https://github.com/spiritbuun/llama-cpp-turboquant-cuda This is a fork of Tom's implementation with CUDA support. Results look promising. 
As per their twitter account spiritbuun. |
Beta Was this translation helpful? Give feedback.
-
|
I tested and turbo3 and turbo4 look much worse than even q4_0 in my testing: ikawrakow/ik_llama.cpp#1509 (comment) I'm running llama-perplexity over entire wiki.test.raw. command and method shown in the linked details. |
Beta Was this translation helpful? Give feedback.
-
seems after Commit c99c230 the model loads properly if the model is fully placed in vram, Thank You |
Beta Was this translation helpful? Give feedback.
-
|
I was having good results with this fork until commit 1010625, since then I can't load the model I used anymore (OOM): FWIW Running at 70 tp/s on a 16GB RTX 5080 (before said commit) vs ~65 with |
Beta Was this translation helpful? Give feedback.
-
Sorry about that. This is fixed now. Please pull and retry. Thanks for testing!! |
Beta Was this translation helpful? Give feedback.
-
Confirmed working, thanks! |
Beta Was this translation helpful? Give feedback.
-
|
So it's already in the main repo of llama.cpp? |
Beta Was this translation helpful? Give feedback.
-
|
no |
Beta Was this translation helpful? Give feedback.
-
|
Is no one else seeing the obvious here? |
Beta Was this translation helpful? Give feedback.
-
|
The most efficient is the e-bit, which can be derived, approximately 2.718. Since 3 is closer to e than 2, it is more efficient. The Soviet Union once promoted ternary computers, but they were not continued for various reasons. |
Beta Was this translation helpful? Give feedback.
-
AND, OR and NOT are operations, not states. States are 0 or 1. |
Beta Was this translation helpful? Give feedback.
-
true, I mistyped. STT transcribed "gates" as "states" |
Beta Was this translation helpful? Give feedback.
-
It was technological limitation at the time, difficult to manufacturer, but a South Korean research team created a ternary logic chip using existing binary chip manufacturing techniques a few years ago, so it's viable now. |
Beta Was this translation helpful? Give feedback.
-
I mean most efficient integer |
Beta Was this translation helpful? Give feedback.
-
Engineering Findings from 8-Model TurboQuant BenchmarkWe independently implemented TurboQuant from scratch (Python/NumPy, 49 tests, distortion matches paper ±15%) and ran systematic benchmarks across 8 models from GPT-2 (124M) to Qwen2.5-7B (7.6B). Sharing findings that may be useful for the llama.cpp integration: Finding 1: K/V Norm DisparityThe paper does not discuss this. Modern LLMs have dramatically different Key vs Value vector magnitudes:
Since quantization error scales with norm squared, K needs far more bits than V. The K/V ratio predicts the optimal bit budget: Finding 2: MSE > Prod for AttentionThe paper recommends TurboQuantProd (QJL residual) for Keys. Our tests show MSE for both K and V works better in practice:
QJL adds variance that softmax amplifies. Low variance (MSE) beats unbiasedness (Prod). Finding 3: Outlier-Aware Mixed Precision~5-20% of K channels (especially Layer 0) have 10-100x larger RMS than median. Storing outlier channels at 8-bit, rest at 3-bit:
Finding 4: Compressed Storage VerifiedActual memory savings: GPT-2 89% reduction, 9x compression, zero PPL impact. RepoFull implementation, benchmarks, and data: https://github.com/scos-lab/turboquant ~2,500 LOC Python, 49 tests, MIT license. Hope these findings help with the llama.cpp integration. |
Beta Was this translation helpful? Give feedback.
-
|
Are you saying they missed the 300% PPL increase that you're seeing? Or did they not do what you think they did? |
Beta Was this translation helpful? Give feedback.
-
|
@Rotatingxenomorph Good question. To clarify: the 300% PPL increase was specifically from using TurboQuantProd (the QJL residual correction method from Theorem 2 of the paper) for Key quantization at b=3 on GPT-2 (head_dim=64). The paper's actual deployment likely uses:
Our finding is that for practical attention computation, the QJL correction's added variance hurts more than its unbiasedness helps — especially at small head_dim and low bit-width. Using MSE quantization for both K and V gives much better results (the same distortion bound applies, just without the unbiasedness guarantee). So it's not that the paper is wrong — their theoretical analysis is correct. It's that the practical implementation benefits from MSE over Prod in the regimes we tested (head_dim=64, b=2-4). At head_dim=128+ with their full system (PolarQuant + mixed precision + outlier handling), Prod may perform better. The K/V norm disparity data is the more actionable finding for llama.cpp — it directly affects how you allocate bits between K and V quantization. |
Beta Was this translation helpful? Give feedback.
-
|
I must just be misunderstanding the whole thing. I thought they applied mse to both k and v and then QJL to the inner product estimator which also applies to both. |
Beta Was this translation helpful? Give feedback.
-
|
I've been working on extending unixsysdev's tq3_0 implementation with V cache support and flash attention. Repo here: https://github.com/animehacker/llama-turboquant What this adds on top of unixsysdev's work: Normalization fix (1/32 → 1/√32 for the asymmetric K-side WHT) 72K context with tq3_0 K+V (4.57x compression) Paper with implementation details: https://oliverchurch.com/turboquant-for-ggml-achieving-4.57x-kv-cache-compression-in-llama.cpp.html |
Beta Was this translation helpful? Give feedback.
-
|
Seems like this tq3 quantization works well. When could it be used on model weights to replace the useless -q3- models? |
Beta Was this translation helpful? Give feedback.
-
Thanks! To clarify, tq3_0 is currently KV cache only- not model weights. It works well for KV because the quantization happens online during inference (no calibration data needed), and the WHT rotation makes any input distribution compressible with a fixed codebook. For weights, the challenge is different. Weights are static so you can afford offline calibration, and formats like q4_K_M already exploit that with per-block scales and importance matrices. Whether WHT + Lloyd-Max beats the existing weight quants at 3-bit is an interesting open question though. The rotation trick might help since it reduces outlier sensitivity, which is exactly what kills q3_K quality. The great thing about this is from my testing (on MY specific setup, 2x3090s), the prompt evaluation runs at many hundreds of tokens per second so even though output is only 3-5 TPS, the input being so fast makes it great for high context situations! |
Beta Was this translation helpful? Give feedback.
-
|
This is the first effort I've seen to apply turboquant to weights |
Beta Was this translation helpful? Give feedback.
-
|
Update Mar 30th 2026: WHT + QJL + MSE is the solution! In @AmesianX 's implementation, PPL decreased after introducing QJL. At first I thought this is due to @AmesianX comments, i.e., The fix was using independent sign patterns for MSE WHT and QJL SRHT. Since the only difference is WHT (Walsh-Hadamard Transform), I implemented another version replace random rotation with WHT (https://github.com/Arclabs001/YATQ/blob/main/turboquant_wht.py) Test Setup
Perplexity Comparison (Random Rotation vs WHT)
Attention Score Metrics Comparison
Observations
Finally, why "random rotation" + QJL makes it worse but WHT + QJL makes it better is still a mystery to me. As in the paper, the author says they used random rotation. (this is infer from claude, maybe explain something) Mar 28th Hey everyone! I just finished reproducing TurboQuant (ICLR 2026) purely in torch. This repo supports real QJL by rewriting whole attention and forward process for QWen3 models. And I found the result independantly: In the same bits budget, k-bit MSE is better than k-1 bit MSE + 1 bit QJL Repo link: https://github.com/arclabs001/YATQ BackgroundTurboQuant proposes a clever way to quantize KV caches:
The paper claims QJL eliminates quantization bias, which sounds great in theory. So I implemented both stages and ran extensive tests. The Surprising PartQJL actually hurts performance in practice. Here's what I found on Qwen3-1.7B (4K context), top-1 token consistency rate drops:
MSE-only consistently wins on Top-1 token matching. The gap is huge at low bits and still noticeable at 8-bit. What's Going On?The theory says QJL = no bias. That's true! But here's the trade-off:
QJL eliminates bias but explodes variance. And for attention, variance is worse than bias! Why? Softmax is tolerant to uniform bias: But variance randomly perturbs each score, which messes up Top-K ranking: So you get "unbiased" estimates that give you the wrong Top-1 token more often. Another Thing: Both Keys and Values Don't Need QJLI also tested whether V should use QJL. Short answer: nope.
Values only do weighted sum, so softmax naturally averages out per-vector errors. QJL wastes 1 bit on useless residual info. My TakeawayFor KV cache quantization:
The implementation is open source if anyone wants to dig deeper or challenge these findings: https://github.com/arclabs001/YATQ Would love to hear thoughts from the community! Did I miss something? Are there scenarios where QJL actually shines? |
Beta Was this translation helpful? Give feedback.
-
|
Why not to compress the weights? For small quants there are very few values per 4/3 bits (16 or 8), it means there are a lot of equal values. Very simple encoding with bit strings easily reduce model's size twice or even more. It requires some computation to uncompress, but it is done in cache and takes a small time when inference is not compute bound, but memory throughput, so there is an extra time for decompression. Prompt processing will be a bit slower, but token generation increases twice or more. A big leap to ignore it. |
Beta Was this translation helpful? Give feedback.
-
TurboQuant CUDA Optimized & BenchedSummaryFull optimized CUDA implementation of TurboQuant KV cache compression for llama.cpp, targeting NVIDIA GPUs (SM86+). All 4 turbo types implemented with native flash attention, parallel SET_ROWS encoding, and aggressive kernel optimizations. Validated across 4 GPUs spanning 3 architecture generations (Ampere, Ada, Blackwell). Repo: Madreag/turbo3-cuda branch Forked from TheTom/llama-cpp-turboquant ( Key results (RTX 5090, Qwen 3.5 27B Q6_K):
1. Decode Performance (RTX 5090, 27B Q6_K)
Measured with 2. Prefill Context Scaling (tok/s)
Prefill auto-dequants turbo→fp16 and uses MMA/TILE kernels. All types track q8_0 with negligible overhead. 3. Quality: Perplexity (wikitext-103, 50 chunks)
turbo3/turbo4 stay within 3% of q8_0 even at 32K. turbo2/turbo1.5 delta grows with context — inherent to 2-bit quantization (proven by threshold control test: PPL is bit-identical at 1e-2 and 1e-6 thresholds). Note: PPL baseline differs between wikitext-103 (used here, q8_0=6.1825) and wikitext-2 (used in README, q8_0=6.759) — this is expected from different evaluation corpora. Relative ranking is consistent (turbo4 < turbo3 < turbo2 < turbo1.5), though absolute deltas differ between corpora. 4. Quality: KL Divergence vs f16 (100 prompts, 27B Q6_K)
5. Sparse V Skip OptimizationSkips V dequantization for positions with negligible attention weights. Control test proves zero quality impact:
Skip rates (Qwen3-1.7B, direct attention weight measurement):
6. Norm Correction (TheTom turbo3 + spiritbuun turbo4)Rescales reconstructed vector norm to match original magnitude. Impact:
7. Asymmetric K/V Quality Matrix (PPL ctx=512)
V type dominates PPL (columns vary more than rows). K=turbo3/V=turbo3 ≈ K=q8_0/V=turbo3. 8. Impact of CUDA Kernel OptimizationsMeasured by comparing the base TurboQuant implementation ( RTX 5090 (27B Q6_K)
RTX 3090 (9B Q8_0)
RTX 4090M (9B Q8_0)
Pattern: Short context is identical (weight-loading bound). Optimizations show at 32K+ where KV bandwidth dominates: +13-69% improvement across 4 GPUs. Advantage grows with context depth (64K: +34-47%). turbo4 benefits most because its larger KV amplifies the unoptimized dequant cost. PPL (3090): q8_0 identical (9.3731). turbo4/turbo3 within noise. Optimized turbo2 PPL = 9.61 vs base 9.84 (2.3% better). 9. Per-GPU Performance DetailsRTX 3090 Ti (SM86, OC +2200 mem, Qwen 3.5 9B Q8_0)Decode speed (
turbo2 beats q8_0 by 5.3% at 32K (81.58 vs 77.44). turbo2 64K = 72.79 where q8_0 OOMs. PPL (base vs optimized): q8_0 identical (8.5249). turbo3 within noise (+0.2%). NIAH (25 tests, 4K-64K, max_tokens=4000): q8_0=turbo3=turbo2=92%, turbo1.5=100%. 4090M (SM89, 16 GB, Qwen 3.5 9B Q8_0)Speed in Section 8 above. PPL (base vs optimized): q8_0 identical (9.3737). turbo3/turbo4 within noise. NIAH (max_tokens=4000): q8_0=turbo3=100%, turbo2=95%, turbo1.5=50%. 10. Cross-GPU Validation (1,351+ iterations, zero failures)
11. NIAH (Needle-in-a-Haystack)RTX 5090 (Qwen 3.5 9B Q8_0, max_tokens=4000):
Earlier testing data (q8_0=85%, turbo3=70%) was caused by Qwen 3.5 thinking model exhausting max_tokens=1000. Raising to 4000 resolves all token exhaustion failures. RTX 3090 Ti (25 tests 4K-64K, max_tokens=4000): q8_0=turbo3=turbo2=92%, turbo1.5=100%. With sufficient token budget, all types converge — remaining failures at 32K/64K depth 10% are model-specific, not turbo degradation. RTX 3090 (20 tests 4K-32K, max_tokens=4000): q8_0=100%, turbo3=100%, turbo2=95% (1 failure at 32K/10%), turbo1.5=60%. RTX 4090M (20 tests 4K-32K, max_tokens=4000): q8_0=100%, turbo3=100%, turbo2=95%, turbo1.5=50%. Previous 90% scores at max_tokens=2000 were token exhaustion — resolved with 4000. turbo1.5 improved from 35% to 50% but remains degraded (real 2-bit quality issue). 12. Optimizations Applied
13. Limitations
14. Configuration Recommendations
15. Attribution
Repo: Madreag/turbo3-cuda branch |
Beta Was this translation helpful? Give feedback.
-
|
Congrats on the release, this is incredible work. +69% on the 3090 at 32K is great. The asymmetric K/V matrix independently confirming V dominance from CUDA is exactly the kind of cross-validation we need. Thank you for the attribution, really appreciate it. Let's keep it up team! |
Beta Was this translation helpful? Give feedback.
-
|
Was able to validate the optimizations on my 5090 rig. Great work! |
Beta Was this translation helpful? Give feedback.
-
|
Amazing. As we move into the experimental phase with these kernels, I’ve been reflecting on how TurboQuant could serve as the 'structural glue' for an even broader efficiency stack. Would this combination make sense in the future?
Each of these pieces multiply a lot models performance. If we eventually combine all those, we might be looking at a significant leap in local LLM feasibility. Would this potentially let us fit 70B+ models into commodity hardware with let's say 16GB VRAM, without losing much speed or intelligence? Quick AI-assisted calculations lead me to 2000x performance improvements. Seems a bit mind-blowing. Am I oversimplifying that synergy? Would llama.cpp be able to run all this? Has anyone else explored this 'Full-Stack Efficiency' horizon? |
Beta Was this translation helpful? Give feedback.
-
|
Speaking of bitnet, their claim alone sounds drastic enough to make commodity hardware viable.
|
Beta Was this translation helpful? Give feedback.
-
You might find this interesting as well from the Kimi team: |
Beta Was this translation helpful? Give feedback.
-
|
Except, recurrent memory comes at a qualitative cost, I don't think moving
away from transformers is necessary.
We already support SSM/Hymba and you can run LFM style models today.
Besides this can continuous context can also be emulated externally via
continuous tree batching + pruning outside the model boundary - for
instance, see
https://docs.lloyal.ai/learn/introduction - uses kv seqIds for exactly that
for static transformers.
…On Wed, 1 Apr 2026 at 7:08 pm, Jairo Llopis ***@***.***> wrote:
Amazing.
As we move into the experimental phase with these kernels, I’ve been
reflecting on how TurboQuant could serve as the 'structural glue' for an
even broader efficiency stack. Would this combination make sense in the
future?
- *BitNet (1.58b)* at the training phase: Ternary-native models using
int sums instead of float multiplications.
- *TurboQuant* at the fine-tuning phase: Ensuring that data
(especially the KV Cache) flows between memory and compute at sub-3bit
precision without bottlenecks.
- *LFM* (Liquid Foundation Models) as the 'Brain Logic': Moving away
from static Transformer blocks toward a continuous, fluid state. This could
theoretically allow for near-infinite context with sub-linear memory growth.
Each of these pieces multiply a lot models performance. If we eventually
combine all those, we might be looking at a significant leap in local LLM
feasibility. Would this potentially let us fit 70B+ models into commodity
hardware with let's say 16GB VRAM, without losing much speed or
intelligence?
Quick AI-assisted calculations lead me to 2000x performance improvements.
Seems a bit mind-blowing. Am I oversimplifying that synergy? Would
llama.cpp be able to run all this? Has anyone else explored this
'Full-Stack Efficiency' horizon?
—
Reply to this email directly, view it on GitHub
<#20969?email_source=notifications&email_token=BZQNGKHSJFX3FF4D3NTJJTD4TTFAZA5CNFSNUABIM5UWIORPF5TWS5BNNB2WEL2ENFZWG5LTONUW63SDN5WW2ZLOOQXTCNRUGA2TGNBQUZZGKYLTN5XKM3LBNZ2WC3FFMV3GK3TUVRTG633UMVZF6Y3MNFRWW#discussioncomment-16405340>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/BZQNGKGGC3V5AFXBGTYH4DL4TTFAZAVCNFSM6AAAAACW6IA4XKVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTMNBQGUZTIMA>
.
You are receiving this because you are subscribed to this thread.Message
ID: ***@***.***
com>
|
Beta Was this translation helpful? Give feedback.
-
TurboQuant v1.3.0 — Bulletproof head_dim Detection + All Reported Issues FixedHi all, v1.3.0 addresses every issue reported in this thread. > Build Notice: Previous v1.3.0 binaries had incorrect CUDA architecture (SM 52 default) and missing CUDA runtime DLLs on Windows. All binaries have been pulled and are being rebuilt with SM 75/80/86/89/90/120 support. New binaries uploading soon. Apologies for the inconvenience. Release: https://github.com/AmesianX/TurboQuant/releases/tag/v1.3.0 Issues Resolved
Key Change: head_dim Detection Completely RedesignedThe old code only read New detection uses a 5-level priority cascade with cross-validation:
All signals are logged for diagnostics: Critical Discovery: n_embd/n_head is WRONG for many models
turbo4-K PPL explosion was not reproduced in our fork. Our turbo4-K gives PPL 6.73 (+7.5% vs F16 6.26) on Qwen3-30B-A3B — completely normal. After reviewing @TheTom's turbo4-resurrection paper, the PPL explosion in his fork was caused by 7 kernel-level bugs (SET_ROWS turbo3/turbo4 packing mismatch, missing QJL steps, etc.), not head_dim misdetection. Our implementation was built independently and does not share these bugs. Benchmark (v1.3.0, Qwen3-30B-A3B Q4_K_M, DGX Spark GB10)
@fritolays @TheTom @sztlink @modderBUG — would appreciate retesting with v1.3.0 on your hardware once builds are up. The standalone Windows build (
Release: https://github.com/AmesianX/TurboQuant/releases/tag/v1.3.0 |
Beta Was this translation helpful? Give feedback.
-
[proceeds to list 5 signals] Is the comment incorrect, or the list short of one signal? |
Beta Was this translation helpful? Give feedback.
-
|
@dagbdagb Good catch — fixed. It's a 5-level priority cascade (P1→P5), each cross-validated against other available signals. Updated the comment. Thanks! |
Beta Was this translation helpful? Give feedback.
-
|
@sztlink Good news — v1.3.0 handles this. head_dim=80 (Qwen3-4B) now gets caught by the power-of-2 validation and falls back to q8_0 with a clear warning instead of silently failing. The WHT kernel still requires pow2 dimensions, but at least users get an explicit message and the model runs normally with standard quantization. Also, your RTX 4090 benchmark on Qwen3-30B-A3B is great data — thank you. With v1.3.0 the head_dim detection is much more robust (P1→P5 cascade caught that n_embd/n_head=64 is wrong for this model, actual head_dim=128). Would be great to see your numbers with the new build when you get a chance. |
Beta Was this translation helpful? Give feedback.
-
|
@yajo Interesting idea. TurboQuant operates on the KV cache (inference-time activation compression), while BitNet operates on model weights (training-time). They are orthogonal — in principle you could have BitNet weights + TurboQuant KV cache. The challenge would be that BitNet models already have very low-precision activations, so the WHT rotation noise budget is tighter. Worth exploring but no data yet. |
Beta Was this translation helpful? Give feedback.
-
v1.3.0 Benchmark: Qwen3.5-27B Distilled Model (Claude 4.6 Opus Reasoning)@TheTom reported turbo4-K PPL 8.22 on "Qwen3.5-27B distill" — here is our retest with v1.3.0 (P1→P5 head_dim fix applied). Model: Qwen3.5-27B-Claude-4.6-Opus-Reasoning-Distilled Q4_K_M PPL + KV Memory + Speed
Key finding: tbqp3/tbq3 at 5.2x compression with only +1.1% PPL — essentially lossless at head_dim=256 with QJL. This confirms turbo4-K works correctly in our independent implementation. The PPL explosion @TheTom reported in his fork was traced to 7 kernel-level bugs (see his turbo4-resurrection paper), not head_dim misdetection. Our fork does not share these bugs. Scientist Name Transliteration Test ("Pauli Test")Korean has an official national standard (외래어 표기법) for transliterating foreign names — exactly ONE correct answer per name. This makes it an extremely sensitive test for attention precision. Config:
4/6 exact, 2/6 minor variant. The two Release: https://github.com/AmesianX/TurboQuant/releases/tag/v1.3.0 |
Beta Was this translation helpful? Give feedback.
-
|
Is the ""////" output on Qwen3.5-27B-UD" about the model getting stuck in a repetitive loop? Because I get that too on Win 11 RTX 4090 with Sometimes its "!!!!!!!!!!!!!!!!" but mostly "???????????????", same with OpenReasoning-Nemotron-32B-IQ4_XS; Nemotron-3-Nano-30B-A3B-UD-Q4_K_XL oddly stops mid generation, shisa-v2.1-unphi4-14b_Q8_0 and EXAONE-4.0-32B.i1-IQ4_XS.gguf seemed to work fine, |
Beta Was this translation helpful? Give feedback.
-
|
@slashedstar Could you try adding Let us know if the repetitive output still happens with flash attention on. |
Beta Was this translation helpful? Give feedback.
-
Yeah same behaviour with flash-attn on, even without the flag I believe it was auto-detecting it and enabling: |
Beta Was this translation helpful? Give feedback.
-
Test on Ada Lovelace (sm_89) + Large MoE (122B) ResultsTested TurboQuant on hardware and model size not yet covered by the community it seems. Built from @TheTom's fork ( Model: Qwen3.5-122B-A10B Q5_K_S (Unsloth imatrix, 86.4GB, 3 split GGUFs) 1. KV Cache Size (1x82K context, 12 attn layers)
Only 12 of 48 layers use KV cache (the rest are recurrent), so absolute sizes are smaller than a pure transformer, but compression ratios hold per-layer. 2. Decode Performance (2x L40S, 82K ctx)
I used the exact same prompt for all tests. 3. Dual-Slot 2x82K — The UnlockOur production config (q8_0, 1x82K + mmproj) left only 186 MiB free the first GPU. turbo3 changes this:
TurboQuant rotation matrices initialized correctly at 128x128. Both slots verified with concurrent requests, no cross-contamination. 4. VRAM Summary
Notes
Happy to run additional tests if useful, jsut let me know, for example specific context lengths, turbo2 results or other stuff. |
Beta Was this translation helpful? Give feedback.
-
|
This is exactly what we needed. V compression free all the way down to 2-bit, K-side is the only source of decode penalty. Clean confirmation across turbo2/3/4. The whole family is here! You are right that the 29 MiB gap between turbo2 and turbo3 is small on this model because only 12/48 layers use KV cache. On a pure transformer (all layers have KV), the gap scales linearly with layer count. A 27B dense model at 82K context would show a much larger absolute savings from turbo2 vs turbo3. Thank you for running all of these. This is some of the most valuable production data the project has received. Hope you enjoy the extra headroom. More to come. |
Beta Was this translation helpful? Give feedback.
-
|
Just one more follow-up: After your turbo2 suggestion I dug into the mmproj OOM and discovered this env var (in PR #14236). Moving the vision part from GPU0 to GPU1 frees ~870 MiB on GPU0 which removed the entire bottleneck. Result: 2x128K dual-slot + mmproj + vision
From 1x82K single-slot to 2x128K dual-slot. Full model native context on both slots. Vision works, no speed penalty. This is now my new production config. Context scaling with MTMD_BACKEND_DEVICE=CUDA1
Asymmetric q8_0/turbo2 (your earlier request)
I went with q8_0-K/turbo3-V for production (slightly more conservative) but turbo2-V would work equally well it seems. Key insightThe real unlock was this in combination with the asymmetric KV and that |
Beta Was this translation helpful? Give feedback.
-
|
Good to see. From 1x82K single-slot to 2x128K dual-slot with vision, no speed penalty, no quality compromise. Full native context on both slots. The MTMD_BACKEND_DEVICE=CUDA1 find is a great catch. That 19 MiB miss from the previous test was bugging me and you found the real fix. Moving mmproj to GPU1 is the kind of system-level thinking that turns benchmarks into production configs. Thank you for documenting the full context scaling table. Seeing the headroom at each step from 2x80K to 2x128K gives real confidence in the stability of this setup. Going to reference this in our asymmetric K/V paper as the strongest production validation to date. 122B, dual L40S, 2x128K, vision, zero decode penalty. Thanks for the finding I'll update my documents. |
Beta Was this translation helpful? Give feedback.
-
|
Correction: asymmetric q8_0-K / turbo3-V produces corrupt output When deploying the asymmetric config to production (Charles, our Matrix chatbot), we noticed empty/garbled responses. Investigation revealed the model outputs literal U+003F ( The speed measurements I reported earlier are accurate (61.1 t/s, zero TG penalty) the model generates tokens at full speed, they're just wrong tokens. I only caught this when checking actual content in production because my latest benchmarks were speed-only and maybe a bit rushed because of excitement. Symmetric turbo3/turbo3 works correctly. Same fork, same binary, switching from This suggests the K and V turbo3 quantization paths need to be paired, mixing q8_0 K with turbo3 V breaks the attention computation, possibly because the rotation matrices are only applied to turbo3-quantized tensors. What we're running now (production, verified working)
The asymmetric config also has a much larger compute buffer on GPU0 for turbo3-K (2074 MiB vs 858 MiB for q8_0-K), which limits max context with symmetric turbo3 compared to asymmetric. The max / sweet spot now is 2x104K with Updated production config2x104K dual-slot, vision on GPU1, cron jobs verified, interactive chat verified. 58 t/s single, ~6% slower than q8_0 baseline. Very happy with this as a production config nonetheless. Sorry for the incorrect asymmetric data, lesson learned to always verify output content, not just generation speed. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for catching this. I have been investigating this tonight and opened issue #47 to track it. After tracing through the code paths, I found two concrete bugs that affect asymmetric configs: 1. Rotation matrix initialization (llama-kv-cache.cpp:332) — The turbo rotation matrices and InnerQ scale tensors were only allocated when K is a turbo type. With 2. V unpad gated on K type (llama-graph.cpp, 3 instances) — Three V unpad blocks that extract the original head_dim after inverse WHT were checking Honest assessment: your model has head_dim=256 which is already 128-aligned, so the unpad is a no-op and bug #2 alone would not explain the corruption. Bug #1 ensures the rotation infrastructure exists, which may be the missing piece. Every other successful asymmetric test to date has been on head_dim=128 models, so yours is the first to exercise this combination. There may be additional issues I have not found yet. I also added diagnostic logging to the fix so we can see exactly what code paths are being taken. When you run with the fix branch you should see lines like: Fix branch: git fetch origin fix/asymmetric-rotation-init
git checkout fix/asymmetric-rotation-initIf it does not resolve the corruption, please paste back the It is late here so I may not respond until morning, but wanted to get this in your hands as soon as possible. |
Beta Was this translation helpful? Give feedback.
-
|
Tested turboquant fork from @Madreag at 1766c91 on two vGPUs from Tested the following combos: turbo4/turbo4 |
Beta Was this translation helpful? Give feedback.
-
TurboQuant v1.3.0 - Model Sweep@AmesianX here is the new set of tests, thanks again for the fast updates. Environment
Findingsv1.3.0 Head_dim DetectionThe new 5-level priority cascade (P1→P5) resolves all previous auto-detection failures. Phi-4 and DeepSeek, which failed in v1.2.0, now work correctly. GLM-4.7-Flash (head_dim=576) is correctly detected but unsupported, falling back to f16. Qwen3.5 Turbo V Quality IssueQwen3.5 models output only question marks when using turbo types on both K and V. Tested on 27B Q3_K_M and confirmed on 9B Q8_0 — issue is independent of weight quant level. Turbo K + q8_0 V works correctly. Other Qwen models (Qwen2.5-14B) are unaffected. Context vs Speed TradeoffAnalysed across models where TurboQuant provides meaningful context gains beyond q4_0 (DeepSeek-R1-14B, Gemma-3-27B, Magistral-Small-2509, Magistry-24B, Qwen3.5-27B, Llama3.2-24B-A3B): Best context: Best balance: Architecture Summary
Test ResultsClick to view, caution its extensive...
|
Beta Was this translation helpful? Give feedback.
-
|
@fritolays Thank you for this incredible 17-model sweep — 17 models across 11 architectures is exactly the kind of systematic validation we need. Gemma 7.94x, DeepSeek 5.09x on 16GB are great numbers, and confirming P1→P5 fixes Phi-4/DeepSeek detection is very valuable. Your Quick question: Did you build from source or use the release binaries? We discovered the release binaries had incorrect CUDA architectures (SM 52 default instead of SM 75-120). We've pulled those and are rebuilding now. On the Qwen3.5 turbo V "???" issue: We downloaded the exact same model ( No question marks. This strongly suggests the issue is build-related (SM 52 release binary running on SM 89 via JIT emulation), not a code bug. On the 50–75% prompt processing drop: Also likely build-related. On our SM 121 native build, tbqp3/tbq3 prompt processing drops only ~15% vs f16 (94→80 t/s). SM 52 JIT emulation causes massive degradation on compute-heavy WHT kernels. The corrected builds (SM 75/80/86/89/90/120 native) are compiling now — 6 architectures take significantly longer than the previous single-arch build. Will update when ready. Apologies for the build issue. |
Beta Was this translation helpful? Give feedback.
-
|
@AmesianX I used turboquant-bin-win-x64-cuda-cu12.8.zip and had supplied my own DLL's. I've grabbed the new release turboquant-bin-win-x64-cuda-cu12.8.zip + cudart-turboquant-bin-win-cuda-cu12.8-x64.zip. |
Beta Was this translation helpful? Give feedback.
-
|
@AmesianX Actually after some more testing it seems like something else is going on... Let me know if you need me to provide more details. |
Beta Was this translation helpful? Give feedback.
-
|
@TheTom |
Beta Was this translation helpful? Give feedback.
-
|
I'm running across two machines, an rtx 5090 mobile and an rtx 4090 mobile via TB4 bridging |
Beta Was this translation helpful? Give feedback.
-
|
error: static assertion failed: GGML_OP_COUNT has changed - update RPC_PROTO_PATCH_VERSION |
Beta Was this translation helpful? Give feedback.
-
|
Very niche set up lol doing more than I should be allowed to with what I have. |
Beta Was this translation helpful? Give feedback.
-
|
That's a known issue with our fork. We added Quick fix in // Change from:
static_assert(GGML_OP_COUNT == 96, ...);
// To:
static_assert(GGML_OP_COUNT == 97, ...);Also bump Both your 5090 and 4090 machines need to be built from the same fork with this fix, otherwise the RPC protocol version will mismatch. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks, builds completed, now to run some tests |
Beta Was this translation helpful? Give feedback.
-
|
Damn, getting the bug I was getting the other day that was patched upstream in [create_node] invalid tensor: null buffer (id=94197996336480) |
Beta Was this translation helpful? Give feedback.
-
|
Hi,
I have been running tests across different forks on RTX 4070ti with those parameters :
Qwen3.5-35B-A3B-UD-IQ2_XXS.gguf -ngl 99 -fa on --host 0.0.0.0 --port 8081 -np 1 -t 10 --top_p 0.95 --top_k 20, --min_p 0.0 --presence_penalty 0.0 --repeat_penalty 1.0 --temp .98 --jinja -ctk tbqp3 -ctv tbqp3 --ctx-size 130000 -fit off
My testing has been real agentic workload with long running sessions, results so far :
1. WORKS FINE | spiritbuun/llama-cpp-turboquant (https://github.com/spiritbuun )
2. WORKS FINE | Madrear/turbo3-cuda ( https://github.com/Madreag )
3. https://github.com/AmesianX/TurboQuant/ - gets “/////////” after some 10k-20k tokens, tried with smaller context same thing. I did build for the specific CUDA platform ( RTX 4070ti - CMAKE_CUDA_ARCHITECTURES=89-real)
BR
From: AmesianX ***@***.***
Sent: Thursday, April 2, 2026 10:43 AM
To: ggml-org/llama.cpp
Cc: plam40; Manual
Subject: Re: [ggml-org/llama.cpp] TurboQuant - Extreme KV Cache Quantization (Discussion #20969)
@fritolays <https://github.com/fritolays> Follow-up: we tested tbq3_0/tbq3_0 on Qwen3.5-27B-UD-Q4_K_XL on our local build (SM 121 native, DGX Spark GB10) and cannot reproduce the "???" issue. Output is normal:
Hello! I'm doing well, thanks for asking. How about you?
This strongly suggests the issue is build-related (SM 52 vs native). Could you confirm whether you used the release binary or built from source? That will tell us if this is a CUDA arch problem or a real bug we need to fix.
—
Reply to this email directly, view it on GitHub <#20969?email_source=notifications&email_token=BZD7QBWBYBOMLN5H3R5XDH34TYKZNA5CNFSNUABIM5UWIORPF5TWS5BNNB2WEL2ENFZWG5LTONUW63SDN5WW2ZLOOQXTCNRUGE4TINRVUZZGKYLTN5XKM3LBNZ2WC3FFMV3GK3TUVRTG633UMVZF6Y3MNFRWW#discussioncomment-16419465> , or unsubscribe <https://github.com/notifications/unsubscribe-auth/BZD7QBUWVSLB6HDVYNEN45L4TYKZNAVCNFSM6AAAAACW6IA4XKVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTMNBRHE2DMNI> .
You are receiving this because you are subscribed to this thread. <https://github.com/notifications/beacon/BZD7QBXEOYJ7OPRGYHE3DRD4TYKZNBFCNFSM6AAAAACW6IA4XKWGG33NNVSW45C7OR4XAZNRIRUXGY3VONZWS33OINXW23LFNZ2KUY3PNVWWK3TUL5UWJTQA7KFITJTSMVQXG33OUZWWC3TVMFWA.gif> Message ID: ***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
|
I'm on Win 11 RTX 4090 and similar here, after some fixes to make it work on Windows gemini got TheTom/turboquant_plus and Madreag/turbo3-cuda to work but on AmesianX/TurboQuant/releases/tag/v1.3.0 I got the repeating character problem on Qwen3.5-0.8B-IQ4_XS.gguf and OpenReasoning-Nemotron-32B-IQ4_XS (#20969) |
Beta Was this translation helpful? Give feedback.
-
|
@TheTom |
Beta Was this translation helpful? Give feedback.
-
|
Related but different direction: I posted a show-and-tell here on an adaptive TurboQuant + RotateKV K-cache path for Qwen2.5: Calibration-driven per-layer / per-channel K compression, with asymmetric K/V support. A few numbers:
Still a proof of concept and slower on decode, but maybe relevant to people following this thread. |
Beta Was this translation helpful? Give feedback.
-
Ampere (RTX 3080 Ti) Benchmark Data — Mixed KV Parity, PR #36 Validation, PR #43 turbo4 Fix@TheTom — you mentioned wanting CUDA mixed Model: Qwen3.5 9B Q4_K_M (bartowski) Mixed q8_0 × turbo — Speed (t/s)
Mixed q8_0 × turbo — PPL (wikitext-2, 16K context)
Takeaway: On Ampere, mixed PR #36 (signalnine FA optimizations) — Ampere A/BTested
PPL: symmetric turbo3 shifted from 6.6245 to 6.6478 — consistent with auto-asymmetric overriding to q8_0 K (matches the q8_0/turbo3 baseline exactly). Auto-asymmetric is working correctly. Takeaway: PR #36 gives +35% decode on RTX 5090 but only +1% on Ampere. No regression on any config. The shmem LUT and occupancy tuning are Blackwell-optimized — Ampere's smaller shared memory doesn't benefit as much. Safe to merge from an Ampere perspective. PR #43 (Dubascudes turbo4 WHT dequant fix) — ValidatedWe previously reported turbo4 K+V producing PPL 6952 on Llama arch (NeuralDaredevil 8B). Tested PR #43 @
turbo4/turbo4 speed: pp512=3970, pp16384=3707, tg128=101.8 The WHT/dense rotation mismatch in the C reference was indeed the root cause. turbo4 K+V is now functional on CUDA. Hardware NoteRTX 3080 Ti (12GB, SM 86, Ampere, PCIe Gen 3 x16). This is representative of the 30-series/40-series consumer cards that most people are actually running. Happy to run additional configs if useful. |
Beta Was this translation helpful? Give feedback.
-
|
Hi, sharing task-accuracy benchmarks as a complement to PPL — since as @TheTom noted, PPL and generation quality can diverge. We ran a KV cache recall benchmark on Qwen3-14b (no-think mode) using the spiritbuun CUDA fork, testing whether quantization corrupts recall of information stored earlier in context. The method: place a math problem at position 0, fill with N tokens of unrelated text, ask the model to recall and solve. 52 tests: 2×2 matrix mult, 3×3 matrix mult, scalar arithmetic — at 0/200/500/1000 token filler distances.
Key findings:
The benchmark script is open source: https://github.com/eullm/eullm/blob/main/bench/turboquant_math_accuracy.py Tested on EULLM Engine (Ollama-compatible Rust inference server) wrapping the spiritbuun CUDA fork. ctx=16384, temperature=0, num_predict=2048. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
Google Research just posted a blog and paper about a new algorithm that allows quantizing the KV cache down to under 3 bits with close to 0 accuracy loss.
Blog: https://research.google/blog/turboquant-redefining-ai-efficiency-with-extreme-compression/
Paper: https://arxiv.org/pdf/2504.19874
This could be huge if their claims are true and MLX developers are already jumping on this
https://x.com/Prince_Canuma/status/2036611007523512397
Thought I'd share the news here to see if llama.cpp developers would be interested in adding this feature.
Beta Was this translation helpful? Give feedback.
All reactions