Python implementation of TurboQuant (Zandieh et al., ICLR 2026) with original engineering findings on KV cache compression for real-world LLMs.

This is a research companion, not a production tool. We reproduce the paper's algorithm, integrate it with HuggingFace transformers, and document what we learned — especially what the paper doesn't tell you about making it work on real models.

The paper doesn't discuss this. We found that modern LLMs have dramatically different Key vs Value vector magnitudes:

Since quantization error scales with norm squared, K vectors need far more bits than V vectors. Uniform bit allocation is catastrophically wasteful on models with high K/V ratios.

The paper recommends TurboQuantProd for Keys (unbiased inner product) and TurboQuantMSE for Values. Our experiments show MSE for both is better:

Why: TurboQuantProd's QJL residual correction adds variance. Softmax attention amplifies variance more than bias. Low variance (MSE) beats unbiasedness (Prod) in practice.

~5-20% of K channels have 10-100x larger RMS than the median (especially Layer 0). Storing these outlier channels at 8-bit while quantizing the rest at 3-bit achieves:

Mixed precision at 3.6 bits is within 0.1 bits of the paper's 3.5-bit target.

K/V ratio < 10x    → 3-bit uniform works      (GPT-2 family)
K/V ratio 10-60x   → 4.5-5 bit asymmetric     (Phi-2, Qwen-3B)
K/V ratio > 100x   → 5.5+ bit or mixed prec.  (Qwen-1.5B, 7B)
K/V ratio > 1000x  → TurboQuant alone insufficient (Qwen-0.5B)

We implemented actual compressed KV cache storage (not just quantize-dequantize simulation):

49 tests pass. MSE distortion matches paper's theoretical bounds:

Full results: AUTO_BENCHMARK_RESULTS.md

• Not a production tool — Pure Python/NumPy, no GPU kernels, too slow for real inference
• Not the full paper — Missing PolarQuant, per-channel mixed precision tuning
• Not a drop-in replacement — Won't make your local LLM use less VRAM today

• Correct reference implementation — Algorithm verified against paper, 49 tests
• Engineering field notes — What you actually need to know to make KV cache quantization work
• Foundation for GPU kernels — Someone can take these findings and write Triton/CUDA kernels

pip install numpy scipy

# Run tests
pytest turboquant/tests/ -v

# Reproduce paper's distortion table
python -m turboquant.benchmarks.distortion

# Run KV cache benchmark on GPT-2
python turboquant/benchmarks/compressed_test.py gpt2

turboquant/
├── core.py                  — TurboQuantMSE + TurboQuantProd
├── rotation.py              — Random orthogonal rotation
├── scalar_quantizer.py      — Beta distribution optimal quantizer (Lloyd's algorithm)
├── qjl.py                   — Quantized Johnson-Lindenstrauss (1-bit inner product)
├── mixed_precision.py       — Outlier-aware mixed precision quantizer
├── kv_cache.py              — HuggingFace transformers Cache integration
├── compressed_cache.py      — Actual compressed storage (saves memory)
├── utils.py                 — Normalize, random vectors
├── tests/                   — 49 tests
├── benchmarks/              — Distortion, KV cache, mixed precision benchmarks
├── BENCHMARK_RESULTS.md     — Detailed analysis report
├── AUTO_BENCHMARK_RESULTS.md — 8-model progressive benchmark data
└── TURBOQUANT_STLC_SPECIFICATION.md — STLC spec (used to drive implementation)

• Python 3.10+
• NumPy
• SciPy
• pytest (for tests)
• transformers + torch (for KV cache benchmarks)

If you use these findings in your work:

@misc{turboquant-ref-impl,
  title={TurboQuant Reference Implementation: Engineering Insights for KV Cache Compression},
  author={Syn-claude and wuko},
  year={2026},
  url={https://github.com/scos-lab/turboquant}
}

Paper:

@inproceedings{zandieh2026turboquant,
  title={TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate},
  author={Zandieh, Amir and Daliri, Majid and Hadian, Majid and Mirrokni, Vahab},
  booktitle={ICLR},
  year={2026}
}

MIT

Built by scos-lab — Syn-claude + wuko