this is seriously impressive work, full ROCm backend in one PR. and thanks for finding the K/V rotation issue in cpy_k/cpy_v. our Metal path handles rotation differently (graph-side Q rotation + inverse rotation after attention, rather than rotating K/V before cache write), so it may not be the same bug, but i want to verify.

will test locally on M5 Max this afternoon to check for regressions before merging. would also appreciate it if you could run the PPL gate when you get a chance:

Test Results — M5 Max 128GB, Qwen3.5-35B-A3B Q8_0

Build: Clean compile on Metal.

⚠️ turbo3 — CRITICAL REGRESSION

PPL: 181.60 (expected: 6.18). Completely broken on Metal. Do not merge.

Root cause: The cpy_k()/cpy_v() changes add WHT rotation in the graph before cache write. On Metal, kernel_set_rows_turbo3 already performs WHT rotation during quantization (line ~503 in ggml-metal.metal). This causes K/V to be double-rotated, destroying quality.

The fix is correct for ROCm/HIP where SET_ROWS does not rotate, but it needs to be backend-gated:

// Only add graph-side rotation for backends where SET_ROWS doesn't rotate internally
if (k->type == GGML_TYPE_TURBO3_0 || k->type == GGML_TYPE_TURBO4_0) {
    if (/* backend is CUDA/HIP, not Metal */) {
        k_cur = ggml_turbo_wht(ctx, k_cur, 0);
    }
}

Or alternatively, remove the rotation from the Metal SET_ROWS kernel and do it in the graph for all backends (cleaner long-term, but requires Metal kernel changes).

ROCm code review (code-only, can't run on this hardware)

• GPU kernels look structurally correct (dequant, vec_dot, WHT, quantize, GET_ROWS, CPY)
• F16 dequant bridge for FA is a reasonable workaround for HIP FA template issues
• CPU fallback additions are correct
• AMD_SERIALIZE_KERNEL=3 requirement is documented

Recommendation

Do not merge as-is. Breaks Metal turbo3 (PPL 6.18 → 181.6). The rotation needs to be backend-gated, or the architecture needs to agree on where rotation happens (graph vs kernel). Happy to help work through the fix.

Good catch on the double-rotation. Pushed a fix: cpy_k()/cpy_v() rotation is now wrapped in #if !defined(GGML_USE_METAL) so it only applies on CUDA/HIP/CPU where SET_ROWS doesn't rotate internally.

Running PPL gate on our end now (wikitext-2-raw, turbo3 vs FP16 baseline on Qwen3.5-27B Q4_K_M, gfx1100). Will post results when done.

Re: the longer-term architecture question (graph vs kernel rotation) — happy to discuss. For now the preprocessor gate is minimal and correct for both paths. If you prefer moving Metal's kernel rotation into the graph to unify the approach, we can help with that in a follow-up.

PPL Gate Results

Model: Qwen3.5-27B-Claude-4.6-Opus-Reasoning-Distilled Q4_K_M
Hardware: AMD 7900 XTX (gfx1100), ROCm 7.1
Dataset: wikitext-2-raw-v1, ctx=2048
Build: With #if !defined(GGML_USE_METAL) backend gate applied

The +28.8% regression is higher than expected (~1% in your Metal benchmarks). This model is already Q4_K_M weight-quantized, so turbo3 KV quantization stacks on top of existing weight quantization loss. Would be useful to compare with an F16/BF16 weight model to isolate the KV-only contribution.

The Metal regression should be zero with the preprocessor gate since the graph-side rotation is now compiled out for Metal builds.

Multi-model PPL Gate Update

Attempted turbo3 PPL tests on all available GGUFs. Results:

Finding: turbo3 currently only works reliably on Qwen3.5 (head_dim=256). Models with head_dim=128 crash during KV cache initialization. This is likely because the WHT rotation group size (128) matches head_dim exactly, causing edge cases in the graph-side cast/rotation code.

The Qwen3.5 result (PPL +29%) is higher than expected for turbo3, but the model is already Q4_K_M weight-quantized. An F16/BF16 weight model would show the true turbo3-only regression.

Will investigate the head_dim=128 compatibility issue next.

Adaptive WHT + Multi-Head-Dim Support

Pushed 2c3bb08: fixes turbo3 crashes on all head_dim models.

What was broken

Three bugs prevented turbo3 from working on anything other than Qwen3.5 (head_dim=256):

1. Dead dense rotation matrices (128x128 turbo_rotation/turbo_rotation_inv) from the abandoned graph-side matmul approach exhausted the ggml context object pool. Every model crashed at init.

2. CPU dup handler didn't support turbo3 -> F16 cast. The scheduler routes some ops to CPU even with full GPU offload, hitting GGML_ABORT in the dup switch.

3. WHT hardcoded to 128-element groups. Models with head_dim < 256 either crashed (128) or skipped rotation entirely (64, 96).

Fix: Adaptive Group Size

WHT group_size now computed as largest power of 2 dividing head_dim, capped at 128:

Sign arrays truncated to first gs elements. Normalization: 1/sqrt(gs). All backends updated (CUDA/HIP, CPU). Metal unchanged (uses internal SET_ROWS rotation).

Benchmark Results (7900 XTX, ROCm 7.1)

PPL Investigation (head_dim=128)

Mistral-Small PPL with turbo3 is catastrophic (~15000 vs 5.16 F16 baseline). Tested both with and without WHT rotation -- PPL is nearly identical either way, confirming the issue is the 3-bit centroid quantization itself, not the rotation. The centroids may need model-family-specific calibration, or head_dim=128 models may need 4-bit (turbo4) for acceptable quality.

Q8_0 KV cache on the same model gives PPL 5.17 (matching F16), so the model's attention values are perfectly quantizable -- just not at 3 bits.

Next: investigating whether turbo4 (4-bit) resolves the quality gap for head_dim=128 models.

thanks for the thorough work here, especially the adaptive WHT fix and tracking down the dead rotation matrices exhausting the context pool. the multi-model compatibility sweep is exactly the kind of testing that catches real issues.

i'm currently tied up with some other validation runs but will get to testing and review once my bench frees up. the head_dim=128 PPL being catastrophic at turbo3 lines up with what buun's been seeing on the CUDA side too — turbo4 path for head_dim=128 is likely the right direction.

will follow up once i can give this a proper look.

Root Cause: PolarQuant Reconstruction Model vs Uniform Quantization

Dug deeper into the head_dim=128 PPL catastrophe. It's NOT a bit depth problem:

turbo4 has MORE bits than q4_0 but 1000x worse PPL. The issue is PolarQuant's reconstruction model: value = norm * centroid[idx]. This assumes all elements in a block share the same scale (the block L2 norm) and the centroids are at fixed positions. Uniform quantization (q4_0) adapts its scale and zero point per block.

Tried recalibrating centroids for the actual unit-sphere distribution (d=32) -- the current centroids are tuned for N(0,1/128) which has 2x narrower range than the per-block-normalized distribution. Wider centroids didn't help (PPL got worse, possibly due to CPU/GPU centroid mismatch on layer 0).

The fundamental issue: Mistral-Small's attention heads have non-uniform value distributions where the PolarQuant assumption (values are roughly equi-distributed around the block centroid) breaks down. Some dimensions carry disproportionate signal.

Possible fixes:

1. Hybrid quantization: Use turbo3/turbo4 for Qwen-family models (where it works), fall back to q4_0/q4_1 for others
2. Adaptive reconstruction: Replace norm * centroid[idx] with scale * centroid[idx] + offset (adds 2 bytes per block but adapts like q4)
3. Per-head calibration: Different centroid sets per attention head

Also pushed turbo4 CUDA/HIP implementation (works on Qwen3.5-27B, PPL 2.28) and corrected centroids analysis to the branch. Holding push until you review.

apollosenvy
commented
1 minute ago
Root Cause: PolarQuant Reconstruction Model vs Uniform Quantization
Dug deeper into the head_dim=128 PPL catastrophe. It's NOT a bit depth problem:

KV Type Bits/val Mistral-Small PPL Reconstruction Model
F16 16 5.16 exact
q8_0 8 5.17 scale * idx
q5_0 5 5.18 scale * idx + offset
q4_0 4 5.26 scale * idx
q4_1 4.5 5.21 scale * idx + min
turbo4 4.25 5574 norm * centroid[idx]
turbo3 3.5 15518 norm * centroid[idx]
turbo4 has MORE bits than q4_0 but 1000x worse PPL. The issue is PolarQuant's reconstruction model: value = norm * centroid[idx]. This assumes all elements in a block share the same scale (the block L2 norm) and the centroids are at fixed positions. Uniform quantization (q4_0) adapts its scale and zero point per block.

Tried recalibrating centroids for the actual unit-sphere distribution (d=32) -- the current centroids are tuned for N(0,1/128) which has 2x narrower range than the per-block-normalized distribution. Wider centroids didn't help (PPL got worse, possibly due to CPU/GPU centroid mismatch on layer 0).

The fundamental issue: Mistral-Small's attention heads have non-uniform value distributions where the PolarQuant assumption (values are roughly equi-distributed around the block centroid) breaks down. Some dimensions carry disproportionate signal.

Possible fixes:

Hybrid quantization: Use turbo3/turbo4 for Qwen-family models (where it works), fall back to q4_0/q4_1 for others
Adaptive reconstruction: Replace norm * centroid[idx] with scale * centroid[idx] + offset (adds 2 bytes per block but adapts like q4)
Per-head calibration: Different centroid sets per attention head
Also pushed turbo4 CUDA/HIP implementation (works on Qwen3.5-27B, PPL 2.28) and corrected centroids analysis to the branch. Holding push until you review.

Two things here:

On the adaptive WHT fix (2c3bb08) — really solid work. The dead rotation matrices exhausting the context pool is something I’d hit as well but hadn’t pushed a fix for on CUDA/HIP yet. CPU dup handler and adaptive group sizing are nice catches. Good to see hd64/96/128 actually running.

On the PolarQuant reconstruction analysis — I’d be careful about calling it a fundamental issue just yet. There are still pipeline-level bugs that can produce exactly the kind of catastrophic PPL you’re seeing.

Two data points from my side:

1. On Qwen3.5-27B, your turbo3 is +28.8% vs q8_0. My runs on the same model family are ~+1%. That’s a huge gap on a model where turbo is known to behave well, which suggests something upstream of reconstruction is off.

2. buun is seeing turbo4 at ~+1.2% vs q8_0 on head_dim=128 models on CUDA. So the hd128 gap seems real but small (1–3%), not catastrophic.

I just finished bringing turbo4 back up on Metal and ended up finding several bugs that massively distorted PPL (one SET_ROWS packing issue alone took it from ~679 down to ~6.1). So I’d focus on ruling out pipeline issues first.

A few specific things I’d check:

• SET_ROWS writing the correct turbo4 block layout (shared turbo3/turbo4 paths can easily corrupt data)
• CPU/GPU centroid mismatch (especially if layer 0 falls back to CPU)
• turbo4 pre-rotation guard (buun flagged a missing check for TURBO4_0)

Also, your turbo4 CUDA/HIP result on Qwen (PPL ~2.28) is actually a strong signal that the reconstruction model itself is sound when the pipeline is correct. The interesting question is what differs between that path and the Mistral one beyond head_dim.

Happy to take a look once you’ve had a chance to cross-check — no rush.

Please sync to TOT and I'll review and test.

Synced to TOT — HIP port on rocm-turbo3-v2

Rebased onto your latest feature/turboquant-kv-cache (172fc85). Clean single commit: HIP/ROCm porting for your warp-cooperative turbo3 kernel + turbo2/turbo3 FA templates.

What was needed for HIP

1. Vendor header (hip.h): Added cudaMemcpyToSymbol/FromSymbol mappings. Fixed __shfl_sync, __shfl_xor_sync, __shfl_up_sync, __shfl_down_sync to support 3-arg calls (CUDA defaults width to warpSize, old HIP macros required 4 args). Added __ballot_sync -> __ballot with uint32_t cast.

2. HIP CMakeLists: Added turbo3 and turbo2 FA template instances (turbo3-turbo3, turbo3-q8_0, q8_0-turbo3, turbo2-turbo2, turbo2-q8_0, q8_0-turbo2).

PPL Confirmed

You were right about the pipeline bugs. With the CPU quantize stub fix (53f1298):

PolarQuant IS sound. The catastrophic PPL was 100% the CPU quantize stub zeroing qs/signs on layer 0. My earlier "fundamental limitation" analysis was wrong — thanks for pushing back on that.

Branch is rocm-turbo3-v2 on fork. Ready for review.

TurboQuant+ Temporal Decay -- Implemented

Pushed to rocm-turbo3-v2: position-aware KV cache quality degradation. Old tokens get their signs field zeroed in-place, reducing turbo3 from 8-level to 4-level centroid reconstruction.

How it works

Three dynamic tiers with proportional boundaries:

• Hot (newest 15%): full quality
• Warm (middle 35%): base turbo3 quality
• Cold (oldest 50%): signs zeroed, 4-level reconstruction

Boundaries shift proportionally as context grows. Demotion is a single ggml_turbo_decay op appended to the graph after SET_ROWS. One kernel launch per layer per boundary advance.

Configuration

TURBO_DECAY=balanced          # 15/35/50 (default)
TURBO_DECAY=conservative      # 25/40/35
TURBO_DECAY=aggressive        # 5/25/70
TURBO_DECAY=10,30,60          # custom percentages

Benchmark (Qwen3.5-27B Q4_K_M, 7900 XTX)

PPL (ctx=2048, 4 chunks):

Speed: Zero measurable overhead (within noise).

The PPL hit at 2K is expected -- at short context, the cold tier covers tokens the model still actively attends to. At longer contexts (16K+), the cold tier covers much older tokens with naturally lower attention weight, so the quality impact should decrease. This is the design tradeoff: sacrifice some short-context quality for better long-context compression efficiency.

Implementation (8 commits)

1. Design spec + reviewer feedback fixes
2. Implementation plan
3. Config struct + env var parsing
4. GGML_OP_TURBO_DECAY op
5. CUDA/HIP/CPU decay kernels
6. Wire into cpy_k/cpy_v graph
7. PPL validation
8. Push

Phase 2 (hot tier q8_0 promotion) and Phase 3 (attention-guided decay) designed but deferred.

Tested on M5 Max 128GB (Metal) and M2 Pro 32GB (Metal). Thanks for the work here, some notes below.

What passes (head_dim=128 models)

PPL and speed are clean for standard models:

Build passes on Metal with no warnings.

Issue 1: TURBO_DECAY crashes on Metal

The temporal decay op has CUDA/HIP/CPU kernels but no Metal kernel. When TURBO_DECAY=balanced is set with a GPU-offloaded model, it aborts:

pre-allocated tensor (cache_k_l3 (view) (view)) in a buffer (MTL0) that cannot run the operation (TURBO_DECAY)

Repro:

TURBO_DECAY=balanced ./build/bin/llama-cli \
  -m model.gguf -ngl 99 -fa on \
  -ctk turbo3 -ctv turbo3 \
  -p "Write a 500 word essay about the history of computing" -n 500

This is a crash, not a graceful fallback. The decay op needs either a Metal kernel or a supports_op guard that prevents dispatch to the Metal backend. As-is, any Metal user who sets the env var will hit an abort.

Issue 2: Boundary V modes 5/6/7 removed

The PR removes layer-adaptive modes 5, 6, and 7 (Boundary V) from the constructor. These were shipped in PR #30 and are part of our current feature set. TURBO_LAYER_ADAPTIVE=5 no longer produces a "Boundary V" log line, confirming the code path is gone.

Suggestion

Consider splitting this into separate PRs:

1. ROCm/HIP kernel support (the core contribution, looks solid)
2. Temporal decay (experimental feature, needs Metal kernel + more validation)
3. Non-128 head_dim q8_0 fallback (architectural decision worth discussing separately)

Happy to re-review individual pieces. The ROCm work is great and would be straightforward to land on its own.

Split per your suggestion:

• PR feat: HIP/ROCm support for turbo3/turbo2 (7900 XTX) #31 — ROCm/HIP port only. Single commit, minimal. Rebased on your latest (includes Boundary V). Also excludes D>=576 fattn-tile instances that exceed HIP's 64KB local memory limit.

• Temporal decay — Will open as separate PR once Metal kernel or supports_op guard is added. The decay op shouldn't dispatch to backends that don't implement it.

• head_dim fallback — Your commit 4c4511c14 already handles this. No separate PR needed from our side.

Re: modes 5/6/7 — we didn't remove them. Our branch was based on a commit before PR #30. The new PR #31 is rebased on your latest which includes Boundary V, so modes 5/6/7 are intact.