Quickly did a couple of benchmarks on strix halo (8060s). PP basically unchanged (maybe a little slower), TG quite a bit faster:

PP performance still exhibits the same issue as described in #18725.

Thanks for testing! Those TG numbers are great — 22% on Coder-Next, 21% on 35B-A3B, 18% on 122B-A10B.

The PP issue from #18725 makes sense — this shader only affects the deltanet recurrence layers, and PP throughput is still bottlenecked by the autoregressive token loop in the current kernel. A chunked parallel kernel (Phase 2) would fix that, but it's a much bigger piece of work. Hope to work on it tomorrow.

Updated the PR description with your benchmark numbers. Really helpful to have data from a discrete 8060S alongside the integrated 890M results.

Same 8060S setup. In my testing, although the current PR shows performance improvement compared to Master PR, Qwen3.5-35B0A3B cannot reach that speed (50 token/s), quite strange :(

command:

/home/mark/llama.cpp/llama.cpp-vulkan-gated-delta-net/build/bin/llama-bench 
-m /home/mark/Models/Q8/Qwen3.5-35B-A3B-Q8_0/Qwen3.5-35B-A3B-UD-Q8_K_XL.gguf 
--repetitions 5 --output md --mmap 0 --delay 0 
--n-prompt 512 
--n-gen 128 
--n-depth 0 
--batch-size 512 
--ubatch-size 512 
--cache-type-k f16 --cache-type-v f16 --threads 16 
--n-gpu-layers 99 
--n-cpu-moe 0 --flash-attn 1 --direct-io 1

ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon Graphics (RADV GFX1151) (radv) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
| model                          |       size |     params | backend    | ngl | n_batch | fa | mmap | dio |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ------: | -: | ---: | --: | --------------: | -------------------: |
| qwen35moe 35B.A3B Q8_0         |  45.33 GiB |    34.66 B | Vulkan     |  99 |     512 |  1 |    0 |   1 |           pp512 |        567.96 ± 2.42 |
| qwen35moe 35B.A3B Q8_0         |  45.33 GiB |    34.66 B | Vulkan     |  99 |     512 |  1 |    0 |   1 |           tg128 |         25.87 ± 0.02 |

build: unknown (0)

@IIIIIllllIIIIIlllll The difference in performance is mostly likely caused by the quantization: UD-Q8_K_XL vs Q8_0

Additionally my Minisforum MS-S1 has a very generous power budget (I think the highest of all available boards with 160W short term and 130W long term).

For what it's worth, I'm seeing some crazy numbers here in terms of performance on my R9700s:

These are anecdotal numbers from ad-hoc testing, but they're pretty consistent across prompts - just trying to give an indication of why my jaw dropped.

The single-GPU boost is great, but I had to check the dual-GPU result with Qwen3-Coder-Next a bunch of times; it's genuinely usable in an agentic context now. With Cline at the helm, it never drops below 50t/s even as the context fills up past 180k, and both GPUs have gone from ~63W constant to 70W with spikes up to 180W (still nowhere near the 300W max, but that's on the tensor parallel PR when it's fixed for Vulkan...?).

@ProgenyAlpha - I feel like some of us may owe you a beer or five. I might actually be getting some value out of these cards now!

Thanks for the review — all 7 items addressed. Pipeline declarations moved to a [3][2] array, shader now uses A_TYPE/D_TYPE/FLOAT_TYPE conventions, scale moved to push constants, dead code removed. 16/16 tests passing. Had a couple of these on my list already but your feedback saved me time and caught things I'd have missed.

Pushed the review fixes — all 7 items from @0cc4m addressed. Also ran a full model bench on 890M (integrated):

Qwen3-Coder-Next REAM Q4_K_M (60B)

• PP-512: 157.67 t/s
• TG-128: 20.90 t/s

@digitalscream those dual-GPU numbers are wild — 42% improvement and usable agentic speeds at 180k context is exactly the kind of thing that makes this worth doing. I have a tip jar but I'm doing it for the community so not necessary. I love the feedback from different platforms so feel free to keep testing changes.

@lemmi thanks again for the Strix Halo data. PP issue is upstream of this shader (SSM_CONV workgroup scaling, #18725) so it won't change here.

@IIIIIllllIIIIIlllll lemmi nailed it — UD-Q8_K_XL uses importance-weighted mixed precision which generally outperforms uniform Q8_0 across backends.

Great work @ProgenyAlpha

@ProgenyAlpha thanks for this, 24% to 42% on R9700 is godsent !

if i may , you can have a look at the following for you next adventure
#19890 (reply in thread)
nocompute flag giving 16% improvement, which may indicate scope for improvement on graphics-queue/compute-queue logic for amd vulkan

Note that for the chunked version you'll need to have the changes from #20340

Pushed f16 mixed-precision state — stores the 128-element state array in float16_t, keeps all arithmetic in float32. No precision loss (16/16 tests), lower register pressure.

890M benchmarks (Qwen3-Coder-Next REAM Q4_K_M):

f16 pipeline auto-selects when the device supports shaderFloat16, falls back to f32 otherwise.

@ggerganov saw your note about #20340 — will rebase onto that once it lands. The rq1 → neqk1 broadcast change needs the CPU side too. I'll work on that and have first pass chunked shaders ready and waiting to be tested.

@zedbytes interesting find on the nocompute flag. That's a driver-level queue scheduling optimization — separate from shader work but worth investigating for AMD Vulkan in general.

@digitalscream @lemmi if you get a chance, would be great to see updated numbers with the latest push. The f16 state change gave +8.3% PP on my 890M — curious if the improvement scales differently on discrete cards with more CUs.

@ProgenyAlpha - OK, with the latest push:

So...+7% PP and -0.8% TG for single GPU, negligible changes for dual GPU.

Please split out the chunked support into a follow-up instead of expanding the scope of this PR. Even float16 might be further than step 1 should be.

Wasn't able to see any meaningful changes on 23fbfcb for PP on my end.

Same 8060S setup. In my testing, although the current PR shows performance improvement compared to Master PR, Qwen3.5-35B0A3B cannot reach that speed (50 token/s), quite strange :(

command:

/home/mark/llama.cpp/llama.cpp-vulkan-gated-delta-net/build/bin/llama-bench 
-m /home/mark/Models/Q8/Qwen3.5-35B-A3B-Q8_0/Qwen3.5-35B-A3B-UD-Q8_K_XL.gguf 
--repetitions 5 --output md --mmap 0 --delay 0 
--n-prompt 512 
--n-gen 128 
--n-depth 0 
--batch-size 512 
--ubatch-size 512 
--cache-type-k f16 --cache-type-v f16 --threads 16 
--n-gpu-layers 99 
--n-cpu-moe 0 --flash-attn 1 --direct-io 1

ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon Graphics (RADV GFX1151) (radv) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
| model                          |       size |     params | backend    | ngl | n_batch | fa | mmap | dio |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ------: | -: | ---: | --: | --------------: | -------------------: |
| qwen35moe 35B.A3B Q8_0         |  45.33 GiB |    34.66 B | Vulkan     |  99 |     512 |  1 |    0 |   1 |           pp512 |        567.96 ± 2.42 |
| qwen35moe 35B.A3B Q8_0         |  45.33 GiB |    34.66 B | Vulkan     |  99 |     512 |  1 |    0 |   1 |           tg128 |         25.87 ± 0.02 |

build: unknown (0)

Sorry to bother everyone.
On my strix halo, this commit(a57f2ac) did not result in any performance improvement. @ProgenyAlpha

@0cc4m Done — stripped this PR back to autoregressive-only. Removed chunked shaders, f16 state, and all the infrastructure that came with them. This is now just the base GATED_DELTA_NET op support + review fixes.

Split into two follow-up PRs:

• vulkan: f16 mixed-precision state for GATED_DELTA_NET #20376 — f16 mixed-precision state (depends on this PR)
• vulkan: chunked parallel kernel for GATED_DELTA_NET #20377 — chunked parallel kernels (depends on this PR + llama : enable chunked fused GDN path #20340)

13/13 backend-ops tests passing. Benchmarks unchanged from before since the autoregressive path is the same.

@lemmi @IIIIIllllIIIIIlllll Thanks for retesting. The f16 state improvement looks hardware-dependent — +8.3% PP on my 890M, +7% on @digitalscream's R9700S, but flat on Strix Halo. Might be related to how the register file handles f16 on different RDNA3.5 configs. Either way, f16 is now split out into its own PR (#20376) so it won't hold up the base support.

This PR is stripped back to autoregressive-only per @0cc4m's review — should be ready for another look.

@lemmi heads up — took a look at the PP/ubatch scaling issue from #18725 and put up a fix in #20379. SSM_CONV workgroup dispatch was the bottleneck. +28% at ub2048 on my 890M, degradation cliff gone. Would love your numbers on the 8060S.

Mark the PR as ready when you are done, please.

Rebased on master and fixed the Q/K broadcast to use the interleaved layout from #20340. 13/13 tests passing.

No, that was not a correct rebase.

Saw it immediately after, rebase picked up duplicate commits from master, didn't catch it before pushing. Cleaned up with cherry-pick, should be 6 commits on master now.

Thanks for correct rebase!

EDIT: Checked.
Although I haven't done exhaustive tests, at least there's no obvious failures (Qwen3.5 models seem working correctly).

I'm gonna skip waiting for the CI to unblock other PRs. I checked locally that it works.