feat(aprender-serve): moe_ffn_forward_layer_cuda — M-GPU-MOE-1.1.1#1469
Merged
Conversation
… GPU helper
Per qwen3-moe-forward-gpu-v1 v1.1.0 option D — first concrete GPU
compute for the contract. Mirrors the M32c.2.2.* CPU staging where
per-expert byte slicer + per-expert SwiGLU helper landed BEFORE the
full moe_ffn_forward_layer integration.
What this PR ships
==================
crates/aprender-serve/src/gguf/cuda/expert_swiglu_cuda.rs NEW
pub(crate) fn expert_swiglu_cuda(
executor: &mut crate::cuda::CudaExecutor,
gate_bytes: &[u8], // Q4_K, [intermediate, hidden_dim]
up_bytes: &[u8], // Q4_K, [intermediate, hidden_dim]
down_bytes: &[u8], // Q6_K, [hidden_dim, intermediate]
hidden: &[f32],
hidden_dim: usize,
intermediate: usize,
) -> Result<Vec<f32>>
Body (mirrors CPU sibling moe_ffn_forward_layer per-expert loop):
1. gate_out = executor.q4k_matvec(gate_bytes, hidden, ..., m=intermediate, k=hidden_dim)
2. up_out = executor.q4k_matvec(up_bytes, hidden, ..., m=intermediate, k=hidden_dim)
3. ffn_inner[i] = silu(gate_out[i]) * up_out[i] (CPU element-wise)
4. expert_out = executor.q6k_gemv(down_bytes, ffn_inner, ..., n=hidden_dim, k=intermediate)
+ 2 unit tests (signature drift gate + InvalidShape rejection)
Why "naive per-expert dispatch" is the M-GPU-MOE-1.1.0 baseline
===============================================================
The fused dequant+matmul + sparse expert batching path is M-GPU-MOE-3.
The contract (qwen3-moe-forward-gpu-v1 implementation_stages) stages
correctness before performance:
M-GPU-MOE-1.1.0 (this) Per-expert via existing primitives SHIPPED ✓
- silu via CPU elementwise (small)
- element-wise gate*up via CPU
- matmuls via existing q4k/q6k GPU kernels
M-GPU-MOE-1.1.1 Full forward integration in
OwnedQuantizedModelCuda::forward_qwen3_moe_cuda
(router + per-token loop + per-expert
dispatch + weighted aggregation) PENDING
M-GPU-MOE-1.2 Cosine-vs-CPU parity gate ≥0.99 PENDING
(FALSIFY-QW3-MOE-GPU-PARITY-001)
M-GPU-MOE-2 wgpu fallback PENDING
M-GPU-MOE-3 Fused kernels + sparse batching PENDING
Verification
============
$ cargo check -p aprender-serve --features cuda
✓ Compiles
$ cargo test -p aprender-serve --features cuda --lib expert_swiglu_cuda
test ... ok. 2 passed; 0 failed
$ pv validate contracts/qwen3-moe-forward-gpu-v1.yaml
0 error(s), 0 warning(s)
Refs PR #1462 squash 4495407 (v1.1.0 option D amendment)
Refs PR #1464 (M-GPU-MOE-1.0-redo stub on OwnedQuantizedModelCuda)
Refs M32c.2.2.0 + M32c.2.2.1 (CPU per-expert sub-milestone precedent)
Refs claude-code-parity-apr POC M49 / R10 (P0 elevation + risk row)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…ngle-layer GPU MoE FFN Mirrors CPU sibling moe_ffn_forward_layer (qwen3_moe_load.rs:363) step-for-step: F32 router on CPU, softmax + top-k + renormalize on CPU, per-expert SwiGLU dispatched through expert_swiglu_cuda (M-GPU-MOE-1.1.0), weighted aggregation on CPU. Per qwen3-moe-forward-gpu-v1 v1.1.0 option D: GPU MoE forward path on OwnedQuantizedModelCuda, reusing existing CudaExecutor primitives (q4k_matvec for gate/up, q6k_gemv for down) per expert. Composes the M-GPU-MOE-1.1.0 helper into the layer-level structure that the next stage M-GPU-MOE-1.1.2 (forward_qwen3_moe_cuda full integration) will call once per token per layer. Verification ============ $ cargo check -p aprender-serve --features cuda ✓ Compiles Refs PR #1465 (expert_swiglu_cuda M-GPU-MOE-1.1.0) Refs M32c.2.2.2.0 (CPU sibling moe_ffn_forward_layer precedent) Refs claude-code-parity-apr POC M49 / R10 (P0) Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
9d2dc98 to
c03fe44
Compare
This was referenced May 4, 2026
noahgift
added a commit
that referenced
this pull request
May 4, 2026
…-MOE-1.1.2
Replaces the M-GPU-MOE-1.0-redo stub body with the full forward
integration. forward_qwen3_moe_cuda now mirrors the CPU sibling
OwnedQuantizedModel::forward_qwen3_moe (forward_qwen3_moe.rs)
line-for-line, with one difference: the per-layer FFN section
routes through moe_ffn_forward_layer_cuda which dispatches per-
expert matmuls to self.executor (CudaExecutor) via the
expert_swiglu_cuda helper.
Per qwen3-moe-forward-gpu-v1 v1.1.0 option D — extends the existing
OwnedQuantizedModelCuda CPU-attention + CUDA-FFN pattern (forward_cuda
in cuda.rs:18). Attention path stays on CPU; only FFN matmuls go to
GPU. M-GPU-MOE-3 fuses dispatch into a single sparse-expert kernel
for ~5× throughput.
Signature changes
=================
- &self → &mut self (executor needs mutable for kernel cache)
- _data → data (passed to moe_ffn_forward_layer_cuda for
expert_byte_slice)
Forward body structure (mirrors CPU sibling step-for-step):
1. Embed (CPU) — self.model.embed
2. Per-layer:
2a. Attention norm (CPU) — ops::rms_norm
2b. QKV projection (CPU) — self.model.qkv_matmul
2c. Per-head Q/K RMSNorm + RoPE (M32d Step 5/5b) — ops::apply_per_head_rms_norm
2d. Causal attention + output proj (CPU) — self.model.causal_attention
2e. Residual — element-wise CPU
2f. Pre-FFN norm (CPU) — ops::rms_norm
2g. **MoE FFN on GPU** — moe_ffn_forward_layer_cuda
→ expert_swiglu_cuda
→ self.executor.q4k_matvec
.q6k_gemv
2h. Residual — element-wise CPU
3. Final norm (CPU)
4. LM head — last token (CPU)
Implementation stages updated
=============================
M-GPU-MOE-0 Contract scaffold v1.0.0 SHIPPED ✓
M-GPU-MOE-0.5 v1.1.0 option D amendment SHIPPED ✓
M-GPU-MOE-1.0-redo Stub on OwnedQuantizedModelCuda SHIPPED ✓ (#1464)
M-GPU-MOE-1.1.0 expert_swiglu_cuda helper SHIPPED ✓ (via #1469 squash)
M-GPU-MOE-1.1.1 moe_ffn_forward_layer_cuda SHIPPED ✓ (#1469)
M-GPU-MOE-1.1.2 forward_qwen3_moe_cuda full integ SHIPPED ✓ (THIS PR)
M-GPU-MOE-1.2 Cosine-vs-CPU parity gate ≥0.99 PENDING
(FALSIFY-QW3-MOE-GPU-PARITY-001)
M-GPU-MOE-2 wgpu fallback PENDING
M-GPU-MOE-3 Throughput ≥150 + VRAM ≤ 95% PENDING
Verification
============
$ cargo check -p aprender-serve --features cuda
✓ Compiles
$ cargo test -p aprender-serve --features cuda --lib forward_qwen3_moe_cuda
test ... ok. 1 passed
Refs PR #1469 squash 77b9f0d (helpers landed)
Refs PR #1462 squash 4495407 (v1.1.0 option D amendment)
Refs claude-code-parity-apr POC M49 / R10 (P0 elevation)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Merged
4 tasks
noahgift
added a commit
that referenced
this pull request
May 4, 2026
…-MOE-1.1.2
Replaces the M-GPU-MOE-1.0-redo stub body with the full forward
integration. forward_qwen3_moe_cuda now mirrors the CPU sibling
OwnedQuantizedModel::forward_qwen3_moe (forward_qwen3_moe.rs)
line-for-line, with one difference: the per-layer FFN section
routes through moe_ffn_forward_layer_cuda which dispatches per-
expert matmuls to self.executor (CudaExecutor) via the
expert_swiglu_cuda helper.
Per qwen3-moe-forward-gpu-v1 v1.1.0 option D — extends the existing
OwnedQuantizedModelCuda CPU-attention + CUDA-FFN pattern (forward_cuda
in cuda.rs:18). Attention path stays on CPU; only FFN matmuls go to
GPU. M-GPU-MOE-3 fuses dispatch into a single sparse-expert kernel
for ~5× throughput.
Signature changes
=================
- &self → &mut self (executor needs mutable for kernel cache)
- _data → data (passed to moe_ffn_forward_layer_cuda for
expert_byte_slice)
Forward body structure (mirrors CPU sibling step-for-step):
1. Embed (CPU) — self.model.embed
2. Per-layer:
2a. Attention norm (CPU) — ops::rms_norm
2b. QKV projection (CPU) — self.model.qkv_matmul
2c. Per-head Q/K RMSNorm + RoPE (M32d Step 5/5b) — ops::apply_per_head_rms_norm
2d. Causal attention + output proj (CPU) — self.model.causal_attention
2e. Residual — element-wise CPU
2f. Pre-FFN norm (CPU) — ops::rms_norm
2g. **MoE FFN on GPU** — moe_ffn_forward_layer_cuda
→ expert_swiglu_cuda
→ self.executor.q4k_matvec
.q6k_gemv
2h. Residual — element-wise CPU
3. Final norm (CPU)
4. LM head — last token (CPU)
Implementation stages updated
=============================
M-GPU-MOE-0 Contract scaffold v1.0.0 SHIPPED ✓
M-GPU-MOE-0.5 v1.1.0 option D amendment SHIPPED ✓
M-GPU-MOE-1.0-redo Stub on OwnedQuantizedModelCuda SHIPPED ✓ (#1464)
M-GPU-MOE-1.1.0 expert_swiglu_cuda helper SHIPPED ✓ (via #1469 squash)
M-GPU-MOE-1.1.1 moe_ffn_forward_layer_cuda SHIPPED ✓ (#1469)
M-GPU-MOE-1.1.2 forward_qwen3_moe_cuda full integ SHIPPED ✓ (THIS PR)
M-GPU-MOE-1.2 Cosine-vs-CPU parity gate ≥0.99 PENDING
(FALSIFY-QW3-MOE-GPU-PARITY-001)
M-GPU-MOE-2 wgpu fallback PENDING
M-GPU-MOE-3 Throughput ≥150 + VRAM ≤ 95% PENDING
Verification
============
$ cargo check -p aprender-serve --features cuda
✓ Compiles
$ cargo test -p aprender-serve --features cuda --lib forward_qwen3_moe_cuda
test ... ok. 1 passed
Refs PR #1469 squash 77b9f0d (helpers landed)
Refs PR #1462 squash 4495407 (v1.1.0 option D amendment)
Refs claude-code-parity-apr POC M49 / R10 (P0 elevation)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
noahgift
added a commit
that referenced
this pull request
May 4, 2026
…-MOE-1.1.2 (#1477) Replaces the M-GPU-MOE-1.0-redo stub body with the full forward integration. forward_qwen3_moe_cuda now mirrors the CPU sibling OwnedQuantizedModel::forward_qwen3_moe (forward_qwen3_moe.rs) line-for-line, with one difference: the per-layer FFN section routes through moe_ffn_forward_layer_cuda which dispatches per- expert matmuls to self.executor (CudaExecutor) via the expert_swiglu_cuda helper. Per qwen3-moe-forward-gpu-v1 v1.1.0 option D — extends the existing OwnedQuantizedModelCuda CPU-attention + CUDA-FFN pattern (forward_cuda in cuda.rs:18). Attention path stays on CPU; only FFN matmuls go to GPU. M-GPU-MOE-3 fuses dispatch into a single sparse-expert kernel for ~5× throughput. Signature changes ================= - &self → &mut self (executor needs mutable for kernel cache) - _data → data (passed to moe_ffn_forward_layer_cuda for expert_byte_slice) Forward body structure (mirrors CPU sibling step-for-step): 1. Embed (CPU) — self.model.embed 2. Per-layer: 2a. Attention norm (CPU) — ops::rms_norm 2b. QKV projection (CPU) — self.model.qkv_matmul 2c. Per-head Q/K RMSNorm + RoPE (M32d Step 5/5b) — ops::apply_per_head_rms_norm 2d. Causal attention + output proj (CPU) — self.model.causal_attention 2e. Residual — element-wise CPU 2f. Pre-FFN norm (CPU) — ops::rms_norm 2g. **MoE FFN on GPU** — moe_ffn_forward_layer_cuda → expert_swiglu_cuda → self.executor.q4k_matvec .q6k_gemv 2h. Residual — element-wise CPU 3. Final norm (CPU) 4. LM head — last token (CPU) Implementation stages updated ============================= M-GPU-MOE-0 Contract scaffold v1.0.0 SHIPPED ✓ M-GPU-MOE-0.5 v1.1.0 option D amendment SHIPPED ✓ M-GPU-MOE-1.0-redo Stub on OwnedQuantizedModelCuda SHIPPED ✓ (#1464) M-GPU-MOE-1.1.0 expert_swiglu_cuda helper SHIPPED ✓ (via #1469 squash) M-GPU-MOE-1.1.1 moe_ffn_forward_layer_cuda SHIPPED ✓ (#1469) M-GPU-MOE-1.1.2 forward_qwen3_moe_cuda full integ SHIPPED ✓ (THIS PR) M-GPU-MOE-1.2 Cosine-vs-CPU parity gate ≥0.99 PENDING (FALSIFY-QW3-MOE-GPU-PARITY-001) M-GPU-MOE-2 wgpu fallback PENDING M-GPU-MOE-3 Throughput ≥150 + VRAM ≤ 95% PENDING Verification ============ $ cargo check -p aprender-serve --features cuda ✓ Compiles $ cargo test -p aprender-serve --features cuda --lib forward_qwen3_moe_cuda test ... ok. 1 passed Refs PR #1469 squash 77b9f0d (helpers landed) Refs PR #1462 squash 4495407 (v1.1.0 option D amendment) Refs claude-code-parity-apr POC M49 / R10 (P0 elevation) Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Architecture (option D, naive per-expert dispatch)
```
gate_bytes/up_bytes/down_bytes = expert_byte_slice(...)
expert_out = expert_swiglu_cuda(executor, ..., hidden, ...) (GPU)
```
Test plan
Stacked on
PR #1465 (M-GPU-MOE-1.1.0 expert_swiglu_cuda helper)
🤖 Generated with Claude Code