docs(ship-007): §40 — SHIP-007 root cause LOCALIZED to GPU path (CPU is correct, MODEL-1 shippable today)#1111
Merged
Conversation
…; CPU is correct Live evidence on canonical 7B teacher (RTX 4090): apr run --no-gpu → "2 + 2 equals" (CORRECT) apr run (default) → "ampiezza = 1" (WRONG) Same model, same prompt, same greedy sampling. The bug is in the GPU dispatch chain. Falsification matrix executed live (all FAIL, bug persists): - APR_SKIP_FP8_WARMUP=1 → "ampiezza = 1" - REALIZR_NO_FP8_CACHE=1 → "ampiezza = 1" - SKIP_CUDA_GRAPH=1 → "ampiezza = 1" - FP8_PREFILL=0 FP8_DECODE=0 → "ampiezza = 1" So the bug is NOT in: - FP8 JIT warming - FP8 weight cache - CUDA graph capture - FP8 prefill or decode kernels Remaining bug surface on the GPU path: 1. Q4K → F32 dequantization (PMAT-333 log: 28GB F32 dequantized). CPU path uses Q4K-fused directly; GPU path dequantizes first. 2. Weight layout transpose for GPU upload (LAYOUT-001/002 risk). 3. wgpu vs CUDA dispatch interplay. KEY SHIPPING IMPLICATION: MODEL-1 is shippable TODAY via CPU path. Two policy options (§40.6): Option A (immediate ship, GPU disabled by default): - Default `apr run` to `--no-gpu` until SHIP-007 fix lands - 5 MODEL-1 PARTIALs auto-discharge → coverage 20+28 (42% DISCHARGED) - MODEL-1 SHIPS TODAY Option B (block ship until GPU fix): - Hold MODEL-1 ship until §40.5 H1/H2/H3 → root-cause fix - Estimated 1-3 days - MODEL-1 ships in 1-3 days with full GPU support Five-whys (§40.5): 1. Why isn't `apr run` correct on GPU? "ampiezza = 1" instead of "2 + 2 equals". 2. Why? GPU FP8/CUDA dispatch path corrupts the forward computation. 3. Why does CPU path work? CPU uses Q4K-fused SIMD, preserves precision. 4. Why was this not localized earlier? §17/§23/§27 chain bisected forward_traced's F32-only path — yet another path that doesn't exercise GPU dispatch. 5. What's the fix? §40.5 H1/H2/H3 falsifiers localize WITHIN GPU path. Spec v2.81.0 → v2.85.0. Coverage scoreboard 15+33 pending Option A/B decision; A flips to 20+28 immediately. This is the highest-leverage spec amendment in the §17→§40 chain because it identifies a CPU-path workaround that ships MODEL-1 TODAY, while leaving the GPU bug as a known-issue follow-up. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
4 tasks
noahgift
added a commit
that referenced
this pull request
May 13, 2026
…-SHIP-CPU-004 (#1114) Adds a "Known Issue (SHIP-007)" callout to README Quick Start that recommends `--no-gpu` for the canonical 7B Q4K teacher (paiml/qwen2.5-coder-7b-apache-q4k-v1). Per SPEC-SHIP-TWO-001 §40 (PR #1111) + apr-cli-model-1-ship-via-cpu-v1.yaml (PR #1113), the GPU dispatch path on this specific model currently produces gibberish output ("ampiezza = 1") while the CPU path produces correct mathematical reasoning ("2 + 2 equals"). This satisfies FALSIFY-MODEL-1-SHIP-CPU-004 (user-facing docs warn about GPU known-issue): $ grep -rE -- '(--no-gpu|SHIP-007|GPU known issue|use --no-gpu)' \ README.md docs/ apr-cookbook/ README.md:> **Known Issue (SHIP-007)**: For the canonical 7B Q4K teacher README.md:> (paiml/qwen2.5-coder-7b-apache-q4k-v1), use `--no-gpu` until the README.md:> apr run paiml/qwen2.5-coder-7b-apache-q4k-v1 "What is 2+2?" --no-gpu Five-whys (consistent with §40): 1. Why does the README need this warning? Users running `apr run` on the canonical 7B Q4K teacher get gibberish without `--no-gpu`. 2. Why? The default GPU dispatch path has SHIP-007 (GPU FP8/dequant defect). 3. Why now? PR #1113 contract requires user-facing docs warning per FALSIFY-MODEL-1-SHIP-CPU-004 — this PR satisfies that gate. 4. Why this README placement? Quick Start is the highest-traffic section; users see it first. 5. What removes the warning? GPU fix lands → contract bumps to v2.0.0 → README warning becomes obsolete and gets removed in same PR. Spec ref: §40.6 Option A (PR #1111). Contract ref: apr-cli-model-1-ship-via-cpu-v1 (PR #1113). Coverage: contributes to MODEL-1 SHIP gate completeness. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
noahgift
added a commit
that referenced
this pull request
May 13, 2026
…Option A (#1113) Authors a new provable contract that codifies the SPEC-SHIP-TWO-001 §40.6 Option A shipping decision: MODEL-1 (paiml/qwen2.5-coder-7b-apache-q4k-v1) IS shippable today via `apr run --no-gpu`. Live evidence (RTX 4090, lambda-labs): $ apr run /mnt/.../qwen2.5-coder-7b-instruct-q4k.apr \ --prompt "What is 2+2?" --max-tokens 5 --temperature 0 \ --skip-contract --no-gpu Output: "2 + 2 equals" ✓ FALSIFY-MODEL-1-SHIP-CPU-001 PASS (contains "equals") Contract structure: - 3 equations: cpu_path_correctness (PASSES today), gpu_path_known_issue (acknowledges defect tracked in §40), gpu_fix_obligation (durable closure mandate). - 6 falsification tests: -001 CPU correctness, -002 §40 in spec, -003 pv validate, -004 user-facing docs warn about GPU, -005 semver signals scope, -006 spec→contract back-reference. - 5 proof_obligations + 2 kani harnesses. - Contract validates clean via `pv validate` (0 errors, 0 warnings). Methodology compliance per `feedback_fix_root_cause_never_route_around.md`: This contract is NOT a workaround. It documents reality (CPU works, GPU has known bug), creates a falsifiable gate that catches CPU regressions, and MANDATES that the GPU bug remain visible in the spec until fixed: - v1.0.0: MODEL-1 ships CPU-only - v2.0.0: MODEL-1 ships CPU+GPU (requires gpu_path_known_issue closure) - Closing the GPU bug requires either: (a) GPU passes cpu_path_correctness gate (b) GPU dispatch is removed/deprecated (c) New hypothesis identified + spec amendment Five-whys (consistent with §40.5): 1. Why isn't MODEL-1 shipped today? Because we lacked a contract-backed verdict that "MODEL-1 produces correct output via SOME inference path". 2. Why? Because the §17/§23/§27/§38 chain was bisecting the wrong path, leaving the actual CPU correctness un-codified. 3. Why now? §40.4 + §40.5 + #1112 H1+H2 falsifiers narrowed the bug to GPU dispatch (H3); CPU is empirically correct. 4. Why this contract NOW? Per the user directive to ship + use contracts; MODEL-1 is shippable TODAY with a v1.0.0 SHIP-via-CPU contract. 5. What's next? On gpu_fix_obligation closure (a/b/c), bump v1.0.0 → v2.0.0 and 5 MODEL-1 PARTIALs auto-discharge. Spec ref: §40.6 Option A. PR cascade: #1105/#1107/#1108/#1109/#1110/#1111/#1112 (this is the SHIP gate that builds on top of §40 localization). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
§40 spec amendment documenting that SHIP-007 is localized to the GPU dispatch path; the CPU path produces correct output. MODEL-1 is shippable TODAY via `apr run --no-gpu`.
Live evidence (RTX 4090, canonical 7B teacher)
Falsification matrix (executed live, all bugs persist)
Bug surface narrowed to:
Plain progress on shipping models
MODEL-1 is shippable TODAY via CPU path. §40.6 presents two policy options:
What this means for the §17→§40 hypothesis chain
This is the highest-leverage spec amendment in the chain because it:
§17, §23, §27 were bisecting a path (F32-reference) that didn't even reach the bug. Now we know exactly where to look.
Five-whys (codified §40.5)
Methodology adherence
Test plan
🤖 Generated with Claude Code