…=GGUF (trace point mismatch) — v2.76 → v2.77

§31 hypothesized that APR's qkv_bias was the divergence introducer (std=10.24).
Live byte-compare via diag_compare_qkv_bias.rs proves this WRONG:

| Tensor | APR mean | APR std | GGUF mean | GGUF std | max diff | RMS diff |
|--------|---------:|--------:|----------:|---------:|---------:|---------:|
| q_bias | 0.127345 | 3.258061 | 0.127345 | 3.258061 | 0.000000 | 0.000000 |
| k_bias | 1.549082 | 29.464621 | 1.549082 | 29.464621 | 0.000000 | 0.000000 |
| v_bias | 0.005926 | 0.186556 | 0.005926 | 0.186556 | 0.000000 | 0.000000 |

APR ≡ GGUF byte-for-byte. The std=10.24 IS Qwen2.5-7B's actual trained
qkv_bias values; both formats store/load them correctly.

So why the 9× layer-0 qkv std gap (APR=10.33 vs GGUF=1.14)?
**TRACE-CAPTURE-POINT MISMATCH.**

GGUF (gguf/inference/forward/traced.rs:144):
  - qkv_stats captured AFTER scratch_attention_block writes scratch.qkv
  - BUT BEFORE the per-Q/K/V bias add at results.rs:216-226
  - = PRE-BIAS measurement → std=1.14

APR (apr_transformer/pmat-260.rs:331-334):
  - matmul writes `qkv` then `add_bias(qkv, bias)` in-place
  - Trace captured AFTER bias add
  - = POST-BIAS measurement → std=10.33

Both forward passes correctly apply qkv_bias. The 9× gap exists only in
the TRACE STATISTICS, not in the actual computation.

Where IS the actual SHIP-007 bug then? Per existing trace, layer 3 ffn_gate
diverges 1.36× (APR=1.92 vs GGUF=1.41), which silu non-linearly amplifies
to 18× ffn_swigl ratio. That's STILL where the bug surface is — exactly as
§28 originally said.

§30's investigation (which refuted §28) only tested LAYER 0 QKV matmul.
LAYER-3 FFN ffn_gate matmul is a DIFFERENT code path. PR E v3 scope:

  Run §31-style bisection AT LAYER 3 with the proper trace capture points,
  comparing APR sub-FFN stats vs GGUF sub-FFN stats (both captured at
  matched points per PR #1066/#1067 forward_traced sub-FFN slots).

Methodology lesson (§32.5): when stat-bisection finds a "smoking gun,"
ALWAYS verify with byte-level comparison against the reference. Stats can
mislead when measurement points differ. Toyota Way: verify physical state
(byte equality), not just symptoms (statistical gaps).

Files:
- crates/aprender-serve/examples/diag_compare_qkv_bias.rs (re-runnable)
- evidence/ship-007-qkv-bisection-2026-04-27/diag_compare_qkv_bias.txt
- §32 spec section (6 subsections)
- §31 marked SUPERSEDED in spec
- Header v2.76.0 → v2.77.0

Coverage scoreboard unchanged (15+33). Will flip when LAYER-3-specific
bisection localizes the actual divergence point.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

…s ARE byte-identical APR=GGUF

Per §32.4 next-step ("layer-3 sub-FFN bisection"), ran a byte-level
comparison of layer-3 Q4K weight bytes between the canonical 7B teacher's
.apr and .gguf files. Result:

  ffn_gate.weight Q4K (38,191,104 bytes)  →  ✓ APR ≡ GGUF byte-for-byte
  ffn_up.weight   Q4K (38,191,104 bytes)  →  ✓ APR ≡ GGUF byte-for-byte
  ffn_down.weight Q4K (38,191,104 bytes)  →  ✓ APR ≡ GGUF byte-for-byte
  layer-0 ffn_gate Q4K (sanity)           →  ✓ APR ≡ GGUF byte-for-byte

So the layer-3 ffn_gate divergence (APR std=1.92 vs GGUF std=1.41 = 1.36×
ratio per existing trace) does NOT come from differing weight bytes.

This eliminates the GGUF→APR converter as the bug surface for layer 3.
Combined with §30 (layer-0 QKV matmul kernel agrees) and §32 (qkv_bias
byte-identical), the elimination chain is now:

  - QKV matmul kernel: ✓ correct (§30)
  - QKV bias bytes: ✓ correct (§32)
  - Layer-3 FFN weight bytes: ✓ correct (this commit)

The remaining hypothesis: cumulative layer-by-layer F32 precision drift
through residual connections. APR layer 0 output std=0.40 vs GGUF=0.36
(10% drift). Layer 1 input = layer 0 output. After 3 layers of accumulating
~1% Q4K rounding diffs, layer-3 ffn_gate input is sufficiently different
between the two formats to push silu into different saturation regions,
producing the 18× ffn_swigl ratio.

Note for the apr trace --payload sub-FFN telemetry: GGUF's forward_traced
in PR A (#1081, merged) currently leaves the 4 sub-FFN slots at default
(zero). PR B (#1082, still DIRTY) clones scratch_swiglu_ffn_traced to
populate them. The existing apr-trace.txt and gguf-trace.txt evidence
files (2026-04-27) were generated when PR B was applied locally to the
binary — those numbers are valid but require PR B to land on main for
reproducibility.

Files added:
- crates/aprender-serve/examples/diag_compare_layer3_ffn.rs (re-runnable)
- evidence/ship-007-qkv-bisection-2026-04-27/diag_compare_layer3_ffn.txt

Coverage scoreboard unchanged. Investigation continues; PR E v3 scope
narrows further.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>