…ETRAIN-INIT-LOAD-003)

Bisects the §61 val_loss > ln(vocab) anomaly. Empirical findings on
lambda-vector RTX 4090:

H4 ROOT CAUSE #1: BF16 dtype mislabel
======================================

The OLD `qwen2.5-coder-0.5b-instruct-fp16.apr` (May-4 import) tags
its tensors with dtype=F16 in the APR v2 header — but the SOURCE
HF safetensors `model.safetensors` uses dtype=BF16. When the loader
sees dtype=F16, it dequantizes via `f16_to_f32`, producing values
that diverge from the BF16-correct decode.

Element-0 cross-check on `model.norm.weight`:
  Safetensors source (BF16-decoded): 7.5625, 8.0, 7.21875, ...
  Old APR (loaded as F16):           7.0625, 7.125, 7.0, ...
  Fresh APR (loaded as BF16):        7.5625, 8.0, 7.21875, ...  ← matches source

Element-0 cross-check on `model.layers.0.self_attn.q_proj.bias`:
  Safetensors (BF16):  0.0674, -0.0859, 0.1104, -0.0605, ...
  Old APR (F16):       (different, distorted)
  Fresh APR (BF16):    0.0674, -0.0859, 0.1104, -0.0605, ...  ← matches source

Fix: re-import Qwen safetensors via current `apr import`. The current
`StreamingWriter::add_raw_f16_tensor` correctly preserves BF16 (line
100-104 of streaming_writer.rs). The old APR was created with a buggy
import path that mis-tagged BF16 as F16.

H4 ROOT CAUSE #2: STILL OPEN
=============================

Even with correct BF16-decoded weights (fresh APR), val_loss at step 1
is **18.55** — still above ln(vocab)=17.21 (uniform-over-vocab
baseline). The dtype fix moved the dial slightly (was 19.80) but did
not resolve the sub-random predictions issue.

Remaining hypotheses for the residual gap:
  H4B (layout): some tensor's row/col-major orientation may differ
       between Qwen export and aprender::Transformer expectations
  H4D (forward path): cuBLAS / CudaBlock forward may produce
       wrong logits despite correct weight values
  Other: tied embedding fall-through path (`lm_head: None` →
       embed_tokens reuse) may have a sign or scale issue

This PR ships the diagnostic infrastructure that PROVED root cause #1
and provides the foundation for bisecting the remaining gap.

What this PR ships
===================

`falsify_h4_init_stats_qwen_embed_norm_sensible` — a host-gated
diagnostic test that:
  - Loads the Qwen 0.5B init APR (prefers fresh, falls back to legacy)
  - Reports tensor stats (mean, std, min, max) for embed_tokens,
    final norm, per-layer norms, q/k/v projections, mlp gates
  - Asserts sensible bounds (embed std ∈ [0.005, 0.5], norm in
    [0.01, 100], etc.)
  - Dumps element-0 values for cross-comparison with safetensors
    source

Industrial validation example output:
  embed_tokens.weight: mean=0.00014, std=0.0152, range[-0.196, 0.128]
                       — sensible HF LLaMA init scale
  model.norm.weight:   mean=7.46, std=0.84, range[-2.28, 17.38]
                       — Qwen-typical (final norm scaled up)
  q_proj.bias L0:      mean=0.03, std=7.88, range[-65.5, 128]
                       — Qwen-typical (large attention biases)

Five-Whys
==========

1. Why was the OLD Qwen APR tagged F16? Created by a buggy import
   path that didn't pass `is_bf16` flag through to the writer.
   Fixed in current apr-cli but the artifact is preserved on disk.
2. Why does the fresh APR not fully fix val_loss? The dtype fix
   makes loaded values match safetensors, but val_loss=18.55 still
   exceeds ln(vocab)=17.21 — meaning forward path or some other
   tensor is still producing sub-random predictions.
3. Why didn't existing falsifiers catch the dtype mislabel? No
   falsifier asserted "loaded values match safetensors source
   element-by-element". The PMAT-187 NaN/Inf/explosive-mean check
   passes because BF16-as-F16 distortion produces values that are
   neither NaN nor unusually large.
4. Why ship the diagnostic before the full H4 fix? The diagnostic
   itself proves H4 root cause #1 and provides the bisection
   foundation for #2. Per `feedback_falsifier_first_cascade_pattern.md`,
   1 PR ≈ 1 falsifier discharge. The dtype-mislabel discharge is
   real progress.
5. Why does the operator need to know? They have an old Qwen APR
   on disk that mis-decodes silently. With this PR's diagnostic
   they can verify before training; without it, the silent error
   wastes ~17 hours of GPU time per cycle (per §60 evidence).

Quality gates (all green)
==========================

- cargo test -p aprender-train --lib falsify_h4_init_stats: PASS
- cargo test -p aprender-train --lib: 7585+ tests PASS
- cargo clippy -p aprender-train --lib -- -D warnings: clean
  (--tests has 4 PRE-EXISTING errors on main; not introduced by this PR)
- rustfmt --check: clean

SHIP-TWO impact
================

- MODEL-1 ship %: unchanged at 91%
- MODEL-2 ship %: unchanged at 57% — H4 root cause #1 found and fix
  available (use fresh APR), but val_loss still >ln(vocab). The
  next-cycle bisection (H4B or H4D) is now well-targeted.
- §60 H1C cascade: FULLY CLOSED per #1598
- §61 evidence: 5g.1-v2 corpus is 7.42 bits entropy / 0% unk
- This PR closes part of PMAT-CODE-PRETRAIN-INIT-LOAD-003 (task #22)

Out-of-scope follow-ups
========================

PMAT-CODE-PRETRAIN-INIT-LOAD-004 (H4 residual cascade):
  - Bisect H4B (layout): forward-pass element-wise compare against
    HF Qwen2 reference at each layer
  - Bisect H4D (forward path): instrument cuBLAS GEMM outputs
    against a CPU reference matmul
  - Fix root cause; flip MODEL-2 ship % 57% → ≥58%

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

…rfaces H4 (PMAT-CODE-PRETRAIN-INIT-LOAD-003) (#1600)

Records the full discharge of PMAT-CODE-PRETRAIN-FINETUNE-LIVE-003
(task #21) and the new H4 defect surface that the honest data
exposed.

Two artifacts:

1. **5g.1 re-encode SUCCESS** — `apr tokenize encode-corpus` with
   PR #1598's upfront vocab-format detection produced a real Python
   corpus from the 3.0 GB JSONL source:
     - 1,241.7 M tokens
     - 405,944 documents
     - 126 shards × 10 M tokens each
     - Shard-0 first 32K: entropy 7.42 bits / 17.21 max; 3324 distinct
       tokens; **0% unk** (was 99.99% unk in §60's broken corpus)
   The data-bug from §60 is fully closed.

2. **5g.2 LIVE dispatch surfaces H4** — Re-running fine-tune from
   Qwen 0.5B init on the now-real corpus aborted at GATE-TRAIN-005:
     - 500-step run: val_loss = 11.55 at epoch 0 (> 10.0 threshold)
     - 1-step diagnostic: val_loss = 19.80 (> ln(vocab) = 17.21)
   val_loss > ln(vocab) means the model assigns LESS than uniform
   probability to true tokens — *worse than random init*. The Qwen
   init weights load (PR #1579's populate-coverage fix is in main)
   but produce sub-random predictions.

Five-Whys

1. Why was val_loss = 19.80 at step 1? Industry baseline for Qwen
   0.5B zero-shot on Python is ~1.5–3.0; uniform random over vocab
   is ln(151643) = 17.21. 19.80 > 17.21 means the model is
   *anti-aligned* with held-out tokens.
2. Why anti-aligned despite Qwen init being loaded? Some structural
   component of the init pipeline is broken at a layer that PR #1579
   doesn't cover.
3. Four hypotheses for H4:
     A. Tied weights — `tie_word_embeddings: true` on Qwen 0.5B; if
        populate writes embed_tokens but doesn't propagate to
        lm_head (or writes them separately to random buffers),
        forward predictions are random while embeddings are correct.
     B. Layout mismatch — GGUF/APR are row-major (tensor-layout-v1);
        if init APR's lm_head is column-major, matmul produces
        wrong logits.
     C. Norm scale — RMSNorm weights loaded but rms_norm_eps mismatch
        cascades through forward.
     D. Residual stream — some block's residual contributes zero from
        an uninitialized buffer.
4. Why ship the diagnosis but not the H4 fix? Each hypothesis is its
   own falsifier-discharge cascade per `feedback_falsifier_first_cascade_pattern.md`.
   Multi-PR scope.
5. Why does this matter for ship %? FALSIFY-005 status flips from
   NUMERICALLY-PASSED-METHODOLOGY-SUSPECT (pre-§61, fake pass on
   broken corpus) to RED-WITH-METHODOLOGICALLY-HONEST (post-§61,
   real defect on real corpus). The honest RED is itself progress
   — the contract now reports the binding defect.

SHIP-TWO impact

- MODEL-1 ship %: unchanged at 91% (this is MODEL-2 work)
- MODEL-2 ship %: unchanged at 57% — diagnosis correct, H4 cascade
  is the gate
- §60 H1C (data-bug) cascade: FULLY CLOSED. Encoder works
  end-to-end on real Qwen vocab + real Python corpus.

Closes PMAT-CODE-PRETRAIN-FINETUNE-LIVE-003 (task #21).

Tracking PMAT-CODE-PRETRAIN-INIT-LOAD-003 (H4 cascade) as the next
ship-mover.

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>

…ETRAIN-INIT-LOAD-003) (#1601)

Bisects the §61 val_loss > ln(vocab) anomaly. Empirical findings on
lambda-vector RTX 4090:

H4 ROOT CAUSE #1: BF16 dtype mislabel
======================================

The OLD `qwen2.5-coder-0.5b-instruct-fp16.apr` (May-4 import) tags
its tensors with dtype=F16 in the APR v2 header — but the SOURCE
HF safetensors `model.safetensors` uses dtype=BF16. When the loader
sees dtype=F16, it dequantizes via `f16_to_f32`, producing values
that diverge from the BF16-correct decode.

Element-0 cross-check on `model.norm.weight`:
  Safetensors source (BF16-decoded): 7.5625, 8.0, 7.21875, ...
  Old APR (loaded as F16):           7.0625, 7.125, 7.0, ...
  Fresh APR (loaded as BF16):        7.5625, 8.0, 7.21875, ...  ← matches source

Element-0 cross-check on `model.layers.0.self_attn.q_proj.bias`:
  Safetensors (BF16):  0.0674, -0.0859, 0.1104, -0.0605, ...
  Old APR (F16):       (different, distorted)
  Fresh APR (BF16):    0.0674, -0.0859, 0.1104, -0.0605, ...  ← matches source

Fix: re-import Qwen safetensors via current `apr import`. The current
`StreamingWriter::add_raw_f16_tensor` correctly preserves BF16 (line
100-104 of streaming_writer.rs). The old APR was created with a buggy
import path that mis-tagged BF16 as F16.

H4 ROOT CAUSE #2: STILL OPEN
=============================

Even with correct BF16-decoded weights (fresh APR), val_loss at step 1
is **18.55** — still above ln(vocab)=17.21 (uniform-over-vocab
baseline). The dtype fix moved the dial slightly (was 19.80) but did
not resolve the sub-random predictions issue.

Remaining hypotheses for the residual gap:
  H4B (layout): some tensor's row/col-major orientation may differ
       between Qwen export and aprender::Transformer expectations
  H4D (forward path): cuBLAS / CudaBlock forward may produce
       wrong logits despite correct weight values
  Other: tied embedding fall-through path (`lm_head: None` →
       embed_tokens reuse) may have a sign or scale issue

This PR ships the diagnostic infrastructure that PROVED root cause #1
and provides the foundation for bisecting the remaining gap.

What this PR ships
===================

`falsify_h4_init_stats_qwen_embed_norm_sensible` — a host-gated
diagnostic test that:
  - Loads the Qwen 0.5B init APR (prefers fresh, falls back to legacy)
  - Reports tensor stats (mean, std, min, max) for embed_tokens,
    final norm, per-layer norms, q/k/v projections, mlp gates
  - Asserts sensible bounds (embed std ∈ [0.005, 0.5], norm in
    [0.01, 100], etc.)
  - Dumps element-0 values for cross-comparison with safetensors
    source

Industrial validation example output:
  embed_tokens.weight: mean=0.00014, std=0.0152, range[-0.196, 0.128]
                       — sensible HF LLaMA init scale
  model.norm.weight:   mean=7.46, std=0.84, range[-2.28, 17.38]
                       — Qwen-typical (final norm scaled up)
  q_proj.bias L0:      mean=0.03, std=7.88, range[-65.5, 128]
                       — Qwen-typical (large attention biases)

Five-Whys
==========

1. Why was the OLD Qwen APR tagged F16? Created by a buggy import
   path that didn't pass `is_bf16` flag through to the writer.
   Fixed in current apr-cli but the artifact is preserved on disk.
2. Why does the fresh APR not fully fix val_loss? The dtype fix
   makes loaded values match safetensors, but val_loss=18.55 still
   exceeds ln(vocab)=17.21 — meaning forward path or some other
   tensor is still producing sub-random predictions.
3. Why didn't existing falsifiers catch the dtype mislabel? No
   falsifier asserted "loaded values match safetensors source
   element-by-element". The PMAT-187 NaN/Inf/explosive-mean check
   passes because BF16-as-F16 distortion produces values that are
   neither NaN nor unusually large.
4. Why ship the diagnostic before the full H4 fix? The diagnostic
   itself proves H4 root cause #1 and provides the bisection
   foundation for #2. Per `feedback_falsifier_first_cascade_pattern.md`,
   1 PR ≈ 1 falsifier discharge. The dtype-mislabel discharge is
   real progress.
5. Why does the operator need to know? They have an old Qwen APR
   on disk that mis-decodes silently. With this PR's diagnostic
   they can verify before training; without it, the silent error
   wastes ~17 hours of GPU time per cycle (per §60 evidence).

Quality gates (all green)
==========================

- cargo test -p aprender-train --lib falsify_h4_init_stats: PASS
- cargo test -p aprender-train --lib: 7585+ tests PASS
- cargo clippy -p aprender-train --lib -- -D warnings: clean
  (--tests has 4 PRE-EXISTING errors on main; not introduced by this PR)
- rustfmt --check: clean

SHIP-TWO impact
================

- MODEL-1 ship %: unchanged at 91%
- MODEL-2 ship %: unchanged at 57% — H4 root cause #1 found and fix
  available (use fresh APR), but val_loss still >ln(vocab). The
  next-cycle bisection (H4B or H4D) is now well-targeted.
- §60 H1C cascade: FULLY CLOSED per #1598
- §61 evidence: 5g.1-v2 corpus is 7.42 bits entropy / 0% unk
- This PR closes part of PMAT-CODE-PRETRAIN-INIT-LOAD-003 (task #22)

Out-of-scope follow-ups
========================

PMAT-CODE-PRETRAIN-INIT-LOAD-004 (H4 residual cascade):
  - Bisect H4B (layout): forward-pass element-wise compare against
    HF Qwen2 reference at each layer
  - Bisect H4D (forward path): instrument cuBLAS GEMM outputs
    against a CPU reference matmul
  - Fix root cause; flip MODEL-2 ship % 57% → ≥58%

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>