…und + fixed + first checkpoint — spec v2.65.0 → v2.66.0

User directive: "we should train a model unless the path is broken,
then fix." This section documents the FIRST sustained from-scratch
MODEL-2 training run since the project began. Three real stack bugs
were discovered DURING training and fixed at root.

## Bug 1 — corpus exhaustion silently emits placeholder

`ShardBatchIter::next()→None` after exhaust; `Cuda*StepFn` returned
`(1.0, 1.0)` placeholder, masking exhaustion silently. 5K-step run
showed train_loss=1.0 / wall_s=0.4 for epochs 3-4 (impossible for
real training).

Fix: PR #1073 first commit — `with_wrap_around(true)` opt-in.
Validation: re-ran 5K → 5 epochs of monotonic train_loss
10.111→9.700.

## Bug 2 — early-stop on val noise not stagnation

50K run with #1073 still early-stopped at epoch 5/24 — train_loss
10.01→9.54 monotonic, val_loss bouncing 9.78-9.92 in 0.04 noise band.

Root cause: HELD_OUT_BATCHES=2 (16k tokens, single-batch fluctuation
matched epoch-over-epoch signal) + patience_epochs=2 (terminates on
2 noise epochs).

Fix: PR #1073 second commit — HELD_OUT_BATCHES 2→16 (131k tokens,
8× larger sample); patience_epochs 2→5; min_epochs_before_early_stop
1→3. Validation: tuned 50K showed val_loss decreasing 9.95→9.84→9.78
across first 3 epochs.

## Bug 3 — corpus too small (18M vs Chinchilla 7.4B for 370M)

After fixes 1+2, tuned run revealed overfitting at epoch 3+:
train_loss continues 9.69→9.52, val_loss climbs 9.78→9.92, gap
inverts. Classic small-corpus signature.

Fix not in code — data engineering: pretokenize The Stack v2 Python
(multi-billion tokens). Deferred to next session per
feedback_compute_pre_authorized.md.

## What was produced

Best checkpoint at /mnt/nvme-raid0/runs/model-2-from-scratch-006-50k-tuned/
ckpt/epoch-002.apr:
- APR v2, 219 tensors, 1.39 GiB, checksum VALID
- Architecture: LlamaForCausalLM, name=llama-370m-pretrain
- val_loss=9.78, 49.2M tokens (corpus wrapped 2.7×)

`apr inspect` validates. AC-SHIP2-005 (.apr checkpoint format)
structurally discharged.

## Coverage impact (no tally change)

| Gate | Prior | Post |
|------|-------|------|
| AC-SHIP2-005 | PARTIAL | structurally discharged |
| GATE-TRAIN-005 | PARTIAL | confirmed correct (didn't fire on real run) |
| GATE-TRAIN-001 | PARTIAL | confirmed correct (per-step metrics live) |

Coverage tally formally unchanged; structural verification awaits
contract-level promotion.

## Methodology

Per feedback_fix_root_cause_never_route_around.md: zero route-arounds.
Each bug had a `TrueCause`-style root cause analysis. The placeholder
return on iterator exhaustion was a route-around itself; replaced
with iterator looping. The early-stop policy was data-undersampled;
fixed by widening the val set, not by raising the patience alone.

Spec v2.65.0 → v2.66.0.

Evidence persisted to evidence/model-2-first-training-2026-04-26/
(12 metadata.json files: 5 from broken+fixed 5K runs, 7 from tuned
50K run).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>