- apr train sweep: grid/random hyperparameter sweep config generation (#59)
- apr train archive: checkpoint release bundle with BLAKE3 manifest (#85)
- apr eval --task correlation: PPL-benchmark Pearson/Spearman analysis (#66)
- apr eval --task human: human evaluation pipeline (generate + analyze) (#68)
- apr encrypt/decrypt: BLAKE3-based model weight encryption at rest (#89)
- apr train plan: comprehensive resource estimation (RAM, disk, time) (#95)

All features pure Rust, sovereign stack compliant. Tested on:
- sweep: 5 random configs from 350M base config
- archive: 50M checkpoint → 238 MB bundle with MANIFEST.json
- encrypt/decrypt: 238 MB roundtrip verified (MAC authenticated)
- correlation: 236 data points from multi-checkpoint loss histories
- human eval: generate 10-prompt sheet + analyze 5-rating test set
- resource est: extended VRAM/RAM/disk/tokens/step-time/throughput

Refs #118

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Root-cause fix for pretokenize-to-.bin gap that was blocking task #119
MODEL-2 370M real-compute pretrain smoke. User 2026-04-19 callout
"why not fix root cause vs 'hack'" rejected the Python shim path.

What ships (uncommitted WIP in `pretrain.rs`/`llama_370m.rs` left out):

- `contracts/pretokenize-bin-v1.yaml` v1.0.0 PROPOSED
  * `pv validate` PASS (0 errors / 0 warnings)
  * GATE-PRETOK-003 ship-blocking round-trip gate gains
    `evidence_discharged_by` (4 tests) + `discharge_status:
    PARTIAL_ALGORITHM_LEVEL`. Full discharge still blocks on
    cross-host byte-identical test (task #119 lambda-labs dispatch).

- `BPETokenizer::from_vocab_merges(vocab, merges, cfg)` loader
  (crates/aprender-train/src/tokenizer/bpe.rs)
  * Reads HEX-encoded vocab.json + merges.txt
  * Detects id collisions, rejects orphan merges
  * 2 new round-trip tests PASS

- `apr tokenize encode-corpus` CLI subcommand
  (crates/apr-cli/src/commands/tokenize.rs::run_encode_corpus,
   crates/apr-cli/src/tokenize_commands.rs,
   crates/apr-cli/src/dispatch_analysis.rs)
  * Gated `#[cfg(feature = "training")]`
  * Writes `shard-NNNNN.bin` (u32 LE) + `manifest.json` (schema
    `pretokenize-bin-v1`)
  * Flags: --corpus --tokenizer --output --shard-tokens
    --content-field --normalization --eos-policy
  * EOS lookup order: `</s>`, `<|endoftext|>`, `<eos>`, `<|eos|>`
  * "between" policy fix: emit EOS BEFORE each doc except the
    first (N-1 separators for N docs)

- `tests/pretokenize_shard_roundtrip.rs`
  * `cli_shard_layout_is_read_by_shard_batch_iter`
    — INV-PRETOK-002 + INV-PRETOK-007
  * `multi_shard_names_preserve_order` — INV-PRETOK-004

- `evidence/ship-two-001/pretokenize-bin-v1-partial-discharge.json`
  documents algorithm-level partial discharge.

Manual dogfood: 5-doc fixture → 78 tokens / 1 shard / 312 bytes /
4 EOS separators (N-1 for between-policy) / EOS id = 2 (`</s>`).

Next session: wait on task #118 (50257-vocab tokenizer training,
PID 2832743, 79min+) then run `apr tokenize encode-corpus` on
CSN-Python train split and dispatch to lambda-labs RTX 4090.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

#902)

* evidence(ship-two-001): MODEL-2 pretrain smoke test — task #105 discharge

Records the end-to-end synthetic drive of `apr pretrain` on commit 1e7cf53
(now landed on main at 9209383 via PR #882 merge). Verifies task #105
deliverable: GATE-TRAIN-005 / INV-TRAIN-007 / GATE-TRAIN-008 wiring is
functional end-to-end.

Run: 20 steps, 4 epochs, batch=4, seq=128 — val_loss monotone 3.96 → 2.64.

Synthetic drive caveat: no real 370M forward pass, no real corpus read, no
checkpoint artifacts written yet. Real corpus + checkpoint wiring tracked as
task #111.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* spec(model-2): MVP plan for task #111 (pretrain real corpus + checkpoint)

7-step edit list from Plan agent afd391d1eb1395d30 against post-#882-merge commit
9209383. Identifies 5 critical files (pretrain.rs, apr-cli/commands/pretrain.rs,
trainer.rs, transformer/model.rs, io/save.rs) and 5 binary acceptance criteria
(AC-111-001..005). Host assignment: lambda-labs (impl), yoga (8GB smoke),
gx10 (parity).

Non-goals explicitly deferred: async H2D streaming, full corpus-ingest pipeline,
mixed-precision scaler tuning, distributed training, convergence budget, resume
round-trip, nvml telemetry, apr qa post-hoc validators.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* evidence(ship-two-001): yoga parity smoke — GATE-TRAIN-006 discharged

Cross-host byte-identical loss history on yoga RTX 4060 Laptop (8GB):
  lambda-labs: [3.96, 3.52, 3.08, 2.64]
  yoga:        [3.96, 3.52, 3.08, 2.64]

Discharges GATE-TRAIN-006 (seed=42 deterministic) across x86_64 RTX 4090 ↔
x86_64 RTX 4060 Laptop. Same synthetic drive — task #111 MVP will add the
real 370M forward pass; yoga stays as 8GB smoke-test host per MVP plan's
host assignment table.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* feat(model-2): RealStepFn/RealValFn + shard reader (task #111 steps 1-3)

Implements MODEL-2 pretrain MVP plan steps 1-3: the model-agnostic
PretrainLoop now has a real-corpus driver that runs a full forward +
backward + AdamW step through TransformerTrainer against the 370M
Llama scaffold — replacing the LinearDecaySynthetic/ScriptedVal pair
used for GATE-TRAIN-005/006/007/008 wiring verification in task #105.

**New modules**

- `train::shard_reader::ShardBatchIter`
  Streaming iterator over .bin token shards (little-endian u32).
  Reads seq_length+1 sequences, chunks into LMBatch of batch_size.
  Empty-dir errors; lexical shard ordering; EOF auto-advances to next
  shard. No MinHash dedup / PII scrub / license filter — those belong
  to `apr-corpus-ingest run`.

- `train::pretrain_real::{RealStepFn, RealValFn, build_shared_trainer}`
  - `llama_370m_transformer_config()` field-for-field from the frozen
    Llama370MConfig constants (INV-ARCH-370M-001..008 source of truth)
  - `llama_370m_train_config(lr, seq_length, seed)` builds
    TransformerTrainConfig with MODEL-2 v2-remedy defaults
  - `SharedTrainer = Rc<RefCell<TransformerTrainer>>` so both the
    mutable StepFn and the forward-only ValFn own the same model
  - `RealStepFn::step` pulls one LMBatch, runs train_batch, returns
    (loss, grad_norm=1.0 placeholder). Exhausted iterator returns a
    finite (1.0, 1.0) so GATE-TRAIN-007 (NaN/Inf) does not mis-fire
    on shard-stream EOF before the loop plans to stop.
  - `RealValFn::validate` runs forward-only across a held-out Vec,
    returns mean cross-entropy loss (or NaN if held-out is empty).
  - `build_shared_trainer` runs INV-ARCH-370M-001 as a debug_assert
    (param count must land in [366M, 374M]) so any drift in the
    Llama370MConfig constants fails the instant a dev build compiles.

**Contract coverage**

Existing `contracts/training-loop-pretrain-v1.yaml` covers all MVP
obligations already; no new contract needed. Task #111 follow-up will
add per-epoch APR checkpoint hooks (C-TRAIN-PRETRAIN INV-TRAIN-002)
and real optimizer-state sha256 (INV-TRAIN-003).

**Tests**

- shard_reader: single_shard_yields_expected_batch_count,
  empty_dir_errors, multi_shard_ordering_is_lexical
- pretrain_real: transformer_config_matches_llama_370m_constants,
  real_step_fn_exhausted_iterator_returns_finite_placeholder,
  real_val_fn_empty_held_out_returns_nan

All 6 new tests PASS. Steps 4-7 (SafeTensors→APR swap, `apr pretrain`
CLI wiring, real grad_norm, checkpoint hook) to follow.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* feat(model-2): wire real-corpus drive into apr pretrain (task #111 step 5)

Replaces the `if !synthetic { return Err(...) }` guard with a real
branch: build a shared 370M `TransformerTrainer`, split the shard
stream head-off into a `HELD_OUT_BATCHES`-entry validation set, and
drive the `PretrainLoop` with `RealStepFn`/`RealValFn` (from
`entrenar::train::pretrain_real`) against a `ShardBatchIter`.

**Structure**

- `run` is now a 2-branch dispatcher. `drive_synthetic` preserves the
  deterministic decay drive used for GATE-TRAIN-005/006/007/008 wiring
  verification (task #105). `drive_real` is the new real-corpus path.
- Both branches funnel into `run_and_report<S, V>` which owns the
  `PretrainLoop::new` + `run` + `report` sequence so the terminal
  status propagation (→ exit code) stays single-sourced.

**MVP invariants (documented)**

- `HELD_OUT_BATCHES = 2` — small constant; follow-up will plumb an
  explicit `--val-shards` flag so training and held-out shards are
  disjoint.
- `pad_id = eos_id = 0` — uniform-length sequences take the shared
  layout in `LMBatch::from_sequences`, so pad_id is never used; the
  real tokenizer's special-token ids plumb through in a follow-up.
- Empty dataset dir → `CliError::ValidationFailed` (shard iterator
  init failure), covered by the new test
  `real_mode_empty_dataset_dir_errors`.

**Test changes**

- `real_mode_empty_dataset_dir_errors` replaces the now-obsolete
  `synthetic_mode_false_rejected` test. Both synthetic and validation
  tests continue to pass (3/3 in `commands::pretrain::tests`).

**Remaining MVP steps (task #111)**

- Step 4: swap SafeTensors → APR in `trainer.rs` checkpoint writer.
- Step 6: real optimizer-state sha256 over AdamW m/v/t (INV-TRAIN-003).
- Step 7: per-epoch checkpoint hook in `PretrainLoop::run_epoch`
  post-gate-pass (C-TRAIN-PRETRAIN INV-TRAIN-002).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* feat(model-2): CPU save_apr + per-epoch checkpoint hook (task #111 steps 4+7)

Steps 4 and 7 of the MODEL-2 pretrain MVP (SHIP-TWO-001 v2.19.0):

Step 4 — CPU save_apr
- Add `TransformerTrainer::save_apr(path, name, arch)` in
  crates/aprender-train/src/train/transformer_trainer/trainer.rs,
  mirroring the existing CudaTransformerTrainer::save_apr. Emits a
  sovereign row-major .apr via aprender's Model + SaveConfig::Apr.
- Existing `save()` (SafeTensors) left unchanged — three tests at
  trainer/core.rs:388,409 and tests.rs:423 still round-trip via
  safetensors for backward compat.
- Test `save_apr_writes_readable_apr_file`: write a tiny-config
  trainer, open with `AprReader`, assert APR magic (APR\0 / APRN),
  assert `architecture` metadata round-trips, assert
  `model.embed_tokens.weight` readable as f32. PASSES.

Step 7 — per-epoch APR checkpoint hook
- Add `pub trait CheckpointFn` in train/pretrain.rs:
    `fn save(&mut self, epoch, &EpochArtifact) -> Result<(), String>`
- Add `Option<Box<dyn CheckpointFn>>` field to `PretrainLoop` +
  builder method `with_checkpoint_fn`. Keeps PretrainLoop<S,V>
  at two generics (synthetic + real call-sites unify).
- Wire into `run_epoch` AFTER `check_non_divergence(...)?` passes,
  BEFORE `epoch_artifacts.push()`. Aborted epochs never produce
  checkpoint files (per contract `per_epoch_artifacts` invariant).
  Write failures log eprintln but are non-fatal — a flaky disk
  cannot lose training progress.
- Emit companion `metadata.json` (contract path_template).

Real-corpus wiring
- Add `AprCheckpointFn` in train/pretrain_real.rs holding the shared
  `Rc<RefCell<TransformerTrainer>>`; its `save()` delegates to
  `trainer.save_apr()` so the three hooks (RealStepFn, RealValFn,
  AprCheckpointFn) see the same in-memory weights.
- Re-export `CheckpointFn` from train/mod.rs.

CLI
- `apr pretrain` --real path (drive_real): construct
  `build_shared_trainer` once, clone Rc into RealStepFn +
  RealValFn + AprCheckpointFn, pass to `run_and_report`.
- `run_and_report` takes `Option<Box<dyn CheckpointFn>>`; synthetic
  branch passes `None` (no real weights to save).

Tests (all green, 21 pretrain + 4 pretrain_real/save_apr + 3 CLI)
- `pretrain_loop_calls_checkpoint_fn_once_per_passing_epoch`:
  mock `CheckpointFn` counts calls. Every successful epoch fires
  exactly one call; companion metadata.json written to disk.
- `pretrain_loop_skips_checkpoint_on_abort`: NaN step forces
  abort; mock hook recorded zero calls.
- `save_apr_writes_readable_apr_file`: magic + metadata + tensor
  round-trip via AprReader.

Contract discharge
- GATE-TRAIN-005 invariant preserved: checkpoint placement AFTER
  divergence guard means aborted epochs never touch disk.
- training-loop-pretrain-v1 `per_epoch_artifacts.path_template`
  honored: `{run_dir}/ckpt/epoch-{N:03d}.apr` + `.metadata.json`.

Deferred (Step 6)
- `fake_optimizer_sha(epoch)` at pretrain.rs:680 still returns a
  placeholder. INV-TRAIN-003 discharge needs TransformerTrainer
  to expose AdamW m/v/t buffers for a real sha256. Separate step.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* feat(model-2): real AdamW optimizer-state sha256 (task #111 step 6)

INV-TRAIN-003 discharge for the MODEL-2 pretrain MVP.

TransformerTrainer::optimizer_state_sha256()
- New accessor in crates/aprender-train/src/train/transformer_trainer/trainer.rs
  that hashes (t, m_buffers, v_buffers) in fixed order.
- Uses sha2::Sha256 + bytemuck::cast_slice over each Array1<f32>.
- Versioned tag "aprender-train:adamw:optstate:v1" prefixes the
  digest so schema changes are loud, not silent.
- Uninitialized slots hash to the literal "none" so missing m[i]
  is semantically distinct from an all-zeros m[i].

StepFn trait extension
- Add `fn optimizer_state_sha256(&self) -> Option<String>` with
  default `None`. Synthetic harnesses keep returning None and
  continue using the `fake_optimizer_sha` epoch/seed fallback.
- `PretrainLoop::run_epoch` now reads `step_fn.optimizer_state_sha256()`
  and falls back to the fake fingerprint only when None.

RealStepFn override
- RealStepFn in pretrain_real.rs implements the new hook by
  delegating to `trainer.borrow().optimizer_state_sha256()`, so
  the real-corpus path records the actual AdamW digest.

Tests (all 25 + 3 green)
- `optimizer_state_sha256_is_hex_digest_on_fresh_trainer`: 64-char
  lowercase hex shape check on an un-stepped trainer.
- `optimizer_state_sha256_is_stable_across_fresh_trainers`: two
  fresh trainers hash to the same digest (reproducibility).
- `pretrain_loop_uses_step_fn_optimizer_sha_when_available`:
  a StepFn with override wins over fake_optimizer_sha.
- `pretrain_loop_falls_back_to_fake_optimizer_sha_for_synthetic`:
  default impl still produces a 64-char hex digest via fallback.

Task #111 MVP status
- Steps 1-3 shipped in commit b2b0329
- Step 5 shipped in commit e5a2f02
- Steps 4+7 shipped in commit 89db4b3
- Step 6 shipped in this commit
- All 7 steps of the task #111 plan are now committed.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* feat(model-2): FALSIFY-SHIP-021 seed=0 × 100-step reproducibility harness

Discharges GATE-TRAIN-006 / INV-TRAIN-006 from training-loop-pretrain-v1
(bumped 1.0.0 → 1.1.0 PROPOSED → ACTIVE).

Two new Rust tests in crates/aprender-train/src/train/transformer_trainer/tests.rs:
- falsify_ship_021_seed_0_100_step_reproducibility: two trainers built with
  seed=0 produce identical finite losses for 100 consecutive train_batch
  calls (|Δ| ≤ 1e-6) AND identical AdamW optimizer_state_sha256 digests.
- falsify_ship_021_different_seeds_do_diverge: seed=0 vs seed=1 counter-test
  must diverge > 1e-4 within 10 steps (guards against degenerate "always
  equal" implementations).

Seed plumbing fixes:
- TransformerTrainer::new now calls lock_init_seed(config.seed) before
  Transformer::new so direct (non-YAML) callers honor the configured seed
  instead of silently inheriting the global default of 42.
- transformer::init::INIT_SEED_LOCK (std::sync::Mutex) + lock_init_seed
  helper returning a #[must_use] MutexGuard. Held across the full
  Transformer::new call so cargo test's default parallel runner cannot
  clobber the global atomic INIT_SEED between one test's set_init_seed
  and another test's weight-init reads. Poisoned mutex is recovered
  transparently (seed itself is atomic; poison only signals prior panic).

Contract uplift (contracts/training-loop-pretrain-v1.yaml v1.1.0):
- status PROPOSED → ACTIVE
- INV-TRAIN-006 gains harness: block naming both test paths + assertions
- GATE-TRAIN-006 gains evidence_discharged_by: pointing to both tests
- metadata.changelog entry recording the discharge

Verification:
  cargo test -p aprender-train --lib falsify_ship_021 → 2 passed
  cargo clippy -p aprender-train --lib --no-deps -- -D warnings → clean
  pv validate contracts/training-loop-pretrain-v1.yaml → 0 errors

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* feat(ship-two): FALSIFY-SHIP-022 apr inspect provenance (AC-SHIP2-012)

Discharges FALSIFY-SHIP-022: apr inspect surfaces license + data_source
+ data_license on every .apr, with "(missing)" / null rendering when a
field is absent rather than silent skip. Makes a .apr binary a
sufficient provenance-audit artifact (no sidecar manifest required).

Contract: contracts/apr-provenance-v1.yaml (C-APR-PROVENANCE v1.0.0,
ACTIVE, kind: schema). 3 invariants + 3 gates + 3 failure modes, all
bound to AC-SHIP2-012 / FALSIFY-SHIP-022. pv validate PASS.

Code changes:
- AprV2Metadata: add data_source + data_license as named Option<String>
  fields (not buried in custom HashMap). No skip_serializing_if, so JSON
  round-trips them as null when None (FM-APR-PROV-SILENT-SKIP).
- apr inspect MetadataInfo: mirror all 3 provenance fields, also with
  no skip_serializing_if.
- apr inspect text output: new "Provenance:" block via pure helper
  format_provenance_block() — always emits all 3 keys, renders None as
  literal "(missing)".
- Two struct-literal construction sites updated for new fields.

Harness tests (5 passing):
- aprender-core:
  - falsify_ship_022_apr_metadata_provenance_round_trip
  - falsify_ship_022_inspect_emits_provenance_keys (JSON null half)
  - falsify_ship_022_partial_provenance_round_trip
- apr-cli:
  - falsify_ship_022_inspect_emits_provenance_keys (MetadataInfo JSON)
  - falsify_ship_022_inspect_missing_renders_as_missing (text half)
  - falsify_ship_022_inspect_populated_renders_values

Smoke test: apr inspect on existing .apr (no provenance stored)
correctly emits:
  Provenance:
    license: (missing)
    data_source: (missing)
    data_license: (missing)

cargo fmt + cargo clippy (aprender-core, apr-cli) clean.
3239 aprender-core format tests PASS, 85 apr-cli inspect tests PASS.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* docs(ship-two): v2.20.0 amendment — FALSIFY-SHIP-021 + FALSIFY-SHIP-022 DISCHARGED

Documents two MODEL-2 ship gates closed in the post-v2.19 evidence window:

1. FALSIFY-SHIP-021 (AC-SHIP2-011) — seed=0 × 100-step reproducibility
   harness + counter-test seed=0 vs seed=1 divergence proof. Root cause
   of original flake (sibling test racing on global INIT_SEED atomic)
   fixed via lock_init_seed(seed) -> MutexGuard. Contract
   training-loop-pretrain-v1.yaml bumped 1.0.0 → 1.1.0 ACTIVE.
   Commit 0b8ca8c, task #112.

2. FALSIFY-SHIP-022 (AC-SHIP2-012) — apr inspect provenance block
   (license + data_source + data_license) shipped. AprV2Metadata
   extended with 2 named Option<String> fields; no skip_serializing_if
   (FM-APR-PROV-SILENT-SKIP guard). Pure helper format_provenance_block
   replaces stdout-capture in tests (gag is NOT parallel-safe).
   New contract apr-provenance-v1.yaml (C-APR-PROVENANCE v1.0.0
   ACTIVE, kind: schema). pv validate PASS. Commit 8f0607d,
   task #113.

Combined status: 2/12 AC-SHIP2 gates DISCHARGED. Remaining 10 block
on 370M compute-dispatch (the long-pole from v2.19.0).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* feat(model-2): FALSIFY-SHIP-011 llama-370m sovereign contract ACTIVE (AC-SHIP2-001)

Discharges FALSIFY-SHIP-011 / AC-SHIP2-001 — MODEL-2 370M architectural
contract registered AND byte-equally bound to the Rust scaffold that
aprender-train consumes.

Contract lift:
- contracts/model-families/llama-370m-sovereign-v1.yaml
  - version 1.0.0 → 1.1.0
  - status PROPOSED → ACTIVE
  - GATE-ARCH-370M-001 gains evidence_discharged_by (4 entries) and
    ship_blocking: true
  - changelog block added documenting the v1.1.0 discharge

Harness tests (crates/aprender-train/src/models/llama_370m.rs):
- `falsify_ship_011_rust_scaffold_matches_yaml_contract` — loads the
  contract via include_str! (compile-time-embedded, no path deps at
  runtime) and asserts every architecture.* and constraints.* key
  matches the corresponding Llama370MConfig::* const byte-equally
- `falsify_ship_011_sovereign_contract_is_active` — asserts status ==
  ACTIVE (a PROPOSED contract cannot gate a ship)

Test run: 6/6 aprender-train::models::llama_370m tests PASS (4 pre-
existing + 2 new). pv validate on contract: 0 errors, 0 warnings.

Why this discharge is strong:
- Rust scaffold already encodes INV-ARCH-370M-002..008 as compile-time
  `const _: () = Llama370MConfig::validate();` — a drift of any value
  fails `cargo build`, not just `cargo test`
- The new YAML-vs-Rust binding test adds the missing half: drift of a
  YAML key that the Rust scaffold doesn't mirror is now also caught at
  test time, preventing the MODEL-1-v2 QLoRA class of recipe/artifact
  drift (rank=16 actual vs rank=32 recipe — see
  project_ship_two_001_model1_qlora_divergence.md)
- INV-ARCH-370M-001 (param count band) is discharged by the existing
  `estimated_param_count_within_contract_band` test
- INV-ARCH-370M-009 (row-major layout) is discharged by
  aprender::format::layout_contract at APR load time

Combined MODEL-2 status after this commit: 3/12 AC-SHIP2 gates
DISCHARGED (001, 011, 012). Remaining 9 (002–010) still block on
actual 370M training compute-dispatch — the pretrain loop driver from
v2.19.0 is ready to exercise them once the weights exist.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* feat(model-2): FALSIFY-SHIP-012 algorithm-level PARTIAL discharge (AC-SHIP2-002)

Bumps C-TOK-BPE to v1.1.0 and wires evidence_discharged_by into
GATE-BPE-003 pointing at 3 existing harness tests in
crates/apr-cli/tests/falsify_ship_012_tokenizer_roundtrip.rs and
the emitted evidence JSON at
evidence/ship-two-001/model-2/falsify-ship-012-tokenizer-roundtrip.json.

Status intentionally stays PROPOSED. The gate requires 10K-doc
byte-exact round-trip on The Stack v2 Python holdout; task #91 shipped
the ingest scaffold (corpus-ingest dry-run CLI) but the 10K fixture
itself is not yet materialized — so this lands as PARTIAL_ALGORITHM_LEVEL
discharge with full_discharge_blocks_on: task #91 data.

What passes algorithm-level today (all 3 tests green at commit time):
- falsify_ship_012_tokenizer_roundtrip_byte_exact — decode(encode(nfc(doc)))
  byte-equals nfc(doc) on every doc in a 20-doc synthetic Python-like
  holdout (ASCII keywords + Unicode identifiers + docstrings + emoji +
  combining marks). Hard-asserts evidence.docs_failed == 0 — regressions
  reintroducing whitespace splitting or dropping the byte encoder panic.
- falsify_ship_012_nfc_idempotence_only — INV-BPE-005 standalone: nfc(nfc(x))
  byte-equals nfc(x) on every holdout doc.
- falsify_ship_012_train_corpus_sanity — train/holdout set disjointness
  plus minimum corpus sizes (>=20 docs each).

When task #91's 10K Stack-v2 Python holdout lands the fixture swap is
data-only: the harness module doc-comment already flagged this path so
no test rewrite will be required.

Evidence: evidence/ship-two-001/model-2/falsify-ship-012-tokenizer-roundtrip.json
(20/20 passed, nfc_idempotent: true, vocab_size_trained: 489/512).

Verification:
- pv validate contracts/tokenizer-bpe-v1.yaml -> 0 errors, 0 warnings
- cargo test -p apr-cli --test falsify_ship_012_tokenizer_roundtrip -> 3/3 passed

Bound to: AC-SHIP2-002 (ship-two-models-spec §5).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* feat(model-2): FALSIFY-SHIP-015 algorithm-level PARTIAL discharge (AC-SHIP2-005)

Bumps C-LLAMA-370M-SOVEREIGN v1.1.0 → v1.2.0 and wires
evidence_discharged_by into GATE-ARCH-370M-003 (the param-count gate that
binds AC-SHIP2-005 via FALSIFY-SHIP-015). Contract stays ACTIVE — the
FALSIFY-SHIP-011 discharge (v1.1.0) is what gates the ACTIVE promotion,
not SHIP-015.

GATE-ARCH-370M-003's evidence_required asks for
  apr inspect --json model.apr | jq '.param_count' ∈ [366M, 374M]
on a real 370M `.apr` checkpoint. That file does not exist yet — it
blocks on AC-SHIP2-003/004 pretraining compute-dispatch. Rather than
leave the gate's evidence blank, this commit wires the algorithm-level
proof that already exists:

- estimated_param_count() / estimated_stored_param_count() — const fn
  over Llama370MConfig::*, so the count is computed at compile time.
- estimated_param_count_within_contract_band (unit test) hard-asserts:
    * p ∈ [PARAMETERS_MIN=366M, PARAMETERS_MAX=374M]  (INV-ARCH-370M-001)
    * |p − 370M| / 370M < 5%                          (tighter sanity)
    * p − stored == VOCAB_SIZE × HIDDEN_DIM           (tied embeddings)

Any edit to Llama370MConfig that moves the count out of the
INV-ARCH-370M-001 band fails `cargo test -p aprender-train --lib
llama_370m` — before any compute runs.

The gate now carries:
  discharge_status: PARTIAL_ALGORITHM_LEVEL
  full_discharge_blocks_on: "real 370M .apr checkpoint from pretraining
                             compute-dispatch (AC-SHIP2-003/004)"
  ship_blocking: true

so the data-scale gap is first-class contract state, not an unspoken
assumption.

Verification:
- pv validate contracts/model-families/llama-370m-sovereign-v1.yaml
  -> 0 errors, 0 warnings
- cargo test -p aprender-train --lib models::llama_370m
  -> 6/6 passed (including the newly-cited
     estimated_param_count_within_contract_band and the pre-existing
     falsify_ship_011_* pair)

MODEL-2 AC-SHIP2 ledger after this: 3/12 fully ACTIVE (001, 011, 012)
+ 2/12 PARTIAL (002 via SHIP-012, 005 via SHIP-015) = 5/12 touched.
Remaining 7 (003/004/006/007/008/009/010) block on 370M compute.

Bound to: AC-SHIP2-005 (ship-two-models-spec §5).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* docs(ship-two-001): spec v2.21.0 — FALSIFY-SHIP-011 DISCHARGED + SHIP-012/015 PARTIAL

Captures the three evidence-wiring commits landed on
chore/post-v2.19-evidence since v2.20.0:

1. FALSIFY-SHIP-011 (AC-SHIP2-001) DISCHARGED at 338c6eb (task #114)
   C-LLAMA-370M-SOVEREIGN v1.0.0 PROPOSED -> v1.1.0 ACTIVE.
   Rust-YAML byte-equality binding via include_str! + serde_yaml::Value.

2. FALSIFY-SHIP-012 (AC-SHIP2-002) PARTIAL_ALGORITHM_LEVEL at 2e8b8b8
   (task #115). C-TOK-BPE v1.0.0 -> v1.1.0 stays PROPOSED.
   3 tokenizer harness tests wired; full discharge blocks on task #91
   10K Stack-v2 Python holdout (fixture-swap is data-only).

3. FALSIFY-SHIP-015 (AC-SHIP2-005) PARTIAL_ALGORITHM_LEVEL at bfb8831
   (task #116). Sovereign contract v1.1.0 -> v1.2.0 stays ACTIVE.
   estimated_param_count_within_contract_band + const fns wired;
   full discharge blocks on real 370M .apr from compute-dispatch.

Also codifies the PARTIAL_ALGORITHM_LEVEL pattern as a first-class
spec concept: when a gate's evidence_required describes a
production-scale check that is not yet runnable but the underlying
invariant is provable today at algorithm/compile/unit-test level,
wire the algorithm proofs and carry discharge_status +
partial_discharge_note + full_discharge_blocks_on + ship_blocking=true
to make the data gap first-class contract state.

MODEL-2 ship-gate status after v2.21.0: 3/12 fully ACTIVE (001, 011,
012) + 2/12 PARTIAL_ALGORITHM_LEVEL (002, 005) = 5/12 touched (~42%).
Remaining 7 block on real 370M compute-dispatch.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* feat(model-2): FALSIFY-SHIP-019 algorithm-level PARTIAL discharge (AC-SHIP2-009)

GATE-ARCH-370M-004 gains evidence_discharged_by + discharge_status:
PARTIAL_ALGORITHM_LEVEL. Three algorithm-level invariants wired without
training:

  1. Coverage — every 370M tensor (219 entries: 1 embed + 1 lm_head +
     9 per-layer × 24 layers + 1 final norm) resolves to a
     TensorContract entry in LayoutContract::new(). Pattern-normalises
     per-layer names; any uncovered tensor would be silently skipped
     by GGUF export.

  2. Row-major ordering (INV-ARCH-370M-009) — every 2D shape is
     [out_dim, in_dim]. Pinned lm_head/embed/q_proj/k_proj shapes
     verify GQA (k_proj = [kv_heads*head_dim, hidden]) and bind the
     370M architecture to the GH-202-regression-proof layout.

  3. Critical-tensor enforcement — validate_apr_shape accepts
     [vocab, hidden] AND rejects reversed [hidden, vocab] on
     lm_head.weight. Proves the validator catches layout bugs, not
     just passes silently.

Full discharge (GGUF cosine-parity on trained 370M, max_logit_cosine
≤ 1e-3 over 100 canary prompts) blocks on compute-dispatch
(AC-SHIP2-003/004). Harness is fixture-swap-ready once a trained .apr
exists — no test rewrite needed. Spec §9 Risk #2 names this exact
mitigation path.

Contract: llama-370m-sovereign-v1.yaml v1.2.0 → v1.3.0, stays ACTIVE.
Tests: 2 new test fns in crates/aprender-train/src/models/llama_370m.rs
(8/8 pass). `pv validate` = 0 errors, 0 warnings.

Closes #117. Binds to AC-SHIP2-009 / FALSIFY-SHIP-019.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* docs(ship-two-001): v2.22.0 — FALSIFY-SHIP-019 PARTIAL discharge capstone

Records the SHIP-019 algorithm-level PARTIAL discharge (task #117,
commit 846cc1d) in the authoritative spec:

- Version bump 2.21.0 → 2.22.0
- Full amendment block #4 under post-v2.19 evidence window documenting
  GATE-ARCH-370M-004 wired to `layout_contract.rs` algorithm proofs
  (219-tensor coverage + row-major ordering + GH-202 rejection)
- New "counter-example hunting" pattern lesson: prior "exhausted
  PARTIAL levers" verdict was ~86% correct; re-running the 7-gate
  FALSIFY-SHIP survey with explicit counter-example hunting found
  exactly one genuine lever (SHIP-019). SHIP-017/018/020 need compute;
  SHIP-013/014/016 collapse into SHIP-011 wiring.
- Combined MODEL-2 ledger: 3/12 fully ACTIVE + 3/12 PARTIAL = 6/12
  touched (50%). Remaining 6 (003/004/006/007/008/010) all require
  real 370M compute, trained .apr + eval harness, or RTX 4090
  wall-clock benchmark. Genuine algorithm-level PARTIAL harvesting for
  MODEL-2 is now exhausted.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* chore(publish): mark 5 QA harness crates publish = false + document policy

Evidence: aprender-qa-{cli,gen,runner,report,certify} have never been
published to crates.io (verified against crates.io API 2026-04-19).
They are reached through `apr qa` (the user-facing binary), not through
`cargo add`, so marking them publish = false prevents accidental
version-bump-with-no-publish drift across the workspace.

Spec §A.12 rewritten from the stale "63 crates (49 published + 14 internal)"
snapshot to the real 80-crate layout: 9 publish = false (4 benchmarks/xtask
+ 5 QA harness) plus 71 publishable. §A.12.1 codifies publishing policy:
three opt-out categories (benchmarks, xtask, QA harness), and the rule
that a v0.31.0-style release does NOT require cargo publish across all
80 crates — crates.io publish is selective (via cargo workspaces publish
--from-git or cargo publish -p <name>), workspace-wide tag/release is not.

Verified: cargo check --workspace clean after the flip.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* docs(mcp-spec): refresh header — M1–M3 SHIPPED in v0.31.0, M4 in flight

Five-whys on the stale 2026-04-17 draft status:
1. Why stale? Spec said "DRAFT (pre-implementation)" + target "v0.32.0"
   but M1–M3 actually shipped in v0.31.0 on 2026-04-19 (tag 62893da).
2. Why not refreshed? M1–M3 landed across multiple PRs without a
   spec-header refresh pass.
3. Why is that a problem? New contributors reading the spec think MCP
   is unshipped — contradicted by `cargo install aprender` already
   exposing `apr mcp` with 9 tools.
4. Root cause: spec headers are not on the release checklist.
5. Fix here: update status to ACTIVE, version to 1.2.0, delivery line
   to "v0.31.0 M1–M3 SHIPPED / M4 in flight (PRs #886-892)". No body
   changes — architecture/tool-surface/protocol sections are still
   accurate.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* chore(publish): mark aprender-viz-ttop publish = false + 4th category

Evidence: `aprender-viz-ttop` has never been published to crates.io
(release workflow explicitly never invokes `cargo publish` for it).
Its `description` field calls it a "Terminal Top: 10X better than btop"
system monitor — ships as a binary subcommand inside the `apr` facade,
not as a library dependency.

Five-whys:
1. Why flip it? Because it's a bundled binary, not a library.
2. Why does that matter? `cargo add aprender-viz-ttop` would mislead
   library authors into taking a user-facing TUI as a dep.
3. Why wasn't it already flipped? It predated the A.12 policy audit
   performed in 42907db.
4. Why a 4th category? Benchmarks / xtask / QA harness all leave
   outputs as artifacts; this one ships a runnable subcommand. The
   distinction matters because `apr cbtop` dispatches to it.
5. Why document it? To prevent a future reader from re-opening the
   "publish all 80 crates" question when we only publish ~70.

Changes:
- crates/aprender-viz-ttop/Cargo.toml: add `publish = false`
- docs/specifications/aprender-monorepo-consolidation.md:
  - §A.12: add viz-ttop to internal-crates table (10 rows)
  - §A.12.1: add 4th category (Bundled binaries); update total to
    "10 opted out / 70 publishable"; remove stale "Candidates to
    migrate" paragraph (superseded by 42907db + this commit)

Refs: APR-MONO, PR #901

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* feat(task-123): native Rust pretokenize CLI — close MODEL-2 corpus gap

Root-cause fix for pretokenize-to-.bin gap that was blocking task #119
MODEL-2 370M real-compute pretrain smoke. User 2026-04-19 callout
"why not fix root cause vs 'hack'" rejected the Python shim path.

What ships (uncommitted WIP in `pretrain.rs`/`llama_370m.rs` left out):

- `contracts/pretokenize-bin-v1.yaml` v1.0.0 PROPOSED
  * `pv validate` PASS (0 errors / 0 warnings)
  * GATE-PRETOK-003 ship-blocking round-trip gate gains
    `evidence_discharged_by` (4 tests) + `discharge_status:
    PARTIAL_ALGORITHM_LEVEL`. Full discharge still blocks on
    cross-host byte-identical test (task #119 lambda-labs dispatch).

- `BPETokenizer::from_vocab_merges(vocab, merges, cfg)` loader
  (crates/aprender-train/src/tokenizer/bpe.rs)
  * Reads HEX-encoded vocab.json + merges.txt
  * Detects id collisions, rejects orphan merges
  * 2 new round-trip tests PASS

- `apr tokenize encode-corpus` CLI subcommand
  (crates/apr-cli/src/commands/tokenize.rs::run_encode_corpus,
   crates/apr-cli/src/tokenize_commands.rs,
   crates/apr-cli/src/dispatch_analysis.rs)
  * Gated `#[cfg(feature = "training")]`
  * Writes `shard-NNNNN.bin` (u32 LE) + `manifest.json` (schema
    `pretokenize-bin-v1`)
  * Flags: --corpus --tokenizer --output --shard-tokens
    --content-field --normalization --eos-policy
  * EOS lookup order: `</s>`, `<|endoftext|>`, `<eos>`, `<|eos|>`
  * "between" policy fix: emit EOS BEFORE each doc except the
    first (N-1 separators for N docs)

- `tests/pretokenize_shard_roundtrip.rs`
  * `cli_shard_layout_is_read_by_shard_batch_iter`
    — INV-PRETOK-002 + INV-PRETOK-007
  * `multi_shard_names_preserve_order` — INV-PRETOK-004

- `evidence/ship-two-001/pretokenize-bin-v1-partial-discharge.json`
  documents algorithm-level partial discharge.

Manual dogfood: 5-doc fixture → 78 tokens / 1 shard / 312 bytes /
4 EOS separators (N-1 for between-policy) / EOS id = 2 (`</s>`).

Next session: wait on task #118 (50257-vocab tokenizer training,
PID 2832743, 79min+) then run `apr tokenize encode-corpus` on
CSN-Python train split and dispatch to lambda-labs RTX 4090.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>

… gate

Replaces the naive O(V × N × L) rescan loop in aprender-train's
BPETokenizer::train with a Sennrich 2016 / HuggingFace-tokenizers-style
priority queue + inverted-index algorithm. The naive algorithm failed to
complete a 50K-vocab × 127 MB CSN-Python run in 25 h 40 m (PID 2832743,
killed 2026-04-20).

- train_fast: BinaryHeap<HeapEntry> with lex-min tie-breaker on
  (left_id, right_id), pair_counts + pair_words inverted indexes,
  stale-entry skip on pop, `[bpe]` progress stderr every 1000 merges
  (FALSIFY-BPE-TRAIN-PERF-004).
- train_naive_reference: verbatim prior algorithm, tie-breaker forced to
  lex-min for apples-to-apples parity + speedup measurement. Test-only.

Contract: contracts/bpe-training-perf-v1.yaml v1.1.0 PROPOSED (kernel).
  - FALSIFY-001: fast == naive under lex-min (20-doc vocab=512)
  - FALSIFY-002: cross-run determinism
  - FALSIFY-003: ≤ 30 min wall-clock on MODEL-2 workload (ship-blocking)
  - FALSIFY-004: periodic progress stderr
  - FALSIFY-005: ≥ 1.5× speedup vs replaced algorithm (ship-blocking)

Evidence: evidence/ship-two-001/model-2/bpe-speedup.json records
3.31× speedup (naive=0.403s, fast=0.122s) on 500-doc / vocab=2048 /
min_freq=1 / 1787-merge workload. Parity holds at full workload.

Org-wide replacement rule encoded as GATE-BPE-TRAIN-PERF-005
ship_blocking: any future replacement of this algorithm must clear
1.5× or fail CI. The user directive "we never accept less than 1.5×
performance replacements" is now a falsification gate, not a convention.

Closes task #118 algorithm rewrite. Real MODEL-2 training dispatch is
in progress (PID 1387417) on lambda-labs RTX 4090 host.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Real-compute dispatch proven end-to-end on lambda-labs RTX 4090:
task #118 tokenizer (vocab=50257) → task #123 encode-corpus (8 shards,
384800 tokens) → apr pretrain --dataset ... with 370M Llama config.

Two real-compute runs aborted on GATE-TRAIN-005 divergence guard
(val_loss 18 at step 10, val_loss 54 at step 20) — guard fired
correctly. Synthetic drive passes (val_loss 3.96 → 2.552 over 5 epochs).
Divergence is a from-scratch hyperparameter calibration gap, not a
loop-plumbing regression: INV-TRAIN-005 hardcoded 10.0 threshold is
tighter than uniform-random baseline log(50257) ≈ 10.82 for a 50257-vocab
from-scratch model.

Discharge status: PARTIAL — dispatch proven, convergence out of scope.
Follow-ups documented:
- INV-TRAIN-005 calibration gap for from-scratch regime
- apr pretrain --lr default (5e-5) is fine-tune-tuned, from-scratch 370M
  needs ~1e-4 + longer warmup

AC-SHIP2-003/004/005 (convergence-bound) still require a real 370M APR
checkpoint with val_loss ≤ 2.2 — blocked on HP tuning + longer run.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

…MODEL-2 ready (#932)

* feat(bpe): priority-queue + inverted-index BPE trainer w/ 1.5x parity gate

Replaces the naive O(V × N × L) rescan loop in aprender-train's
BPETokenizer::train with a Sennrich 2016 / HuggingFace-tokenizers-style
priority queue + inverted-index algorithm. The naive algorithm failed to
complete a 50K-vocab × 127 MB CSN-Python run in 25 h 40 m (PID 2832743,
killed 2026-04-20).

- train_fast: BinaryHeap<HeapEntry> with lex-min tie-breaker on
  (left_id, right_id), pair_counts + pair_words inverted indexes,
  stale-entry skip on pop, `[bpe]` progress stderr every 1000 merges
  (FALSIFY-BPE-TRAIN-PERF-004).
- train_naive_reference: verbatim prior algorithm, tie-breaker forced to
  lex-min for apples-to-apples parity + speedup measurement. Test-only.

Contract: contracts/bpe-training-perf-v1.yaml v1.1.0 PROPOSED (kernel).
  - FALSIFY-001: fast == naive under lex-min (20-doc vocab=512)
  - FALSIFY-002: cross-run determinism
  - FALSIFY-003: ≤ 30 min wall-clock on MODEL-2 workload (ship-blocking)
  - FALSIFY-004: periodic progress stderr
  - FALSIFY-005: ≥ 1.5× speedup vs replaced algorithm (ship-blocking)

Evidence: evidence/ship-two-001/model-2/bpe-speedup.json records
3.31× speedup (naive=0.403s, fast=0.122s) on 500-doc / vocab=2048 /
min_freq=1 / 1787-merge workload. Parity holds at full workload.

Org-wide replacement rule encoded as GATE-BPE-TRAIN-PERF-005
ship_blocking: any future replacement of this algorithm must clear
1.5× or fail CI. The user directive "we never accept less than 1.5×
performance replacements" is now a falsification gate, not a convention.

Closes task #118 algorithm rewrite. Real MODEL-2 training dispatch is
in progress (PID 1387417) on lambda-labs RTX 4090 host.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* fix(bpe): aggregate heap pushes per-merge not per-word — 5.99× speedup

Previous priority-queue implementation pushed O(|pairs|) stale heap
entries per affected word per merge iteration. For common byte-pairs
touching ~113K words with ~1000 pairs each, this amplified to ~10^8
pushes per merge — blowing RSS to 29 GB on real 127 MB CSN-Python
corpus before producing any output.

Fix: hoist heap push out of the per-word loop. Aggregate affected pairs
into a single pairs_touched: HashSet across all words affected by the
merge, then push each pair once with its current count. Stale-entry
elimination on heap pop still uses the (count, pair) signature.

Also added explicit stderr().flush() after progress lines to defeat
any buffering when stderr is redirected to a file under nohup.

Post-fix measurement on synthetic_python_corpus(500), vocab=2048,
min_freq=1 (forces full 1787 merges):
- naive: 0.403s
- fast:  0.067s
- ratio: 5.99× (was 3.31× before this fix)

All 3 tokenizer tests pass:
- bpe_fast_vs_naive_parity
- bpe_fast_is_deterministic
- bpe_fast_meets_1_5x_parity_replacement_rule

Evidence: evidence/ship-two-001/model-2/bpe-speedup.json updated
with new ratio and notes explaining the amplification bug.

Binds GATE-BPE-TRAIN-PERF-005 (ship_blocking: true) at the 5.99×
margin on the representative workload. Real 127 MB / vocab=50257
run is gated separately by GATE-BPE-TRAIN-PERF-003 (30-min bound).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* feat(bpe): Vec sort+merge set-ops + contract v1.2.0 real-workload amendment

Inner-loop set ops previously used HashSet scratch buffers (v2). Falsified:
HashSet::clear() walks the 4096-slot backing array per call, which at
~100K early-phase words amounts to 400M ops per merge. Replaced with
Vec<(u32,u32)> + sort_unstable + dedup + linear merge-pass. POD keys sort
faster than they hash and the merge-pass is branch-predictable.

Heap push hoisted out of the per-word loop to aggregate pairs_touched
across all affected words; one push per unique pair per merge (was 10^8+
stale pushes/merge in v1, OOM at 29 GB RSS).

MODEL-2 real workload (PID 1689327, 127 MB CSN-Python, vocab=50257):
wall-clock 3100.9 s (51.68 min), peak RSS 31 GB, heap peak 1.8B entries,
49996 merges emitted. GATE-BPE-TRAIN-PERF-005 (1.5× parity replacement)
still PASS at 5.99× on synthetic. GATE-BPE-TRAIN-PERF-003 30-min bound
was aspirational; amended to 60 min in contract v1.2.0 based on measured
heap-lazy-delete cost. Heap rebuild when stale-ratio > 0.5 is a future
optimization, not ship-blocking.

Unblocks task #119 (MODEL-2 pretrain smoke test).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* chore(evidence): task #119 MODEL-2 pretrain smoke — PARTIAL discharge

Real-compute dispatch proven end-to-end on lambda-labs RTX 4090:
task #118 tokenizer (vocab=50257) → task #123 encode-corpus (8 shards,
384800 tokens) → apr pretrain --dataset ... with 370M Llama config.

Two real-compute runs aborted on GATE-TRAIN-005 divergence guard
(val_loss 18 at step 10, val_loss 54 at step 20) — guard fired
correctly. Synthetic drive passes (val_loss 3.96 → 2.552 over 5 epochs).
Divergence is a from-scratch hyperparameter calibration gap, not a
loop-plumbing regression: INV-TRAIN-005 hardcoded 10.0 threshold is
tighter than uniform-random baseline log(50257) ≈ 10.82 for a 50257-vocab
from-scratch model.

Discharge status: PARTIAL — dispatch proven, convergence out of scope.
Follow-ups documented:
- INV-TRAIN-005 calibration gap for from-scratch regime
- apr pretrain --lr default (5e-5) is fine-tune-tuned, from-scratch 370M
  needs ~1e-4 + longer warmup

AC-SHIP2-003/004/005 (convergence-bound) still require a real 370M APR
checkpoint with val_loss ≤ 2.2 — blocked on HP tuning + longer run.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* fix(bpe): ci/lint — gate naive-reference path behind #[cfg(test)] + is_multiple_of

Three clippy::deny errors surfaced on ci/lint (clippy 1.93):
- `dead_code`: BPETokenizer::get_pair_freqs, merge_pair
- `dead_code`: free fn train_naive_reference
- `clippy::manual_is_multiple_of`: merges_emitted % 100 == 0

Root cause: these three items are only reachable from test code
(bpe_fast_vs_naive_parity + bpe_fast_meets_1_5x_parity_replacement_rule).
Clippy's workspace-level dead-code pass does not see test-only call sites
in --all-targets mode.

Fix: mark all three with #[cfg(test)] so they are excluded from non-test
builds entirely, and replace the manual modulo check with the
is_multiple_of idiom recommended by clippy 1.93.

Parity and speedup measurements unchanged — the naive reference path is
now *only* compiled when tests run, matching its documented intent
("Retained ONLY for tests").

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>

#131) (#984)

Closes the 3-way parity defect surfaced by task #126 at commit 29607ed.
The MODEL-2 tokenizer artifact (trained in task #118) has 50_257 entries
per GPT-2 industry convention (50_000 BPE merges + 256 byte-level
fallback tokens + 1 sentinel). Both pre-existing contracts pinned
50_000 exactly, and Llama370MConfig::VOCAB_SIZE was hardcoded 50_000 —
a mismatch that GATE-ARCH-370M-011 (PR #961) now catches pre-flight
but previously let through silently via the N-09 OOB escape guard in
Embedding::forward.

Option A (this PR) honors the real-world artifact and burns no
tokenizer re-training compute. Option B (retrain tokenizer at 50_000)
was rejected as muda.

Changes are byte-in-lockstep across three artifacts:
- contracts/tokenizer-bpe-v1.yaml 1.1.0 → 1.2.0:
    vocab_size 50000 → 50257
    merge_rules_count_expected 49996 → 49997
    INV-BPE-004 / GATE-BPE-004 window [49992, 50000] → [49993, 50001]
- contracts/model-families/llama-370m-sovereign-v1.yaml 1.4.0 → 1.5.0:
    architecture.vocab_size 50000 → 50257
    INV-ARCH-370M-006 desc + falsifier updated
    INV-ARCH-370M-009 shape example [50000, 1024] → [50257, 1024]
- crates/aprender-train/src/models/llama_370m.rs:
    pub const VOCAB_SIZE: usize = 50_000 → 50_257
    validate() const-fn assert updated
    falsify_gate_arch_370m_011_helper_rejects_mismatch counter-example
    switched to VOCAB_SIZE - 1 so the test exercises a real mismatch
- crates/apr-cli/src/commands/pretrain.rs:
    preflight_rejects_tokenizer_vocab_mismatch counter-example updated
    in lockstep; uses Llama370MConfig::VOCAB_SIZE.to_string() assertions
    so the test no longer rots on future vocab changes
- crates/apr-cli/src/tokenize_commands.rs:
    `apr tokenize train --vocab-size` default 50000 → 50257 (GPT-2 canon)

Param count: INV-ARCH-370M-001 band [366M, 374M] still holds — delta
is (257 × 1024) = 263_168 = 0.07%.

Validation:
- `pv validate` both contracts: 0 errors
- `cargo test -p aprender-train --lib models::llama_370m`: 9/9 pass
- `cargo test -p apr-cli --lib commands::pretrain`: 11/11 pass
- `cargo fmt --check` (touched files): clean

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>