feat(ship-two-001): SPEC v2.19.0 — teacher shipped + MODEL-2 scaffold + pre-upload gates#882
Merged
Merged
Conversation
Bump all 78 workspace crates from 0.30.0 to 0.31.0. 13,026 tests passing. Clean workspace build verified. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…-050) Ship three contract-first invariants eliminating unconditional diagnostics from the decode/prefill hot path on Qwen2.5-Coder-1.5B Q4_K_M (RTX 4090): - F-DECODE-HOTPATH-001: remove 7 per-token /tmp fs::write calls 184 -> 382 tok/s, 2.07x speedup, 0.6x -> 1.3x Ollama (Grade D -> B) - F-DECODE-HOTPATH-002: remove realizr#198 first-5/6-token eprintlns 381.7 -> 391.8 tok/s (+2.6%), parity 1.30x -> 1.33x - F-DECODE-HOTPATH-003: gate PMAT-450 prefix-cache eprintlns on config.trace parity 1.33x -> 1.36x Grade B (5 sites across generate_1.rs + generate_2.rs) - Class-of-bug sweep: remove PAR-050 first-15-tokens eprintln from legacy generate_full_cuda_with_cache path (apr trace --gpu fallback, apr run CLI) Forest-level invariant: zero unconditional eprintln/fs::write in decode/prefill boundary. All diagnostics gated behind config.trace (per-gen) or OnceLock env vars (per-token: DECODE_TIMING, GPU_DEBUG, KV_FINGERPRINT). Disclosure: generate_2.rs also carries previously-unstaged SPEC-MOE-APR-001 serial-prefill branch and Config::head_dim() refactor (6 call sites) from a parallel work thread; they are bundled here because the HP-series changes overlap the same hunks. Contracts (new): - contracts/decode-hot-path-zero-syscalls-v1.yaml - contracts/decode-hot-path-first-tokens-diagnostic-v1.yaml - contracts/decode-hot-path-prefix-cache-diagnostic-v1.yaml Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…phed per-op hotspots
Five-whys + falsification tests documenting a measurement-methodology gap
in 'apr profile --granular':
(1) 'Decode throughput: 429 tok/s' comes from a graph-captured
generate_gpu_resident run (production path).
(2) 'Per-Operation Hotspots' table comes from a second pass with
SKIP_CUDA_GRAPH=1 set (kernel.rs:371) so BrickProfiler can attach
cudaEventRecord hooks per kernel.
Consequence: the headline 'AttentionScore 37.5%' figure includes ~2µs
of per-launch host overhead per call that graph replay eliminates.
Fusion ROI estimates based on (num_kernels * launch_overhead) are
6-7x too optimistic for the graphed path.
Proof obligations:
- Hotspot table header labeled 'ungraphed' / 'SKIP_CUDA_GRAPH'
- Graph dispatch cost reported as separate line
(graphed_us/token - sum_kernel_compute_us/token) / num_graph_nodes
- Fusion savings estimator uses graph-node overhead, not launch overhead
Blocker cleared for: picking next 1.5x Ollama parity lever with
trustworthy measurements. apr qa currently reports 1.47x parity on
Qwen2.5-Coder-1.5B Q4_K_M — 98% of target, within noise of 1.5x.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…le output
Satisfies FALSIFY-PROF10-001 (contracts/profile-graph-vs-per-op-methodology-v1.yaml).
Three output changes prevent users from picking fusion targets based on
an ungraphed measurement and trigger-happy WARNING messages:
1. 'Per-Operation Hotspots' header now reads 'Per-Operation Hotspots
(ungraphed — SKIP_CUDA_GRAPH=1)'. Also adds a 3-line footnote under
the table noting per-op times include ~1-2µs/call launch overhead
and should not be used to estimate graphed-mode fusion wins.
2. 'Category Summary' renamed 'Category Summary (ungraphed)' — the
percentages are computed from ungraphed kernel times.
3. 'Kernel Launch Overhead' renamed 'Non-Kernel Host Overhead'. The
metric is (graphed_per_token_decode − ungraphed_per_token_kernel_sum),
which captures argmax sync, H2D/D2H copies, graph-replay dispatch,
and uncounted kernels — NOT launches that would benefit from fusion.
Rewording the WARNING:
before: 'WARNING: >20% overhead — consider kernel fusion' (red)
after: '>40%: investigate sampling sync / graph replay' (red)
'>20%: per-token sync or missed instrumentation' (yellow)
'<=20%: kernels dominate decode time' (green)
Verified on Qwen2.5-Coder-1.5B-Instruct Q4_K_M / RTX 4090:
- Decode throughput: 439.5 tok/s (unchanged, within noise)
- Non-Kernel Host Overhead: 862µs (37.9% of decode time)
- Hotspot table correctly labeled ungraphed
Phase 2 (compute graph-node dispatch cost as separate metric) deferred
to a follow-up; requires exposing graph node count from CudaExecutor.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Contract-first design for the next lever on the path to 1.5x Ollama
parity. Currently 1.47x on Qwen2.5-Coder-1.5B Q4_K_M / RTX 4090 per
apr qa, and F-PROFILE-010 labels 37.9% of graphed decode time (862
microseconds/token) as Non-Kernel Host Overhead.
Five-Whys pinpoints reduces.rs:255 `stream.synchronize()` + 4-byte
copy_to_host in `gpu_argmax` as the stall: each token requires a
full GPU drain before the CPU can compute the next embedding and
start the next H2D. The graph replay cannot queue its successor
until the host advances.
Proposed fix (three coupled changes):
1. Upload embedding weights to GPU, add gpu_embed_lookup kernel
(Q4K gather or F16 strided copy).
2. Recapture decode graph with embed-lookup as FIRST node and
argmax as LAST node; graph input becomes a u32 token_id_buf.
3. Batch stop-token detection every N=8 tokens via async D2H
ring-buffer, trimming overshoot before returning to caller.
Falsification:
- FALSIFY-GRS-001: token-for-token parity against pre-change greedy
decode (bit-identical, not just similar text).
- FALSIFY-GRS-002: apr qa Ollama parity median across 3 runs >= 1.50x.
- FALSIFY-GRS-003: stop-token latency bounded to N tokens past actual.
Risk R-GRS-004 explicitly permits the contract to falsify its own
ROI estimate: if throughput does not cross 1.5x after implementation,
the 862us overhead was mostly compute (not sync stall) and the lever
choice was wrong.
Related: F-PROFILE-010 (methodology), F-DECODE-HOTPATH-001/002/003
(per-token diagnostic hygiene).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
DECODE_TIMING=1 dogfooding run on Qwen2.5-Coder-1.5B Q4_K_M /
RTX 4090 falsifies the contract's why_4 before any implementation:
[DECODE-TIMING] pos=N: embed=0-1us gpu=2250-3344us total~2500us
CPU embed_into is sub-microsecond. The 862us "Non-Kernel Host
Overhead" measured by F-PROFILE-010 is NOT host-side sync waiting;
it is graph-internal dispatch overhead.
862us / 647 graph nodes = 1.33us per graph-node dispatch
(vs initially assumed ~0.3us/node in F-PROFILE-010)
GPU-resident token flow would save at most 1-2us per token (the
trivial CPU work between iterations), not the 862us the contract
targeted. R-GRS-004 explicitly permitted this falsification.
The real lever (revealed by the falsification): kernel fusion to
reduce graph node count. At 1.33us/node, fusing 100 kernels saves
133us/token (~6% throughput). This becomes the next contract.
Status changed PROPOSED -> FALSIFIED. Implementation plan retained
for historical reference. Contract-first design caught the wrong
premise in 5 minutes of dogfooding instead of hours of coding.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…flight gate
Closes SHIP-TWO-001 §12.7 pre-upload defense-in-depth. Three new gates
fire BEFORE any network I/O, catching the exact class of bug that
produced a 30.46 GiB F32 safetensors artifact against an fp16 manifest:
FALSIFY-PM-007 — safetensors header dtype Poka-Yoke
FALSIFY-PUB-EXTRA-009 — corrupt manifest sha256 aborts publish
FALSIFY-PUB-EXTRA-010 — preflight_validate_manifest defined + called 3x
before any publish_format
New `apr validate-manifest` subcommand (contracts/publish-manifest-v1.yaml
v1.1.0) + pre-flight gate in scripts/ship-two-001/ex-04-upload-hf.sh.
End-to-end verified on real 15 GiB teacher artifact (evidence/ship-two-001/
ex-04-preflight-gate-smoketest.json).
Contracts:
- publish-manifest-v1.yaml v1.0.0 → v1.1.0 (+ PM-007)
- apr-cli-publish-extra-v1.yaml v1.1.0 → v1.2.0 (+ -009/-010)
- apr-model-qa-v1.yaml v1.0.0 → v1.1.0 (+ --require-golden-output)
Spec:
- SPEC-SHIP-TWO-001 v2.0.0 → v2.5.0
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds the third-format ship-blocker gate mirroring PM-007
(safetensors) — catches manifests that lie about GGUF quantization
before 30+ GiB of mis-labelled bytes move across the network.
Design: predominant non-float tensor type is the authoritative
signal. `general.file_type` is retained as a fallback only — real
llama.cpp quantize output (e.g. our 8 GiB teacher GGUF) has shipped
with stale ftype=0 despite fully Q4_K tensors, so trusting the
metadata_kv field would force a false FAIL on an artifact every
inference engine happily consumes.
New API:
• read_gguf_signature(path) — reads metadata_kv + tensor_metadata
• predominant_quant_type(counts) — majority non-float type,
falling back to majority float only when all tensors are float
• expected_ggml_tensor_type(quant) — manifest string → GGML type
(ggml-common.h enum ggml_type)
• ggml_type_name(t) — u32 → "Q4_K" etc.
Verified on real artifact:
$ apr validate-manifest paiml-qwen2.5-coder-7b-apache-q4k-v1-gguf.yaml \
--artifact qwen2.5-coder-7b-instruct-q4k.gguf
[PASS] FALSIFY-PM-008: predominant tensor type = 12 (Q4_K)
matches quantization 'q4_k'
(note: general.file_type=0=ALL_F32 is stale)
Tests: 15 unit tests (10 original ftype-only + 5 new for
tensor-authoritative path, including the real teacher scenario
of Q4_K tensors + stale ftype=0).
Contract: publish-manifest-v1.yaml v1.2.1 — PM-008 entry rewritten
to describe tensor-authoritative semantics.
Refs: SPEC-SHIP-TWO-001 §12.7.2 (ship-blocker class)
Closes: task #69
Contributor
Author
Extension: FALSIFY-PM-008 (GGUF tensor-type Poka-Yoke)Pushed Design: tensor types are authoritativeReal-world finding: our 8 GiB So PM-008:
Verified on real artifactUnit tests15 total (
Contract
|
…ritative Amends SPEC-SHIP-TWO-001 to v2.6.0 documenting the PM-008 design pivot from ftype-only to tensor-type-authoritative GGUF validation. Why: real-world teacher GGUF (PM-008 discharge run, 2026-04-18) ships with `general.file_type=0` (ALL_F32) despite every weight being Q4_K. PM-008's first version would have false-FAILed the teacher. Revised version trusts per-tensor ggml_type histogram; stale ftype is surfaced as a diagnostic note, not a ship-blocker. Changes: - Frontmatter: 2.5.0 → 2.6.0 + v2.6.0 amendment paragraph - §12.7: paragraph describing PM-008 two-tier authority + llama.cpp bug - §12.7.2: table row for predominant-tensor-type mismatch FAIL condition - §11.1: contract bump v1.1.0 → v1.2.1 (publish-manifest-v1.yaml) - Test count 21 → 36 Memory: `feedback_gguf_file_type_stale.md` captures the ftype stale footgun for future GGUF audits. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Contributor
Author
Extension 3: v2.6 Spec Amendment (c2729a6)Spec What changed:
Why the pivot: Real-world teacher GGUF ships with PR now ready for review — all 3 commits (PM-007 + PM-008 + v2.6 spec) tell one coherent story. |
Closes the three-format ship symmetry (PM-007 safetensors + PM-008 gguf +
PM-009 apr) — every shipped format now has a pre-flight Poka-Yoke gate
that aborts BEFORE any network I/O when the staged artifact disagrees
with the manifest.
v1.0 scope (pragmatic MVP): verify first 4 bytes of .apr artifact match
one of APR\0 / APRN / APR1 / APR2 (the four APR magic variants recognised
by aprender-registry::format::parse_apr_header). Catches "GGUF renamed
.apr" or "safetensors staged as .apr" ship-blockers.
Expansion path (v1.1, deferred): parse APR v2 tensor index via
aprender::format::v2::AprV2Reader, compute predominant non-float dtype,
compare to manifest.quantization (symmetric to PM-008 tensor authority).
Defer until real-world FAIL justifies complexity — same discipline as
PM-008's initial ftype-only scope.
Changes:
- crates/apr-cli/src/commands/validate_manifest.rs
- Added check_apr_magic + read_apr_magic + apr_magic_name + ascii_or_hex
- Wired into run() dispatch after PM-008
- Module docstring updated (9 gates listed)
- +9 unit tests: 3 happy-path magic variants, 2 DEFERRED paths,
3 FAIL paths (GGUF-as-APR, safetensors-as-APR, empty file),
1 name-table coverage
- contracts/publish-manifest-v1.yaml v1.2.1 → v1.3.0
- Added FALSIFY-PM-009 entry with authoritative magic list
- Changelog updated
- docs/specifications/aprender-train/ship-two-models-spec.md v2.6.0 → v2.7.0
- Frontmatter + v2.7.0 amendment paragraph
- §12.7 PM-009 description with dogfood verdict
- §12.7.2 table row for .apr magic mismatch FAIL
- §11.1 contract reference bumped to v1.3.0
- Test count 36 → 45
Dogfood: real teacher .apr (8 GiB, qwen2.5-coder-7b-instruct-q4k.apr)
verdict PASS ("apr magic = APR\0 (v2) (valid)").
Tests: cargo test -p apr-cli --lib validate_manifest → 45/45 PASS
(15 PM-008 + 9 PM-007 + 9 PM-009 + 12 other).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Contributor
Author
Extension 4 — FALSIFY-PM-009 APR magic-bytes Poka-Yoke (ec60b5c)Closes the three-format ship symmetry. Every shipped format now has a pre-flight Poka-Yoke gate that aborts BEFORE any network I/O when the staged artifact disagrees with the manifest:
v1.0 scope (pragmatic MVP)Verify first 4 bytes of const APR_MAGICS: &[&[u8; 4]] = &[
b"APR\0", // v2 (canonical)
b"APRN", // v1
b"APR1",
b"APR2",
];Catches Expansion path (v1.1, deferred)Parse APR v2 tensor index via Dogfood verdictReal 8 GiB teacher artifact ( Changes
TestsBreakdown: 15 PM-008 + 9 PM-007 + 9 PM-009 + 12 misc. ex-04-upload-hf.sh wiringNo script change needed — Three-format ship symmetry — DONEEvery 🤖 Generated with Claude Code |
Three-format end-to-end dogfood of FALSIFY-PM-001..009 on the staged teacher artifacts — supersedes the v1 smoketest which captured only PM-001..007 (before PM-008 GGUF tensor-type authority and PM-009 APR magic-bytes existed). Per-format verdict (all PASS on 2026-04-18 via canonical release binary /mnt/nvme-raid0/targets/aprender/release/apr, commit ec60b5c): | Format | Size | Active gates | Deferred | |--------------|----------|------------------------------------------------|-----------------| | .apr | 8.0 GiB | PM-001/-002/-004/-005/-006/-009 | -003/-007/-008 | | .safetensors | 15.2 GiB | PM-001/-002/-004/-005/-006/-007 | -003/-008/-009 | | .gguf | 8.0 GiB | PM-001/-002/-004/-005/-006/-008 | -003/-007/-009 | Every format hit its format-specific binary-layer gate: .safetensors → PM-007 (198 F16 weight tensors, 141 F32 norm/bias exempt) .gguf → PM-008 (predominant tensor = Q4_K, stale file_type=0 noted) .apr → PM-009 (magic = APR\0 (v2)) PM-008 tensor-authority demonstration: teacher .gguf declares general.file_type=0 (ALL_F32) in its metadata but every weight tensor is actually Q4_K (ggml_type=12). Under an ftype-only design PM-008 would have falsely ship-blocked the teacher. The tensor-authoritative design (per feedback_gguf_file_type_stale.md) surfaces the stale ftype as a diagnostic note and returns PASS — exactly the intended behavior. Three-format ship symmetry: COMPLETE. Every SHIP-TWO-001 release format now has a pre-flight Poka-Yoke gate that aborts BEFORE any network I/O when the staged artifact's binary layer disagrees with its manifest. Changes: - evidence/ship-two-001/ex-04-preflight-gate-smoketest-v2.json (NEW) - docs/specifications/aprender-train/ship-two-models-spec.md §12.7 evidence reference bumped v1 → v2 Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Previously the "Lessons codified as contracts" table only pointed at publish-manifest-v1.yaml v1.0 (the base schema). v1.1 / v1.2 / v1.3 each encode a distinct lesson from the pre-flight Poka-Yoke work: - v1.1 (PM-007) — safetensors header dtype must match manifest - v1.2 (PM-008) — trust per-tensor ggml_type histogram, not stale general.file_type (llama.cpp ships Q4_K files with ftype=0) - v1.3 (PM-009) — .apr artifact magic-bytes must match (three-format ship symmetry with PM-007/PM-008) Each row names the exact ship-blocker it prevents so a reader can tell why the contract exists without spelunking the changelog. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Contributor
Author
Extension 5 — Real-artifact dogfood v2 + §13.3 lessons tableTwo follow-up commits on top of Extension 4 (PM-009): 2cab38e — Real-artifact dogfood v2 evidenceThree-format end-to-end dogfood of FALSIFY-PM-001..009 on the staged teacher artifacts, run from the canonical release binary (commit ec60b5c, built
Every format's binary-layer gate fires:
PM-008 tensor-authority demonstration: the teacher Evidence: 8e2edfe — §13.3 "Lessons codified as contracts" expansionThe retrospective table previously only pointed at
PR-wide summary so far (5 extensions on this branch)
Three-format ship symmetry COMPLETE. Every 🤖 Generated with Claude Code |
… blocker
Two findings from real HF_TOKEN EX-04 upload attempt (2026-04-18):
1. SCRIPT FIX: apr publish uses --message, not --commit-message.
Live HF_TOKEN caught this typo in ex-04-upload-hf.sh:119 that
dry-runs did not. Single-line fix; verified via apr publish --dry-run.
2. ARCHITECTURAL BLOCKER (documented): All three teacher formats
exceed HF Hub's 5 GiB HTTP API limit. HF preupload returns
uploadMode:lfs with empty upload_url + chunk_urls, signaling
client must use git-lfs batch API or hf_transfer custom agent.
Neither is implemented in apr publish.
reject_oversized_file (hf_hub/upload.rs:283) aborts with
NetworkError. Pre-flight gates (PM-001..009) cannot catch this
since 5 GiB is a destination-side property.
Sizes: .apr 8.0 GiB, .gguf 8.0 GiB, .safetensors 15.2 GiB
Teacher: Qwen2.5-Coder-7B-Instruct Q4_K (smallest meaningful precision).
Full Five-Whys + options table (A-E) in
evidence/ship-two-001/ex-04-five-whys-lfs-5gb-blocker.md.
Recommended path A+C (subject to operator decision):
A) Add apr export --max-shard-size 4G for .safetensors
(uses HF's native sharding index convention)
C) For .apr/.gguf: publish to self-hosted S3 mirror only;
manifest.artifact_url_mirror already supports this.
This preserves the three-format ship promise while respecting
HF's 5 GiB limit without waiting for full LFS batch API support.
Ship is blocked on this architectural decision — no network I/O
path exists today for >5 GiB single-file HF Hub uploads.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Contributor
Author
Extension 6 — EX-04 live HF_TOKEN run: script fix + architectural blocker discoveredCommit: (pushed following this comment) What happenedOperator supplied Two things surfaced — one trivial, one load-bearing. Finding 1 — trivial script typo (fixed)
This was a real bug that pre-flight gates could not catch (they validate Finding 2 — 5 GiB HF Hub upload blocker (architectural)After the Root cause ( All three teacher formats exceed the 5 GiB threshold:
Q4_K is already the smallest meaningful precision for a 7 B model. Why pre-flight gates could not catch this
Five-Whys + Options A-EFull analysis (Five-Whys chain + 5-row options table) committed to Options surfaced (summarized):
Recommended path: A+C combined —
Ship statusBLOCKED on operator decision before SHIP-TWO-001 can proceed to EX-05/06/07. Task #59 and #63 metadata updated to PR stateThis extension keeps the branch scope true to its name (FALSIFY-PM-007 |
…001) EX-04 discovered that all three SHIP-TWO-001 teacher artifacts (.apr 8 GiB / .gguf 8 GiB / .safetensors 15.2 GiB) exceed HF Hub's 5 GiB HTTP preupload threshold, triggering reject_oversized_file() in hf_hub/upload.rs. The fix is not sharding (workaround) and not a self-hosted S3 mirror (not sovereign — still AWS-dependent). The fix is to implement HF Hub's actual current large-file protocol: Xet. This commit is DbPC research + contract only (no code changes yet): - contracts/apr-publish-hf-large-file-v1.yaml v1.0.0 (NEW, 596 lines) 10 falsification gates: file-size dispatch, Xet token acquisition, chunk size bounds (8 KiB min / 64 KiB avg / 128 KiB max), xorb size bound (<=64 MiB), strict shard-after-xorbs ordering, content-addressable idempotency (was_inserted:false / result:0 = success), retry taxonomy (429/500/503/504 retry; 400/403/404 abort), Xet hash-string encoding (8-byte-reversed hex, not naive hex), LFS pointer git commit after Xet upload, three-format dogfood end-to-end. - docs/specifications/aprender-train/ship-two-models-spec.md v2.7.0 -> v2.8.0 New section §12.8 (Large-File Upload via Xet): rejected-paths table, normative protocol summary (8 lifecycle steps), FALSIFY-PUB-LFS-001..010 registry, implementation plan (xet-core crates, sync->async bridge via block_on, edit sites), sovereignty position, falsification verdicts. Xet protocol source of truth: huggingface.co/docs/xet/index v1.0.0 Rust reference impl: github.com/huggingface/xet-core Apache-2.0 v1.4.3 Crates planned: hf-xet, xet-client, xet-data, xet-core-structures. Next (Phase 2, separate PR): replace reject_oversized_file with XetUploader::upload in crates/aprender-core/src/hf_hub/upload.rs, add `xet` sub-feature to aprender-core, dogfood EX-04 upload. Refs: evidence/ship-two-001/ex-04-five-whys-lfs-5gb-blocker.md Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Contributor
Author
Extension 7 — Correction: proper HF Hub large-file path (not A+C)Supersedes the Extension 6 recommendation (A+C sharding + self-hosted S3).
Committed this push (b486ec7):
Research anchors:
Key Xet invariants codified in the contract:
Sovereignty posture (§12.8.5): we ship through HF Hub for discovery Next commit (separate scope, implementation phase):
No code changes in this commit — contract + spec only. Phase 1 (DbPC) complete. |
Wires the HF Xet content-addressable protocol into `apr publish` via
the `hf-xet` crate (HF's reference impl, Apache-2.0). Replaces the
v1.0.0 `reject_oversized_file` hard-abort with a live upload path so
the SHIP-TWO-001 teacher (8 GiB .apr / 15 GiB .safetensors / 8 GiB
.gguf) can ship to the Hugging Face Hub.
Discharges FALSIFY-PUB-LFS-001 (file-size dispatch) and -002 (token
refresh URL shape) with 4 deterministic unit tests. Phases 3–7 of the
Xet protocol (chunking, dedup, xorb/shard CAS upload, hash encoding)
are delegated wholesale to hf-xet 1.5.1 — our wrapper is 178 lines.
- Cargo.toml: hf-xet = "1.5.1" in [workspace.dependencies]
- aprender-core/Cargo.toml: optional hf-xet dep + `xet` sub-feature
(activates hf-hub-integration + hf-xet)
- apr-cli/Cargo.toml: `xet = ["hf-hub", "aprender/xet"]` forwarder;
appended to `full`
- aprender-core/src/hf_hub/xet.rs: NEW (178 lines). Exports
`HF_XET_THRESHOLD_BYTES`, `should_use_xet` (pure fn), and
`build_token_refresh_url`. Behind `xet`, also exports `XetUploader`
which wraps `XetSessionBuilder → new_upload_commit →
with_token_refresh_url → build_blocking → upload_from_path_blocking
→ commit_blocking`.
- aprender-core/src/hf_hub/mod.rs: added `pub mod xet` +
`HfHubError::XetUpload(String)` + `HfHubError::PartialUpload {
cas_success, commit_success, detail }` variants with Display impls.
- aprender-core/src/hf_hub/upload.rs: `reject_oversized_file` (with
its broken `apr export --max-shard-size` recommendation) DELETED;
replaced by `upload_via_xet` (tempfile materialize + XetUploader)
and `reject_needs_xet_feature` (clear error when built without
`--features xet`). Dispatch in `upload_via_lfs` routes files > 5
GiB via `super::super::xet::should_use_xet`.
- contracts/apr-publish-hf-large-file-v1.yaml: bumped to v1.1.0,
status IMPLEMENTED, changelog + implementation_plan updated to
reflect actual wiring (single `hf-xet` dep vs. originally planned
four crates; blocking API obviates the planned tokio bridge).
Sovereignty preserved: the Xet protocol is HF-open-speced (v1.0.0);
reference impl is Apache-2.0; no AWS dependency introduced. Bytes
still mirrored to self-hosted S3 via manifest.artifact_url_mirror for
stacks that prefer not to transit HF.
Tests: 36/36 hf_hub tests pass under `--features xet`; 4 new xet
module tests discharge FALSIFY-PUB-LFS-001/002.
Follow-up: live EX-04 upload against `paiml/qwen2.5-coder-7b-apache-
q4k-v1` is still blocked on HF_TOKEN in the dogfood environment.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Contributor
Author
F-PUB-LFS-001 Phase 2 — Xet upload path landed (commit 18fd953)Following the Phase 1 contract (b486ec7) and the rejected A+C recommendation, Phase 2 wires the real Hugging Face Xet protocol into What shipped
Total: 8 files, +1090 / -142 lines. Architectural simplification vs. v1.0.0 contractThe contract originally planned four separate crates ( session
.new_upload_commit()?
.with_token_refresh_url(token_refresh_url, headers)
.build_blocking()?
.upload_from_path_blocking(path, Sha256Policy::Compute)?
.commit_blocking()?makes our wrapper 178 lines and delegates phases 3–7 (chunking, dedup, xorb CAS upload, shard upload, hash encoding) entirely to the Hugging Face reference implementation. The token-refresh URL pattern means we never construct or cache raw access tokens — we just hand Falsification gates discharged
Test resultsBaseline Sovereignty postureStill preserved. Xet is an HF-open-speced protocol (v1.0.0) with an Apache-2.0 reference impl. No AWS dependency introduced. Bytes may still mirror to a self-hosted S3 bucket via Off-by-default; opt-in binary size
Next (Phase 3)
Related tickets
🤖 Generated with Claude Code |
Phase 2 of F-PUB-LFS-001 landed in 18fd953 (PR #882). Update spec status from "CONTRACT DRAFTED — IMPLEMENTATION PENDING" to "XET UPLOAD PATH IMPLEMENTED — AWAITING LIVE EX-04 DOGFOOD". FALSIFY-PUB-LFS-001 (file-size dispatch) and -002 (token-refresh URL shape) discharged by 4 unit tests in hf_hub::xet. -003..-009 inherited from hf-xet 1.5.1 (HF's Apache-2.0 reference impl). -010 (three-format dogfood) still pending HF_TOKEN. Contract apr-publish-hf-large-file-v1.yaml bumped to v1.1.0 / IMPLEMENTED in the earlier commit; this commit just syncs the specification document that references it. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Static evidence that the canonical apr binary at /mnt/nvme-raid0/targets/aprender/release/apr (55 MB, tag 0.31.0 + 9b081e5) links the full hf-xet 1.5.1 runtime. Enumerates which falsification gates are discharged statically vs delegated to hf-xet vs still pending live dogfood: - PUB-LFS-001/-002: DISCHARGED STATICALLY (4 unit tests) - PUB-LFS-003..009: DELEGATED to hf-xet 1.5.1 subsystems - PUB-LFS-010: PENDING HF_TOKEN (three-format live upload) Symbol-linkage section lists the hf-xet symbols confirmed present in the release binary via `strings`, including our own literal " GiB) via hf-xet (>5 GiB path)" from upload.rs::upload_via_xet. This does NOT close SHIP-TWO-001 — it documents the static ready state so the only remaining blocker is HF_TOKEN availability in the ship environment for the live EX-04 dogfood. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Contributor
Author
Phase 2 complete — canonical binary + evidence
Gate status
Symbol-linkage proof (from Next action: needs The only remaining blocker on closing SHIP-TWO-001 is a paiml-org write token in the ship environment so we can execute the three-format live dogfood:
…against Merge housekeeping: branch is DIRTY against main (10 MCP M3 commits landed in parallel — overlaps in |
…001 observable)
Adds `format_upload_route(size)` to the `apr publish --dry-run` path
so every listed file is annotated with the exact HF Hub code path it
will take:
- `[→ HTTP LFS (≤5 GiB)]` ≤ 5 GiB (normal multipart path)
- `[→ Xet CAS (>5 GiB)]` > 5 GiB, built with --features xet
- `[✗ would FAIL: rebuild with --features xet]` > 5 GiB without xet
This exercises FALSIFY-PUB-LFS-001 (file-size dispatch) from the CLI
without needing HF_TOKEN, closing a gap in EX-04 pre-upload
verification. A reviewer can now verify routing by eye from a dry-run
log, and we have a fast signal if the dispatch gate regresses.
Dogfood:
$ truncate -s 1G small.safetensors
$ truncate -s 6G big.safetensors
$ apr publish . paiml/test --dry-run
- .../big.safetensors (6442.5 MB) [→ Xet CAS (>5 GiB)]
- .../small.safetensors (1073.7 MB) [→ HTTP LFS (≤5 GiB)]
Tests (publish_tests.rs):
- dry_run_route_partitions_at_5_gib_exactly — boundary
- dry_run_route_above_5_gib_reports_xet_when_enabled — xet feature on
- dry_run_route_above_5_gib_flags_missing_xet_feature — xet feature off
Also:
- Updates spec §12.8.3 / §12.8.4 / §12.8.6 to reflect v1.1.0
IMPLEMENTED state (edit-sites tree matches reality; implementation
plan annotated with what actually shipped vs what v1.0.0
anticipated). Contract `apr-publish-hf-large-file-v1.yaml` was
already bumped to v1.1.0 in 18fd953.
Does NOT unblock FALSIFY-PUB-LFS-010 — that still needs HF_TOKEN for
live three-format dogfood.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Contributor
Author
Added: dry-run dispatch visibility (commit
|
| Build | ≤ 5 GiB | > 5 GiB |
|---|---|---|
--features cuda,xet |
[→ HTTP LFS (≤5 GiB)] |
[→ Xet CAS (>5 GiB)] |
default (hf-hub only, no xet) |
[→ HTTP LFS (≤5 GiB)] |
[✗ would FAIL: rebuild with --features xet] |
Tests
3 new unit tests in publish_tests.rs pin the output strings:
dry_run_route_partitions_at_5_gib_exactlydry_run_route_above_5_gib_reports_xet_when_enabled(xet feature)dry_run_route_above_5_gib_flags_missing_xet_feature(default feature)
Also included in this commit
Spec v2.8.1 §12.8.3 / §12.8.4 / §12.8.6 brought into alignment with the IMPLEMENTED state:
- §12.8.4 "Implementation plan" → "Implementation (shipped 2026-04-18, commit
18fd9536e)" - Edit-sites tree now matches actual file layout (
xet.rssingle file, not 4-filexet/module) - Two extra falsification rows recording the v2.8.1 post-conditions (binary default footprint unchanged; unit tests green)
Still blocked
- FALSIFY-PUB-LFS-010 (three-format live dogfood) — needs
HF_TOKEN - PR still
DIRTYagainstmain(MCP M3 overlaps) — needs merge/rebase before landing
Ran `apr publish /mnt/nvme-raid0/models/ship-two-001 paiml/qwen2.5-coder-7b-apache-q4k-v1 --dry-run` against the actual SHIP-TWO-001 teacher directory using the canonical release binary at /mnt/nvme-raid0/targets/aprender/release/apr (features: cuda,xet; tag 0.31.0 5ca162e). All three teacher artifacts route correctly to the Xet CAS path: - qwen2.5-coder-7b-instruct-q4k.apr (8035.6 MB) -> Xet CAS - qwen2.5-coder-7b-instruct-q4k.gguf (8037.1 MB) -> Xet CAS - qwen2.5-coder-7b-instruct-q4k.safetensors (15231.9 MB) -> Xet CAS Discharges FALSIFY-PUB-LFS-001 (file_size_dispatch) against the real teacher sizes — not just sparse test fixtures. The dispatch gate in format_upload_route correctly calls should_use_xet(size). Does NOT discharge FALSIFY-PUB-LFS-010 (three_format_dogfood) — that requires HF_TOKEN + a live upload against paiml/qwen2.5-coder-7b-apache-q4k-v1 and will be captured in ex-04-xet-upload.{log,json}. Ref: contracts/apr-publish-hf-large-file-v1.yaml v1.1.0 Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Contributor
Author
Phase 2 evidence: live-on-teacher dry-run passLanded commit Dry-run against the actual SHIP-TWO-001 teacher directory Canonical binary: All three teacher artifacts route correctly to the Xet CAS path:
Evidence files (
Gates status
Contract: Next blocker: HF_TOKEN in ship environment for live three-format dogfood against Note on merge state: PR is still |
Spec previously cited only the static wiring proof (commit ee63828). After landing live-on-teacher dry-run evidence (commit 18f8b56), update §12.8.4 bullet 5 to list both pre-live evidence files and their respective commit hashes: (a) ex-04-xet-phase2-wiring.json @ ee63828 — binary linkage proof (b) ex-04-xet-dryrun-teacher.* @ 18f8b56 — dispatch on 3 real teacher artifacts (.apr/.gguf/.safetensors) Live EX-04 remains blocked on HF_TOKEN; the log+verify evidence paths for the live upload are unchanged. FALSIFY-PUB-LFS-001 is now discharged against real teacher sizes, not just synthetic fixtures. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add Normalization::{None,NFC} enum + TokenizerConfig.normalization
(default None, #[serde(default)] for backward-compat). BPETokenizer
now routes text through preprocess() — NFC first, then optional
lowercase — so both train() and encode() share the same normalization
pipeline.
Why NFC before lowercase: char::to_lowercase() is not closed over
non-NFC input for every grapheme, so normalizing first keeps the
pipeline deterministic for composed vs decomposed variants of the
same visible text. Tests lock this in:
- test_bpe_nfc_composed_decomposed_parity — café U+00E9 and café
(e + U+0301) must encode to identical IDs under NFC.
- test_bpe_without_nfc_composed_decomposed_diverge — without NFC
they MUST diverge (live falsification witness for INV-TOK-003).
Contracts:
- C-TOK-BPE-001 INV-TOK-003 (tokenizer-bpe-v1.yaml) — mandates NFC
for MODEL-2; composed/decomposed drift between training-time
corpus prep and inference-time input is the exact failure mode.
- C-DATA-THESTACK-PYTHON INV-DATA-007 (dataset-thestack-python-v1)
— requires NFC-normalized UTF-8 in every shard before dedup.
Adds unicode-normalization 0.1 to aprender-train deps. 11/11
tokenizer tests pass (2 new + 9 preserved).
Task: #89 (SHIP-TWO-001 MODEL-2 P0 blocker)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds two scaffolding pieces for SHIP-TWO-001 MODEL-2 pretraining: 1. `apr tokenize train` (task #90) — new subcommand trains a BPE tokenizer from a JSONL corpus (file or directory). Walks `.jsonl`, extracts `content` field, applies `--normalization nfc` (default) via `unicode-normalization::nfc`, calls the BPE trainer, emits `vocab.json` + `merges.txt`. JSON mode round-trips all params. `--min-frequency` is accepted for contract parity but threading to the underlying trainer is owned by a follow-up — the `aprender-core` BpeTokenizer this CLI currently calls has no public `min_frequency` parameter; a later patch will switch the CLI to `aprender-train::tokenizer::BPETokenizer` (which does, and which already has the NFC plumbing from commit b0e0a28). 3 unit tests pass: happy-path JSONL file, directory walk, unknown-normalization rejection. 2. `apr-corpus-ingest` (task #91) — new binary at `src/bin/apr-corpus-ingest.rs` (+517 LOC) providing `plan` and `validate-contract` subcommands over `C-DATA-THESTACK-PYTHON` v1.0.0. `plan` reads the contract, validates 7 invariants + 5 falsification tests + 5 gates are present (with correct INV-DATA-*/FALSIFY-DATA-*/GATE-DATA-* prefixes), asserts the 6 required top-level keys (source, license_whitelist, pii_scrub, deduplication, split, budget), and emits `./output/dry-run-manifest.yaml` with TODO placeholders + UTC timestamp. `validate-contract` is exit-code-only. Hard constraints honored: NO network, NO writes outside `./output/`, only workspace `serde`/`serde_yaml`/`anyhow`/`clap` deps. Does NOT touch `aprender-train/` or `aprender-core/`. 2 unit tests pass. Adds `anyhow = { workspace = true }` + `unicode-normalization = "0.1"` + `[[bin]] apr-corpus-ingest` to apr-cli Cargo.toml. Tasks: #90 (tokenize CLI) + #91 (corpus ingest scaffold) Contracts: - C-TOK-BPE-001 (tokenizer-bpe-v1.yaml) — BPE train + NFC - C-DATA-THESTACK-PYTHON (dataset-thestack-python-v1.yaml) — corpus Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
All three P0 MODEL-2 blockers identified in the v2.14.0 readiness audit are now closed: 1. Task #89 (commit b0e0a28) — BPE NFC patch with falsification pair (composed/decomposed café parity + witness test). 2. Task #90 (commit 512ea51) — `apr tokenize train` subcommand: walks JSONL corpus, applies NFC, trains BPE, emits vocab.json + merges.txt. 3 unit tests pass. --min-frequency accepted for contract parity but not threaded (follow-up to switch CLI to aprender-train BPE). 3. Task #91 (commit 512ea51) — `apr-corpus-ingest` binary with plan/validate-contract subcommands over C-DATA-THESTACK-PYTHON v1.0.0. Validates 6 top keys, 7 INV-DATA-*, 5 FALSIFY-DATA-*, 5 GATE-DATA-*. 2 unit tests pass. NO network, output confined to ./output/. Revised MODEL-2 readiness: 10-14 days to first pretraining loss curve (up from v2.14.0's 5-7d estimate now that the 370M Llama arch implementation is the clear gating path). Contracts: C-TOK-BPE-001, C-DATA-THESTACK-PYTHON Tasks: #89 #90 #91 Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…ublish::execute dispatch_analysis.rs:1035 invoked commands::publish::execute with 10 args but the function signature grew to 12 (manifest: Option<&Path>, extra_files: &[PathBuf]) for F-PUBLISH-EXTRA-001. Pass None/&[] defaults at the dispatch call site so the non-manifest path still type-checks. A follow-up will thread --manifest + extra files through the ToolCommands::Publish enum variant for full feature parity. Zero-Tolerance fix per SHIP-TWO-001 spec §3 row #8: the branch must always compile; no staged hacks or #[ignore] escape valves. Refs: #96 (broken-branch unblock), #63 (F-PUBLISH-EXTRA-001 parent) Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…r is dense-only Under `--features cuda` the workspace failed to compile because `generate_2.rs:490` referenced `moe_gate_weight` on `OwnedQuantizedLayer`, but that struct never holds MoE expert weights (it's the GGUF-dense path). MoE dispatch lives in `apr_transformer::AprTransformerLayer`, a different code path entirely — so the SPEC-MOE-APR-001 prefill branch can statically resolve `is_moe = false` here without changing behavior. Minimal fix: replace the broken field access with `false` + cite the routing invariant in the comment. Fixes: task #96 (branch-unblock per Zero-Tolerance spec §3 row #8) Verify: `cargo check -p aprender-serve --lib --features cuda` — clean. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…vidence Contract: contracts/tokenizer-bpe-v1.yaml (C-TOK-BPE) Gate: GATE-BPE-003 / INV-BPE-003 / INV-BPE-005 Spec: docs/specifications/aprender-train/ship-two-models-spec.md §5 Task: #98 Discharge harness for GATE-BPE-003 (byte-exact round-trip on held-out Python-like corpus) and INV-BPE-005 (NFC idempotence). Ships with a 20-doc synthetic fixture until the real TheStack-Python 10K-doc holdout is materialized (task #91); swapping the fixture for the real corpus is a data-only change. What the harness does: - Trains aprender::text::tokenize::BpeTokenizer on a 20-doc Python-idiom train corpus (imports, class defs, lambdas, match statements) - Evaluates byte-exact `decode(encode(text)) == text` on a disjoint 20-doc holdout (ASCII + Unicode identifiers, docstrings, numeric literals, byte strings, tab indent, combining marks, emoji in comments) - Emits evidence JSON to $APR_EVIDENCE_DIR when set, otherwise asserts in-process. Soft-asserts round-trip failures (known defect — see below) but hard-asserts NFC idempotence so CI stays green per the monorepo Andon rule. Current evidence (committed): - vocab_size_trained: 232 / vocab_size_requested: 512 - docs_passed: 1/20, docs_failed: 19/20 - failing_doc_indices: [0,1,3,4,5,...,19] - nfc_idempotent: true - passed: false (FALSIFY-SHIP-012 **OPEN** — ship-blocker for MODEL-2) Root cause (preliminary): aprender-core BPE likely drops whitespace/ indentation at the pretokenize→join boundary. Only the bare ASCII literal `"x = 42"` round-trips. Fix tracked as a separate P0 task. Files: - crates/apr-cli/tests/falsify_ship_012_tokenizer_roundtrip.rs (+200) - evidence/ship-two-001/falsify-ship-012-tokenizer-roundtrip.json (+36) - crates/apr-cli/Cargo.toml (+2, dev-dep unicode-normalization="0.1") Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Brings in upstream via merge commit to keep PR #882 mergeable without rewriting 33 commits of SHIP-TWO-001 audit history (FALSIFY-PM-007, FALSIFY-PM-008, FALSIFY-PM-009, FALSIFY-SHIP-012 evidence). Conflicts resolved: - crates/apr-cli/Cargo.toml: keep aprender@0.31.0 (pm-007 is workspace version); add aprender-mcp@0.31.0 line from main (bumped from main's 0.30.0 pin to match workspace). - Cargo.lock: took main's version, then regenerated via `cargo check -p apr-cli --lib` to re-resolve pm-007's new unicode-normalization dev-dep (FALSIFY-SHIP-012 harness). Upstream brings in (from main): - MCP M3 complete: codegen schemas for 7 remaining tools, notifications/ progress for apr.finetune, notifications/cancelled SIGTERM→SIGKILL - provable-contracts-macros → aprender-contracts-macros workspace migration (#895) - apr.finetune synchronous wrapper (8th Phase-1 MCP tool) Verified: `cargo check -p apr-cli --lib` passes in 35s after merge. Task: #95
…DEL-2
Adds `Tokenizer`, `ModelFamilyVariant`, `TrainingLoop`, `PretrainingCorpus`
variants to `aprender_contracts::schema::ContractKind`. Rebuilds `pv` so
it accepts these kinds in `metadata.kind:` instead of rejecting them as
"unknown variant".
## Why
`contracts/tokenizer-bpe-v1.yaml` (C-TOK-BPE for MODEL-2 370M), the
MODEL-2 variant contracts, the training-loop contract, and
`contracts/dataset-thestack-python-v1.yaml` all declare non-kernel kinds
that the schema previously rejected. Without this, the enforced
dogfooding rule ("pv not bash for contracts") could not apply to
MODEL-2 — every MODEL-2 contract would fall back to ad-hoc YAML parsing
which is MUDA.
Prior task #97 marked completed but was never committed. Reopened and
done properly.
## Provability
All 4 new kinds are **exempt from PROVABILITY-001** (same treatment as
`Registry`, `ModelFamily`, `Pattern`, `Schema`). Their validation
gates are byte-exact tests (round-trip for `Tokenizer`, schedule
monotonicity for `TrainingLoop`, checksum/shard layout for
`PretrainingCorpus`) rather than Kani harnesses — provability
requirements would be category errors for data-shaped contracts.
## Tests
- `contract_kind_display` covers all 9 variants → kebab-case string
- `non_kernel_kinds_exempt_from_provability` exercises the 8 non-kernel
kinds → `requires_proofs() == false`, `provability_violations()` empty
- Both tests pass: `cargo test -p aprender-contracts --lib contract_kind
non_kernel_kinds` → 3 passed
## Verification
- `cargo install --path crates/aprender-contracts-cli --force --locked`
replaces `/home/noah/.cargo/bin/pv` with the new binary
- `pv validate contracts/tokenizer-bpe-v1.yaml` previously failed with
"unknown variant 'tokenizer'"; now advances past kind validation
(remaining failures are contract-file data bugs filed separately)
Task: #97
…hestack-python Both MODEL-2 contracts were missing `metadata.description` which the `aprender-contracts` schema requires. `pv validate` failed with: error: Failed to parse YAML: metadata: missing field `description` Promoted the first paragraph of each contract's top-level `summary` block into `metadata.description` (one-sentence form). No invariant or gate changes — purely schema compliance. Verification: - `pv validate contracts/tokenizer-bpe-v1.yaml` → "Contract is valid." - `pv validate contracts/dataset-thestack-python-v1.yaml` → "Contract is valid." Unblocks contract-first dogfooding for SHIP-TWO-001 MODEL-2 work (tokenizer trainer, corpus ingest, downstream training loop). Follow-up from: task #97 (ContractKind extension) Refs: #99 (BPE round-trip fix will now reference this validated contract)
Promotes top-level summary: into metadata.description: for: - contracts/training-loop-pretrain-v1.yaml - contracts/model-families/llama-370m-sovereign-v1.yaml Same schema fix pattern as commit 1d32c76 (tokenizer-bpe-v1.yaml, dataset-thestack-python-v1.yaml). aprender-contracts requires metadata.description for non-Kernel kinds too. Dogfooded via pv validate (aprender-contracts-cli): $ for f in contracts/tokenizer-bpe-v1.yaml \ contracts/training-loop-pretrain-v1.yaml \ contracts/model-families/llama-370m-sovereign-v1.yaml \ contracts/dataset-thestack-python-v1.yaml; do pv validate "$f" done → all 4: "0 error(s), 0 warning(s). Contract is valid." All 4 MODEL-2 contracts now pv-validate green. PROPOSED→ACTIVE promotion remains gated on per-contract evidence (e.g. tokenizer blocked on #99 BPE round-trip repair). Refs: SHIP-TWO-001, task #100
**aprender-contracts schema**: adds serde `alias` attributes on
`FalsificationTest` so pre-APR-MONO contracts using legacy field
vocabulary still deserialize under pv validate:
rule ← alias "description"
prediction ← alias "expected"
test ← alias "command"
if_fails ← alias "fails_if"
This is the least-invasive unification of the pre-Phase-2b
provable-contracts schema with the legacy contract files.
**Contract YAML parse fixes** (independently broken before this commit):
- contracts/eval-sharding-v1.yaml:
multi-line changelog list item now uses literal block scalar ("|")
so embedded colons ("Evidence:", "Status:") don't misparse as
mapping keys.
- contracts/eval-harness-humaneval-v1.yaml:
- line 81: `target_claim: >= student_primary (…)` was being read as
a folded scalar because `>` is YAML's fold indicator. Now quoted.
- line 174: `expected: |abs(…) <= 0.6` had a block-scalar indicator
glued to the content without a newline. Now quoted scalar.
Effect on lint::gates::tests::load_contracts_real (test was ALREADY
red before this commit): 3 contracts advance further before failing.
eval-sharding and publish-manifest now surface a deeper
`proof_obligations[0]: missing field 'type'` legacy-vocabulary error
that needs schema harmonization — filed as follow-up.
Dogfooded via pv validate (aprender-contracts-cli):
$ cargo install --path crates/aprender-contracts-cli --force --locked
$ pv validate contracts/eval-sharding-v1.yaml
# advances past YAML parse; proof_obligations still legacy → follow-up
Refs: SHIP-TWO-001, spec audit ad6b3411a7c141e8b
`aprender::text::tokenize::BpeTokenizer` was char-level with a whitespace pre-tokenizer (`text.split_whitespace()` in both `train` and `encode`), so any input with indentation, tabs, multiple spaces, newlines, or multi-byte UTF-8 codepoints round-tripped with bytes lost or mangled — 19/20 Python-like SHIP-012 holdout docs failed. Switched to GPT-2-style byte-level BPE on the `train` / `encode` paths, using the existing `aprender::text::bpe::bytes_to_unicode` mapping (every byte 0..=255 maps to a unique printable Unicode codepoint). Decode reverses the mapping and UTF-8-decodes the recovered bytes, so `decode(encode(text)) == text` holds by construction per tokenizer-bpe-v1 INV-BPE-003 / INV-BPE-007. A new `byte_level: bool` field gates the new path: - `train` / `train_with_special_tokens` → `true` (new default) - `from_vocab` → `false` (preserves the </w>-word-marker back-compat surface for existing in-repo fixtures) - `from_huggingface` → auto-detected from vocab keys (`Ġ` present → true) Evidence: evidence/ship-two-001/model-2/falsify-ship-012-tokenizer-roundtrip.json docs_passed=20/20, vocab_size_trained=489, nfc_idempotent=true. The SHIP-012 harness was flipped from soft-warning to hard assertion on `docs_failed == 0` so any regression into whitespace splitting will break CI instead of silently regressing the round-trip contract. Regression checks: - cargo test -p aprender-core --lib text::tokenize → 113 pass - cargo test -p apr-cli --test falsification_tokenizer_data → 12 pass (includes TOK-001/001b/004/006/008/009/010 — the BPE contract suite) - cargo test -p apr-cli --lib tokenize → 16 pass Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
… (task #101) Closes blocker for `pv validate` across 760 repo contracts. schema/types.rs: - Add Safety + Liveness variants to ObligationType (28 total, up from 26) - ProofObligation: alias `statement`/`verification` → `property`/`formal`; rename `type` → obligation_type; all default-on for legacy compat - FalsificationTest: alias `description` → rule; `expected` → prediction; `fails_if` → if_fails - Contract.equations: custom polymorphic deserializer accepts either map `{id: Equation}` or sequence `[{id, ...}]` — legacy contracts use the list form; new contracts use the map form. Both coexist. - Equation.formula / QaGate.{id,name}: serde default for empty-body case probar_gen/{mod,wired}.rs + explain.rs: - Exhaustive match arms + Display for Safety + Liveness lint/gates.rs: - collect_yaml_files skips publish-manifests/ (PublishManifest ≠ Contract) contracts/ (6 legacy YAMLs uplifted to metadata: block form): - decode-gpu-resident-sampling-v1 - decode-hot-path-first-tokens-diagnostic-v1 - decode-hot-path-prefix-cache-diagnostic-v1 - eval-harness-humaneval-v1 - eval-sharding-v1 - profile-graph-vs-per-op-methodology-v1 - publish-manifest-v1 Tests: load_contracts_real + parse_missing_metadata_returns_error both PASS (1368/1371 aprender-contracts lib tests green). Remaining 3 failures are downstream lint/provability-gate content checks (empty formula:, missing kani_harnesses, falsifications < proof_obligations on the same 6 legacy contracts) that were already red on pristine main — separate scope from parser harmonization. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Three concurrent sub-agent lanes closed in one compute window against non-overlapping surfaces, validating the monorepo sub-agent workflow. #102 — Backfill 8 legacy contracts with formula + kani_harnesses + falsification parity. 22 ERROR findings → 0, lint_passes_on_real_contracts green. Dogfooded via `pv validate`: 8/8 clean (1 advisory SCHEMA-013). No bash/grep workaround needed; honors "pv not bash for contracts" MEMORY.md directive. #103 — Thread --min-frequency through apr-cli tokenize → entrenar BPE. Call-site swapped from aprender-core BPE to aprender-train BPE via train_bpe_via_entrenar helper; TokenizerConfig::bpe().with_min_frequency now honored. Public vocab()/merges() accessors added. New falsification test run_train_honors_min_frequency_pruning asserts singleton byte-pairs pruned at threshold 2. Closes v2.15.0 §1 Known gap. 17 tokenize tests green. #104 — gx10 third-party capacity gate PASS. llama.cpp on teacher GGUF: 38.0 tok/s decode (prompt eval 509 tok/s, 7.7 GiB VRAM) vs 30 tok/s threshold = 26.7% margin; 2.45× the forbidden 15.5 tok/s NF4 fallback. Zero-Tolerance §3 row #8 preserved. Evidence JSON committed. Follow-ups flagged: - gx10 decode drift (46 → 38.0 tok/s) worth tracking - gx10 disk 95% full — cleanup before MODEL-2 7B training - eval-sharding qa_gate SCHEMA-013 advisory (non-blocking) Spec bumped to v2.18.0. Task #105 (370M pretraining loop wiring) is now the sole long-pole item. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add qa_gate block to contracts/eval-sharding-v1.yaml closing the last advisory from the #102 backfill. pv validate now clean 8/8 legacy contracts with 0 errors 0 warnings. must_pass: FALSIFY-SHARD-001 (completeness) + SHARD-003 (determinism, discharged live yoga vs gx10 2026-04-18). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…ocker wired (task #105) Ships the pretrain loop driver for SHIP-TWO-001 MODEL-2, closing the sole long-pole task remaining on that epic. Implements every required gate and invariant from contracts/training-loop-pretrain-v1.yaml. aprender-train::train::pretrain (NEW, 963 LOC): * StepMetrics — exactly the 6 fields required by per_step_metrics (step, train_loss, grad_norm, lr, tokens_per_sec, gpu_util_pct) * EpochMetadata — all 9 required_fields for per_epoch_artifacts * EpochArtifact — builds ckpt/epoch-{N:03d}.apr paths per contract * check_non_divergence — GATE-TRAIN-005 ship-blocker (MANDATORY, UNCONFIGURABLE). val_loss[N] > 2.0 × val_loss[N-1] → hard abort. Directly addresses MODEL-1 v2 silent-divergence defect (memory/project_ship_two_001_model1_qlora_divergence.md). * check_numerical_stability — INV-TRAIN-007 NaN/Inf guard, validated BEFORE logging so poisoned metrics never reach the step log. * PretrainConfig::model_2_defaults() — LR=5e-5, seed=42 (the exact remedy from the MODEL-1 v2 post-mortem). * PretrainLoop<S: StepFn, V: ValFn> — trait-object drive lets the full gate surface be exercised via synthetic step/val functions while the 370M forward pass in llama_370m.rs is still scaffold. * 15 falsification tests: divergence-doubling, epoch-zero blowup, exact-2.0x boundary (ALLOWED), NaN train_loss, Inf grad_norm, happy-path decreasing loss, seed reproducibility, contract-path template, warmup-cosine LR schedule boundaries. apr-cli commands/pretrain.rs (NEW) + mod.rs / dispatch / extended_commands: * apr pretrain — synthetic drive by default (non-synthetic path returns ValidationFailed with pointer to the 370M follow-up). * 12 flags: --dataset, --tokenizer, --run-dir, --lr, --num-steps, --warmup-steps, --batch-size, --seq-length, --steps-per-epoch, --seed, --target-val-loss, --synthetic, --json. * abort_to_err attributes every abort to its contract gate ID (GATE-TRAIN-005 / 007 / 008) so operators see which gate fired in the shell exit status. * 3 CLI tests: happy-path end-to-end, synthetic=false rejection, invalid target_val_loss rejection. Verification: pv validate contracts/training-loop-pretrain-v1.yaml → 0 errors. cargo test -p aprender-train --lib train::pretrain → 15/15 pass. cargo test -p apr-cli --lib commands::pretrain → 3/3 pass. cargo test -p aprender-train --lib train:: → 947/947 pass (no regressions). Non-goals (documented in code): does NOT train an actual MODEL-2 checkpoint; does NOT wire the 370M forward pass; follow-up ticket needed once llama_370m.rs gains real compute. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…n (task #108) Root cause: contracts/model-families/llama-370m-sovereign-v1.yaml is a ModelFamilyVariant CONTRACT (starts with `contract_id:`) co-located with its parent family for documentation, not a ModelFamily REGISTRY entry. The 5 directory iterators in aprender-core's format module were treating it as a registry entry and failing with "missing required field: family" — 32 workspace-test failures in CI run 24614757928. Fix: all iterators now skip files whose first line matches `contract_id:` (model-family registry YAMLs all start with `metadata:`). Discriminator verified via corpus scan — no false positives. Paths fixed: - format/parsing.rs (load_family_registry public API) - format/model_family_contract_falsify.rs (falsify_mf_*) - format/metadata_bounds_contract_falsify.rs (falsify_mb_*) - format/tokenizer_vocab_contract_falsify.rs (falsify_tv_*) - format/converter_types_tests_parity.rs (parity test) Test verification: - cargo test -p aprender-core --lib format:: → 13031 passed, 0 failed - full suite re-green after 32→0 failures Doesn't touch MODEL-2 pretrain loop commit 9a5af3a (task #105) or the ci/lint workspace ambiguity (separate follow-up). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…dening - Task #105 CLOSED: MODEL-2 pretrain loop driver landed via sub-agent (commit 9a5af3a, 6 files +1379 LOC). GATE-TRAIN-005, INV-TRAIN-007, GATE-TRAIN-008 all wired. `apr pretrain` CLI gated by training feature. MODEL-1 v2 remedies baked into PretrainConfig::model_2_defaults() (LR=5e-5, rank=32, seed=42). - Task #108 CLOSED: 5 directory iterators in aprender-core/src/format/ hardened to skip ModelFamilyVariant contracts (contract_id: at root discriminator). 32→0 workspace-test regressions. 13031 passed locally. - Task #109 SPLIT OUT: ci/lint workspace package ambiguity (transitive aprender@0.27 deps via realizar/renacer/trueno/entrenar/bashrs/pacha). Needs [patch.crates-io] restoration or path-dep migration. Separate lane, not blocking SHIP-TWO-001 MODEL-2 work. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Root-cause correction: task #109 was originally framed as an aprender@0.27 workspace ambiguity, but inspection of CI lint job 71975024364 revealed the actual failure was cargo fmt --check diffs across 33 files in apr-cli, aprender-cgp, aprender-contracts-cli, aprender-contracts-macros, aprender-core, aprender-data, aprender-explain, aprender-present-cli, aprender-present-core, aprender-present-terminal, aprender-profile, aprender-registry, aprender-train, and aprender-viz. Sub-agent a6ea86f0d2d89eafb misdiagnosed as transitive dep pulling aprender@0.27.8 — that duplicate exists via apr-cli's published aprender-profile@0.29.0 dep but is NOT the active lint blocker. Fix: cargo fmt --all. Content unchanged — only whitespace/wrap normalized. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Rust 1.93.0 introduced clippy::doc_overindented_list_items. The module header in crates/aprender-train/src/models/llama_370m.rs (landed via task #105) uses a 30-column aligned table-style invariant list whose continuation lines are now flagged. Surgical fix: #![allow(clippy::doc_overindented_list_items)] at the top of the file. Rewriting continuation lines to 2-space indentation would destroy the alignment that makes the invariants readable at a glance. Unblocks ci/lint on feat/pm-007-preflight-poka-yoke (task #109). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…contract (#110) FALSIFY-CLI-002/005 failed on feat/pm-007-preflight-poka-yoke: `apr --help` exposed 59 commands but registered_commands() had 57. Two commands were missing from the Rust-side mirror: - validate-manifest (AC-EX-004 tool, task #61) — added in earlier slice but never wired into the test vec. - pretrain (task #105 MODEL-2 pretrain loop) — added to CLI but never added to the contract YAML or the test vec. Fix: - Add pretrain entry to contracts/apr-cli-commands-v1.yaml (category: training). - Add validate-manifest + pretrain to registered_commands() in the test. - cfg_attr allow(unused_mut) on the vec for non-`code`-feature builds. Local: `cargo test -p apr-cli --test cli_commands` — 6/6 PASS. `pv validate contracts/apr-cli-commands-v1.yaml` — 0 errors, 0 warnings. Unblocks task #95 (open PR against main for SHIP-TWO-001 work). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
noahgift
added a commit
that referenced
this pull request
Apr 18, 2026
…arge Records the end-to-end synthetic drive of `apr pretrain` on commit 1e7cf53 (now landed on main at 9209383 via PR #882 merge). Verifies task #105 deliverable: GATE-TRAIN-005 / INV-TRAIN-007 / GATE-TRAIN-008 wiring is functional end-to-end. Run: 20 steps, 4 epochs, batch=4, seq=128 — val_loss monotone 3.96 → 2.64. Synthetic drive caveat: no real 370M forward pass, no real corpus read, no checkpoint artifacts written yet. Real corpus + checkpoint wiring tracked as task #111. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
9 tasks
noahgift
added a commit
that referenced
this pull request
Apr 18, 2026
…int) 7-step edit list from Plan agent afd391d1eb1395d30 against post-#882-merge commit 9209383. Identifies 5 critical files (pretrain.rs, apr-cli/commands/pretrain.rs, trainer.rs, transformer/model.rs, io/save.rs) and 5 binary acceptance criteria (AC-111-001..005). Host assignment: lambda-labs (impl), yoga (8GB smoke), gx10 (parity). Non-goals explicitly deferred: async H2D streaming, full corpus-ingest pipeline, mixed-precision scaler tuning, distributed training, convergence budget, resume round-trip, nvml telemetry, apr qa post-hoc validators. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
noahgift
added a commit
that referenced
this pull request
Apr 19, 2026
…olicy (#901) * evidence(ship-two-001): MODEL-2 pretrain smoke test — task #105 discharge Records the end-to-end synthetic drive of `apr pretrain` on commit 1e7cf53 (now landed on main at 9209383 via PR #882 merge). Verifies task #105 deliverable: GATE-TRAIN-005 / INV-TRAIN-007 / GATE-TRAIN-008 wiring is functional end-to-end. Run: 20 steps, 4 epochs, batch=4, seq=128 — val_loss monotone 3.96 → 2.64. Synthetic drive caveat: no real 370M forward pass, no real corpus read, no checkpoint artifacts written yet. Real corpus + checkpoint wiring tracked as task #111. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * spec(model-2): MVP plan for task #111 (pretrain real corpus + checkpoint) 7-step edit list from Plan agent afd391d1eb1395d30 against post-#882-merge commit 9209383. Identifies 5 critical files (pretrain.rs, apr-cli/commands/pretrain.rs, trainer.rs, transformer/model.rs, io/save.rs) and 5 binary acceptance criteria (AC-111-001..005). Host assignment: lambda-labs (impl), yoga (8GB smoke), gx10 (parity). Non-goals explicitly deferred: async H2D streaming, full corpus-ingest pipeline, mixed-precision scaler tuning, distributed training, convergence budget, resume round-trip, nvml telemetry, apr qa post-hoc validators. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * evidence(ship-two-001): yoga parity smoke — GATE-TRAIN-006 discharged Cross-host byte-identical loss history on yoga RTX 4060 Laptop (8GB): lambda-labs: [3.96, 3.52, 3.08, 2.64] yoga: [3.96, 3.52, 3.08, 2.64] Discharges GATE-TRAIN-006 (seed=42 deterministic) across x86_64 RTX 4090 ↔ x86_64 RTX 4060 Laptop. Same synthetic drive — task #111 MVP will add the real 370M forward pass; yoga stays as 8GB smoke-test host per MVP plan's host assignment table. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat(model-2): RealStepFn/RealValFn + shard reader (task #111 steps 1-3) Implements MODEL-2 pretrain MVP plan steps 1-3: the model-agnostic PretrainLoop now has a real-corpus driver that runs a full forward + backward + AdamW step through TransformerTrainer against the 370M Llama scaffold — replacing the LinearDecaySynthetic/ScriptedVal pair used for GATE-TRAIN-005/006/007/008 wiring verification in task #105. **New modules** - `train::shard_reader::ShardBatchIter` Streaming iterator over .bin token shards (little-endian u32). Reads seq_length+1 sequences, chunks into LMBatch of batch_size. Empty-dir errors; lexical shard ordering; EOF auto-advances to next shard. No MinHash dedup / PII scrub / license filter — those belong to `apr-corpus-ingest run`. - `train::pretrain_real::{RealStepFn, RealValFn, build_shared_trainer}` - `llama_370m_transformer_config()` field-for-field from the frozen Llama370MConfig constants (INV-ARCH-370M-001..008 source of truth) - `llama_370m_train_config(lr, seq_length, seed)` builds TransformerTrainConfig with MODEL-2 v2-remedy defaults - `SharedTrainer = Rc<RefCell<TransformerTrainer>>` so both the mutable StepFn and the forward-only ValFn own the same model - `RealStepFn::step` pulls one LMBatch, runs train_batch, returns (loss, grad_norm=1.0 placeholder). Exhausted iterator returns a finite (1.0, 1.0) so GATE-TRAIN-007 (NaN/Inf) does not mis-fire on shard-stream EOF before the loop plans to stop. - `RealValFn::validate` runs forward-only across a held-out Vec, returns mean cross-entropy loss (or NaN if held-out is empty). - `build_shared_trainer` runs INV-ARCH-370M-001 as a debug_assert (param count must land in [366M, 374M]) so any drift in the Llama370MConfig constants fails the instant a dev build compiles. **Contract coverage** Existing `contracts/training-loop-pretrain-v1.yaml` covers all MVP obligations already; no new contract needed. Task #111 follow-up will add per-epoch APR checkpoint hooks (C-TRAIN-PRETRAIN INV-TRAIN-002) and real optimizer-state sha256 (INV-TRAIN-003). **Tests** - shard_reader: single_shard_yields_expected_batch_count, empty_dir_errors, multi_shard_ordering_is_lexical - pretrain_real: transformer_config_matches_llama_370m_constants, real_step_fn_exhausted_iterator_returns_finite_placeholder, real_val_fn_empty_held_out_returns_nan All 6 new tests PASS. Steps 4-7 (SafeTensors→APR swap, `apr pretrain` CLI wiring, real grad_norm, checkpoint hook) to follow. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat(model-2): wire real-corpus drive into apr pretrain (task #111 step 5) Replaces the `if !synthetic { return Err(...) }` guard with a real branch: build a shared 370M `TransformerTrainer`, split the shard stream head-off into a `HELD_OUT_BATCHES`-entry validation set, and drive the `PretrainLoop` with `RealStepFn`/`RealValFn` (from `entrenar::train::pretrain_real`) against a `ShardBatchIter`. **Structure** - `run` is now a 2-branch dispatcher. `drive_synthetic` preserves the deterministic decay drive used for GATE-TRAIN-005/006/007/008 wiring verification (task #105). `drive_real` is the new real-corpus path. - Both branches funnel into `run_and_report<S, V>` which owns the `PretrainLoop::new` + `run` + `report` sequence so the terminal status propagation (→ exit code) stays single-sourced. **MVP invariants (documented)** - `HELD_OUT_BATCHES = 2` — small constant; follow-up will plumb an explicit `--val-shards` flag so training and held-out shards are disjoint. - `pad_id = eos_id = 0` — uniform-length sequences take the shared layout in `LMBatch::from_sequences`, so pad_id is never used; the real tokenizer's special-token ids plumb through in a follow-up. - Empty dataset dir → `CliError::ValidationFailed` (shard iterator init failure), covered by the new test `real_mode_empty_dataset_dir_errors`. **Test changes** - `real_mode_empty_dataset_dir_errors` replaces the now-obsolete `synthetic_mode_false_rejected` test. Both synthetic and validation tests continue to pass (3/3 in `commands::pretrain::tests`). **Remaining MVP steps (task #111)** - Step 4: swap SafeTensors → APR in `trainer.rs` checkpoint writer. - Step 6: real optimizer-state sha256 over AdamW m/v/t (INV-TRAIN-003). - Step 7: per-epoch checkpoint hook in `PretrainLoop::run_epoch` post-gate-pass (C-TRAIN-PRETRAIN INV-TRAIN-002). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat(model-2): CPU save_apr + per-epoch checkpoint hook (task #111 steps 4+7) Steps 4 and 7 of the MODEL-2 pretrain MVP (SHIP-TWO-001 v2.19.0): Step 4 — CPU save_apr - Add `TransformerTrainer::save_apr(path, name, arch)` in crates/aprender-train/src/train/transformer_trainer/trainer.rs, mirroring the existing CudaTransformerTrainer::save_apr. Emits a sovereign row-major .apr via aprender's Model + SaveConfig::Apr. - Existing `save()` (SafeTensors) left unchanged — three tests at trainer/core.rs:388,409 and tests.rs:423 still round-trip via safetensors for backward compat. - Test `save_apr_writes_readable_apr_file`: write a tiny-config trainer, open with `AprReader`, assert APR magic (APR\0 / APRN), assert `architecture` metadata round-trips, assert `model.embed_tokens.weight` readable as f32. PASSES. Step 7 — per-epoch APR checkpoint hook - Add `pub trait CheckpointFn` in train/pretrain.rs: `fn save(&mut self, epoch, &EpochArtifact) -> Result<(), String>` - Add `Option<Box<dyn CheckpointFn>>` field to `PretrainLoop` + builder method `with_checkpoint_fn`. Keeps PretrainLoop<S,V> at two generics (synthetic + real call-sites unify). - Wire into `run_epoch` AFTER `check_non_divergence(...)?` passes, BEFORE `epoch_artifacts.push()`. Aborted epochs never produce checkpoint files (per contract `per_epoch_artifacts` invariant). Write failures log eprintln but are non-fatal — a flaky disk cannot lose training progress. - Emit companion `metadata.json` (contract path_template). Real-corpus wiring - Add `AprCheckpointFn` in train/pretrain_real.rs holding the shared `Rc<RefCell<TransformerTrainer>>`; its `save()` delegates to `trainer.save_apr()` so the three hooks (RealStepFn, RealValFn, AprCheckpointFn) see the same in-memory weights. - Re-export `CheckpointFn` from train/mod.rs. CLI - `apr pretrain` --real path (drive_real): construct `build_shared_trainer` once, clone Rc into RealStepFn + RealValFn + AprCheckpointFn, pass to `run_and_report`. - `run_and_report` takes `Option<Box<dyn CheckpointFn>>`; synthetic branch passes `None` (no real weights to save). Tests (all green, 21 pretrain + 4 pretrain_real/save_apr + 3 CLI) - `pretrain_loop_calls_checkpoint_fn_once_per_passing_epoch`: mock `CheckpointFn` counts calls. Every successful epoch fires exactly one call; companion metadata.json written to disk. - `pretrain_loop_skips_checkpoint_on_abort`: NaN step forces abort; mock hook recorded zero calls. - `save_apr_writes_readable_apr_file`: magic + metadata + tensor round-trip via AprReader. Contract discharge - GATE-TRAIN-005 invariant preserved: checkpoint placement AFTER divergence guard means aborted epochs never touch disk. - training-loop-pretrain-v1 `per_epoch_artifacts.path_template` honored: `{run_dir}/ckpt/epoch-{N:03d}.apr` + `.metadata.json`. Deferred (Step 6) - `fake_optimizer_sha(epoch)` at pretrain.rs:680 still returns a placeholder. INV-TRAIN-003 discharge needs TransformerTrainer to expose AdamW m/v/t buffers for a real sha256. Separate step. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat(model-2): real AdamW optimizer-state sha256 (task #111 step 6) INV-TRAIN-003 discharge for the MODEL-2 pretrain MVP. TransformerTrainer::optimizer_state_sha256() - New accessor in crates/aprender-train/src/train/transformer_trainer/trainer.rs that hashes (t, m_buffers, v_buffers) in fixed order. - Uses sha2::Sha256 + bytemuck::cast_slice over each Array1<f32>. - Versioned tag "aprender-train:adamw:optstate:v1" prefixes the digest so schema changes are loud, not silent. - Uninitialized slots hash to the literal "none" so missing m[i] is semantically distinct from an all-zeros m[i]. StepFn trait extension - Add `fn optimizer_state_sha256(&self) -> Option<String>` with default `None`. Synthetic harnesses keep returning None and continue using the `fake_optimizer_sha` epoch/seed fallback. - `PretrainLoop::run_epoch` now reads `step_fn.optimizer_state_sha256()` and falls back to the fake fingerprint only when None. RealStepFn override - RealStepFn in pretrain_real.rs implements the new hook by delegating to `trainer.borrow().optimizer_state_sha256()`, so the real-corpus path records the actual AdamW digest. Tests (all 25 + 3 green) - `optimizer_state_sha256_is_hex_digest_on_fresh_trainer`: 64-char lowercase hex shape check on an un-stepped trainer. - `optimizer_state_sha256_is_stable_across_fresh_trainers`: two fresh trainers hash to the same digest (reproducibility). - `pretrain_loop_uses_step_fn_optimizer_sha_when_available`: a StepFn with override wins over fake_optimizer_sha. - `pretrain_loop_falls_back_to_fake_optimizer_sha_for_synthetic`: default impl still produces a 64-char hex digest via fallback. Task #111 MVP status - Steps 1-3 shipped in commit b2b0329 - Step 5 shipped in commit e5a2f02 - Steps 4+7 shipped in commit 89db4b3 - Step 6 shipped in this commit - All 7 steps of the task #111 plan are now committed. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat(model-2): FALSIFY-SHIP-021 seed=0 × 100-step reproducibility harness Discharges GATE-TRAIN-006 / INV-TRAIN-006 from training-loop-pretrain-v1 (bumped 1.0.0 → 1.1.0 PROPOSED → ACTIVE). Two new Rust tests in crates/aprender-train/src/train/transformer_trainer/tests.rs: - falsify_ship_021_seed_0_100_step_reproducibility: two trainers built with seed=0 produce identical finite losses for 100 consecutive train_batch calls (|Δ| ≤ 1e-6) AND identical AdamW optimizer_state_sha256 digests. - falsify_ship_021_different_seeds_do_diverge: seed=0 vs seed=1 counter-test must diverge > 1e-4 within 10 steps (guards against degenerate "always equal" implementations). Seed plumbing fixes: - TransformerTrainer::new now calls lock_init_seed(config.seed) before Transformer::new so direct (non-YAML) callers honor the configured seed instead of silently inheriting the global default of 42. - transformer::init::INIT_SEED_LOCK (std::sync::Mutex) + lock_init_seed helper returning a #[must_use] MutexGuard. Held across the full Transformer::new call so cargo test's default parallel runner cannot clobber the global atomic INIT_SEED between one test's set_init_seed and another test's weight-init reads. Poisoned mutex is recovered transparently (seed itself is atomic; poison only signals prior panic). Contract uplift (contracts/training-loop-pretrain-v1.yaml v1.1.0): - status PROPOSED → ACTIVE - INV-TRAIN-006 gains harness: block naming both test paths + assertions - GATE-TRAIN-006 gains evidence_discharged_by: pointing to both tests - metadata.changelog entry recording the discharge Verification: cargo test -p aprender-train --lib falsify_ship_021 → 2 passed cargo clippy -p aprender-train --lib --no-deps -- -D warnings → clean pv validate contracts/training-loop-pretrain-v1.yaml → 0 errors Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat(ship-two): FALSIFY-SHIP-022 apr inspect provenance (AC-SHIP2-012) Discharges FALSIFY-SHIP-022: apr inspect surfaces license + data_source + data_license on every .apr, with "(missing)" / null rendering when a field is absent rather than silent skip. Makes a .apr binary a sufficient provenance-audit artifact (no sidecar manifest required). Contract: contracts/apr-provenance-v1.yaml (C-APR-PROVENANCE v1.0.0, ACTIVE, kind: schema). 3 invariants + 3 gates + 3 failure modes, all bound to AC-SHIP2-012 / FALSIFY-SHIP-022. pv validate PASS. Code changes: - AprV2Metadata: add data_source + data_license as named Option<String> fields (not buried in custom HashMap). No skip_serializing_if, so JSON round-trips them as null when None (FM-APR-PROV-SILENT-SKIP). - apr inspect MetadataInfo: mirror all 3 provenance fields, also with no skip_serializing_if. - apr inspect text output: new "Provenance:" block via pure helper format_provenance_block() — always emits all 3 keys, renders None as literal "(missing)". - Two struct-literal construction sites updated for new fields. Harness tests (5 passing): - aprender-core: - falsify_ship_022_apr_metadata_provenance_round_trip - falsify_ship_022_inspect_emits_provenance_keys (JSON null half) - falsify_ship_022_partial_provenance_round_trip - apr-cli: - falsify_ship_022_inspect_emits_provenance_keys (MetadataInfo JSON) - falsify_ship_022_inspect_missing_renders_as_missing (text half) - falsify_ship_022_inspect_populated_renders_values Smoke test: apr inspect on existing .apr (no provenance stored) correctly emits: Provenance: license: (missing) data_source: (missing) data_license: (missing) cargo fmt + cargo clippy (aprender-core, apr-cli) clean. 3239 aprender-core format tests PASS, 85 apr-cli inspect tests PASS. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * docs(ship-two): v2.20.0 amendment — FALSIFY-SHIP-021 + FALSIFY-SHIP-022 DISCHARGED Documents two MODEL-2 ship gates closed in the post-v2.19 evidence window: 1. FALSIFY-SHIP-021 (AC-SHIP2-011) — seed=0 × 100-step reproducibility harness + counter-test seed=0 vs seed=1 divergence proof. Root cause of original flake (sibling test racing on global INIT_SEED atomic) fixed via lock_init_seed(seed) -> MutexGuard. Contract training-loop-pretrain-v1.yaml bumped 1.0.0 → 1.1.0 ACTIVE. Commit 0b8ca8c, task #112. 2. FALSIFY-SHIP-022 (AC-SHIP2-012) — apr inspect provenance block (license + data_source + data_license) shipped. AprV2Metadata extended with 2 named Option<String> fields; no skip_serializing_if (FM-APR-PROV-SILENT-SKIP guard). Pure helper format_provenance_block replaces stdout-capture in tests (gag is NOT parallel-safe). New contract apr-provenance-v1.yaml (C-APR-PROVENANCE v1.0.0 ACTIVE, kind: schema). pv validate PASS. Commit 8f0607d, task #113. Combined status: 2/12 AC-SHIP2 gates DISCHARGED. Remaining 10 block on 370M compute-dispatch (the long-pole from v2.19.0). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat(model-2): FALSIFY-SHIP-011 llama-370m sovereign contract ACTIVE (AC-SHIP2-001) Discharges FALSIFY-SHIP-011 / AC-SHIP2-001 — MODEL-2 370M architectural contract registered AND byte-equally bound to the Rust scaffold that aprender-train consumes. Contract lift: - contracts/model-families/llama-370m-sovereign-v1.yaml - version 1.0.0 → 1.1.0 - status PROPOSED → ACTIVE - GATE-ARCH-370M-001 gains evidence_discharged_by (4 entries) and ship_blocking: true - changelog block added documenting the v1.1.0 discharge Harness tests (crates/aprender-train/src/models/llama_370m.rs): - `falsify_ship_011_rust_scaffold_matches_yaml_contract` — loads the contract via include_str! (compile-time-embedded, no path deps at runtime) and asserts every architecture.* and constraints.* key matches the corresponding Llama370MConfig::* const byte-equally - `falsify_ship_011_sovereign_contract_is_active` — asserts status == ACTIVE (a PROPOSED contract cannot gate a ship) Test run: 6/6 aprender-train::models::llama_370m tests PASS (4 pre- existing + 2 new). pv validate on contract: 0 errors, 0 warnings. Why this discharge is strong: - Rust scaffold already encodes INV-ARCH-370M-002..008 as compile-time `const _: () = Llama370MConfig::validate();` — a drift of any value fails `cargo build`, not just `cargo test` - The new YAML-vs-Rust binding test adds the missing half: drift of a YAML key that the Rust scaffold doesn't mirror is now also caught at test time, preventing the MODEL-1-v2 QLoRA class of recipe/artifact drift (rank=16 actual vs rank=32 recipe — see project_ship_two_001_model1_qlora_divergence.md) - INV-ARCH-370M-001 (param count band) is discharged by the existing `estimated_param_count_within_contract_band` test - INV-ARCH-370M-009 (row-major layout) is discharged by aprender::format::layout_contract at APR load time Combined MODEL-2 status after this commit: 3/12 AC-SHIP2 gates DISCHARGED (001, 011, 012). Remaining 9 (002–010) still block on actual 370M training compute-dispatch — the pretrain loop driver from v2.19.0 is ready to exercise them once the weights exist. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat(model-2): FALSIFY-SHIP-012 algorithm-level PARTIAL discharge (AC-SHIP2-002) Bumps C-TOK-BPE to v1.1.0 and wires evidence_discharged_by into GATE-BPE-003 pointing at 3 existing harness tests in crates/apr-cli/tests/falsify_ship_012_tokenizer_roundtrip.rs and the emitted evidence JSON at evidence/ship-two-001/model-2/falsify-ship-012-tokenizer-roundtrip.json. Status intentionally stays PROPOSED. The gate requires 10K-doc byte-exact round-trip on The Stack v2 Python holdout; task #91 shipped the ingest scaffold (corpus-ingest dry-run CLI) but the 10K fixture itself is not yet materialized — so this lands as PARTIAL_ALGORITHM_LEVEL discharge with full_discharge_blocks_on: task #91 data. What passes algorithm-level today (all 3 tests green at commit time): - falsify_ship_012_tokenizer_roundtrip_byte_exact — decode(encode(nfc(doc))) byte-equals nfc(doc) on every doc in a 20-doc synthetic Python-like holdout (ASCII keywords + Unicode identifiers + docstrings + emoji + combining marks). Hard-asserts evidence.docs_failed == 0 — regressions reintroducing whitespace splitting or dropping the byte encoder panic. - falsify_ship_012_nfc_idempotence_only — INV-BPE-005 standalone: nfc(nfc(x)) byte-equals nfc(x) on every holdout doc. - falsify_ship_012_train_corpus_sanity — train/holdout set disjointness plus minimum corpus sizes (>=20 docs each). When task #91's 10K Stack-v2 Python holdout lands the fixture swap is data-only: the harness module doc-comment already flagged this path so no test rewrite will be required. Evidence: evidence/ship-two-001/model-2/falsify-ship-012-tokenizer-roundtrip.json (20/20 passed, nfc_idempotent: true, vocab_size_trained: 489/512). Verification: - pv validate contracts/tokenizer-bpe-v1.yaml -> 0 errors, 0 warnings - cargo test -p apr-cli --test falsify_ship_012_tokenizer_roundtrip -> 3/3 passed Bound to: AC-SHIP2-002 (ship-two-models-spec §5). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat(model-2): FALSIFY-SHIP-015 algorithm-level PARTIAL discharge (AC-SHIP2-005) Bumps C-LLAMA-370M-SOVEREIGN v1.1.0 → v1.2.0 and wires evidence_discharged_by into GATE-ARCH-370M-003 (the param-count gate that binds AC-SHIP2-005 via FALSIFY-SHIP-015). Contract stays ACTIVE — the FALSIFY-SHIP-011 discharge (v1.1.0) is what gates the ACTIVE promotion, not SHIP-015. GATE-ARCH-370M-003's evidence_required asks for apr inspect --json model.apr | jq '.param_count' ∈ [366M, 374M] on a real 370M `.apr` checkpoint. That file does not exist yet — it blocks on AC-SHIP2-003/004 pretraining compute-dispatch. Rather than leave the gate's evidence blank, this commit wires the algorithm-level proof that already exists: - estimated_param_count() / estimated_stored_param_count() — const fn over Llama370MConfig::*, so the count is computed at compile time. - estimated_param_count_within_contract_band (unit test) hard-asserts: * p ∈ [PARAMETERS_MIN=366M, PARAMETERS_MAX=374M] (INV-ARCH-370M-001) * |p − 370M| / 370M < 5% (tighter sanity) * p − stored == VOCAB_SIZE × HIDDEN_DIM (tied embeddings) Any edit to Llama370MConfig that moves the count out of the INV-ARCH-370M-001 band fails `cargo test -p aprender-train --lib llama_370m` — before any compute runs. The gate now carries: discharge_status: PARTIAL_ALGORITHM_LEVEL full_discharge_blocks_on: "real 370M .apr checkpoint from pretraining compute-dispatch (AC-SHIP2-003/004)" ship_blocking: true so the data-scale gap is first-class contract state, not an unspoken assumption. Verification: - pv validate contracts/model-families/llama-370m-sovereign-v1.yaml -> 0 errors, 0 warnings - cargo test -p aprender-train --lib models::llama_370m -> 6/6 passed (including the newly-cited estimated_param_count_within_contract_band and the pre-existing falsify_ship_011_* pair) MODEL-2 AC-SHIP2 ledger after this: 3/12 fully ACTIVE (001, 011, 012) + 2/12 PARTIAL (002 via SHIP-012, 005 via SHIP-015) = 5/12 touched. Remaining 7 (003/004/006/007/008/009/010) block on 370M compute. Bound to: AC-SHIP2-005 (ship-two-models-spec §5). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * docs(ship-two-001): spec v2.21.0 — FALSIFY-SHIP-011 DISCHARGED + SHIP-012/015 PARTIAL Captures the three evidence-wiring commits landed on chore/post-v2.19-evidence since v2.20.0: 1. FALSIFY-SHIP-011 (AC-SHIP2-001) DISCHARGED at 338c6eb (task #114) C-LLAMA-370M-SOVEREIGN v1.0.0 PROPOSED -> v1.1.0 ACTIVE. Rust-YAML byte-equality binding via include_str! + serde_yaml::Value. 2. FALSIFY-SHIP-012 (AC-SHIP2-002) PARTIAL_ALGORITHM_LEVEL at 2e8b8b8 (task #115). C-TOK-BPE v1.0.0 -> v1.1.0 stays PROPOSED. 3 tokenizer harness tests wired; full discharge blocks on task #91 10K Stack-v2 Python holdout (fixture-swap is data-only). 3. FALSIFY-SHIP-015 (AC-SHIP2-005) PARTIAL_ALGORITHM_LEVEL at bfb8831 (task #116). Sovereign contract v1.1.0 -> v1.2.0 stays ACTIVE. estimated_param_count_within_contract_band + const fns wired; full discharge blocks on real 370M .apr from compute-dispatch. Also codifies the PARTIAL_ALGORITHM_LEVEL pattern as a first-class spec concept: when a gate's evidence_required describes a production-scale check that is not yet runnable but the underlying invariant is provable today at algorithm/compile/unit-test level, wire the algorithm proofs and carry discharge_status + partial_discharge_note + full_discharge_blocks_on + ship_blocking=true to make the data gap first-class contract state. MODEL-2 ship-gate status after v2.21.0: 3/12 fully ACTIVE (001, 011, 012) + 2/12 PARTIAL_ALGORITHM_LEVEL (002, 005) = 5/12 touched (~42%). Remaining 7 block on real 370M compute-dispatch. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat(model-2): FALSIFY-SHIP-019 algorithm-level PARTIAL discharge (AC-SHIP2-009) GATE-ARCH-370M-004 gains evidence_discharged_by + discharge_status: PARTIAL_ALGORITHM_LEVEL. Three algorithm-level invariants wired without training: 1. Coverage — every 370M tensor (219 entries: 1 embed + 1 lm_head + 9 per-layer × 24 layers + 1 final norm) resolves to a TensorContract entry in LayoutContract::new(). Pattern-normalises per-layer names; any uncovered tensor would be silently skipped by GGUF export. 2. Row-major ordering (INV-ARCH-370M-009) — every 2D shape is [out_dim, in_dim]. Pinned lm_head/embed/q_proj/k_proj shapes verify GQA (k_proj = [kv_heads*head_dim, hidden]) and bind the 370M architecture to the GH-202-regression-proof layout. 3. Critical-tensor enforcement — validate_apr_shape accepts [vocab, hidden] AND rejects reversed [hidden, vocab] on lm_head.weight. Proves the validator catches layout bugs, not just passes silently. Full discharge (GGUF cosine-parity on trained 370M, max_logit_cosine ≤ 1e-3 over 100 canary prompts) blocks on compute-dispatch (AC-SHIP2-003/004). Harness is fixture-swap-ready once a trained .apr exists — no test rewrite needed. Spec §9 Risk #2 names this exact mitigation path. Contract: llama-370m-sovereign-v1.yaml v1.2.0 → v1.3.0, stays ACTIVE. Tests: 2 new test fns in crates/aprender-train/src/models/llama_370m.rs (8/8 pass). `pv validate` = 0 errors, 0 warnings. Closes #117. Binds to AC-SHIP2-009 / FALSIFY-SHIP-019. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * docs(ship-two-001): v2.22.0 — FALSIFY-SHIP-019 PARTIAL discharge capstone Records the SHIP-019 algorithm-level PARTIAL discharge (task #117, commit 846cc1d) in the authoritative spec: - Version bump 2.21.0 → 2.22.0 - Full amendment block #4 under post-v2.19 evidence window documenting GATE-ARCH-370M-004 wired to `layout_contract.rs` algorithm proofs (219-tensor coverage + row-major ordering + GH-202 rejection) - New "counter-example hunting" pattern lesson: prior "exhausted PARTIAL levers" verdict was ~86% correct; re-running the 7-gate FALSIFY-SHIP survey with explicit counter-example hunting found exactly one genuine lever (SHIP-019). SHIP-017/018/020 need compute; SHIP-013/014/016 collapse into SHIP-011 wiring. - Combined MODEL-2 ledger: 3/12 fully ACTIVE + 3/12 PARTIAL = 6/12 touched (50%). Remaining 6 (003/004/006/007/008/010) all require real 370M compute, trained .apr + eval harness, or RTX 4090 wall-clock benchmark. Genuine algorithm-level PARTIAL harvesting for MODEL-2 is now exhausted. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * chore(publish): mark 5 QA harness crates publish = false + document policy Evidence: aprender-qa-{cli,gen,runner,report,certify} have never been published to crates.io (verified against crates.io API 2026-04-19). They are reached through `apr qa` (the user-facing binary), not through `cargo add`, so marking them publish = false prevents accidental version-bump-with-no-publish drift across the workspace. Spec §A.12 rewritten from the stale "63 crates (49 published + 14 internal)" snapshot to the real 80-crate layout: 9 publish = false (4 benchmarks/xtask + 5 QA harness) plus 71 publishable. §A.12.1 codifies publishing policy: three opt-out categories (benchmarks, xtask, QA harness), and the rule that a v0.31.0-style release does NOT require cargo publish across all 80 crates — crates.io publish is selective (via cargo workspaces publish --from-git or cargo publish -p <name>), workspace-wide tag/release is not. Verified: cargo check --workspace clean after the flip. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * docs(mcp-spec): refresh header — M1–M3 SHIPPED in v0.31.0, M4 in flight Five-whys on the stale 2026-04-17 draft status: 1. Why stale? Spec said "DRAFT (pre-implementation)" + target "v0.32.0" but M1–M3 actually shipped in v0.31.0 on 2026-04-19 (tag 62893da). 2. Why not refreshed? M1–M3 landed across multiple PRs without a spec-header refresh pass. 3. Why is that a problem? New contributors reading the spec think MCP is unshipped — contradicted by `cargo install aprender` already exposing `apr mcp` with 9 tools. 4. Root cause: spec headers are not on the release checklist. 5. Fix here: update status to ACTIVE, version to 1.2.0, delivery line to "v0.31.0 M1–M3 SHIPPED / M4 in flight (PRs #886-892)". No body changes — architecture/tool-surface/protocol sections are still accurate. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * chore(publish): mark aprender-viz-ttop publish = false + 4th category Evidence: `aprender-viz-ttop` has never been published to crates.io (release workflow explicitly never invokes `cargo publish` for it). Its `description` field calls it a "Terminal Top: 10X better than btop" system monitor — ships as a binary subcommand inside the `apr` facade, not as a library dependency. Five-whys: 1. Why flip it? Because it's a bundled binary, not a library. 2. Why does that matter? `cargo add aprender-viz-ttop` would mislead library authors into taking a user-facing TUI as a dep. 3. Why wasn't it already flipped? It predated the A.12 policy audit performed in 42907db. 4. Why a 4th category? Benchmarks / xtask / QA harness all leave outputs as artifacts; this one ships a runnable subcommand. The distinction matters because `apr cbtop` dispatches to it. 5. Why document it? To prevent a future reader from re-opening the "publish all 80 crates" question when we only publish ~70. Changes: - crates/aprender-viz-ttop/Cargo.toml: add `publish = false` - docs/specifications/aprender-monorepo-consolidation.md: - §A.12: add viz-ttop to internal-crates table (10 rows) - §A.12.1: add 4th category (Bundled binaries); update total to "10 opted out / 70 publishable"; remove stale "Candidates to migrate" paragraph (superseded by 42907db + this commit) Refs: APR-MONO, PR #901 Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
noahgift
added a commit
that referenced
this pull request
Apr 19, 2026
#902) * evidence(ship-two-001): MODEL-2 pretrain smoke test — task #105 discharge Records the end-to-end synthetic drive of `apr pretrain` on commit 1e7cf53 (now landed on main at 9209383 via PR #882 merge). Verifies task #105 deliverable: GATE-TRAIN-005 / INV-TRAIN-007 / GATE-TRAIN-008 wiring is functional end-to-end. Run: 20 steps, 4 epochs, batch=4, seq=128 — val_loss monotone 3.96 → 2.64. Synthetic drive caveat: no real 370M forward pass, no real corpus read, no checkpoint artifacts written yet. Real corpus + checkpoint wiring tracked as task #111. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * spec(model-2): MVP plan for task #111 (pretrain real corpus + checkpoint) 7-step edit list from Plan agent afd391d1eb1395d30 against post-#882-merge commit 9209383. Identifies 5 critical files (pretrain.rs, apr-cli/commands/pretrain.rs, trainer.rs, transformer/model.rs, io/save.rs) and 5 binary acceptance criteria (AC-111-001..005). Host assignment: lambda-labs (impl), yoga (8GB smoke), gx10 (parity). Non-goals explicitly deferred: async H2D streaming, full corpus-ingest pipeline, mixed-precision scaler tuning, distributed training, convergence budget, resume round-trip, nvml telemetry, apr qa post-hoc validators. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * evidence(ship-two-001): yoga parity smoke — GATE-TRAIN-006 discharged Cross-host byte-identical loss history on yoga RTX 4060 Laptop (8GB): lambda-labs: [3.96, 3.52, 3.08, 2.64] yoga: [3.96, 3.52, 3.08, 2.64] Discharges GATE-TRAIN-006 (seed=42 deterministic) across x86_64 RTX 4090 ↔ x86_64 RTX 4060 Laptop. Same synthetic drive — task #111 MVP will add the real 370M forward pass; yoga stays as 8GB smoke-test host per MVP plan's host assignment table. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat(model-2): RealStepFn/RealValFn + shard reader (task #111 steps 1-3) Implements MODEL-2 pretrain MVP plan steps 1-3: the model-agnostic PretrainLoop now has a real-corpus driver that runs a full forward + backward + AdamW step through TransformerTrainer against the 370M Llama scaffold — replacing the LinearDecaySynthetic/ScriptedVal pair used for GATE-TRAIN-005/006/007/008 wiring verification in task #105. **New modules** - `train::shard_reader::ShardBatchIter` Streaming iterator over .bin token shards (little-endian u32). Reads seq_length+1 sequences, chunks into LMBatch of batch_size. Empty-dir errors; lexical shard ordering; EOF auto-advances to next shard. No MinHash dedup / PII scrub / license filter — those belong to `apr-corpus-ingest run`. - `train::pretrain_real::{RealStepFn, RealValFn, build_shared_trainer}` - `llama_370m_transformer_config()` field-for-field from the frozen Llama370MConfig constants (INV-ARCH-370M-001..008 source of truth) - `llama_370m_train_config(lr, seq_length, seed)` builds TransformerTrainConfig with MODEL-2 v2-remedy defaults - `SharedTrainer = Rc<RefCell<TransformerTrainer>>` so both the mutable StepFn and the forward-only ValFn own the same model - `RealStepFn::step` pulls one LMBatch, runs train_batch, returns (loss, grad_norm=1.0 placeholder). Exhausted iterator returns a finite (1.0, 1.0) so GATE-TRAIN-007 (NaN/Inf) does not mis-fire on shard-stream EOF before the loop plans to stop. - `RealValFn::validate` runs forward-only across a held-out Vec, returns mean cross-entropy loss (or NaN if held-out is empty). - `build_shared_trainer` runs INV-ARCH-370M-001 as a debug_assert (param count must land in [366M, 374M]) so any drift in the Llama370MConfig constants fails the instant a dev build compiles. **Contract coverage** Existing `contracts/training-loop-pretrain-v1.yaml` covers all MVP obligations already; no new contract needed. Task #111 follow-up will add per-epoch APR checkpoint hooks (C-TRAIN-PRETRAIN INV-TRAIN-002) and real optimizer-state sha256 (INV-TRAIN-003). **Tests** - shard_reader: single_shard_yields_expected_batch_count, empty_dir_errors, multi_shard_ordering_is_lexical - pretrain_real: transformer_config_matches_llama_370m_constants, real_step_fn_exhausted_iterator_returns_finite_placeholder, real_val_fn_empty_held_out_returns_nan All 6 new tests PASS. Steps 4-7 (SafeTensors→APR swap, `apr pretrain` CLI wiring, real grad_norm, checkpoint hook) to follow. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat(model-2): wire real-corpus drive into apr pretrain (task #111 step 5) Replaces the `if !synthetic { return Err(...) }` guard with a real branch: build a shared 370M `TransformerTrainer`, split the shard stream head-off into a `HELD_OUT_BATCHES`-entry validation set, and drive the `PretrainLoop` with `RealStepFn`/`RealValFn` (from `entrenar::train::pretrain_real`) against a `ShardBatchIter`. **Structure** - `run` is now a 2-branch dispatcher. `drive_synthetic` preserves the deterministic decay drive used for GATE-TRAIN-005/006/007/008 wiring verification (task #105). `drive_real` is the new real-corpus path. - Both branches funnel into `run_and_report<S, V>` which owns the `PretrainLoop::new` + `run` + `report` sequence so the terminal status propagation (→ exit code) stays single-sourced. **MVP invariants (documented)** - `HELD_OUT_BATCHES = 2` — small constant; follow-up will plumb an explicit `--val-shards` flag so training and held-out shards are disjoint. - `pad_id = eos_id = 0` — uniform-length sequences take the shared layout in `LMBatch::from_sequences`, so pad_id is never used; the real tokenizer's special-token ids plumb through in a follow-up. - Empty dataset dir → `CliError::ValidationFailed` (shard iterator init failure), covered by the new test `real_mode_empty_dataset_dir_errors`. **Test changes** - `real_mode_empty_dataset_dir_errors` replaces the now-obsolete `synthetic_mode_false_rejected` test. Both synthetic and validation tests continue to pass (3/3 in `commands::pretrain::tests`). **Remaining MVP steps (task #111)** - Step 4: swap SafeTensors → APR in `trainer.rs` checkpoint writer. - Step 6: real optimizer-state sha256 over AdamW m/v/t (INV-TRAIN-003). - Step 7: per-epoch checkpoint hook in `PretrainLoop::run_epoch` post-gate-pass (C-TRAIN-PRETRAIN INV-TRAIN-002). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat(model-2): CPU save_apr + per-epoch checkpoint hook (task #111 steps 4+7) Steps 4 and 7 of the MODEL-2 pretrain MVP (SHIP-TWO-001 v2.19.0): Step 4 — CPU save_apr - Add `TransformerTrainer::save_apr(path, name, arch)` in crates/aprender-train/src/train/transformer_trainer/trainer.rs, mirroring the existing CudaTransformerTrainer::save_apr. Emits a sovereign row-major .apr via aprender's Model + SaveConfig::Apr. - Existing `save()` (SafeTensors) left unchanged — three tests at trainer/core.rs:388,409 and tests.rs:423 still round-trip via safetensors for backward compat. - Test `save_apr_writes_readable_apr_file`: write a tiny-config trainer, open with `AprReader`, assert APR magic (APR\0 / APRN), assert `architecture` metadata round-trips, assert `model.embed_tokens.weight` readable as f32. PASSES. Step 7 — per-epoch APR checkpoint hook - Add `pub trait CheckpointFn` in train/pretrain.rs: `fn save(&mut self, epoch, &EpochArtifact) -> Result<(), String>` - Add `Option<Box<dyn CheckpointFn>>` field to `PretrainLoop` + builder method `with_checkpoint_fn`. Keeps PretrainLoop<S,V> at two generics (synthetic + real call-sites unify). - Wire into `run_epoch` AFTER `check_non_divergence(...)?` passes, BEFORE `epoch_artifacts.push()`. Aborted epochs never produce checkpoint files (per contract `per_epoch_artifacts` invariant). Write failures log eprintln but are non-fatal — a flaky disk cannot lose training progress. - Emit companion `metadata.json` (contract path_template). Real-corpus wiring - Add `AprCheckpointFn` in train/pretrain_real.rs holding the shared `Rc<RefCell<TransformerTrainer>>`; its `save()` delegates to `trainer.save_apr()` so the three hooks (RealStepFn, RealValFn, AprCheckpointFn) see the same in-memory weights. - Re-export `CheckpointFn` from train/mod.rs. CLI - `apr pretrain` --real path (drive_real): construct `build_shared_trainer` once, clone Rc into RealStepFn + RealValFn + AprCheckpointFn, pass to `run_and_report`. - `run_and_report` takes `Option<Box<dyn CheckpointFn>>`; synthetic branch passes `None` (no real weights to save). Tests (all green, 21 pretrain + 4 pretrain_real/save_apr + 3 CLI) - `pretrain_loop_calls_checkpoint_fn_once_per_passing_epoch`: mock `CheckpointFn` counts calls. Every successful epoch fires exactly one call; companion metadata.json written to disk. - `pretrain_loop_skips_checkpoint_on_abort`: NaN step forces abort; mock hook recorded zero calls. - `save_apr_writes_readable_apr_file`: magic + metadata + tensor round-trip via AprReader. Contract discharge - GATE-TRAIN-005 invariant preserved: checkpoint placement AFTER divergence guard means aborted epochs never touch disk. - training-loop-pretrain-v1 `per_epoch_artifacts.path_template` honored: `{run_dir}/ckpt/epoch-{N:03d}.apr` + `.metadata.json`. Deferred (Step 6) - `fake_optimizer_sha(epoch)` at pretrain.rs:680 still returns a placeholder. INV-TRAIN-003 discharge needs TransformerTrainer to expose AdamW m/v/t buffers for a real sha256. Separate step. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat(model-2): real AdamW optimizer-state sha256 (task #111 step 6) INV-TRAIN-003 discharge for the MODEL-2 pretrain MVP. TransformerTrainer::optimizer_state_sha256() - New accessor in crates/aprender-train/src/train/transformer_trainer/trainer.rs that hashes (t, m_buffers, v_buffers) in fixed order. - Uses sha2::Sha256 + bytemuck::cast_slice over each Array1<f32>. - Versioned tag "aprender-train:adamw:optstate:v1" prefixes the digest so schema changes are loud, not silent. - Uninitialized slots hash to the literal "none" so missing m[i] is semantically distinct from an all-zeros m[i]. StepFn trait extension - Add `fn optimizer_state_sha256(&self) -> Option<String>` with default `None`. Synthetic harnesses keep returning None and continue using the `fake_optimizer_sha` epoch/seed fallback. - `PretrainLoop::run_epoch` now reads `step_fn.optimizer_state_sha256()` and falls back to the fake fingerprint only when None. RealStepFn override - RealStepFn in pretrain_real.rs implements the new hook by delegating to `trainer.borrow().optimizer_state_sha256()`, so the real-corpus path records the actual AdamW digest. Tests (all 25 + 3 green) - `optimizer_state_sha256_is_hex_digest_on_fresh_trainer`: 64-char lowercase hex shape check on an un-stepped trainer. - `optimizer_state_sha256_is_stable_across_fresh_trainers`: two fresh trainers hash to the same digest (reproducibility). - `pretrain_loop_uses_step_fn_optimizer_sha_when_available`: a StepFn with override wins over fake_optimizer_sha. - `pretrain_loop_falls_back_to_fake_optimizer_sha_for_synthetic`: default impl still produces a 64-char hex digest via fallback. Task #111 MVP status - Steps 1-3 shipped in commit b2b0329 - Step 5 shipped in commit e5a2f02 - Steps 4+7 shipped in commit 89db4b3 - Step 6 shipped in this commit - All 7 steps of the task #111 plan are now committed. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat(model-2): FALSIFY-SHIP-021 seed=0 × 100-step reproducibility harness Discharges GATE-TRAIN-006 / INV-TRAIN-006 from training-loop-pretrain-v1 (bumped 1.0.0 → 1.1.0 PROPOSED → ACTIVE). Two new Rust tests in crates/aprender-train/src/train/transformer_trainer/tests.rs: - falsify_ship_021_seed_0_100_step_reproducibility: two trainers built with seed=0 produce identical finite losses for 100 consecutive train_batch calls (|Δ| ≤ 1e-6) AND identical AdamW optimizer_state_sha256 digests. - falsify_ship_021_different_seeds_do_diverge: seed=0 vs seed=1 counter-test must diverge > 1e-4 within 10 steps (guards against degenerate "always equal" implementations). Seed plumbing fixes: - TransformerTrainer::new now calls lock_init_seed(config.seed) before Transformer::new so direct (non-YAML) callers honor the configured seed instead of silently inheriting the global default of 42. - transformer::init::INIT_SEED_LOCK (std::sync::Mutex) + lock_init_seed helper returning a #[must_use] MutexGuard. Held across the full Transformer::new call so cargo test's default parallel runner cannot clobber the global atomic INIT_SEED between one test's set_init_seed and another test's weight-init reads. Poisoned mutex is recovered transparently (seed itself is atomic; poison only signals prior panic). Contract uplift (contracts/training-loop-pretrain-v1.yaml v1.1.0): - status PROPOSED → ACTIVE - INV-TRAIN-006 gains harness: block naming both test paths + assertions - GATE-TRAIN-006 gains evidence_discharged_by: pointing to both tests - metadata.changelog entry recording the discharge Verification: cargo test -p aprender-train --lib falsify_ship_021 → 2 passed cargo clippy -p aprender-train --lib --no-deps -- -D warnings → clean pv validate contracts/training-loop-pretrain-v1.yaml → 0 errors Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat(ship-two): FALSIFY-SHIP-022 apr inspect provenance (AC-SHIP2-012) Discharges FALSIFY-SHIP-022: apr inspect surfaces license + data_source + data_license on every .apr, with "(missing)" / null rendering when a field is absent rather than silent skip. Makes a .apr binary a sufficient provenance-audit artifact (no sidecar manifest required). Contract: contracts/apr-provenance-v1.yaml (C-APR-PROVENANCE v1.0.0, ACTIVE, kind: schema). 3 invariants + 3 gates + 3 failure modes, all bound to AC-SHIP2-012 / FALSIFY-SHIP-022. pv validate PASS. Code changes: - AprV2Metadata: add data_source + data_license as named Option<String> fields (not buried in custom HashMap). No skip_serializing_if, so JSON round-trips them as null when None (FM-APR-PROV-SILENT-SKIP). - apr inspect MetadataInfo: mirror all 3 provenance fields, also with no skip_serializing_if. - apr inspect text output: new "Provenance:" block via pure helper format_provenance_block() — always emits all 3 keys, renders None as literal "(missing)". - Two struct-literal construction sites updated for new fields. Harness tests (5 passing): - aprender-core: - falsify_ship_022_apr_metadata_provenance_round_trip - falsify_ship_022_inspect_emits_provenance_keys (JSON null half) - falsify_ship_022_partial_provenance_round_trip - apr-cli: - falsify_ship_022_inspect_emits_provenance_keys (MetadataInfo JSON) - falsify_ship_022_inspect_missing_renders_as_missing (text half) - falsify_ship_022_inspect_populated_renders_values Smoke test: apr inspect on existing .apr (no provenance stored) correctly emits: Provenance: license: (missing) data_source: (missing) data_license: (missing) cargo fmt + cargo clippy (aprender-core, apr-cli) clean. 3239 aprender-core format tests PASS, 85 apr-cli inspect tests PASS. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * docs(ship-two): v2.20.0 amendment — FALSIFY-SHIP-021 + FALSIFY-SHIP-022 DISCHARGED Documents two MODEL-2 ship gates closed in the post-v2.19 evidence window: 1. FALSIFY-SHIP-021 (AC-SHIP2-011) — seed=0 × 100-step reproducibility harness + counter-test seed=0 vs seed=1 divergence proof. Root cause of original flake (sibling test racing on global INIT_SEED atomic) fixed via lock_init_seed(seed) -> MutexGuard. Contract training-loop-pretrain-v1.yaml bumped 1.0.0 → 1.1.0 ACTIVE. Commit 0b8ca8c, task #112. 2. FALSIFY-SHIP-022 (AC-SHIP2-012) — apr inspect provenance block (license + data_source + data_license) shipped. AprV2Metadata extended with 2 named Option<String> fields; no skip_serializing_if (FM-APR-PROV-SILENT-SKIP guard). Pure helper format_provenance_block replaces stdout-capture in tests (gag is NOT parallel-safe). New contract apr-provenance-v1.yaml (C-APR-PROVENANCE v1.0.0 ACTIVE, kind: schema). pv validate PASS. Commit 8f0607d, task #113. Combined status: 2/12 AC-SHIP2 gates DISCHARGED. Remaining 10 block on 370M compute-dispatch (the long-pole from v2.19.0). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat(model-2): FALSIFY-SHIP-011 llama-370m sovereign contract ACTIVE (AC-SHIP2-001) Discharges FALSIFY-SHIP-011 / AC-SHIP2-001 — MODEL-2 370M architectural contract registered AND byte-equally bound to the Rust scaffold that aprender-train consumes. Contract lift: - contracts/model-families/llama-370m-sovereign-v1.yaml - version 1.0.0 → 1.1.0 - status PROPOSED → ACTIVE - GATE-ARCH-370M-001 gains evidence_discharged_by (4 entries) and ship_blocking: true - changelog block added documenting the v1.1.0 discharge Harness tests (crates/aprender-train/src/models/llama_370m.rs): - `falsify_ship_011_rust_scaffold_matches_yaml_contract` — loads the contract via include_str! (compile-time-embedded, no path deps at runtime) and asserts every architecture.* and constraints.* key matches the corresponding Llama370MConfig::* const byte-equally - `falsify_ship_011_sovereign_contract_is_active` — asserts status == ACTIVE (a PROPOSED contract cannot gate a ship) Test run: 6/6 aprender-train::models::llama_370m tests PASS (4 pre- existing + 2 new). pv validate on contract: 0 errors, 0 warnings. Why this discharge is strong: - Rust scaffold already encodes INV-ARCH-370M-002..008 as compile-time `const _: () = Llama370MConfig::validate();` — a drift of any value fails `cargo build`, not just `cargo test` - The new YAML-vs-Rust binding test adds the missing half: drift of a YAML key that the Rust scaffold doesn't mirror is now also caught at test time, preventing the MODEL-1-v2 QLoRA class of recipe/artifact drift (rank=16 actual vs rank=32 recipe — see project_ship_two_001_model1_qlora_divergence.md) - INV-ARCH-370M-001 (param count band) is discharged by the existing `estimated_param_count_within_contract_band` test - INV-ARCH-370M-009 (row-major layout) is discharged by aprender::format::layout_contract at APR load time Combined MODEL-2 status after this commit: 3/12 AC-SHIP2 gates DISCHARGED (001, 011, 012). Remaining 9 (002–010) still block on actual 370M training compute-dispatch — the pretrain loop driver from v2.19.0 is ready to exercise them once the weights exist. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat(model-2): FALSIFY-SHIP-012 algorithm-level PARTIAL discharge (AC-SHIP2-002) Bumps C-TOK-BPE to v1.1.0 and wires evidence_discharged_by into GATE-BPE-003 pointing at 3 existing harness tests in crates/apr-cli/tests/falsify_ship_012_tokenizer_roundtrip.rs and the emitted evidence JSON at evidence/ship-two-001/model-2/falsify-ship-012-tokenizer-roundtrip.json. Status intentionally stays PROPOSED. The gate requires 10K-doc byte-exact round-trip on The Stack v2 Python holdout; task #91 shipped the ingest scaffold (corpus-ingest dry-run CLI) but the 10K fixture itself is not yet materialized — so this lands as PARTIAL_ALGORITHM_LEVEL discharge with full_discharge_blocks_on: task #91 data. What passes algorithm-level today (all 3 tests green at commit time): - falsify_ship_012_tokenizer_roundtrip_byte_exact — decode(encode(nfc(doc))) byte-equals nfc(doc) on every doc in a 20-doc synthetic Python-like holdout (ASCII keywords + Unicode identifiers + docstrings + emoji + combining marks). Hard-asserts evidence.docs_failed == 0 — regressions reintroducing whitespace splitting or dropping the byte encoder panic. - falsify_ship_012_nfc_idempotence_only — INV-BPE-005 standalone: nfc(nfc(x)) byte-equals nfc(x) on every holdout doc. - falsify_ship_012_train_corpus_sanity — train/holdout set disjointness plus minimum corpus sizes (>=20 docs each). When task #91's 10K Stack-v2 Python holdout lands the fixture swap is data-only: the harness module doc-comment already flagged this path so no test rewrite will be required. Evidence: evidence/ship-two-001/model-2/falsify-ship-012-tokenizer-roundtrip.json (20/20 passed, nfc_idempotent: true, vocab_size_trained: 489/512). Verification: - pv validate contracts/tokenizer-bpe-v1.yaml -> 0 errors, 0 warnings - cargo test -p apr-cli --test falsify_ship_012_tokenizer_roundtrip -> 3/3 passed Bound to: AC-SHIP2-002 (ship-two-models-spec §5). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat(model-2): FALSIFY-SHIP-015 algorithm-level PARTIAL discharge (AC-SHIP2-005) Bumps C-LLAMA-370M-SOVEREIGN v1.1.0 → v1.2.0 and wires evidence_discharged_by into GATE-ARCH-370M-003 (the param-count gate that binds AC-SHIP2-005 via FALSIFY-SHIP-015). Contract stays ACTIVE — the FALSIFY-SHIP-011 discharge (v1.1.0) is what gates the ACTIVE promotion, not SHIP-015. GATE-ARCH-370M-003's evidence_required asks for apr inspect --json model.apr | jq '.param_count' ∈ [366M, 374M] on a real 370M `.apr` checkpoint. That file does not exist yet — it blocks on AC-SHIP2-003/004 pretraining compute-dispatch. Rather than leave the gate's evidence blank, this commit wires the algorithm-level proof that already exists: - estimated_param_count() / estimated_stored_param_count() — const fn over Llama370MConfig::*, so the count is computed at compile time. - estimated_param_count_within_contract_band (unit test) hard-asserts: * p ∈ [PARAMETERS_MIN=366M, PARAMETERS_MAX=374M] (INV-ARCH-370M-001) * |p − 370M| / 370M < 5% (tighter sanity) * p − stored == VOCAB_SIZE × HIDDEN_DIM (tied embeddings) Any edit to Llama370MConfig that moves the count out of the INV-ARCH-370M-001 band fails `cargo test -p aprender-train --lib llama_370m` — before any compute runs. The gate now carries: discharge_status: PARTIAL_ALGORITHM_LEVEL full_discharge_blocks_on: "real 370M .apr checkpoint from pretraining compute-dispatch (AC-SHIP2-003/004)" ship_blocking: true so the data-scale gap is first-class contract state, not an unspoken assumption. Verification: - pv validate contracts/model-families/llama-370m-sovereign-v1.yaml -> 0 errors, 0 warnings - cargo test -p aprender-train --lib models::llama_370m -> 6/6 passed (including the newly-cited estimated_param_count_within_contract_band and the pre-existing falsify_ship_011_* pair) MODEL-2 AC-SHIP2 ledger after this: 3/12 fully ACTIVE (001, 011, 012) + 2/12 PARTIAL (002 via SHIP-012, 005 via SHIP-015) = 5/12 touched. Remaining 7 (003/004/006/007/008/009/010) block on 370M compute. Bound to: AC-SHIP2-005 (ship-two-models-spec §5). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * docs(ship-two-001): spec v2.21.0 — FALSIFY-SHIP-011 DISCHARGED + SHIP-012/015 PARTIAL Captures the three evidence-wiring commits landed on chore/post-v2.19-evidence since v2.20.0: 1. FALSIFY-SHIP-011 (AC-SHIP2-001) DISCHARGED at 338c6eb (task #114) C-LLAMA-370M-SOVEREIGN v1.0.0 PROPOSED -> v1.1.0 ACTIVE. Rust-YAML byte-equality binding via include_str! + serde_yaml::Value. 2. FALSIFY-SHIP-012 (AC-SHIP2-002) PARTIAL_ALGORITHM_LEVEL at 2e8b8b8 (task #115). C-TOK-BPE v1.0.0 -> v1.1.0 stays PROPOSED. 3 tokenizer harness tests wired; full discharge blocks on task #91 10K Stack-v2 Python holdout (fixture-swap is data-only). 3. FALSIFY-SHIP-015 (AC-SHIP2-005) PARTIAL_ALGORITHM_LEVEL at bfb8831 (task #116). Sovereign contract v1.1.0 -> v1.2.0 stays ACTIVE. estimated_param_count_within_contract_band + const fns wired; full discharge blocks on real 370M .apr from compute-dispatch. Also codifies the PARTIAL_ALGORITHM_LEVEL pattern as a first-class spec concept: when a gate's evidence_required describes a production-scale check that is not yet runnable but the underlying invariant is provable today at algorithm/compile/unit-test level, wire the algorithm proofs and carry discharge_status + partial_discharge_note + full_discharge_blocks_on + ship_blocking=true to make the data gap first-class contract state. MODEL-2 ship-gate status after v2.21.0: 3/12 fully ACTIVE (001, 011, 012) + 2/12 PARTIAL_ALGORITHM_LEVEL (002, 005) = 5/12 touched (~42%). Remaining 7 block on real 370M compute-dispatch. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat(model-2): FALSIFY-SHIP-019 algorithm-level PARTIAL discharge (AC-SHIP2-009) GATE-ARCH-370M-004 gains evidence_discharged_by + discharge_status: PARTIAL_ALGORITHM_LEVEL. Three algorithm-level invariants wired without training: 1. Coverage — every 370M tensor (219 entries: 1 embed + 1 lm_head + 9 per-layer × 24 layers + 1 final norm) resolves to a TensorContract entry in LayoutContract::new(). Pattern-normalises per-layer names; any uncovered tensor would be silently skipped by GGUF export. 2. Row-major ordering (INV-ARCH-370M-009) — every 2D shape is [out_dim, in_dim]. Pinned lm_head/embed/q_proj/k_proj shapes verify GQA (k_proj = [kv_heads*head_dim, hidden]) and bind the 370M architecture to the GH-202-regression-proof layout. 3. Critical-tensor enforcement — validate_apr_shape accepts [vocab, hidden] AND rejects reversed [hidden, vocab] on lm_head.weight. Proves the validator catches layout bugs, not just passes silently. Full discharge (GGUF cosine-parity on trained 370M, max_logit_cosine ≤ 1e-3 over 100 canary prompts) blocks on compute-dispatch (AC-SHIP2-003/004). Harness is fixture-swap-ready once a trained .apr exists — no test rewrite needed. Spec §9 Risk #2 names this exact mitigation path. Contract: llama-370m-sovereign-v1.yaml v1.2.0 → v1.3.0, stays ACTIVE. Tests: 2 new test fns in crates/aprender-train/src/models/llama_370m.rs (8/8 pass). `pv validate` = 0 errors, 0 warnings. Closes #117. Binds to AC-SHIP2-009 / FALSIFY-SHIP-019. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * docs(ship-two-001): v2.22.0 — FALSIFY-SHIP-019 PARTIAL discharge capstone Records the SHIP-019 algorithm-level PARTIAL discharge (task #117, commit 846cc1d) in the authoritative spec: - Version bump 2.21.0 → 2.22.0 - Full amendment block #4 under post-v2.19 evidence window documenting GATE-ARCH-370M-004 wired to `layout_contract.rs` algorithm proofs (219-tensor coverage + row-major ordering + GH-202 rejection) - New "counter-example hunting" pattern lesson: prior "exhausted PARTIAL levers" verdict was ~86% correct; re-running the 7-gate FALSIFY-SHIP survey with explicit counter-example hunting found exactly one genuine lever (SHIP-019). SHIP-017/018/020 need compute; SHIP-013/014/016 collapse into SHIP-011 wiring. - Combined MODEL-2 ledger: 3/12 fully ACTIVE + 3/12 PARTIAL = 6/12 touched (50%). Remaining 6 (003/004/006/007/008/010) all require real 370M compute, trained .apr + eval harness, or RTX 4090 wall-clock benchmark. Genuine algorithm-level PARTIAL harvesting for MODEL-2 is now exhausted. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * chore(publish): mark 5 QA harness crates publish = false + document policy Evidence: aprender-qa-{cli,gen,runner,report,certify} have never been published to crates.io (verified against crates.io API 2026-04-19). They are reached through `apr qa` (the user-facing binary), not through `cargo add`, so marking them publish = false prevents accidental version-bump-with-no-publish drift across the workspace. Spec §A.12 rewritten from the stale "63 crates (49 published + 14 internal)" snapshot to the real 80-crate layout: 9 publish = false (4 benchmarks/xtask + 5 QA harness) plus 71 publishable. §A.12.1 codifies publishing policy: three opt-out categories (benchmarks, xtask, QA harness), and the rule that a v0.31.0-style release does NOT require cargo publish across all 80 crates — crates.io publish is selective (via cargo workspaces publish --from-git or cargo publish -p <name>), workspace-wide tag/release is not. Verified: cargo check --workspace clean after the flip. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * docs(mcp-spec): refresh header — M1–M3 SHIPPED in v0.31.0, M4 in flight Five-whys on the stale 2026-04-17 draft status: 1. Why stale? Spec said "DRAFT (pre-implementation)" + target "v0.32.0" but M1–M3 actually shipped in v0.31.0 on 2026-04-19 (tag 62893da). 2. Why not refreshed? M1–M3 landed across multiple PRs without a spec-header refresh pass. 3. Why is that a problem? New contributors reading the spec think MCP is unshipped — contradicted by `cargo install aprender` already exposing `apr mcp` with 9 tools. 4. Root cause: spec headers are not on the release checklist. 5. Fix here: update status to ACTIVE, version to 1.2.0, delivery line to "v0.31.0 M1–M3 SHIPPED / M4 in flight (PRs #886-892)". No body changes — architecture/tool-surface/protocol sections are still accurate. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * chore(publish): mark aprender-viz-ttop publish = false + 4th category Evidence: `aprender-viz-ttop` has never been published to crates.io (release workflow explicitly never invokes `cargo publish` for it). Its `description` field calls it a "Terminal Top: 10X better than btop" system monitor — ships as a binary subcommand inside the `apr` facade, not as a library dependency. Five-whys: 1. Why flip it? Because it's a bundled binary, not a library. 2. Why does that matter? `cargo add aprender-viz-ttop` would mislead library authors into taking a user-facing TUI as a dep. 3. Why wasn't it already flipped? It predated the A.12 policy audit performed in 42907db. 4. Why a 4th category? Benchmarks / xtask / QA harness all leave outputs as artifacts; this one ships a runnable subcommand. The distinction matters because `apr cbtop` dispatches to it. 5. Why document it? To prevent a future reader from re-opening the "publish all 80 crates" question when we only publish ~70. Changes: - crates/aprender-viz-ttop/Cargo.toml: add `publish = false` - docs/specifications/aprender-monorepo-consolidation.md: - §A.12: add viz-ttop to internal-crates table (10 rows) - §A.12.1: add 4th category (Bundled binaries); update total to "10 opted out / 70 publishable"; remove stale "Candidates to migrate" paragraph (superseded by 42907db + this commit) Refs: APR-MONO, PR #901 Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat(task-123): native Rust pretokenize CLI — close MODEL-2 corpus gap Root-cause fix for pretokenize-to-.bin gap that was blocking task #119 MODEL-2 370M real-compute pretrain smoke. User 2026-04-19 callout "why not fix root cause vs 'hack'" rejected the Python shim path. What ships (uncommitted WIP in `pretrain.rs`/`llama_370m.rs` left out): - `contracts/pretokenize-bin-v1.yaml` v1.0.0 PROPOSED * `pv validate` PASS (0 errors / 0 warnings) * GATE-PRETOK-003 ship-blocking round-trip gate gains `evidence_discharged_by` (4 tests) + `discharge_status: PARTIAL_ALGORITHM_LEVEL`. Full discharge still blocks on cross-host byte-identical test (task #119 lambda-labs dispatch). - `BPETokenizer::from_vocab_merges(vocab, merges, cfg)` loader (crates/aprender-train/src/tokenizer/bpe.rs) * Reads HEX-encoded vocab.json + merges.txt * Detects id collisions, rejects orphan merges * 2 new round-trip tests PASS - `apr tokenize encode-corpus` CLI subcommand (crates/apr-cli/src/commands/tokenize.rs::run_encode_corpus, crates/apr-cli/src/tokenize_commands.rs, crates/apr-cli/src/dispatch_analysis.rs) * Gated `#[cfg(feature = "training")]` * Writes `shard-NNNNN.bin` (u32 LE) + `manifest.json` (schema `pretokenize-bin-v1`) * Flags: --corpus --tokenizer --output --shard-tokens --content-field --normalization --eos-policy * EOS lookup order: `</s>`, `<|endoftext|>`, `<eos>`, `<|eos|>` * "between" policy fix: emit EOS BEFORE each doc except the first (N-1 separators for N docs) - `tests/pretokenize_shard_roundtrip.rs` * `cli_shard_layout_is_read_by_shard_batch_iter` — INV-PRETOK-002 + INV-PRETOK-007 * `multi_shard_names_preserve_order` — INV-PRETOK-004 - `evidence/ship-two-001/pretokenize-bin-v1-partial-discharge.json` documents algorithm-level partial discharge. Manual dogfood: 5-doc fixture → 78 tokens / 1 shard / 312 bytes / 4 EOS separators (N-1 for between-policy) / EOS id = 2 (`</s>`). Next session: wait on task #118 (50257-vocab tokenizer training, PID 2832743, 79min+) then run `apr tokenize encode-corpus` on CSN-Python train split and dispatch to lambda-labs RTX 4090. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
SHIP-TWO-001 spec driven through v2.5.0 → v2.19.0 over 82 commits. This PR landed as the primary feature branch for the entire Ship-Two-Models epic and now encompasses four major work streams:
1. FALSIFY-PM-007 pre-upload Poka-Yoke (original scope, §12.7)
Three gates fire BEFORE any network I/O, catching the bug class that produced a 30.46 GiB F32 safetensors artifact against an fp16 manifest:
manifest.sha256aborts publish (exit 2) before uploadpublish_format)New
apr validate-manifest --artifactsubcommand (v1.1.0 manifest contract) + pre-flight gate wired intoex-04-upload-hf.sh.2. MODEL-1 Teacher SHIPPED (spec v2.11.0, §12.8)
SHIP-TWO-001-MODEL-1-TEACHERtag at 06a3eae; 3 formats on HF paiml/qwen2.5-coder-7b-apache-q4k-v1apr qa --require-golden-outputpromotes SKIPPED → FAIL; FALSIFY-EX-001 wired3. MODEL-2 370M scaffold LANDED (spec v2.15.0 → v2.19.0)
PretrainConfig::model_2_defaults()(LR=5e-5, rank=32, seed=42 — MODEL-1 v2 divergence remedies)apr pretrainCLI (requires--features training) — wires GATE-TRAIN-005 / INV-TRAIN-007 / GATE-TRAIN-008apr tokenize trainCLI +apr-corpus-ingestbinaryaprender-contracts: ModelFamilyVariant, Tokenizer, TrainingLoop, PretrainingCorpus4. Loader / CI hardening (tasks #101 / #108 / #109 / #110)
pv validatePASS)aprender-core/src/format/hardened to skipcontract_id:ModelFamilyVariant contractscargo fmtclean (33 files reformatted) + Rust 1.93.0clippy::doc_overindented_list_itemsallowvalidate-manifest+pretrainadded to contract + test registryContracts bumped
publish-manifest-v1.yamlapr-cli-publish-extra-v1.yamlapr-model-qa-v1.yaml--require-golden-outputeval-sharding-v1.yamltraining-loop-pretrain-v1.yamltokenizer-bpe-v1.yamldataset-thestack-python-v1.yamlllama-370m-sovereign-v1.yamlapr-cli-commands-v1.yamlpretrainentrySpec
SPEC-SHIP-TWO-001bumped v2.0.0 → v2.19.0.Evidence
evidence/ship-two-001/ex-04-preflight-gate-smoketest.json— FALSIFY-PM-007 pathevidence/ship-two-001/ex-01-teacher-qa.json— golden-output gateevidence/ship-two-001/ex-04-xet-postfix-v1.1.3-discharged.json— live HF uploadevidence/ship-two-001/model-2-pretrain-smoke-test.json— GATE-TRAIN-005/007/008 wiring (synthetic drive)Test plan
cargo test -p apr-cli --test cli_commands— 6/6 PASS (FALSIFY-CLI-001..005)cargo test --workspace --lib— full CI green on 1e7cf53cargo fmt --all --check— cleancargo clippy -- -D warnings— clean on Rust 1.93.0 (+ 1 file-level allow)pv validate contracts/...— all 760 contracts PASSapr pretrainsynthetic smoke test: val_loss 3.96 → 3.52 → 3.08 → 2.64 (monotone)Follow-ups (post-merge)
🤖 Generated with Claude Code