feat(format): apr_v2 stamp_provenance_bytes helper — SHIP-009 full-discharge enabler#1050
Merged
Conversation
…scharge enabler Adds a pure Rust helper to patch `license` / `data_source` / `data_license` fields on an existing APR v2 buffer, returning a re-serialized buffer with the same tensor bytes, the same header flags (QUANTIZED, HAS_VOCAB, HAS_MODEL_CARD, etc.), and the LAYOUT-002 jidoka guard always engaged. ## Why now The shipped MODEL-1 teacher (`paiml/qwen2.5-coder-7b-apache-q4k-v1`, `/mnt/nvme-raid0/models/ship-two-001/qwen2.5-coder-7b-instruct-q4k.apr`) was built at commit `06a3eae38` (spec v2.11.0) — before the `GATE-APR-PROV-001/002/003` provenance-writing gates shipped at commit `8f0607d42` (post-v2.19 evidence branch, task #113). Consequence: `apr inspect` reports `Provenance: license: (missing), data_source: (missing), data_license: (missing)` Which means `GATE-APR-PROV-004` (the algorithm gate for `AC-SHIP1-009`) fails at full-discharge time — the `(None, None, None)` triple trips the "at least one None" counter-example class. SHIP-009 is stuck at PARTIAL_ALGORITHM_LEVEL with no tooling path to close it. This commit closes the tooling gap. The release-cycle portion (re-stamp the teacher → re-upload to HF → refresh publish-manifest sha256 → retrigger EX-04..EX-07) is a separate follow-up; that is full-discharge, not this scaffolding. ## Design - Public surface: `pub fn stamp_provenance_bytes(input: &[u8], patch: &ProvenancePatch) -> Result<Vec<u8>, V2FormatError>` and `pub struct ProvenancePatch { license, data_source, data_license }`. - Empty patch (`!patch.has_any()`) is rejected up-front so callers cannot accidentally rewrite without changing the artifact. - Header flags are carried across via new `AprV2Writer::set_header_flags` (2-LOC addition; LAYOUT_ROW_MAJOR is always OR-ed in regardless of input so the LAYOUT-002 jidoka never disengages). - Tensor bytes are copied verbatim — no quantize/dequantize round-trip. ## Tests (all PASS, 6/6) - `stamp_populates_all_three_fields_when_source_is_unpopulated` — happy path for the exact teacher-gap scenario. - `stamp_preserves_tensor_data_byte_for_byte` — regression guard against accidental f32/bytes round-tripping. - `stamp_preserves_header_flags` — QUANTIZED | HAS_VOCAB survive; also asserts LAYOUT_ROW_MAJOR stays engaged. - `stamp_rejects_empty_patch` — `ProvenancePatch::default()` is rejected with an explicit error message. - `stamp_allows_partial_patch_leaving_other_fields_unchanged` — patching only `data_source` preserves an already-set `license` and leaves `data_license` at `None`. - `stamp_is_idempotent_under_identical_patch` — applying the same patch twice yields byte-identical output. ## What this commit does NOT do - No CLI subcommand yet. Wiring `apr stamp <input.apr> --license X --data-source Y --data-license Z --output <out.apr>` is a follow-up under task #141. - No dogfood run on the 7.48 GiB teacher. The 6 unit tests prove the logic on synthetic buffers; a live run is release-cycle work. - No contract extension (no new `GATE-APR-PROV-005` or similar). `apr-provenance-v1` v1.1.0's existing `GATE-APR-PROV-004` is what gates full discharge; the stamp tool enables the evidence, it is not itself a new contract rule. Spec reference: `docs/specifications/aprender-train/ship-two-models-spec.md` §v2.52.0 atomic next action (2) "Teacher provenance gap". Closes scaffolding portion of task #141. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
216b152 to
3454d04
Compare
…per to CLI (#1051) Wraps `aprender::format::v2::stamp_provenance_bytes` (PR #1050) so the shipped MODEL-1 teacher and any other pre-`GATE-APR-PROV-001..003` `.apr` can have its `license` / `data_source` / `data_license` populated post-hoc directly from the shell. ## Surface ``` apr stamp <FILE> \ --license <SPDX> \ --data-source <URL_OR_IDENTIFIER> \ --data-license <SPDX> \ --output <OUT.apr> \ [--force] \ [--json] ``` At least one of the three provenance flags must be set; an empty patch is rejected up-front with a clear CLI error so callers cannot accidentally produce a no-op rewrite. After writing, the output is re-read and parsed to confirm round-trip integrity (a stamped file that no longer parses is a hard ship-blocker — fail fast). ## Live dogfood (RTX 4090, noah-Lambda-Vector) ``` $ apr stamp /mnt/nvme-raid0/models/ship-two-001/qwen2.5-coder-7b-instruct-q4k.apr \ --license "Apache-2.0" \ --data-source "huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct" \ --data-license "Apache-2.0" \ --output /tmp/stamped.apr \ --json { "command": "stamp", "input_bytes": 8035635524, "output_bytes": 8035635652, "tensor_count": 339, "stamped": { "license": "Apache-2.0", "data_source": "huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct", "data_license": "Apache-2.0" } } $ apr inspect /tmp/stamped.apr | grep -A4 Provenance Provenance: license: Apache-2.0 data_source: huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct data_license: Apache-2.0 ``` Compared to the input artifact: ``` $ apr inspect <input> | grep -A4 Provenance Provenance: license: (missing) data_source: (missing) data_license: (missing) ``` — exactly the gap §v2.52.0 atomic next action (2) called out. Output size grew by 128 bytes (metadata JSON expansion) on a 7.48 GiB file, all 339 tensors preserved, header LAYOUT_ROW_MAJOR flag retained, checksum validates. ## Tests (5/5 PASS) - `stamp_cli_populates_all_three_fields` — end-to-end happy path. - `stamp_cli_rejects_empty_patch` — explicit "at least one" CLI error; output file is NOT created on failure. - `stamp_cli_rejects_missing_input` — surfaces `CliError::FileNotFound`. - `stamp_cli_rejects_existing_output_without_force` — error mentions `--force`; pre-existing content is untouched. - `stamp_cli_overwrites_existing_output_with_force` — `--force` works, resulting file parses as valid APR with patched license. ## What this does NOT include - Re-stamping the published teacher artifact + re-uploading to HF + refreshing the publish-manifest sha256 + retriggering EX-04..EX-07. That is the release-cycle portion of SHIP-009 full discharge — this PR is the tooling. ## Stacking Stacked on `feat/apr-stamp-provenance` (PR #1050). Auto-merge of #1050 will cascade into this branch's base. Spec reference: §v2.52.0 atomic next action (2) "Teacher provenance gap". Closes the CLI portion of task #142 / task #141 follow-up. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
…LI-002/005) Previous commit added the `apr stamp` subcommand but missed two sister entries that the contract-coverage tests require: 1. `contracts/apr-cli-commands-v1.yaml` — added a `stamp` entry under `model_ops` with `requires_model: true` and `side_effects: [filesystem]` to mirror `convert`. 2. `crates/apr-cli/tests/cli_commands.rs::registered_commands()` — added `"stamp"` next to `"convert"` so FALSIFY-CLI-002 (no-unregistered-commands) and FALSIFY-CLI-005 (count-matches) pass. CI on PR #1050 caught both via `apr-cli/tests/cli_commands.rs`: thread 'test_no_unregistered_commands' panicked: FALSIFY-CLI-002: Commands in `apr --help` but not in contract: ["stamp"] thread 'test_command_count_matches' panicked: FALSIFY-CLI-005: Command count mismatch. `apr --help` has 79 commands, contract has 78. Both now PASS locally (`cargo test -p apr-cli --test cli_commands` → 6/6 PASS). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
noahgift
added a commit
that referenced
this pull request
Apr 25, 2026
…re-swap (#1054) SHIP-TWO-001 spec v2.52.0 → v2.53.0: GATE-APR-PROV-004 (AC-SHIP1-009) flipped PARTIAL_ALGORITHM_LEVEL → DISCHARGED on noah-Lambda-Vector RTX 4090 via live `apr stamp` re-stamping of the canonical teacher artifact. First MODEL-1 PARTIAL → DISCHARGED of the cycle. Discharge mechanism: 1. Backed up canonical lambda-labs staging artifact: /mnt/nvme-raid0/models/ship-two-001/qwen2.5-coder-7b-instruct-q4k.apr → .pre-stamp.bak.apr 2. Ran `apr stamp` (PR #1050 helper) with: --license Apache-2.0 --data-source huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct --data-license Apache-2.0 3. Replaced canonical with stamped variant. 4. `apr inspect` and `apr inspect --json` now emit non-(missing) string values for all three provenance fields. 5. `apr diff <pre> <post> --json` confirms tensor byte-identity: structural=0, tensor=0, quantization=0, metadata=0; only file_size category shows the +128 byte delta from the 3 metadata strings. apr stamp's "tensor bytes preserved verbatim" honored. Pre-stamp: sha256=0a854098...c73666, size=8035635524 Post-stamp: sha256=a394dd28...0ddeb28, size=8035635652 (+128 bytes) Files changed: - contracts/apr-provenance-v1.yaml v1.1.0 → v1.2.0 GATE-APR-PROV-004 discharge_status: PARTIAL_ALGORITHM_LEVEL → DISCHARGED discharged_evidence block records: host, binary, pre/post sha256, size delta, provenance fields, tooling chain (apr stamp PR #1050 / apr inspect PR #889 / apr diff), tensor byte-identity proof, backup path, and three deferred irreversible-shipped follow-ups (apr-leaderboard checkpoint update, HF Hub re-upload, publish- manifest sha256 bump) — all require explicit user authorization per `feedback_compute_pre_authorized.md` and stay deferred. - crates/aprender-core/src/format/tests/provenance_tests.rs Drift-prevention test updated: assertion flipped from PARTIAL_ALGORITHM_LEVEL → DISCHARGED, added two new assertions on discharged_evidence.host == "noah-Lambda-Vector" and discharged_evidence.evidence_discharged_by_live[non-empty]. - docs/specifications/aprender-train/ship-two-models-spec.md v2.52.0 → v2.53.0 with full atomic-next-action narrative. Coverage tally: 39 PARTIAL + 6 DISCHARGED → 38 + 7. - evidence/ship-009-full-discharge/discharge-evidence-v1.json (NEW) Self-contained JSON evidence document; all sha256s, sizes, command lines, host pin, binary path, deferred-actions list, and discharge rationale captured for future audit. Verification (all green): - cargo test -p aprender-core --lib falsify_ship_009 — 2/2 passes - cargo test -p aprender-core --lib provenance — 81/81 passes - pv lint contracts/apr-provenance-v1.yaml — PASS Methodological note: this PR uses `apr stamp` + `apr inspect` + `apr diff` exclusively (no `eprintln!`, no bash workaround scripts) per `feedback_apr_trace_not_eprintln.md` and `feedback_pv_not_bash_for_contracts.md`. The dogfooded toolchain proved end-to-end on the 7.5 GiB shipped teacher. Memory: feedback_compute_pre_authorized.md (compute lanes pre-authorized), feedback_post_publish_qa_required.md (HF re-upload requires post-publish QA — deferred to user-confirmed PR), reference_lambda_labs_host_locality.md (this host IS lambda-labs). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Merged
4 tasks
noahgift
added a commit
that referenced
this pull request
May 17, 2026
…AT-690 P0-K extension) (#1757) Extends the existing `apr stamp` command (PR #1050 — provenance fields) to also patch HF identity + architecture family slug in place. Unblocks in-place salvage of pre-P0-K APR checkpoints whose architecture stamps were corrupted by the §82 P0-H fallback. ## Background — SPEC §86 root cause The §85 P2-E live run produced 50 epoch checkpoints (~125 GB total) at best val_loss=4.6227. P2-G v1 attempted to resume from P2-E ep49 and the init eval surfaced val_loss=8.60 — proof that --init silently failed to load the trained weights. Root cause: P2-E's init APR pre-dates P0-K (PR #1742), so the P0-H fallback stamped architecture="LlamaForCausalLM" into the trained checkpoint despite the actual tensors being Qwen2-shaped. `apr pretrain --init` reads the (wrong) architecture stamp and rejects the load. ## What this PR adds Three new CLI flags on `apr stamp`: - `--hf-architecture <CLASS>` (e.g. Qwen2ForCausalLM) — the HF class name. PMAT-690 P0-K's upstream stamp. - `--hf-model-type <SLUG>` (e.g. qwen2) — config.json::model_type. - `--architecture <SLUG>` (e.g. qwen2) — the lowercase family slug that `apr pretrain --init` reads for arch dispatch. **This is the load-bearing field** for §86 salvage — patching just hf_architecture alone won't unblock `apr pretrain --init`. The existing `--license` / `--data-source` / `--data-license` flags are unchanged. The patch struct's `has_any()` gate now accepts any combination of the six fields; at least one must be specified or the stamp is rejected up-front. ## Operator workflow for §86 salvage ```bash # Patch a pre-P0-K Qwen2-actual-Llama-stamped checkpoint in place apr stamp /path/to/p2e-epoch-049.apr \ --architecture qwen2 \ --hf-architecture Qwen2ForCausalLM \ --hf-model-type qwen2 \ -o /path/to/p2e-epoch-049-stamped.apr # Verify apr inspect /path/to/p2e-epoch-049-stamped.apr --quality --json | jq .quality # breakdown.hf_identity should jump 0 → 20 # Now usable as init for resume training: apr pretrain --init /path/to/p2e-epoch-049-stamped.apr ... ``` ## Discharges - §86 SPEC amendment (`evidence/p2g-2026-05-17/section-86-draft.md`) — workaround #2 (in-place restamp) - Salvages ~125 GB of pre-P0-K P2-E checkpoints without a 53-min retrain - Establishes a pattern for in-place metadata patching that future spec amendments can build on (e.g., a `--name` / `--description` extension for model card metadata) ## Tests - 2 new unit tests in `aprender-core::format::v2::stamp` (extends 6 → 6, existing tests adjusted for new struct fields via Default) - 2 new CLI tests in `apr-cli::commands::stamp` (extends 5 → 7): - `stamp_p0k_recovers_pre_p0k_apr_identity` — full §86 use case - `stamp_p0k_partial_hf_architecture_only` — verifies field independence (stamp one without touching others) - All 5,944 apr-cli lib tests pass — 0 regressions - All 13,800 aprender-core lib tests pass — 0 regressions ## Refs - PR [#1742](#1742) (PMAT-690 P0-K base) - PR [#1750](#1750) (P3-A `apr inspect --quality` scorer) - PR [#1754](#1754) (SPEC §85 P2-E findings) - PR #1050 (the original `apr stamp` PR — this extends it) - `docs/specifications/aprender-train/ship-model-2-spec.md §86` (forthcoming) - `evidence/p2g-2026-05-17/section-86-draft.md` (root cause + workaround analysis) - `memory/feedback_upstream_metadata_masquerade.md` (methodology #33) Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary (combined: helper + CLI)
This PR now bundles two commits that land SHIP-009 full-discharge tooling end-to-end:
aprender::format::v2::stamp_provenance_bytes— pure Rust helper to patchlicense/data_source/data_licenseon an existing APR v2 buffer. Tensor bytes and header flags preserved.apr stamp <input.apr> --license X --data-source Y --data-license Z --output <out.apr> [--force] [--json]— CLI subcommand wired over the helper.(PR #1051 was opened against this branch and squash-merged in, so both commits ride together.)
Live dogfood on the actual MODEL-1 teacher (RTX 4090 host, noah-Lambda-Vector)
Verification via
apr inspect:license: (missing), data_source: (missing), data_license: (missing)+128 bytesoverhead (JSON metadata expansion on 7.48 GiB file), LAYOUT_ROW_MAJOR retained, checksum validates.Why this matters
Shipped MODEL-1 teacher (
paiml/qwen2.5-coder-7b-apache-q4k-v1) was built at commit06a3eae38(spec v2.11.0) beforeGATE-APR-PROV-001/002/003shipped at8f0607d42(post-v2.19).GATE-APR-PROV-004rejects the(None, None, None)triple → SHIP-009 stuck at PARTIAL_ALGORITHM_LEVEL. This PR closes the tooling gap.Tests
cargo test -p aprender-core --lib format::v2::stampcargo test -p apr-cli --lib commands::stampPlus 1 live dogfood smoke on the 7.48 GiB shipped teacher.
What this PR does NOT include
That's the release-cycle portion of SHIP-009 full discharge — separate from this tooling PR.
Spec reference
docs/specifications/aprender-train/ship-two-models-spec.md§v2.52.0 atomic next action (2) "Teacher provenance gap".Closes
Tasks #141 (helper scaffolding) + #142 (CLI wiring).
🤖 Generated with Claude Code