Skip to content

feat(format): apr_v2 stamp_provenance_bytes helper — SHIP-009 full-discharge enabler#1050

Merged
noahgift merged 3 commits into
mainfrom
feat/apr-stamp-provenance
Apr 25, 2026
Merged

feat(format): apr_v2 stamp_provenance_bytes helper — SHIP-009 full-discharge enabler#1050
noahgift merged 3 commits into
mainfrom
feat/apr-stamp-provenance

Conversation

@noahgift

@noahgift noahgift commented Apr 24, 2026

Copy link
Copy Markdown
Contributor

Summary (combined: helper + CLI)

This PR now bundles two commits that land SHIP-009 full-discharge tooling end-to-end:

  1. aprender::format::v2::stamp_provenance_bytes — pure Rust helper to patch license / data_source / data_license on an existing APR v2 buffer. Tensor bytes and header flags preserved.
  2. apr stamp <input.apr> --license X --data-source Y --data-license Z --output <out.apr> [--force] [--json] — CLI subcommand wired over the helper.

(PR #1051 was opened against this branch and squash-merged in, so both commits ride together.)

Live dogfood on the actual MODEL-1 teacher (RTX 4090 host, noah-Lambda-Vector)

$ apr stamp /mnt/nvme-raid0/models/ship-two-001/qwen2.5-coder-7b-instruct-q4k.apr \
    --license "Apache-2.0" \
    --data-source "huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct" \
    --data-license "Apache-2.0" \
    --output /tmp/stamped.apr \
    --json
{
  "input_bytes":  8035635524,
  "output_bytes": 8035635652,
  "tensor_count": 339,
  "stamped": {
    "license":      "Apache-2.0",
    "data_source":  "huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct",
    "data_license": "Apache-2.0"
  }
}

Verification via apr inspect:

  • Input (shipped teacher): license: (missing), data_source: (missing), data_license: (missing)
  • Stamped output: all three populated, 339/339 tensors preserved, +128 bytes overhead (JSON metadata expansion on 7.48 GiB file), LAYOUT_ROW_MAJOR retained, checksum validates.

Why this matters

Shipped MODEL-1 teacher (paiml/qwen2.5-coder-7b-apache-q4k-v1) was built at commit 06a3eae38 (spec v2.11.0) before GATE-APR-PROV-001/002/003 shipped at 8f0607d42 (post-v2.19). GATE-APR-PROV-004 rejects the (None, None, None) triple → SHIP-009 stuck at PARTIAL_ALGORITHM_LEVEL. This PR closes the tooling gap.

Tests

Suite Pass
cargo test -p aprender-core --lib format::v2::stamp 6/6
cargo test -p apr-cli --lib commands::stamp 5/5

Plus 1 live dogfood smoke on the 7.48 GiB shipped teacher.

What this PR does NOT include

  • Re-uploading the stamped teacher to HF.
  • Refreshing the publish-manifest sha256.
  • Retriggering EX-04..EX-07 evidence.

That's the release-cycle portion of SHIP-009 full discharge — separate from this tooling PR.

Spec reference

docs/specifications/aprender-train/ship-two-models-spec.md §v2.52.0 atomic next action (2) "Teacher provenance gap".

Closes

Tasks #141 (helper scaffolding) + #142 (CLI wiring).

🤖 Generated with Claude Code

@noahgift noahgift enabled auto-merge (squash) April 24, 2026 21:15
…scharge enabler

Adds a pure Rust helper to patch `license` / `data_source` / `data_license`
fields on an existing APR v2 buffer, returning a re-serialized buffer
with the same tensor bytes, the same header flags (QUANTIZED, HAS_VOCAB,
HAS_MODEL_CARD, etc.), and the LAYOUT-002 jidoka guard always engaged.

## Why now

The shipped MODEL-1 teacher (`paiml/qwen2.5-coder-7b-apache-q4k-v1`,
`/mnt/nvme-raid0/models/ship-two-001/qwen2.5-coder-7b-instruct-q4k.apr`)
was built at commit `06a3eae38` (spec v2.11.0) — before the
`GATE-APR-PROV-001/002/003` provenance-writing gates shipped at commit
`8f0607d42` (post-v2.19 evidence branch, task #113).

Consequence: `apr inspect` reports
  `Provenance: license: (missing), data_source: (missing), data_license: (missing)`

Which means `GATE-APR-PROV-004` (the algorithm gate for `AC-SHIP1-009`)
fails at full-discharge time — the `(None, None, None)` triple trips the
"at least one None" counter-example class. SHIP-009 is stuck at
PARTIAL_ALGORITHM_LEVEL with no tooling path to close it.

This commit closes the tooling gap. The release-cycle portion
(re-stamp the teacher → re-upload to HF → refresh publish-manifest
sha256 → retrigger EX-04..EX-07) is a separate follow-up; that is
full-discharge, not this scaffolding.

## Design

- Public surface: `pub fn stamp_provenance_bytes(input: &[u8], patch:
  &ProvenancePatch) -> Result<Vec<u8>, V2FormatError>` and
  `pub struct ProvenancePatch { license, data_source, data_license }`.
- Empty patch (`!patch.has_any()`) is rejected up-front so callers
  cannot accidentally rewrite without changing the artifact.
- Header flags are carried across via new `AprV2Writer::set_header_flags`
  (2-LOC addition; LAYOUT_ROW_MAJOR is always OR-ed in regardless of
  input so the LAYOUT-002 jidoka never disengages).
- Tensor bytes are copied verbatim — no quantize/dequantize round-trip.

## Tests (all PASS, 6/6)

- `stamp_populates_all_three_fields_when_source_is_unpopulated` —
  happy path for the exact teacher-gap scenario.
- `stamp_preserves_tensor_data_byte_for_byte` — regression guard against
  accidental f32/bytes round-tripping.
- `stamp_preserves_header_flags` — QUANTIZED | HAS_VOCAB survive; also
  asserts LAYOUT_ROW_MAJOR stays engaged.
- `stamp_rejects_empty_patch` — `ProvenancePatch::default()` is rejected
  with an explicit error message.
- `stamp_allows_partial_patch_leaving_other_fields_unchanged` —
  patching only `data_source` preserves an already-set `license` and
  leaves `data_license` at `None`.
- `stamp_is_idempotent_under_identical_patch` — applying the same
  patch twice yields byte-identical output.

## What this commit does NOT do

- No CLI subcommand yet. Wiring `apr stamp <input.apr>
  --license X --data-source Y --data-license Z --output <out.apr>`
  is a follow-up under task #141.
- No dogfood run on the 7.48 GiB teacher. The 6 unit tests prove the
  logic on synthetic buffers; a live run is release-cycle work.
- No contract extension (no new `GATE-APR-PROV-005` or similar).
  `apr-provenance-v1` v1.1.0's existing `GATE-APR-PROV-004` is what
  gates full discharge; the stamp tool enables the evidence, it is
  not itself a new contract rule.

Spec reference: `docs/specifications/aprender-train/ship-two-models-spec.md`
§v2.52.0 atomic next action (2) "Teacher provenance gap".

Closes scaffolding portion of task #141.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
noahgift and others added 2 commits April 25, 2026 09:13
…per to CLI (#1051)

Wraps `aprender::format::v2::stamp_provenance_bytes` (PR #1050) so the
shipped MODEL-1 teacher and any other pre-`GATE-APR-PROV-001..003` `.apr`
can have its `license` / `data_source` / `data_license` populated
post-hoc directly from the shell.

## Surface

```
apr stamp <FILE>                                          \
    --license      <SPDX>                                 \
    --data-source  <URL_OR_IDENTIFIER>                    \
    --data-license <SPDX>                                 \
    --output       <OUT.apr>                              \
    [--force]                                             \
    [--json]
```

At least one of the three provenance flags must be set; an empty patch
is rejected up-front with a clear CLI error so callers cannot
accidentally produce a no-op rewrite. After writing, the output is
re-read and parsed to confirm round-trip integrity (a stamped file that
no longer parses is a hard ship-blocker — fail fast).

## Live dogfood (RTX 4090, noah-Lambda-Vector)

```
$ apr stamp /mnt/nvme-raid0/models/ship-two-001/qwen2.5-coder-7b-instruct-q4k.apr \
    --license "Apache-2.0" \
    --data-source "huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct" \
    --data-license "Apache-2.0" \
    --output /tmp/stamped.apr \
    --json
{
  "command": "stamp",
  "input_bytes": 8035635524,
  "output_bytes": 8035635652,
  "tensor_count": 339,
  "stamped": {
    "license": "Apache-2.0",
    "data_source": "huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct",
    "data_license": "Apache-2.0"
  }
}

$ apr inspect /tmp/stamped.apr | grep -A4 Provenance
  Provenance:
    license: Apache-2.0
    data_source: huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct
    data_license: Apache-2.0
```

Compared to the input artifact:
```
$ apr inspect <input> | grep -A4 Provenance
  Provenance:
    license:      (missing)
    data_source:  (missing)
    data_license: (missing)
```

— exactly the gap §v2.52.0 atomic next action (2) called out.

Output size grew by 128 bytes (metadata JSON expansion) on a 7.48 GiB
file, all 339 tensors preserved, header LAYOUT_ROW_MAJOR flag retained,
checksum validates.

## Tests (5/5 PASS)

- `stamp_cli_populates_all_three_fields` — end-to-end happy path.
- `stamp_cli_rejects_empty_patch` — explicit "at least one" CLI error;
  output file is NOT created on failure.
- `stamp_cli_rejects_missing_input` — surfaces `CliError::FileNotFound`.
- `stamp_cli_rejects_existing_output_without_force` — error mentions
  `--force`; pre-existing content is untouched.
- `stamp_cli_overwrites_existing_output_with_force` — `--force` works,
  resulting file parses as valid APR with patched license.

## What this does NOT include

- Re-stamping the published teacher artifact + re-uploading to HF +
  refreshing the publish-manifest sha256 + retriggering EX-04..EX-07.
  That is the release-cycle portion of SHIP-009 full discharge — this
  PR is the tooling.

## Stacking

Stacked on `feat/apr-stamp-provenance` (PR #1050). Auto-merge of #1050
will cascade into this branch's base.

Spec reference: §v2.52.0 atomic next action (2) "Teacher provenance gap".
Closes the CLI portion of task #142 / task #141 follow-up.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
…LI-002/005)

Previous commit added the `apr stamp` subcommand but missed two
sister entries that the contract-coverage tests require:

1. `contracts/apr-cli-commands-v1.yaml` — added a `stamp` entry under
   `model_ops` with `requires_model: true` and `side_effects:
   [filesystem]` to mirror `convert`.
2. `crates/apr-cli/tests/cli_commands.rs::registered_commands()` —
   added `"stamp"` next to `"convert"` so FALSIFY-CLI-002
   (no-unregistered-commands) and FALSIFY-CLI-005 (count-matches) pass.

CI on PR #1050 caught both via `apr-cli/tests/cli_commands.rs`:

  thread 'test_no_unregistered_commands' panicked:
    FALSIFY-CLI-002: Commands in `apr --help` but not in contract: ["stamp"]

  thread 'test_command_count_matches' panicked:
    FALSIFY-CLI-005: Command count mismatch.
    `apr --help` has 79 commands, contract has 78.

Both now PASS locally (`cargo test -p apr-cli --test cli_commands` →
6/6 PASS).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@noahgift noahgift merged commit 71d23fb into main Apr 25, 2026
18 of 20 checks passed
@noahgift noahgift deleted the feat/apr-stamp-provenance branch April 25, 2026 10:08
noahgift added a commit that referenced this pull request Apr 25, 2026
…re-swap (#1054)

SHIP-TWO-001 spec v2.52.0 → v2.53.0: GATE-APR-PROV-004 (AC-SHIP1-009)
flipped PARTIAL_ALGORITHM_LEVEL → DISCHARGED on noah-Lambda-Vector
RTX 4090 via live `apr stamp` re-stamping of the canonical teacher
artifact. First MODEL-1 PARTIAL → DISCHARGED of the cycle.

Discharge mechanism:
1. Backed up canonical lambda-labs staging artifact:
     /mnt/nvme-raid0/models/ship-two-001/qwen2.5-coder-7b-instruct-q4k.apr
   → .pre-stamp.bak.apr
2. Ran `apr stamp` (PR #1050 helper) with:
     --license Apache-2.0
     --data-source huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct
     --data-license Apache-2.0
3. Replaced canonical with stamped variant.
4. `apr inspect` and `apr inspect --json` now emit non-(missing)
   string values for all three provenance fields.
5. `apr diff <pre> <post> --json` confirms tensor byte-identity:
   structural=0, tensor=0, quantization=0, metadata=0; only
   file_size category shows the +128 byte delta from the 3 metadata
   strings. apr stamp's "tensor bytes preserved verbatim" honored.

Pre-stamp:  sha256=0a854098...c73666, size=8035635524
Post-stamp: sha256=a394dd28...0ddeb28, size=8035635652 (+128 bytes)

Files changed:
- contracts/apr-provenance-v1.yaml v1.1.0 → v1.2.0
  GATE-APR-PROV-004 discharge_status: PARTIAL_ALGORITHM_LEVEL → DISCHARGED
  discharged_evidence block records: host, binary, pre/post sha256,
  size delta, provenance fields, tooling chain (apr stamp PR #1050 /
  apr inspect PR #889 / apr diff), tensor byte-identity proof,
  backup path, and three deferred irreversible-shipped follow-ups
  (apr-leaderboard checkpoint update, HF Hub re-upload, publish-
  manifest sha256 bump) — all require explicit user authorization
  per `feedback_compute_pre_authorized.md` and stay deferred.

- crates/aprender-core/src/format/tests/provenance_tests.rs
  Drift-prevention test updated: assertion flipped from
  PARTIAL_ALGORITHM_LEVEL → DISCHARGED, added two new assertions
  on discharged_evidence.host == "noah-Lambda-Vector" and
  discharged_evidence.evidence_discharged_by_live[non-empty].

- docs/specifications/aprender-train/ship-two-models-spec.md
  v2.52.0 → v2.53.0 with full atomic-next-action narrative.
  Coverage tally: 39 PARTIAL + 6 DISCHARGED → 38 + 7.

- evidence/ship-009-full-discharge/discharge-evidence-v1.json (NEW)
  Self-contained JSON evidence document; all sha256s, sizes,
  command lines, host pin, binary path, deferred-actions list,
  and discharge rationale captured for future audit.

Verification (all green):
  - cargo test -p aprender-core --lib falsify_ship_009 — 2/2 passes
  - cargo test -p aprender-core --lib provenance — 81/81 passes
  - pv lint contracts/apr-provenance-v1.yaml — PASS

Methodological note: this PR uses `apr stamp` + `apr inspect` +
`apr diff` exclusively (no `eprintln!`, no bash workaround scripts)
per `feedback_apr_trace_not_eprintln.md` and
`feedback_pv_not_bash_for_contracts.md`. The dogfooded toolchain
proved end-to-end on the 7.5 GiB shipped teacher.

Memory: feedback_compute_pre_authorized.md (compute lanes
pre-authorized), feedback_post_publish_qa_required.md (HF re-upload
requires post-publish QA — deferred to user-confirmed PR),
reference_lambda_labs_host_locality.md (this host IS lambda-labs).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
noahgift added a commit that referenced this pull request May 17, 2026
…AT-690 P0-K extension) (#1757)

Extends the existing `apr stamp` command (PR #1050 — provenance fields)
to also patch HF identity + architecture family slug in place. Unblocks
in-place salvage of pre-P0-K APR checkpoints whose architecture stamps
were corrupted by the §82 P0-H fallback.

## Background — SPEC §86 root cause

The §85 P2-E live run produced 50 epoch checkpoints (~125 GB total) at
best val_loss=4.6227. P2-G v1 attempted to resume from P2-E ep49 and
the init eval surfaced val_loss=8.60 — proof that --init silently
failed to load the trained weights. Root cause: P2-E's init APR
pre-dates P0-K (PR #1742), so the P0-H fallback stamped
architecture="LlamaForCausalLM" into the trained checkpoint despite
the actual tensors being Qwen2-shaped. `apr pretrain --init` reads
the (wrong) architecture stamp and rejects the load.

## What this PR adds

Three new CLI flags on `apr stamp`:

- `--hf-architecture <CLASS>` (e.g. Qwen2ForCausalLM) — the HF class
  name. PMAT-690 P0-K's upstream stamp.
- `--hf-model-type <SLUG>` (e.g. qwen2) — config.json::model_type.
- `--architecture <SLUG>` (e.g. qwen2) — the lowercase family slug
  that `apr pretrain --init` reads for arch dispatch. **This is the
  load-bearing field** for §86 salvage — patching just hf_architecture
  alone won't unblock `apr pretrain --init`.

The existing `--license` / `--data-source` / `--data-license` flags
are unchanged. The patch struct's `has_any()` gate now accepts any
combination of the six fields; at least one must be specified or the
stamp is rejected up-front.

## Operator workflow for §86 salvage

```bash
# Patch a pre-P0-K Qwen2-actual-Llama-stamped checkpoint in place
apr stamp /path/to/p2e-epoch-049.apr \
  --architecture qwen2 \
  --hf-architecture Qwen2ForCausalLM \
  --hf-model-type qwen2 \
  -o /path/to/p2e-epoch-049-stamped.apr

# Verify
apr inspect /path/to/p2e-epoch-049-stamped.apr --quality --json | jq .quality
# breakdown.hf_identity should jump 0 → 20

# Now usable as init for resume training:
apr pretrain --init /path/to/p2e-epoch-049-stamped.apr ...
```

## Discharges

- §86 SPEC amendment (`evidence/p2g-2026-05-17/section-86-draft.md`) —
  workaround #2 (in-place restamp)
- Salvages ~125 GB of pre-P0-K P2-E checkpoints without a 53-min retrain
- Establishes a pattern for in-place metadata patching that future
  spec amendments can build on (e.g., a `--name` / `--description`
  extension for model card metadata)

## Tests

- 2 new unit tests in `aprender-core::format::v2::stamp` (extends 6
  → 6, existing tests adjusted for new struct fields via Default)
- 2 new CLI tests in `apr-cli::commands::stamp` (extends 5 → 7):
  - `stamp_p0k_recovers_pre_p0k_apr_identity` — full §86 use case
  - `stamp_p0k_partial_hf_architecture_only` — verifies field
    independence (stamp one without touching others)
- All 5,944 apr-cli lib tests pass — 0 regressions
- All 13,800 aprender-core lib tests pass — 0 regressions

## Refs

- PR [#1742](#1742) (PMAT-690 P0-K base)
- PR [#1750](#1750) (P3-A `apr inspect --quality` scorer)
- PR [#1754](#1754) (SPEC §85 P2-E findings)
- PR #1050 (the original `apr stamp` PR — this extends it)
- `docs/specifications/aprender-train/ship-model-2-spec.md §86` (forthcoming)
- `evidence/p2g-2026-05-17/section-86-draft.md` (root cause + workaround analysis)
- `memory/feedback_upstream_metadata_masquerade.md` (methodology #33)

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant