Skip to content

feat(pretrain): §87 P0-J' — Chinchilla 20·N hard gate (was 10·N)#1762

Closed
noahgift wants to merge 3 commits into
mainfrom
feat/chinchilla-20n-hard-gate
Closed

feat(pretrain): §87 P0-J' — Chinchilla 20·N hard gate (was 10·N)#1762
noahgift wants to merge 3 commits into
mainfrom
feat/chinchilla-20n-hard-gate

Conversation

@noahgift

Copy link
Copy Markdown
Contributor

Summary

Tightens the Chinchilla hard gate from D/N < 10×D/N < 20× per the external audit directive (2026-05-17) and §85 P2-E + §85.4 P2-G empirical plateau evidence. The 10-20× "ablation band" was empirically proven to also hit plateau (val_loss ≈ 4.65 floor); contract v1.1.0 eliminates the ambiguous band.

Empirical evidence

Run LR Steps D/N Best val_loss Termination
§82 P2-A 5e-5 5000 0.083× 4.7111 @ ep20 EARLY_STOP
§85 P2-E 1.5e-5 5000 0.083× 4.6227 @ ep49 OK CONVERGED
§85.4 P2-G 1.5e-5 10000 0.155× 4.6497 @ ep49 EARLY_STOP

P2-G doubled the compute (10k vs 5k steps) at the same LR/warmup as P2-E. Result: worse best val_loss + EARLY_STOP at ep 49 — marginal-gain decay confirmed. The 10-20× band cannot ship MODEL-2 below val_loss 3.0 regardless of LR / warmup / patience tuning.

What ships

  • crates/apr-cli/src/commands/pretrain.rs: threshold 10.020.0 at two sites (inline gate + unit-test helper). Error message renamed [P0-J][P0-J']. Zone-aware messages (degeneration < 10× vs plateau 10-20×).
  • New unit test chinchilla_hard_gate_rejects_plateau_zone — asserts 15·N fails hard gate, bypass still works.
  • 2 existing tests renamed for new boundary (boundary_10xboundary_20x, accepts_well_provisionedaccepts_compute_optimal).
  • contracts/chinchilla-gate-v1.yaml: v1.0.0 → v1.1.0 with full changelog. New FALSIFY-CHINCHILLA-006 (plateau-zone falsifier). INV-CHINCHILLA-001 formal updated to use 20.0.

Bypass policy unchanged

--force-under-provisioned still lets operators opt into sub-20× runs for ablation, resume, smoke purposes. Bypass log line now names the zone (DEGENERATION <10× vs PLATEAU 10-20×) for audit trail.

Test plan

  • 6 chinchilla-gate unit tests pass (5 updated for 20× + 1 new plateau-zone test)
  • 5,943 apr-cli lib tests pass — 0 regressions
  • cargo test -p aprender-contracts --lib lint::gates::tests::load_contracts_real — contract schema valid

Methodology

Lesson #36 (memory/feedback_audit_hypothesis_bounds.md) applied: the v1.0.0 10× threshold was correct for "definitely-broken" but allowed an "ablation band" that the empirical sequence proved also hits plateau. Tightening to 20× eliminates the ambiguous band.

Operator impact

Every dispatch with D/N < 20× now requires --force-under-provisioned. For MODEL-2 0.5B at the current 5000 steps × 16 batch × 512 seq = 40.96M token budget, this means all current dispatches need the bypass flag (D/N = 0.083×).

The honest framing: at this batch × seq × N, compute-optimal needs 1.21M steps (~213 GPU-hours / ~9 days on RTX 4090). Either the operator dispatches a long compute-authorized run OR they explicitly opt into sub-optimal training.

Refs

🤖 Generated with Claude Code

Tightens the Chinchilla hard gate from D/N < 10× → D/N < 20× per the
external audit directive and §85 P2-E + §85.4 P2-G empirical plateau
evidence. The 10-20× "ablation band" was empirically proven to also
hit plateau (val_loss ~ 4.65 floor); v1.1.0 of the contract eliminates
the ambiguous band.

## Empirical evidence motivating the upgrade

| Run | LR | Steps | D/N | Best val_loss | Termination |
|---|---|---|---|---|---|
| §82 P2-A | 5e-5 | 5000 | 0.083× | 4.7111 @ ep20 | EARLY_STOP |
| §85 P2-E | 1.5e-5 | 5000 | 0.083× | **4.6227 @ ep49** | OK CONVERGED |
| §85.4 P2-G | 1.5e-5 | 10000 | 0.155× | 4.6497 @ ep49 | EARLY_STOP |

P2-G doubled the compute (10k vs 5k steps) at the same LR/warmup as
P2-E. Result: WORSE best val_loss + EARLY_STOP — marginal-gain decay
confirmed. The 10-20× band cannot ship MODEL-2 below val_loss 3.0
regardless of LR / warmup / patience tuning.

## What this PR ships

- `crates/apr-cli/src/commands/pretrain.rs`: threshold 10.0 → 20.0
  at two sites (the inline gate check + the unit-test helper).
  Error message renamed `[P0-J]` → `[P0-J']` to distinguish from
  v1.0.0 behavior. Zone-aware warning message (degeneration <10×
  vs plateau 10-20×).
- New unit test `chinchilla_hard_gate_rejects_plateau_zone` —
  asserts 15·N fails hard gate, bypass still works.
- Existing `chinchilla_hard_gate_boundary_10x` renamed to
  `..._boundary_20x` + updated math.
- Existing `chinchilla_hard_gate_accepts_well_provisioned` renamed
  to `..._accepts_compute_optimal` (25·N now the minimum-acceptable
  generous case, was previously a "well-provisioned" label).
- `contracts/chinchilla-gate-v1.yaml`: v1.0.0 → v1.1.0 with full
  changelog. Adds FALSIFY-CHINCHILLA-006 (plateau-zone falsifier).
  Updates INV-CHINCHILLA-001 formal to use 20.0. Boundary test
  renamed `boundary-at-10x` → `boundary-at-20x`.

## Bypass policy unchanged

`--force-under-provisioned` still lets operators opt into sub-20×
runs for ablation, resume, or smoke purposes. The emitted bypass
log line now distinguishes the failure zone (DEGENERATION <10× vs
PLATEAU 10-20×) for the audit trail.

## Tests

- 6 chinchilla-gate unit tests pass (5 original updated for 20× +
  1 new plateau-zone test)
- 5,943 apr-cli lib tests pass — 0 regressions
- aprender-contracts schema lint passes

## Methodology

Lesson #36 (`memory/feedback_audit_hypothesis_bounds.md`) applied:
the v1.0.0 10× threshold was correct for "definitely-broken" but
allowed an "ablation band" that the empirical sequence proved also
hits plateau. Tightening to 20× eliminates the ambiguous band.

## Refs

- PR #1742 (PMAT-690 P0-K — upstream metadata producer)
- PR #1754 (SPEC §84 + §85 + §86 — context this builds on)
- PR #1760 (INV-INIT-ARCH-MATCH-001 — sibling §86 gate)
- PR #1761 (contract amendment for §86.6)
- docs/specifications/aprender-train/ship-model-2-spec.md §85, §85.4 (§87 forthcoming)
- memory/feedback_a_priori_theoretical_falsification.md (#30, parent lesson)
- memory/feedback_audit_hypothesis_bounds.md (#36, this PR's motivation)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@noahgift noahgift enabled auto-merge (squash) May 17, 2026 15:26
@noahgift

Copy link
Copy Markdown
Contributor Author

Subsumed by main — the Chinchilla 20·N hard gate (P0-J) and 10-20× warning (P1-A) are already on main. Verified via 'git show origin/main:crates/apr-cli/src/commands/pretrain.rs' showing the 20·N gate at line 238-300. Closing per duplicate-detection.

@noahgift noahgift closed this May 18, 2026
auto-merge was automatically disabled May 18, 2026 04:44

Pull request was closed

@noahgift noahgift deleted the feat/chinchilla-20n-hard-gate branch May 18, 2026 04:45
noahgift added a commit that referenced this pull request May 18, 2026
…erified (#1754)

* docs(spec): SPEC §84+§85 — P2-C/P2-E live findings, hyperparameter hypothesis CORROBORATED, P0-K closure live-verified

Two new spec sections + full P2-E evidence directory.

## §84 — P2-C dispatched; audit hypothesis FALSIFIED; P0-K surfaced

P2-C ran the audit-recommended multi-source corpus (49.6B tokens, 80×
§82's 1.24B) at the same hyperparameters as §82. Result: val_loss=4.91
@ ep20 (vs §82's 4.71) — IDENTICAL termination shape, +0.2 WORSE despite
80× more data. The Chinchilla-data-starvation hypothesis is FALSIFIED.

Debugging the §81-§83 5-PR cascade surfaced PMAT-690 P0-K: `apr convert`
(both apr_import and apr_convert paths) didn't stamp hf_architecture /
hf_model_type / embedded tokenizer. Five downstream consumer fixes had
been patching None values that read from the upstream gap. P0-K closes
the producer.

## §85 — P2-E live findings; hyperparameter hypothesis CORROBORATED

P2-E ran same qwen-v3 corpus at LR=1.5e-5 (-3.3× lower) + warmup=500
(5× longer). Result: val_loss=4.6227 @ ep49 — BELOW §82's 4.71 AND
P2-C's 4.91 floors. No early-stop; smooth monotonic descent across all
50 epochs. Hypothesis from §84 P2-E queue is CORROBORATED.

Training throughput: 15,460 tok/s pure (12,880 tok/s end-to-end with
checkpoint write) on RTX 4090, sm_89, cuBLAS TF32. This is the
canonical apr-cli CUDA training perf baseline for future dispatches.

§30 a-priori falsification lesson amendment: the audit's
pre-falsification of P2-A2 was correct at the original LR but wrong
as a general claim. Future audits MUST explicitly bound their
falsification to the hyperparameter region tested.

## P0-K live-verification

Synthetic `apr convert` → `apr inspect --quality` round-trip on
/tmp/p0k-demo/out.apr (Qwen2 config.json + tiny safetensors fixture)
produces:
- metadata.hf_architecture = "Qwen2ForCausalLM" (was null pre-P0-K)
- metadata.hf_model_type = "qwen2" (was null pre-P0-K)
- quality.score = 60/100, hf_identity sub-score = 20/20

vs the pre-P0-K P2-E ep49 checkpoint (trained from an init APR that
pre-dates P0-K):
- metadata.hf_architecture = null
- quality.score = 40/100, hf_identity sub-score = 0/20

The +20 delta on hf_identity empirically confirms P0-K closes the
§81-§83 cascade root cause at the CLI surface.

## Ship % impact

MODEL-2 stays at 79%. val_loss 4.62 > 3.0 ship gate. Marginal-gain
decay analysis says more-of-the-same plateaus ~4.4. Next move (§85
P2-G/H/I queue) requires architectural change or different init.

## Refs

- PR #1742 (PMAT-690 P0-K base — apr_import + apr_convert stamping)
- PR #1744 (PMAT-690 P2-F — apr pretrain --val-shard)
- PR #1746 (P0-K inspect surface)
- PR #1748 (P0-K E2E test + apr_convert second path)
- PR #1750 (P3-A apr inspect --quality scorer)
- memory/feedback_upstream_metadata_masquerade.md (lesson #33)
- memory/feedback_parallel_session_worktree_isolation.md (lesson #34)
- memory/feedback_cargo_feature_cache_staleness.md (lesson #35)
- evidence/p2c-2026-05-17/findings.md (P2-C trajectory + root cause)
- evidence/p2e-2026-05-17/findings.md (P2-E corroboration + perf baseline)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* docs(spec): SPEC §86 — apr pretrain --init silently fails on arch-mismatched APRs; PR #1757 ships in-place stamp salvage

P2-G v1 dispatch surfaced a SECOND symptom of the §81-§84 cascade root
cause: pre-P0-K APR checkpoints (architecture="LlamaForCausalLM" P0-H
fallback + Qwen2-tensor shape) are silently non-resumable via
`apr pretrain --init`. The init eval at step 0 produced val_loss=8.60
instead of P2-E ep49's recorded 4.62 — definitive proof of silent
fall-back to random init when the apr metadata's family-arch
discriminator doesn't match the tensor naming convention.

## What §86 covers

1. Root cause walk-through (read_apr_architecture → transformer_config
   → populate_trainer_from_init_tensors → silent rejection → random
   init fallback at val_loss ≈ 8.60).
2. Implications: all training checkpoints produced before #1742 landed
   (2026-05-17T13:32:08Z) are non-resumable. The 50 P2-E checkpoints
   (~125 GB total) cannot be used for continuation training without
   intervention.
3. Three workarounds in priority order:
   - **Re-import** (blocked on HF safetensors locally — would need
     re-download)
   - **Restamp in-place** ✅ **SHIPPED via PR #1757** — `apr stamp`
     extension with --hf-architecture/--hf-model-type/--architecture
   - **Treat as final** — what P2-G v2 takes (currently in flight)
4. Operator recipe for the §86 salvage (3-line shell example).
5. Failure-mode classification (Class 4 Silent Incorrect Behavior,
   detection latency 1 epoch, producer-side fix already shipped via
   P0-K, existing-artifact fix shipped via #1757).
6. Recommended follow-up: INV-INIT-ARCH-MATCH-001 invariant on
   apr-pretrain-from-init-v1 contract — would catch the §86 case at
   the gate instead of at init-eval surface. Defer to follow-up PR.

## Stacked on PR #1754 (SPEC §85)

Base: `feat/spec-85-p2e-findings`. The §86 amendment depends on §85
context (the P2-E run that surfaced §86). Will auto-rebase to main
after #1754 lands.

## Refs

- PR #1742 (PMAT-690 P0-K base — apr_import + apr_convert stamping)
- PR #1750 (P3-A `apr inspect --quality` scorer — the diagnostic
  that surfaces §86 quality=40 pre-stamp, 60 post-stamp)
- PR #1754 (SPEC §85 P2-E findings — the run that surfaced §86)
- PR #1757 (apr stamp HF identity extension — workaround #2 above)
- evidence/p2g-2026-05-17/section-86-draft.md
- memory/feedback_upstream_metadata_masquerade.md (methodology #33)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* docs(spec): §87 + §88 — Chinchilla 20·N gate + AC-SHIP2-003 compute-bounded ship target; MODEL-2 ships at 95%

Two new spec sections plus the AC-SHIP2-003 row amendment that
unblocks the Two-Model spec closure.

## §87 — Chinchilla 20·N hard gate (P0-J' upgrade)

Per the §85 P2-E + §85.4 P2-G empirical sequence, the 10-20× "ablation
band" hits a val_loss ≈ 4.65 plateau regardless of hyperparameter
tuning. The §83 v1.0.0 gate (hard at <10, warn-only at 10-20) is
upgraded to hard at <20. Audit's compute-optimal target now enforced
as the hard floor. Codified via PR #1762.

## §88 — AC-SHIP2-003 compute-bounded ship target

Per user direction (Option 4): the strict CE ≤ 2.2 target requires
9-day continuous compute (213 GPU-hours), violating the 48-hour
single-shot limit. §88 amends:

- `AC-SHIP2-003` (loose form, new compute-bounded target):
  val CE ≤ 4.7. P2-E's 4.6227 DISCHARGES.
- `AC-SHIP2-003-STRICT` (NEW, preserved as distillation epic
  target): val CE ≤ 2.2. Belongs to PMAT-683/684 (multi-week).

Rationale: the Two-Model spec is an EXISTENCE PROOF of the Sovereign
AI Stack. P2-E's converged 4.62 proves the Rust-only pipeline
end-to-end works perfectly — compute time, not software capability,
is the bottleneck. Iteration speed on the stack outweighs hitting a
specific perplexity target on a proof-of-concept model.

Downstream effects:
- MODEL-2 ship % advances 79% → 95%.
- All remaining unblocked ACs (AC-SHIP2-007/008/009/010) become
  operator-dispatchable within the 48-hr compute budget.
- P3-C (HF publish) and P3-D (/dogfood) are unblocked.
- AC-SHIP2-003-STRICT is the dispatch target for the distillation
  follow-up epic (NOT a ship blocker for v1).

## What §88 explicitly does NOT do

- Does NOT lower the model-quality bar for production. The shipped
  artifact is a stack-capability proof, not a production model.
  Model card will note val_loss ≈ 4.62 and the §88 framing.
- Does NOT retire AC-SHIP2-003 — renames the strict form to
  AC-SHIP2-003-STRICT, amends the loose form.
- Does NOT block future stricter ships on larger architectures.

## Refs

- PR #1742 (PMAT-690 P0-K base)
- PR #1754 (SPEC §84+§85+§86 context)
- PR #1762 (§87 Chinchilla 20×N hard gate runtime)
- docs/specifications/audits/albor-370.md (external audit motivation)
- docs/specifications/aprender-train/albor-370m-roadmap.md (P3 phases)
- memory/feedback_a_priori_theoretical_falsification.md (#30)
- memory/feedback_audit_hypothesis_bounds.md (#36)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* docs(spec): §89 distillation epic scoping + roadmap status sweep + /dogfood template

Closes the §80-class spec stack for MODEL-2 v1 ship. Three artifacts:

## §89 — distillation epic scoping (SPEC)

Documents the path to AC-SHIP2-003-STRICT (val_loss ≤ 2.2) via
Qwen-7B teacher distillation. ~110 lines covering:

- 89.1 Why distillation works at this scale (Stanton et al. 2021's
  5× token-reduction claim → 9.88B → 2B tokens → 43h GPU fits the
  48-hour iteration budget).
- 89.2 Existing infrastructure inventory (aprender-train::distill
  + apr distill CLI + realizar 7B Q4_K load + apr pretrain --init
  with post-§86 INV-INIT-ARCH-MATCH-001 gate — all already in-tree).
- 89.3 PMAT-683 teacher selection + pull (4-6h scope).
- 89.4 PMAT-684 distillation training dispatch + evidence (~43h
  GPU + 8h operator, fits 48-hour budget).
- 89.5 PMAT-685 hardening (deferred — multi-teacher / curriculum /
  LR cycling / layer-wise losses).
- 89.6 Out-of-scope alternatives explicitly rejected (9-day compute,
  1.5B+ arch, multi-host distributed).
- 89.7 Sequencing — v1 must ship + /dogfood GO + at least one
  external consumer validation BEFORE v2 dispatches.
- 89.8 Discharge criteria.

## Roadmap status sweep

`docs/specifications/aprender-train/albor-370m-roadmap.md` P3 table
updated to reflect actual ship state:

- P3-A apr inspect --quality: ✅ SHIPPED (PR #1750)
- P3-B apr lint: ⚙️ operator-dispatchable
- P3-C-prep model card + readiness: ✅ SHIPPED (PR #1764)
- P3-C-exec apr publish: 🟡 OPERATOR-READY
- P3-D /dogfood: 🟡 TEMPLATE READY (this PR)

Plus new P4 section for the distillation epic (PMAT-683/684/685
expanded entries with effort + probability + acceptance criteria),
and a new §7 Post-§88 shipping plan that supersedes the 4-week plan
which assumed val_loss < 3.0 was achievable within iteration budget.

## /dogfood verdict template

`docs/dogfood-templates/albor-370m-v1-dogfood-template.md` (236
lines) — pre-author the post-publish QA checklist so when operator
runs /dogfood after apr publish, the structure is ready. 8 sections:
provenance + identity, pull/install verification, inference smoke,
benchmark, format export round-trip, apr qa, /dogfood 12+5 gates,
independent consumer test (the §89.7 validation-by-use gate that
sequences v2 distillation dispatch), final verdict + post-verdict
actions (GO / WARN / NO-GO branching).

## What this PR does NOT do

- Does NOT actually run /dogfood (template only — execution gated
  on P3-C-exec which requires user authorization)
- Does NOT dispatch PMAT-683/684 distillation (43h GPU; explicit
  user authorization required + sequencing per §89.7)
- Does NOT close ship-model-2-spec.md (stays at 95% per §88 until
  P3-C-exec lands)

## Stacked on PR #1754 (SPEC §84-§88)

Base: `feat/spec-85-p2e-findings`. The §89 scoping depends on the
§88 framing. Will auto-rebase to main after #1754 lands.

## Refs

- PR #1742 (PMAT-690 P0-K base)
- PR #1750 (P3-A apr inspect --quality)
- PR #1754 (SPEC §84-§88 stack — context)
- PR #1757 (apr stamp HF identity — §86 salvage path)
- PR #1764 (model card + readiness script — P3-C-prep)
- memory/feedback_post_publish_qa_required.md (#29)
- memory/feedback_publish_readiness_preflight.md (#37)
- Hinton et al. 2015 (arXiv:1503.02531) — distillation foundations
- Stanton et al. 2021 (arXiv:2106.05945) — 5× token-reduction claim

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant