feat(pretrain): task #124 — regime-aware INV-TRAIN-005 epoch-zero cap#936
Merged
Merged
Conversation
The hardcoded val_loss[0] ≤ 10.0 cap was calibrated for MODEL-1 finetuning,
but any from-scratch pretraining starts at ln(vocab_size) ≈ 10.82 for
vocab=50257 — every legitimate cold-start would trip the divergence abort.
Contract training-loop-pretrain-v1 v1.1.0 → v1.2.0:
- INV-TRAIN-005 / GATE-TRAIN-005 cap becomes regime-dependent:
* finetune: literal 10.0 (MODEL-1 lesson preserved)
* from_scratch: 2.0 × ln(vocab_size) (permits near-random baseline)
- GATE-TRAIN-005.evidence_discharged_by lists 4 Rust tests
- `pv validate` green (0 errors)
Rust:
- New TrainingRegime enum { Finetune, FromScratch { vocab_size } }
serde-tagged "kind" for YAML/JSON interop; default Finetune.
- epoch_zero_val_loss_limit() returns regime-dependent cap.
- check_non_divergence(epoch, history, ®ime) threaded through
PretrainConfig, run_epoch, test_config, 4 existing call sites.
- 3 new falsifier tests:
* from_scratch permits near-random baseline (val_loss[0]=18, vocab=50257)
* from_scratch aborts above 2·ln(vocab) (val_loss[0]=25)
* regime cap formula lock (2.0 × ln(vocab_size))
CLI:
- apr pretrain PretrainConfig literal defaults to Finetune;
task #125 will wire --mode {finetune|from-scratch} to flip regime
alongside LR / warmup / target_val_loss.
Gates: 22/22 pretrain tests PASS, 3/3 CLI tests PASS, 1371 contract
tests PASS, clippy+fmt clean.
Closes: #124
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
training-loop-pretrain-v1v1.1.0 → v1.2.0:INV-TRAIN-005/GATE-TRAIN-005epoch-zero cap is now regime-dependent (finetunekeeps the MODEL-1 literal 10.0;from_scratchuses2.0 × ln(vocab_size)≈ 21.64 for vocab=50257).pv validategreen.TrainingRegimeenum threaded throughcheck_non_divergence,PretrainConfig,run_epoch,test_config, and all 4 existing call sites. DefaultFinetune.2.0 × ln(vocab_size).apr pretraindefaultsregime: Finetune; task Integrate trueno-rag for enhanced text/document ML #125 will wire--mode {finetune|from-scratch}to flip regime alongside LR / warmup / target_val_loss.Why
The hardcoded
val_loss[0] ≤ 10.0cap was calibrated for MODEL-1 finetuning. Any from-scratch pretrain starts at uniform-random cross-entropy =ln(vocab_size)≈ 10.82 for vocab=50257, so every legitimate cold start trips the divergence abort. This is a blocker for task #119-onwards real-compute pretrain dispatches.Test plan
pv validate contracts/training-loop-pretrain-v1.yaml— 0 errors, 0 warningscargo test -p aprender-contracts --lib— 1371 passcargo test -p aprender-train --lib pretrain::— 22 pass (incl. 3 new)cargo test -p apr-cli --lib pretrain— 3 passcargo clippy -p aprender-train --lib --no-deps -- -D warnings— cleancargo fmt --all -- --check— cleanci / gate+workspace-testgreenCloses #124
🤖 Generated with Claude Code