Skip to content

NTK Eval + Overtone Init (val_bpb=1.2160)#59

Closed
notapplica wants to merge 1 commit intoopenai:mainfrom
notapplica:submission/ntk-eval-overtone-init
Closed

NTK Eval + Overtone Init (val_bpb=1.2160)#59
notapplica wants to merge 1 commit intoopenai:mainfrom
notapplica:submission/ntk-eval-overtone-init

Conversation

@notapplica
Copy link
Copy Markdown
Contributor

@notapplica notapplica commented Mar 19, 2026

Summary

  • Overtone embedding init: SVD-based spectral shaping of tied embeddings to power-law decay
  • Phase-transition residual mixing: sigmoid-scheduled resid_mix init across layers
  • NTK-aware RoPE scaling: train@1024, eval@2048 with dynamic base frequency adjustment
  • AdamW weight decay 0.01, warmdown 2500 steps, tied embedding LR 0.10

Multi-seed results (3 seeds, p = 0.0012)

Seed Steps ms/step Pre-quant BPB Post-quant BPB val_loss (nats) Artifact
1337 13024 46.07 1.2210 1.2166 2.0542 15.78MB
42 13239 45.32 1.2211 1.2163 2.0537 15.79MB
7 13230 45.35 1.2188 1.2152 2.0519 15.80MB
Mean 1.2203 1.2160 2.0533
  • Improvement over baseline: 0.0194 nats (3.9x the 0.005 threshold)
  • t-statistic: 20.67, df=2, p = 0.0012
  • All 3 runs individually beat SOTA by >0.005 nats
  • All 3 artifacts under 16MB

Note on hardware

Runs were done on Modal 8xH100 SXM (NVLink). Python 3.12, PyTorch <3.

Test plan

  • Post-quant val_bpb verified via int8+zlib roundtrip
  • Artifact under 16MB (all 3 seeds)
  • Runs within 600s wallclock
  • train_gpt.py compiles and runs from records folder
  • Multiple seed runs: p = 0.0012 (3 seeds, t=20.67)

@pbanuru @claude
Add NTK Eval + Overtone Init submission (1.2160 BPB)
382fadf 
Train@1024 with overtone embedding init and phase-transition residual
mixing, eval@2048 with NTK-aware dynamic RoPE scaling. Mean val_bpb
1.2160 across 3 seeds (p=0.0012 for 0.0194-nat improvement over baseline).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@notapplica notapplica closed this Mar 19, 2026
lolrazh added a commit to lolrazh/parameter-golf that referenced this pull request Mar 20, 2026
@lolrazh @claude
Log experiment openai#59: TTT on Thunder H100 PCIe
a36c8ec 
PCIe ~50% slower than SXM (258 vs 386 steps). TTT didn't improve
BPB on undertrained model. SWA with 5 snapshots worked correctly.
Need Flash Attention + more steps before TTT becomes useful.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
lolrazh added a commit to lolrazh/parameter-golf that referenced this pull request Mar 20, 2026
@lolrazh @claude
Log experiments openai#59-60: Thunder H100 PCIe runs
a0b7e8a 
openai#59: 5-min + TTT, 258 steps, TTT didn't improve undertrained model
openai#60: 10-min no TTT, 515 steps, best prequant 1.4038, sliding eval incomplete

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
lolrazh added a commit to lolrazh/parameter-golf that referenced this pull request Mar 20, 2026
@lolrazh @claude
Enhance session logging with detailed experiment results and updated …
05c7d28 
…run queue

- Documented outcomes of experiments openai#59-60, including performance metrics and observations.
- Updated the run queue to reflect current competition state and configurations for A100 production runs.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@notapplica @pbanuru