Skip to content

Non-record: 11L EMA + TTT(20ep,freeze=0) + 15-run ablation study — val_bpb=1.1213 (3-seed)#398

Open
felipe-parodi wants to merge 1 commit intoopenai:mainfrom
felipe-parodi:submission/11L-EMA-TTT20ep-1.1213
Open

Non-record: 11L EMA + TTT(20ep,freeze=0) + 15-run ablation study — val_bpb=1.1213 (3-seed)#398
felipe-parodi wants to merge 1 commit intoopenai:mainfrom
felipe-parodi:submission/11L-EMA-TTT20ep-1.1213

Conversation

@felipe-parodi
Copy link
Copy Markdown

@felipe-parodi felipe-parodi commented Mar 22, 2026

Record: 11L EMA + TTT(20ep) — val_bpb: 1.1213

val_bpb = 1.1213 (sliding window stride=64, best seed 1337) | 15.53 MB artifact | 8xH100 SXM, 600s

Key Finding: EMA + Aggressive TTT with All Blocks Unfrozen

EMA(0.997) weight averaging combined with aggressive test-time training (20 epochs SGD, lr=0.008, all blocks unfrozen) outperforms Tight SWA + VE128 approaches (PR #388, 1.1231).

Results (3-seed, 8xH100 SXM)

Seed Steps Sliding BPB (s64) Artifact
1337 7386 1.1213 15.53 MB
42 7411 1.1221 15.51 MB
2025 7386 1.1228 15.53 MB

Mean: 1.1221 | Std: 0.0008

Critical Discoveries (15-run ablation)

  1. TTT_FREEZE_BLOCKS=0 is essential — freezing early blocks during aggressive TTT creates internal inconsistency (quant gap 5x worse)
  2. Late QAT is counterproductive with aggressive TTT
  3. XSA removed — saves ~1.4ms/step, yielding ~130 more training steps
  4. PPM-C eval blending hurts strong models — classical compression adds noise when neural model is already strong
  5. Memory tokens, aggressive warmdown, gradient-guided quant — all documented negative results

Run Command

pip install zstandard flash_attn_3 --find-links https://windreamer.github.io/flash-attention3-wheels/cu128_torch291

SEED=1337 NUM_LAYERS=11 BIGRAM_VOCAB_SIZE=2048 XSA_LAST_N=0 \
EMA_ENABLED=1 EMA_DECAY=0.997 SWA_ENABLED=0 \
ROPE_DIMS=16 LN_SCALE=1 LATE_QAT=0 \
TTT_ENABLED=1 TTT_LR=0.008 TTT_EPOCHS=20 TTT_MOMENTUM=0.9 TTT_FREEZE_BLOCKS=0 \
MUON_WD=0.04 ADAM_WD=0.04 \
MATRIX_LR=0.025 SCALAR_LR=0.025 TIED_EMBED_LR=0.035 \
MUON_MOMENTUM=0.99 MUON_MOMENTUM_WARMUP_START=0.92 \
MUON_MOMENTUM_WARMUP_STEPS=1500 WARMDOWN_ITERS=3000 \
ITERATIONS=9000 MAX_WALLCLOCK_SECONDS=600 EVAL_STRIDE=64 \
torchrun --standalone --nproc_per_node=8 train_gpt.py

See the full README in the submission folder for detailed architecture, training config, TTT analysis, and the complete 15-run ablation table.

@notapplica notapplica mentioned this pull request Mar 22, 2026
Open
@leloykun leloykun mentioned this pull request Mar 22, 2026
Open
samuelczhao added a commit to samuelczhao/parameter-golf that referenced this pull request Mar 22, 2026
@samuelczhao
Aggressive TTT defaults (20 epochs, all blocks, lr=0.008) per PR open…
1985065 
…ai#398 findings; disable XSA for throughput
mrdavtan added a commit to mrdavtan/parameter-golf that referenced this pull request Mar 22, 2026
@mrdavtan
Add aggressive TTT script (matches PR openai#398 strategy)
6d047ea
mrdavtan added a commit to mrdavtan/parameter-golf that referenced this pull request Mar 22, 2026
@mrdavtan
Cap TTT at 7 epochs (our TTT is single-GPU, 73s/epoch vs openai#398's…
08f3dcb 
… DDP 14.8s/epoch)
robinojw pushed a commit to robinojw/parameter-golf that referenced this pull request Mar 22, 2026
@robinoutside
Add PR openai#398 train_gpt.py (1.1213 BPB) as reference and V5 scaffold
4403708 
PR openai#398: 11L EMA + TTT(20ep, freeze=0), no XSA, no Late QAT
- Best seed 1.1213 BPB, 3-seed mean 1.1221
- 7386 steps at ~81ms/step
- Has: FA3, NTK RoPE, MTP, TTT, (B,S,H,D) layout
- Missing: memory tokens, magnitude pruning, late-K passthrough
robinojw pushed a commit to robinojw/parameter-golf that referenced this pull request Mar 22, 2026
@robinoutside
V5: add Cautious Muon, magnitude pruning, allow_in_graph to PR openai…
3b45b93 
…#398 base

Built on PR openai#398 (1.1213 BPB). Three targeted improvements:

1. Cautious Muon: mask Muon updates that disagree with gradient
   direction (~1.47x convergence speedup, 2 lines, zero risk)

2. Magnitude pruning (5% default): zero smallest weights before
   quantization, improves zstd compression ratio by 5-15%

3. allow_in_graph + cache_size_limit=32: safer torch.compile
   with FA3 custom ops and 11-block guard specialization

Respects PR openai#398 negative results: no memory tokens, no Late QAT.
@abaybektursun abaybektursun mentioned this pull request Mar 22, 2026
Open
robinojw pushed a commit to robinojw/parameter-golf that referenced this pull request Mar 22, 2026
@robinoutside
V6: minimal delta from PR openai#398 -- Cautious Muon + GPTQ-lite + m…
f95541f 
…agnitude pruning
robinojw pushed a commit to robinojw/parameter-golf that referenced this pull request Mar 22, 2026
@robinoutside
V6: add EMA + TTT + sliding window eval + PR openai#398 defaults
53c1113
sjp611 added a commit to sjp611/parameter-golf that referenced this pull request Mar 22, 2026
@sjp611
Record: 11L EMA + AdamW TTT 10ep (mean val_bpb=1.1027)
aef01ee 
Replace SGD with AdamW for test-time training. 3-line diff from PR openai#398.
Mean val_bpb 1.1027 (3-seed), best 1.0992. Beats prior SOTA 1.1213 by -0.019.
@sjp611 sjp611 mentioned this pull request Mar 22, 2026
Closed
ThomAub pushed a commit to ThomAub/parameter-golf that referenced this pull request Mar 22, 2026
@claude
Add critical TTT data leakage caveat and correct causal TTT protocol
4cbf170 
Many TTT submissions (openai#136, openai#152, openai#254, openai#264, openai#338, openai#398, openai#417, openai#421, openai#442)
flagged as potentially invalid for adapting on eval tokens BEFORE scoring them.
Added correct score-then-adapt protocol with implementation guide.

https://claude.ai/code/session_01M5XTtyz2Zdq5BDeh9qNn9y
JoeProAI pushed a commit to JoeProAI/parameter-golf that referenced this pull request Mar 22, 2026
@Snowplan
Record: SwiGLU + XSA4 + U-Net + AdamW TTT (3-seed mean val_bpb=1.0672)
8bd231a 
Architecture discovered via GEPA (Gemini-driven evolutionary search).
SwiGLU FFN, Star-ReLU, U-Net skip gates, BigramHash 8192, XSA4.
AdamW TTT (lr=0.0005, 10ep) from @sjp611 (openai#442).
EMA, RoPE, LN Scale, QAT from @felipe-parodi (openai#398) and @fbedev (openai#410).

3-seed results: 1.06733 / 1.06833 / 1.06580
Mean: 1.06715, Std: 0.00104

Built by @joepro with AI agents via OpenClaw.
Compute provided by Modal.
@JoeProAI JoeProAI mentioned this pull request Mar 22, 2026
Closed
JoeProAI added a commit to JoeProAI/parameter-golf that referenced this pull request Mar 22, 2026
@JoeProAI
Record: SwiGLU + XSA4 + U-Net + AdamW TTT (3-seed mean val_bpb=1.0672)
d1e5693 
Architecture discovered via GEPA (Gemini-driven evolutionary search).
SwiGLU FFN, Star-ReLU, U-Net skip gates, BigramHash 8192, XSA4.
AdamW TTT (lr=0.0005, 10ep) from @sjp611 (openai#442).
EMA, RoPE, LN Scale, QAT from @felipe-parodi (openai#398) and @fbedev (openai#410).

3-seed results: 1.06733 / 1.06833 / 1.06580
Mean: 1.06715, Std: 0.00104

Built by @joepro with AI agents via OpenClaw.
Compute provided by Modal.
mrdavtan added a commit to mrdavtan/parameter-golf that referenced this pull request Mar 22, 2026
@mrdavtan
Fix DDP TTT: contiguous slicing + ReduceOp.AVG (matches openai#398 pa…
d8f6c27 
…ttern)

Root cause: per-sequence indexing from permuted indices was ~100x slower
than contiguous val_tokens slicing. Each GPU now takes a contiguous
shard and iterates sequentially, matching openai#398's working implementation.
mohosy pushed a commit to mohosy/parameter-golf that referenced this pull request Mar 23, 2026
@claude
Update to frontier stack: SwiGLU + XSA4 + EMA + U-Net + AdamW TTT
af190d5 
Major rewrite based on latest meta (PRs openai#398, openai#442, openai#462):
- SwiGLU FFN with Star-ReLU (hidden=1792)
- U-Net skip connections with learned gating
- EMA (decay=0.9985) replacing SWA
- AdamW TTT (legal score-first protocol)
- Partial RoPE (16 dims)
- LN Scale (1/sqrt(layer_idx+1))
- BigramHash(8192) + SmearGate
- GPTQ-lite quantization
- DDP compile fix for multi-GPU

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@mohosy mohosy mentioned this pull request Mar 23, 2026
Open
mrdavtan added a commit to mrdavtan/parameter-golf that referenced this pull request Mar 23, 2026
@mrdavtan
Record: Cosine TTT scheduling with per-layer lr (mean val_bpb=1.0970,…
c8e034d 
… 3 seeds)

AdamW TTT with cosine lr decay over 30 epochs and per-layer lr groups
(3x for MLP output projections, 0.5x for input projections). 34 TTT
configurations tested. FINDINGS.md documents 31 experiments including
negative results on codebook quantization, symmetry-transport, layer
dropping, focal loss, and KL divergence TTT.

Builds on PRs openai#162, openai#180, openai#77, openai#398, openai#442, openai#417, openai#315.
@mrdavtan mrdavtan mentioned this pull request Mar 23, 2026
Closed
@felipe-parodi felipe-parodi changed the title Record: 11L EMA + TTT(20ep,freeze=0) — val_bpb=1.1213 (3-seed mean 1.1221) Non-record: 11L EMA + TTT(20ep,freeze=0) + 15-run ablation study — val_bpb=1.1213 (3-seed) Mar 23, 2026
@felipe-parodi
Copy link
Copy Markdown
Author

felipe-parodi commented Mar 23, 2026 •
edited
Loading

Converting to non-record per TTT ruling (Issue #402). The README documents a 15-run ablation (memory tokens, causal TTT, PPM-C blending, grad-guided quant, aggressive warmdown — all negative results at the frontier) and the freeze_blocks=0 finding for aggressive TTT. Working on a non-TTT submission.

@ndokutovich ndokutovich mentioned this pull request Mar 23, 2026
Closed
@amaljithkuttamath amaljithkuttamath mentioned this pull request Mar 23, 2026
Draft
@abaybektursun abaybektursun mentioned this pull request Mar 23, 2026
Closed
@andrewbaggio1 andrewbaggio1 mentioned this pull request Mar 23, 2026
Closed
4 tasks
lukacf added a commit to lukacf/parameter-golf-submission that referenced this pull request Mar 23, 2026
@lukacf @claude
Record: 0.978 BPB — Goldfish ML Autonomous Research (100ep Cosine TTT)
aec8d7f 
3-seed mean: 0.9789 BPB (sliding window stride=64)
Best seed: 0.9779 (seed 7)
Std: 0.0015

Key innovation: Autonomous ML research methodology.
AI coding agent discovered cosine LR scaling for TTT in a single
2-hour session — 7 experiments from hypothesis to record.

Technical: CosineAnnealingLR over 100 TTT epochs (3-line change).
Architecture: PR openai#398/openai#442 base (11L, int6+zstd, 15.51MB).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
lukacf added a commit to lukacf/parameter-golf-submission that referenced this pull request Mar 23, 2026
@lukacf
Record: 0.978 BPP — Goldfish ML Autonomous Research (100ep Cosine TTT)
5c2901a 
3-seed mean: 0.9789 BPB (sliding window stride=64)
Best seed: 0.9779 (seed 7)
Std: 0.0015

Key innovation: Autonomous ML research methodology.
AI coding agent discovered cosine LR scaling for TTT in a single
2-hour session — 7 experiments from hypothesis to record.

Technical: CosineAnnealingLR over 100 TTT epochs (3-line change).
Architecture: PR openai#398/openai#442 base (11L, int6+zstd, 15.51MB).
@lukacf lukacf mentioned this pull request Mar 23, 2026
Closed
@NotADevIAmaMeatPopsicle NotADevIAmaMeatPopsicle mentioned this pull request Mar 23, 2026
Closed
This was referenced Mar 25, 2026
Closed
Closed
Closed
Closed
haikosys pushed a commit to haikosys/parameter-golf that referenced this pull request Mar 30, 2026
@claude
MEGA TABLE + 6 new PR codebases (openai#414 openai#417 openai#442 ope…
37e8e37 
…nai#400 openai#369 openai#398)

KEY DISCOVERY: PR#414 stacks EMA + Tight SWA together (-0.0006 BPB free)
GPTQ should be per-ROW not per-matrix (-0.0006 BPB)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@felipe-parodi