Seed 1337: 81.86ms, 1.1241 bpb, 15.83MB
Seed 42: 81.88ms, 1.1253 bpb, 15.82MB
Seed 2025: 81.86ms, 1.1247 bpb, 15.80MB
Mean: 81.87ms, 1.1247 bpb

Also adds GPTQ-lite (PR openai#414's per-row optimal clip percentile search)
for improved int6 quantization quality.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Three stacked innovations:
1. QAT/GPTQ mismatch fix: QAT STE uses percentile clip (0.9995)
   matching GPTQ-lite export, instead of row_max. Model trains
   against the actual quantization it'll face.
2. TrigramHash(2048, dim=48): 3-token n-gram patterns, additive
   to BigramHash. Inspired by PR #440. ~100KB extra params.
3. EMA-SWA blend (80/20): combine exponential and uniform averaging.
   SWA data already collected but unused in PR #414.

Plus proven TTT burst from v1.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Legal score-first TTT (PR openai#461 recipe) on openai#414 stack + Parameter Banking + Parallel Muon.
Pre-TTT: 1.1234, post-TTT: 1.1213 (-0.0021). TTT eval: 400s. Artifact: 15.84 MB.

Every token scored BEFORE model adapts (inference_mode enforced).
SGD+momentum, 3 epochs/32K chunk, freeze first 2 blocks, cosine LR decay.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Legal score-first TTT (PR openai#461 recipe) applied to openai#414 stack with
Parameter Banking + Parallel Muon (first introduced in PR openai#399).

Pre-TTT: 1.1234, post-TTT: 1.1213 (-0.0021). TTT eval: 400s.
Artifact: 15.84 MB. Seed 1337, 8×H100 SXM, PyTorch 2.9.1+cu128.

Every token scored BEFORE model adapts (inference_mode enforced).
SGD+momentum(0.9), 3 epochs/32K chunk, freeze first 2 blocks.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Legal score-first TTT (PR openai#461 recipe) applied to openai#414 stack with
Parameter Banking + Parallel Muon (first introduced in PR openai#399).

Pre-TTT: 1.1234, post-TTT: 1.1213 (-0.0021). TTT eval: 400s.
Artifact: 15.84 MB. Seed 1337, 8×H100 SXM, PyTorch 2.9.1+cu128.

Every token scored BEFORE model adapts (inference_mode enforced).
SGD+momentum(0.9), 3 epochs/32K chunk, freeze first 2 blocks.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Updated CLAUDE.md and idea bank with:
- Current valid leaderboard (PR openai#414 at 1.1233 is the real leader)
- TTT legality analysis (full-val TTT ruled invalid, score-first legal)
- New techniques to adopt: GPTQ-lite, backout, U-Net skips, value residual,
  catalytic residuals, gated attention
- Phased experiment roadmap: parity -> zero-cost arch -> novel quant -> training
- Dead ends confirmed since openai#332: PPM-C, SwiGLU, depth recurrence

Legal score-first TTT (PR openai#461 recipe) applied to openai#414 stack with
Parameter Banking + Parallel Muon (first introduced in PR openai#399).

Pre-TTT: 1.1234, post-TTT: 1.1213 (-0.0021). TTT eval: 400s.
Artifact: 15.84 MB. Seed 1337, 8×H100 SXM, PyTorch 2.9.1+cu128.

Every token scored BEFORE model adapts (inference_mode enforced).
SGD+momentum(0.9), 3 epochs/32K chunk, freeze first 2 blocks.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Based on PR openai#414's exact train_gpt.py (11L, EMA, XSA, PartialRoPE,
LNScale, VE128, GPTQ-lite, QAT@0.15, warmdown=3500, int6+zstd-22).

Added legal score-first TTT from PR openai#461/openai#473 protocol:
- SGD + momentum 0.9, lr=0.002 with cosine decay
- 3 epochs per 32K token chunk
- Freeze blocks 0-1
- Score each chunk BEFORE training on it (inference_mode)
- Expected ~0.002 bpb improvement over base

Strategy shift: reproduce proven frontier instead of iterating on
our custom stack. PR openai#414 achieves 1.1233 on 3 seeds; adding
legal TTT should push to ~1.121.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…icating PR openai#414)

Legal score-first TTT (PR openai#461 recipe) applied to openai#414 stack with
Parameter Banking + Parallel Muon (first introduced in PR openai#399).

Pre-TTT: 1.1234, post-TTT: 1.1213 (-0.0021). TTT eval: 400s.
Artifact: 15.84 MB. Seed 1337, 8×H100 SXM, PyTorch 2.9.1+cu128.

Every token scored BEFORE model adapts (inference_mode enforced).
SGD+momentum(0.9), 3 epochs/32K chunk, freeze first 2 blocks.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…T stack)

- PR openai#414 repro: 1.1286 BPB warm cache (5903 steps at 98ms)
- stride=32 vs stride=64: negligible (0.0001 BPP)
- VRL net negative on openai#414 stack (+0.0012 BPP)
- catalytic residuals experiment in progress

…time

- XSA on all 11 layers: 1.1268 vs baseline 1.1286 (-0.0018)
- catalytic residuals: neutral on openai#414 stack
- backout connection: slightly negative
- stride=32: negligible gain over stride=64

All arch tweaks tested on openai#414 stack:
- XSA-all: -0.0007 BPP (keeping)
- Gated attn, catalytic, VRL, backout: all neutral/negative
- Pod throughput (98-104ms vs 82ms) is the dominant bottleneck
- Moving to quantization improvements

Carefully implemented based on signalrush SOTA (#414, 1.1233 BPB):
- 11L: num_layers = 11 (up from 9)
- XSA4: post-attention geometric subtraction on last 4 layers (0 new params)
- EMA: exponential moving average decay=0.997, updated every step from step 0
- Late QAT: quantization-aware training enabled when LR < 15% threshold
- GPTQ-lite: per-row quantization with 99.99% clip percentile

EMA state initialized before training loop, updated after each optimizer step,
applied to weights at end of training. Late QAT uses class-level flag with inline STE.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

… local test script

- 11 layers, XSA on last 4, int6 quantization + zstd-22
- EMA(0.997), GPTQ-lite, Tight SWA, Late QAT@0.15
- Partial RoPE 16/64, LN Scale 1/sqrt(layer+1)
- SmearGate + BigramHash(2048,128), VE128 on layers 9,10
- Muon WD=0.04, momentum=0.99, matrix_lr=0.025
- SDPA fallback (no FA3), batch 786K, seq 2048
- add zstandard to Modal image

…#414)

…ression moonshots falsified

Benchmarked FA2 vs cuDNN SDPA on H100: FA2 is 0.73ms per call.
11 layers x 2 = 16ms of 98ms step. FA3 would save ~8ms max.
Gap to PR openai#414 (82ms) is likely hardware variation.
Added pod setup script. Updated experiment priorities.

- replace relu(x)^2 with leaky_relu(x, 0.5)^2
- PR openai#493 reaches 1.1309 with partial stack using this activation
- untried on full openai#414 stack — could give -0.002 to -0.005 BPB
- zero param cost, zero speed overhead

…d mean)

Legal score-first TTT (PR openai#461 recipe) + BigramHash(3072) + freeze=0
on openai#414 stack with Parameter Banking + Parallel Muon (PR openai#399).

3-seed results (BIGRAM=3072, 3ep, freeze=0, SGD+mom=0.9):
  Seed 1337: 1.1204 bpb, 413s TTT, 15.98 MB
  Seed 42:   1.1216 bpb, 406s TTT, 15.99 MB
  Seed 2025: 1.1221 bpb, 405s TTT, 15.99 MB
  Mean:      1.1214 (std 0.0009)

All artifacts under 16MB. All eval times under 600s.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

- Based on PR openai#414 SOTA (1.1228 BPB)
- LeakyReLU(0.5)^2 activation (proven frontier technique)
- XSA on all 11 layers (PR openai#609 approach)
- Full Hessian-aware GPTQ quantization with column reordering
- Binary-search selective pruning to fit under 16MB
- Auto-detects GPU: uses FlashAttention 3 + torch.compile on H100,
  falls back to PyTorch SDP + no compile on T4/P100/L4
- 1500 lines exactly

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

- technique-card-head: align-items center, flex-wrap for badge overflow
- All technique card sub-elements: explicit text-align center
- combo-chip: centered text on all child elements
- Top 3 neural leaderboard: corrected types from PR titles
  - openai#549: Neural + TTT (LeakyReLU² + Legal Score-First TTT)
  - openai#414: Neural (EMA + GPTQ-lite, no TTT)
  - openai#641: Neural, Non-Record (Binary U-Net)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…ai#414 tuning)