- replace relu(x)^2 with leaky_relu(x, 0.5)^2
- PR openai#493 reaches 1.1309 with partial stack using this activation
- untried on full openai#414 stack — could give -0.002 to -0.005 BPB
- zero param cost, zero speed overhead

Key changes from studying PR openai#505 (1.1181) and openai#486 (1.0887):
- train_batch_tokens: 524K → 786K (all top entries use this)
- bigram_hash_buckets: 4096 → 8192 (PR openai#505 uses 8192, openai#493 uses 10240)
- grad_clip_norm: 0.3 → 0.0 (PR openai#505 disables clipping)
- Star-ReLU and TrigramHash enabled in all run scripts

…stopher-Lee-McClendon

…bpb 1.1178

3-seed mean: 1.1178 BPB (std 0.0005), ~15.75 MB artifact, 8×H100 SXM.

Novel contribution: Late Soft-Round QAT — replaces STE identity surrogate
with sigmoid soft-round in the backward pass during the final 2% of training,
giving bin-aware gradients that settle weights onto int6 grid points.

Built on PR openai#414 (base model), PR openai#461 (TTT recipe), PR openai#493 (LeakyReLU²).

- Interleaved draft tokens: soft predictions placed between real tokens
  for 1-2 token lookahead via standard causal attention
- SmearGate and BigramHash naturally gain future context on interleaved seq
- Bigram noise curriculum: drafts anneal from GT to realistic noise
- Two-pass eval: pass 1 generates drafts, pass 2 refines with interleaving
- LeakyReLU(0.5)² activation toggle (free -0.003 BPB from PR openai#493)
- W&B logging (opt-in via WANDB_PROJECT env var)
- Sweep runner with 13 configs covering baselines, draft variants, and ablations

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

LeakyReLU(0.5)²: preserves negative gradient flow through MLP while
maintaining non-negative output. ~0.003 BPB improvement per PR openai#493.

Legal TTT (test-time training): at eval time, split val tokens into
32K-token chunks, score each chunk under inference_mode(), then train
on the already-scored chunk with SGD. Gives ~0.0025 BPB improvement
per PR openai#461. Score-first protocol guarantees no future information
leaks into scored tokens.

…stopher-Lee-McClendon

…stopher-Lee-McClendon

…stopher-Lee-McClendon

…stopher-Lee-McClendon

…stopher-Lee-McClendon

…stopher-Lee-McClendon