Summary

Base model optimization campaign on 8xH100 (600s wallclock). Rat Rod Green achieves 1.1129 BPB sliding window / 0.4489 BPB n-gram eval — our strongest neural model to date.

Config: Parallel Muon (PR#609) + XSA-all-11 + BigramHash 2048 + RoPE 16 + SWA 50 + entropy-adaptive n-gram eval (orders 2-9). Steps: 6882 at 87.20ms/step.

Ablation log (2026-03-26 → 2026-03-27)

All experiments A/B tested against v1 (1.1129). Nine levers tested — none beat baseline:

Key findings

• 87ms/step is near H100 hardware ceiling — gains must come from algorithmic changes
• Loss modification doesn't work for our architecture — both Siphon (ensemble loss) and complement weighting failed. Model learns best with uniform CE.
• Warmdown shape matters — 2000 >> 3500 >> 5000 at 200s, pending 600s validation

Next steps — pressing into novel territory

• Warmdown schedule shapes (Jitter/Swirl/Cascade — zero ms/step cost)
• N-gram eval parameter sweep on existing checkpoints (free BPB)
• Learned mixer head (frontier technique from PR#834/#859)
• Architecture exploration: gated attention, MTP variants

🤖 Generated with Claude Code