Summary

• 11-layer 512d GPT with PR#414's 10-technique stack + LeakyReLU(0.5)² activation
• GPTQ int6 quantization: Hessian-guided column-wise quantization replacing naive per-row rounding, reducing quantization error by 33.6% (Hessian-weighted MSE)
• SGD test-time training (TTT): Continues training on validation data in a causal (score-first) manner with cosine LR decay, adapting last 9/11 layers

Key Results

Techniques

Architecture (PR#414 stack): XSA4, EMA, U-Net skip, SmearGate, BigramHash, PartialRoPE, LNScale, ValueEmbed, LateQAT, SWA

Novel contributions:

1. LeakyReLU(0.5)² replacing GELU² — saves 0.0026 bpb by improving gradient flow
2. GPTQ int6 — data-dependent quantization using 256 calibration samples, block-128 updates
3. SGD TTT — Simple SGD (lr=0.002, momentum=0.9) with cosine schedule over 900 chunks of 32K tokens, 3 epochs/chunk

Compression

zstd level-21 with long-distance matching (LDM) for model artifact compression.

Files

• train_gpt.py — Full training + GPTQ + TTT evaluation pipeline
• eval_gptq.py — Standalone GPTQ evaluation script
• eval_ttt.py — Standalone TTT evaluation script
• submission.json — Structured results metadata
• train.log — Complete training log
• README.md — Detailed writeup with technique descriptions and ablations

See records/track_10min_16mb/2026-03-24_GPTQ_TTT/README.md for full details.