Research contribution (not a record submission). I wanted to see if nGPT could be made competitive under Parameter Golf constraints after PR #831 dismissed it. Turns out it can — and the investigation turned up some findings that might be useful to others.

val_bpb: 1.1502 (sliding window, 8xH200, 560s)

Findings

• PR Research: Why Novel Architectures Fail at 16MB — Throughput-Quantization Co-optimization #831's failure was three init bugs, not architectural incompatibility. Fixing zero-init projections, adding learnable logit scaling, and not normalizing embeddings takes nGPT from 1.69 to 1.27 BPB.
• torch.compile has a precision compounding bug with sequential L2 normalization. Inductor fuses through float() casts; across 86 normalize calls the bf16 errors compound catastrophically. Fix: wrap normalize in an opaque autograd.Function via allow_in_graph. Generalizes beyond nGPT. (Large numeric divergence for torch compile vs eager in bf16 pytorch/pytorch#168126)
• Post-dequant renormalization cuts the quantization gap from +0.35 to +0.008 BPB. Three lines of code.
• nGPT's compression advantage vanishes at full training length — undertrained weights compress well because they're close to orthogonal init, not because of the architecture.
• Paper design choices (signed alpha, s_z scaling) hurt at 5000 steps — the optimizer doesn't have time to exploit the extra degrees of freedom.
• TTT is incompatible with renorm dequantization — produces NaN at every LR tested.

Hardware

MIT ORCD cluster, 8xH200 SXM, CUDA 12.4, PyTorch 2.6. 560s wallclock.

Full writeup with code, tables, and ablations in the README.