val_bpb = 1.1574 — Top-ranked valid submission at time of posting

Baseline: 1.2244. Improvement: -0.067 BPB / -0.120 nats. 15.98MB artifact. 600s training + 240s sliding window eval on 8xH100 SXM.

This is the third PR in a series (see also PR #61 at 1.2154 and PR #96 at 1.1764), each building on systematic experimentation across ~60 runs.

What's Novel

1. Train@2048 matches train@4096 when using sliding window eval. I tested both extensively — training at 2048 gives identical BPB to training at 4096 (1.1764 vs 1.1765) when evaluated with sliding window. The window already provides long context at eval; the model just needs to learn local patterns. Training at 2048 is strictly better because it gets more optimization steps in the 10-minute budget. Other top submissions train at 1024 or 4096 — 2048 is the overlooked sweet spot.

2. GRAD_CLIP_NORM=0.3 stabilizes long-sequence training. I swept from 0.0 to 1.0 and found a narrow optimum at 0.3 for train@2048. The mechanism: longer sequences produce higher gradient variance, and tight clipping (0.3 vs the common 1.0) stabilizes without over-constraining. Full sweep data in experiment log.

3. Batch=786K is optimal for train@2048. Swept from 393K to 1M. The default 524K is suboptimal; 786K balances gradient noise against step count.

4. Selective precision preservation. Two techniques stacked: (a) FP16 tied embedding — the dual-role embedding/output matrix is disproportionately damaged by quantization, discovered via systematic warmdown-quantization analysis in PR #61; (b) Late-K passthrough — last 2 layers' key projections kept in fp16 for sharper late-layer attention.

5. Stride=256 beats stride=64. Counter to other submissions using stride=64, I found stride=256 gives slightly better BPP (1.1574 vs 1.1579) at 4x less eval time. Diminishing returns from smaller strides.

Approach

Int6 post-training quantization (per-row scaling, ±31 range) compresses weight matrices ~25% smaller than int8, freeing artifact space to triple the MLP hidden dimension (1024 → 1536). 21.8M parameters in 15.98MB.

Training

TRAIN_SEQ_LEN=2048 TRAIN_BATCH_TOKENS=786432 MLP_HIDDEN=1536
MATRIX_LR=0.02 SCALAR_LR=0.02 TIED_EMBED_LR=0.03
MUON_MOMENTUM=0.99 MUON_MOMENTUM_WARMUP_START=0.92
MUON_MOMENTUM_WARMUP_STEPS=1500 WARMDOWN_ITERS=3000 GRAD_CLIP_NORM=0.3

Evaluation

Sliding window with stride=256 at eval_seq_len=2048. Every scored token sees 1792+ tokens of context. 240s eval time on 8xH100 (well within the separate 10-min eval budget).

Progression

Full experiment log with ~60 runs, sweep data, and negative results available in the repository.