Non-record submission. val_bpb=1.1329 (3-seed mean, std=0.0006). 11L 512d int6+zstd-22, 15.3 MB, 8xH100 SXM. 25 experiments in FINDINGS.md.

Findings not tested elsewhere

Codebook quantization (#24): K-means K=256 gives 87% lower reconstruction MSE than int6 per-row, but 25% larger artifact under zstd-22. Higher byte entropy in codebook indices compresses less efficiently than clamped int6 values. Other K values and codecs not tested.

Magnitude pruning (#23): Zeroing smallest 3% of weights increased our artifact by 728KB under zstd-22. 1% and 5% were neutral. Non-monotonic on our checkpoint; other weight distributions may differ.

Multi-token prediction (#14): Auxiliary t+2 head at 0.5× loss weight: +0.0018 BPP, 3% slower. Other weightings and scales not tested.

Embedding SVD (#25): Rank-64 explains 41.9% of variance on tok_emb (1024×512). Linear low-rank factorization not viable at this vocabulary size. Nonlinear methods not tested.

Also documented: Depth recurrence failure (#13), QAT under torch.compile (#2b, credit @152334H), int5 gap (#15), curriculum learning (#11), optimizer coverage (#16), and 14 others.

Reproduction

git clone https://github.com/mrdavtan/parameter-golf.git && cd parameter-golf && git checkout int6-3xMLP-pr
pip install flash-attn --no-cache-dir --no-build-isolation && pip install zstandard sentencepiece huggingface_hub
python3 data/cached_challenge_fineweb.py --variant sp1024
SEED=7 QAT=0 TTT_MAX_STEPS=500 TTT_FREEZE_BLOCKS=1 TRAIN_SEQ_LEN=2048 EVAL_SEQ_LEN=2048 \
UNET_SKIPS=1 ROPE_DIMS=16 LN_SCALE=1 ROPE_BASE=10000 EVAL_STRIDE=32 DOC_ISOLATED_EVAL=0 \
LATE_K_FP16=0 FP16_EMBED_EXPORT=0 \
torchrun --standalone --nproc_per_node=8 records/track_10min_16mb/2026-03-21_11L_XSA_EMA_TTT/train_gpt.py

Builds on PRs #162, #77, #180 and modded-nanogpt.