Summary

Non-record submission with two negative results under the 16MB artifact cap:

• Preliminary MoE negative result: a 2-expert soft-routing MoE (2 × 1.5x MLP) underperforms the dense control throughout the observed training window. I added moe_train_partial.log, the surviving partial 8xH100 SXM log; the RunPod pod died at step 2000, so the MoE conclusion should be interpreted as preliminary rather than a fully converged final result.
• Leaderboard-relevant multi-bit quantization comparison: the dense control reaches 1.1456 val_bpb, which is within 0.0028 BPB of the March 20, 2026 leaderboard leader (1.1428). On that same trained dense model, int5 MLP costs +0.0068 BPB while int4 MLP costs +0.0655 BPB, making aggressive quantization unattractive for MoE expansion at this scale.

Included evidence

• README.md with updated explanation and MoE-vs-dense checkpoint table
• submission.json with updated metadata
• train.log for the dense control / quantization comparison
• moe_train_partial.log for the surviving MoE run
• train_gpt.py
• quant_comparison.png

Quantization Comparison Results

MoE Observed Checkpoints