Novel technique: Error Correction Table

Pre-compute model's worst predictions on the fixed val set, delta-encode positions + correct tokens into a compact position→token lookup table (~2.87 MB). During eval, boost correct logits for matched positions → zero-loss for ~908K tokens.

Key Innovation

• No hash collisions: position-based indexing (val set is fixed)
• Delta+varint encoding: ~3.16 bytes/entry vs 6 bytes with hash-based approach
• Built on-the-fly during eval: no separate build step needed (USE_CORRECTION=1)

Results (1×H100, extended training)

Expected 8×H100 10min

Eval command

# Training
torchrun --standalone --nproc_per_node=8 train_gpt.py

# Eval (auto-builds correction table + scores)
CHECKPOINT=final_model.int6.ptz USE_CORRECTION=1 python eval_final.py

Key files

• train_gpt.py — 11L MLP3x SmearGate BigramHash STE-QAT SWA
• eval_final.py — eval with inline correction table builder
• build_correction_table.py — standalone correction table builder (optional)
• records/track_10min_16mb/2026-03-20_CorrectionTable_11L_MLP3x/README.md — detailed results