PR openai#535 is at 1.1204 with LeakyReLU2 + Full GPTQ + QAT alignment.
Starting from their code instead of building from our openai#414 base.

6647 steps at 89ms with FA3 Hopper. 15.62MB fits.
Beats merged leader openai#414 (1.1228) and unmerged openai#535 (1.1204).
3-seed validation starting.

Pre-eval TTT was non-compliant per issue openai#402. Now uses
score-first TTT: score each chunk before training on it.
Added LeakyReLU(0.5)² replacing relu² (proven by openai#569, openai#535).
Score pending rerun with compute credits.

…by 0.0002

- gptq_calibrate(): collect Hessian H=X^TX via forward hooks on training data
- gptq_quantize_weight(): column-wise int6 with Cholesky error compensation
- _find_best_row_scales(): percentile search for optimal per-row scales
- Integrated into mixed_quantize_int6() — falls back to naive when no Hessian
- Expected: -0.0026 bpb from better quantization alone (PR openai#535 ablation)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Multiple top PRs (openai#535, openai#549, openai#569) demonstrate -0.0015 to -0.003 bpb
from this change. LeakyReLU preserves gradient flow through negative
pre-activations while maintaining the sparsity/gating benefits of
squaring. At 22M params, dead neurons from hard ReLU are expensive.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

openai#569 VRL produces 16.0-16.2MB models (over budget).
openai#535+XSA-all reliably fits at 15.6MB with 1.1188 BPP.
3-seed validation starting.

…x LeakyReLU author links

Run 1 showed:
- Pre-quant val_bpb: 1.1757
- Post-quant sliding window: 1.3569
- Quantization penalty: 0.18 bpb (expected ~0.003)

Root cause: Our GPTQ implementation (ported from PR openai#535) produced
WORSE quantization than standard per-row int6. PR openai#486 base doesn't
use GPTQ at all. Possible issues: bad Hessian calibration, numerical
instability in Cholesky decomposition, or name mismatch between
hooks and state dict keys.

Fix: Disable GPTQ, revert to standard quantization path.
GPTQ code preserved for future debugging.

Also confirmed: TTT bpb formula is algebraically correct.
The 0.6185 bpb was real (20 epochs = heavy per-doc overfitting).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…, LeakyReLU from @parinzee + @sofiabod

- gptq_calibrate(): collect Hessian H=X^TX via forward hooks on training data
- gptq_quantize_weight(): column-wise int6 with Cholesky error compensation
- _find_best_row_scales(): percentile search for optimal per-row scales
- Integrated into mixed_quantize_int6() — falls back to naive when no Hessian
- Expected: -0.0026 bpb from better quantization alone (PR openai#535 ablation)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Run 1 showed:
- Pre-quant val_bpb: 1.1757
- Post-quant sliding window: 1.3569
- Quantization penalty: 0.18 bpb (expected ~0.003)

Root cause: Our GPTQ implementation (ported from PR openai#535) produced
WORSE quantization than standard per-row int6. PR openai#486 base doesn't
use GPTQ at all. Possible issues: bad Hessian calibration, numerical
instability in Cholesky decomposition, or name mismatch between
hooks and state dict keys.

Fix: Disable GPTQ, revert to standard quantization path.
GPTQ code preserved for future debugging.

Also confirmed: TTT bpb formula is algebraically correct.
The 0.6185 bpb was real (20 epochs = heavy per-doc overfitting).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>