Summary

Pre-quant val_bpb: 1.2035 on 1xH100 (7900 steps). Projected ~1.19 post-quant with sliding window. Non-record — awaiting 8xH100 compute.

12-layer transformer with Low-Rank Q factorization (r=128) and Quantization-Aware Training, developed through a 3-stage research pipeline that started with applied math prototyping on Apple Silicon.

Technique Stack

Research Pipeline (what makes this interesting)

Stage 1: Apple Silicon prototyping

Started from cross-disciplinary ideas — contraction mappings (DEQ/Banach), Lyapunov stability, FTLE from fluid mechanics. Prototyped on 1MB data subsets:

• Layer sharing (depth recurrence): 3 shared blocks = 9 unique at 1/3 params. Validated that trained shared blocks become contractive (Lyapunov exponent ≈ -0.008).
• FTLE sensitivity tracking: Built gradient-based row sensitivity tracker to guide per-row quantization precision.
• Finding: At tiny scale (256d, 1.45M params), more iterations with fewer params beats fewer iterations with more params at equal wall time.

Stage 2: A100 validation (TACC Lonestar6)

• Layer sharing abandoned at 512d — costs 0.09 BPB vs unique layers. The 16MB budget already fits enough unique parameters, so sharing doesn't unlock anything.
• Integrated BigramHash + SmearGate → 0.094 BPB improvement over baseline.
• Best A100 result: 1.3260 BPB (9L, zstd-22, sliding window s1024).

Stage 3: H100 refinement

• Implemented Low-Rank Q + 12L + QAT + FTLE + Stride-OGD.
• Clean negative result: FTLE per-row precision does NOT help. Uniform int-N has both lower RMSE and smaller compressed size at every bit width. Mixed int4-int8 values have higher entropy → worse zstd compression.
• Pre-quant: 1.2035 BPB at 7900 steps (est. ~7800 steps on 8xH100 in 10min).

What we'd do with a $500 RunPod grant

Phase 1: 8xH100 validation of 12L + Low-Rank Q + QAT → expect ~1.17-1.19 BPB post-quant.

Phase 2: Hyperparameter sweep — WD (0.03-0.05), LR (0.020-0.035), Muon momentum (0.95-0.99), SWA cadence (every 25-100 steps).

Phase 3: Novel combinations — QAT with int5-MLP/int6-attn mixed quant (nobody has combined QAT with PR #180's mixed scheme), 13 layers with Low-Rank Q savings, fix Stride-OGD speed.

Phase 4: 3-seed validation + submission packaging.

Phase 5: Frontier exploration — NTK-RoPE 4096 at eval, adaptive per-layer Q rank, BitNet b1.58 (ternary weights for 5x params in same space).

Negative Results (saving others time)

1. FTLE per-row precision is a dead end. At every bit width (int5 through int8), uniform per-row quantization has both lower reconstruction error AND smaller compressed size than FTLE-guided mixed precision. The intuition: mixing different bit widths per row increases the entropy of the quantized values, defeating zstd compression.

2. Layer sharing doesn't help at 512d. The 16MB budget fits ~22M unique params with int6. Sharing saves artifact space that isn't needed, while costing 0.09 BPB from reduced per-layer specialization.

3. Stride-OGD needs batched gradient computation. Tracking gradients through full [batch, seq_len, vocab] logits tensors is prohibitively slow. A batched or approximate approach is needed.

Test Plan

• 1xH100 training (7900 steps, pre-quant 1.2035)
• FTLE ablation (uniform beats FTLE at all bit widths)
• QAT integration (6% overhead, clean training)
• 8xH100 full run (awaiting compute)
• 3-seed statistical validation
• Post-quant + sliding window scoring