Summary

Implements online sequence-level curriculum learning that scores and filters sequences within each batch by unigram entropy, following a V-shaped difficulty schedule aligned with LR warmdown and SWA phases. Zero extra parameters. Built upon PR #623.

Motivation

Standard training feeds random batches regardless of training phase. In a 600-second window (~1100 steps), the model benefits from different data at different stages:

• Early training (high LR): easy sequences → stable gradients, fast initial convergence
• Mid training: hard sequences → push the model's frontier while LR is still meaningful
• Late training (SWA region): easy sequences → coherent checkpoint averaging

Method

Per-sequence difficulty score — unigram entropy:

H(s) = -Σ p_s(t) · log₂(p_s(t)) for each sequence s of length 2048

V-shaped target — maps training progress to difficulty percentile d ∈ [0,1]:

d(step) = step / (0.45 · T) if step ≤ 0.45·T
d(step) = 1 - (step/T - 0.45) / (1 - 0.45) otherwise

Selection: Load 2× sequences per batch, sort by entropy, select the half centered around percentile d(step). The V-shape completes within each batch — no dependence on shard ordering.

Results

• val_bpb: 1.3557 (post int6+zstd, 1×H100, seed=42)
• Pre-quant: 1.3280 | Quant penalty: 0.0277
• 1,021 steps in 600s (588 ms/step) | 15.25MB artifact
• Run on 1×H100 due to compute constraints

Observation

Worse than baseline (1.3345). The 2× oversampling adds ~50ms/step overhead (588ms vs 540ms), costing ~80 training steps. The curriculum signal doesn't compensate for lost steps. Implication: curriculum at this scale must be zero-overhead (precomputed ordering, not runtime filtering).