The Frugendorff

6 unique transformer blocks, each applied twice in sequence, yielding 12 effective layers from 6 stored parameter sets. The parameter savings are redirected to a 4x MLP expansion.

val_bpb: 1.1478 | 15.19 MB | 8xH100 SXM, 600s, seed 1337

Architecture

Orthogonal loop position embeddings (QR-initialized). U-Net skip connections within each loop iteration. SmearGate, BigramHash, shared value embeddings, XSA on last 2 blocks. Tied embeddings, logit softcap 30.

Training Pipeline

Muon (matrices) + AdamW (embeddings, scalars) · SWA · Late QAT (int6, scale < 0.15) · Training data replay (2 epochs, last 100 batches) · Self-distillation (EMA teacher, 50 steps) · EMA application · int6 + zstd export

Results

Observations

A few properties of this approach that may warrant further investigation:

• The shared weights can be evaluated with fewer loops at inference time, trading quality for speed without retraining
• The parameter efficiency leaves ~0.8 MB of headroom in the artifact budget, which could accommodate additional capacity
• Alternating between full-depth and reduced-depth forward passes during training appeared to act as a form of regularization, though we have not yet isolated the effect
• The recursive structure may compose well as a component within conventional architectures, applied selectively to a subset of layers

Compliance

No test-time training on validation data. Training replay and self-distillation operate on training data only.