The Frugendorff: Recursive Weight Sharing Under Extreme Compression

Research submission documenting the full arc of fractal weight sharing — from the original Frugendorff (6x2 symmetric, 1.1478) through the Micro Crawler (4f+2cx2, 1.1325), including systematic ablations that map the tradeoffs of recursion under compression.

This is not a SOTA submission. It is a research direction exploring whether recursive weight reuse can improve compute-per-byte efficiency in size-constrained language models.

Best Result

• val_bpb: 1.1325 (sliding window, stride 64) — Micro Crawler, cad0 (no double-firing)
• val_bpb: 1.1355 — Micro Crawler + bidirectional PD, cad2
• val_bpb: 1.1478 — Original Frugendorff Squared (6x2 uniform)
• 8xH100 SXM, 600s, seed 1337

Architecture Evolution

Stage 1: Frugendorff Squared (1.1478 BPB)

6 unique blocks x 2 loops = 12 effective depth. All blocks shared. MLP 4x enabled by parameter savings.

Stage 2: Micro Crawler (1.1377 -> 1.1325 BPB)

Asymmetric sharing — 4 unique flat blocks + 2 shared crawler blocks. Flat section trains with zero gradient conflict. Only the crawler pair shares weights.

Stage 3: Persistent Deliberation

The crawler's two orthogonal firings deliberate through a learned gate. Critical discovery: the gate needs bidirectional gradient flow — consensus_ref as nn.Parameter, not a detached buffer. Gradients flow IN (loss -> ref) and OUT (ref -> blocks) on every step.

PD showed mid-training advantages (+0.007 BPP ahead at steps 5000-7000) but the gains were fragile under EMA smoothing.

Cadence Ablation (H1 + H2)

Systematic sweep of C/N ratio (double-fire vs single-fire steps) across two architectures at 0.25 scale (150s, 8xH100) and full scale (600s).

H1: 4f+2cx2 Cadence Sweep (0.25 scale)

H1: Full Scale Confirmation (600s, production script)

H2: 3f+3cx2 Cadence Sweep (0.25 scale)

H4: Crawler Bank at U-Net Bottleneck

Shared block at the encoder/decoder bottleneck of GS v7: learns better per step (+0.016 BPP at step 1500) but loses on final sliding BPP (1.2371 vs 1.2145 control) due to 14% fewer steps.

Key Findings

What Works

1. Asymmetric > uniform sharing. 4f+2cx2 beats 6x2 by 0.010 BPP. Isolate gradient conflict to minimal shared blocks.
2. GPTQ is essential for shared weights. Quant gap drops from 0.0146 -> 0.0070.
3. MLP 4x is the primary quality driver — weight sharing is the compression technique that enables it.

Challenges with Current Recursion Implementation

The current double-firing mechanism shows real per-step learning benefit (crawler bank: +0.016 per step) but is challenged under wallclock constraints:

• Compute cost: C-steps are ~2x FLOP, reducing total steps by 10-20%
• EMA instability: Double-firing creates weight oscillation EMA can't track (gap: 0.105 at cad1 vs 0.053 at cad4)
• Quantization sensitivity: Quant gap scales with reuse frequency (0.030 at cad1 -> 0.006 at cad4)
• val@500 identical across cadences (1.384 +/- 0.0004) — C-steps are neutral per step
• Deeper stacks amplify these issues: 3f+3cx2 always worse than 4f+2cx2, with 6x2 cad1 going backwards after step 500

These are implementation-specific challenges, not fundamental limits. A cheaper recurrence mechanism (lightweight adapter loops, partial-block refire, amortized recursion) could capture the per-step learning benefit without the wallclock and EMA penalties.

Transferable Findings

1. EMA instability from parameter reuse — any weight-tied architecture (Universal Transformers, LoRA, MoE) suffers EMA tracking degradation proportional to reuse frequency
2. Training dynamics -> quantization robustness — how parameters are updated during training directly affects quant quality. 5x quant gap reduction from cad1 to cad4
3. Asymmetric parameter allocation — more unique + fewer shared is strictly better than balanced sharing

Full Results Table

Compliance

No test-time training on validation data. Training replay and self-distillation operate on training data only. All evaluation follows score-first protocol per issue #402.

Generated with Claude Code