Summary

Adds a learned dynamic chunking layer (inspired by H-Net) to the standard transformer baseline. The chunker predicts soft boundaries between adjacent token embeddings and blends neighbors where boundaries are low — a differentiable approximation of H-Net's hard chunking.

val_bpb: 1.3587 | 1×RTX 5090, 600s | TTT enabled

• Nearly matches baseline (1.3577) with only ~263K extra parameters
• Demonstrates learned tokenization is viable for parameter-constrained models
• Implements one of OpenAI's explicitly requested research directions (H-net tokenization)
• Setting HNET_ENABLED=0 produces identical behavior to base script

Changes

• DynamicChunker module: boundary prediction + soft blending
• Inserted after embedding norm, before transformer blocks
• New env vars: HNET_ENABLED, HNET_LAYERS

Test plan

• Verified on 1×RTX 5090 (600s wallclock)
• Control run with HNET_ENABLED=0 matches baseline
• Model fits in 16MB after int8+zlib compression

🤖 Generated with Claude Code