Record: 11L 512d Int8+Zlib Baseline

val_bpb: 1.2135 (3-seed mean) | 15.54 MB (mean) | 8xH100 SXM, 599s

Summary

Baseline train_gpt.py with NUM_LAYERS=11 (up from the default 9). All other hyperparameters are stock defaults. This submission demonstrates the baseline architecture properly scaled with additional depth on 8xH100 SXM hardware.

Changes from Naive Baseline

Results (3 seeds, 8xH100 SXM)

Mean: 1.2135 | Std: 0.0003

Architecture

• 11 transformer layers, 512-dim, 8 heads (4 KV heads, GQA)
• 2x MLP expansion (1024 hidden)
• U-Net skip connections (5 encoder, 6 decoder)
• Tied embeddings, logit softcap=30.0
• Vocab size 1024 (SentencePiece BPE)
• Muon optimizer, int8+zlib quantization
• Total artifact: 15,541,950 bytes (well under 16MB cap)

Run Command

NUM_LAYERS=11 SEED=1337 \
DATA_PATH=./data/datasets/fineweb10B_sp1024/ \
TOKENIZER_PATH=./data/tokenizers/fineweb_1024_bpe.model \
VOCAB_SIZE=1024 \
torchrun --standalone --nproc_per_node=8 train_gpt.py

Notes

This is a non-SOTA submission demonstrating baseline scaling. We have several novel techniques in development (TTT, GPTQ-lite, SmearGate, BigramHash, PolarQuant, hybrid RNN-attention architectures) that we plan to submit as improved records.

Checklist

• README.md with detailed explanation
• submission.json with metadata
• Training logs for 3 seeds
• train_gpt.py (stock baseline, compiles and runs)
• All seeds under 16MB and under 10 minutes on 8xH100 SXM