11L Int5-MLP + TTT-SGD + SmearGate + SWA (1.1455 BPB)#264
Open
stukenov wants to merge 1 commit intoopenai:mainfrom
Open
11L Int5-MLP + TTT-SGD + SmearGate + SWA (1.1455 BPB)#264stukenov wants to merge 1 commit intoopenai:mainfrom
stukenov wants to merge 1 commit intoopenai:mainfrom
Conversation
11-layer model with mixed int5/int6 quantization, full-model SGD test-time training, SmearGate, BigramHash, SWA, and OrthoInit. Single seed result (1337): val_bpb=1.1455 Artifact: 15.94 MB (under 16MB limit) 3-seed validation in progress.
|
ttt with int5 mlp is a sick combo, how long does your ttt take during eval? tryna figure out if 3 epochs is worth it over 2 |
Author
|
@notapplica i am need runpod credits for 3-seed validation |
HyperPotatoNeo
added a commit
to HyperPotatoNeo/parameter-golf
that referenced
this pull request
Mar 21, 2026
Stacks XSA (PR openai#265), EMA weight averaging (PR openai#287), Int5-MLP (PR openai#264), MuonWD=0.04 tuned from PR openai#162, seq_len=2048, 11 layers, BigramHash(2048), SmearGate, OrthoInit (PR openai#135), Late-K FP16 on final layer. Single-seed result (seed=1337), ~8903 steps on 8xH100.
ThomAub
pushed a commit
to ThomAub/parameter-golf
that referenced
this pull request
Mar 22, 2026
Many TTT submissions (openai#136, openai#152, openai#254, openai#264, openai#338, openai#398, openai#417, openai#421, openai#442) flagged as potentially invalid for adapting on eval tokens BEFORE scoring them. Added correct score-then-adapt protocol with implementation guide. https://claude.ai/code/session_01M5XTtyz2Zdq5BDeh9qNn9y
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Techniques
Architecture
11L / 512d / 8h / 4kv (GQA) / MLP 3x / relu^2 / 2048 seq_len / 26.67M params
Results
Trained on 8xH100 SXM, 600s wallclock, 115ms/step.
Test plan