Non-record: Meta-TTT + Cache/OGD Eval Stacking + Tokenizer Ablation#384
Open
anantdgoel wants to merge 1 commit intoopenai:mainfrom
Open
Non-record: Meta-TTT + Cache/OGD Eval Stacking + Tokenizer Ablation#384anantdgoel wants to merge 1 commit intoopenai:mainfrom
anantdgoel wants to merge 1 commit intoopenai:mainfrom
Conversation
Three novel research contributions with controlled ablations: 1. MAML Meta-TTT (negative: -0.085 BPB, hyperparams too aggressive) 2. Eval-time cache mixture + OGD vocab bias (-0.004 BPB additive on TTT) 3. Tokenizer optimization (null: -5.7% tokens/byte, no BPB gain) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Non-record: Meta-TTT + Cache/OGD Eval Stacking + Tokenizer Ablation
val_bpb: 1.2882 (sliding window) | 13.2 MB | 1xA40 Secure, 2000 steps, 524K batch
Three novel research directions with controlled ablations, each exploring techniques no other submissions have investigated.
Contributions
1. MAML-style Meta-TTT (negative result)
First-order MAML during training to optimize initialization for TTT adaptation. Controlled A/B test (same 524K batch, same arch):
2. Eval-time technique stacking (positive result)
Unigram cache mixture + online gradient descent on vocab bias, stacked on SGD TTT:
3. Tokenizer optimization (null result)
First submission to modify the tokenizer. BPE with
split_digits=False, max_len=64gives -5.7% fewer tokens/byte at v8192, but no BPB improvement (+0.0006 worse). Longer merged tokens are harder to predict per-token, offsetting compression gains. Explains why community converged on stock v1024.Files
README.md— Full writeup with ablation tablessubmission.json— Metadatatrain_gpt.py— Modified script with cache mixture, OGD, Meta-TTTtrain.log— Training log from A40 control run