Skip to content

Non-record: Competitive Stack + Phonetic Tokenization Exploration (val_bpb=1.2055, 4xH100)#454

Open
nalediym wants to merge 1 commit intoopenai:mainfrom
nalediym:submission/competitive-stack-v2
Open

Non-record: Competitive Stack + Phonetic Tokenization Exploration (val_bpb=1.2055, 4xH100)#454
nalediym wants to merge 1 commit intoopenai:mainfrom
nalediym:submission/competitive-stack-v2

Conversation

@nalediym
Copy link
Copy Markdown

Summary

  • Competitive stack: int6 STE QAT, BigramHash, SmearGate, OrthoInit, MLP 3x
  • val_bpb: 1.2055 (sliding window, stride=64) on 4xH100 SXM
  • Includes IPA phonetic tokenization research (interesting negative result: phonetic encoding provides marginal gains in isolation but is subsumed by modern training techniques)
  • Known issue: model 19.6MB with int8 export, needs int6 export for 16MB compliance

See records/track_non_record_16mb/2026-03-20_CompetitiveStack/README.md for details.

…l_bpb=1.2055)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Si6gma added a commit to Si6gma/parameter-golf that referenced this pull request Mar 22, 2026
- DEEP_RESEARCH_PROMPT.md: Copy-paste prompts for Claude
- RECENT_PR_ANALYSIS.md: Analysis of latest PRs (openai#442-openai#454)
- Research priorities: Catalytic Residuals, Late QAT, 12L
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant