EleutherAI (@AiEleuther) / X

EleutherAI

929 posts

EleutherAI

@AiEleuther

A non-profit research lab focused on interpretability, alignment, and ethics of AI. Creators of Pythia, VQGAN-CLIP, and using SAEs for interp

Joined August 2022

Pinned
EleutherAI
@AiEleuther
Jun 6, 2025
Can you train a performant language models without using unlicensed text? We are thrilled to announce the Common Pile v0.1, an 8TB dataset of openly licensed and public domain text. We train 7B models for 1T and 2T tokens and match the performance similar models like LLaMA 1&2
179K
EleutherAI
@AiEleuther
Mar 2, 2023
Over the past two and a half years, EleutherAI has grown from a group of hackers on Discord to a thriving open science research community. Today, we are excited to announce the next step in our evolution: the formation of a non-profit research institute.
blog.eleuther.ai
The View from 30,000 Feet: Preface to the Second EleutherAI Retrospective
(Some of) what we've been up to for the past year-and-a-half at EleutherAI.
346K
EleutherAI
@AiEleuther
Mar 21, 2023
Introducing the release 2.0 of GPT-NeoX, the open-source Megatron-DeepSpeed based library used to train GPT-NeoX-20B and the Pythia model suite.
GitHub - EleutherAI/gpt-neox: An implementation of model parallel autoregressive transformers on...
From github.com
79K
EleutherAI
@AiEleuther
Apr 19, 2023
The most common question we get about our models is "will X fit on Y GPU?" This, and many more questions about training and inferring with LLMs, can be answered with some relatively easy math. By @QuentinAnthon15, @BlancheMinerva, and @haileysch__
blog.eleuther.ai
Transformer Math 101
We present basic math related to computation and memory usage for transformers
79K
EleutherAI
@AiEleuther
May 23, 2023
Everyone knows that transformers are synonymous with large language models… but what if they weren’t? Over the past two years @BlinkDL_AI and team have been hard at work scaling RNNs to unprecedented scales. Today we are releasing a preprint on our work
arxiv.org
RWKV: Reinventing RNNs for the Transformer Era
Transformers have revolutionized almost all natural language processing (NLP) tasks but suffer from memory and computational complexity that scales quadratically with sequence length. In contrast,...
120K
EleutherAI
@AiEleuther
Dec 16, 2022
What do LLMs learn over the course of training? How do these patterns change as you scale? To help answer these questions, we are releasing a Pythia, suite of LLMs + checkpoints specifically designed for research on interpretability and training dynamics!
GitHub - EleutherAI/pythia: The hub for EleutherAI's work on interpretability and learning dynamics
From github.com
116K
EleutherAI
@AiEleuther
Aug 11, 2024
📕Today, we'd like to draw attention to the EleutherAI cookbook! (github.com/EleutherAI/coo…) The cookbook contains practical details and utilities that go into working with real models! Such as: 🧵
GitHub - EleutherAI/cookbook: Deep learning for dummies. All the practical details and useful...
From github.com
49K
EleutherAI
@AiEleuther
Sep 27, 2022
As part of our work to democratize and promote access to language model technology worldwide, the Polyglot team at EleutherAI is conducting research on multilingual and non-English NLP. We are excited to announce their first models: Korean LLMs with 1.3B and 3.8B parameters.
EleutherAI
@AiEleuther
Mar 9, 2023
We have been getting emails from confused individuals trying to access api.eleuther.ai recently. That webpage doesn’t exist, because we don’t have an API. One of them finally clued us in to why: apparently ChatGPT suggests it for trying out our models.
49K
EleutherAI
@AiEleuther
May 24, 2024
Excited to share our new paper, Lessons From The Trenches on Reproducible Evaluation of Language Models! In it, we discuss common challenges we’ve faced evaluating LMs, and how our library the Evaluation Harness is designed to mitigate them 🧵 arxiv.org/abs/2405.14782
71K
EleutherAI
@AiEleuther
Sep 7, 2023
ggml is a deeply impressive project, and much of its success is likely ascribable to @ggerganov's management. Managing large-scale branching collaborations is a very challenging task (one we hope to improve at!), and Georgi deserves huge props for how he handles it.
Georgi Gerganov
@ggerganov
Aug 31, 2023
The ggml roadmap is progressing as expected with a lot of infrastructural development already completed We now enter the more interesting phase of the project - applying the framework to practical problems and doing cool stuff on the Edge
39K
EleutherAI
@AiEleuther
Jul 21, 2023
We applaud @Meta’s continued push to openly license their models with #Llama2 having the most permissive license yet. However we are extremely sad to see Meta continue to spread misinformation about the licensing of the model: LLaMA 2 is not open source
Meta’s LLaMa license is not Open Source
From opensource.org
64K
EleutherAI
@AiEleuther
Feb 29, 2024
We’re excited to be collaborating on a new *resource release* to help provide an on-ramp for new open model developers: the Foundation Model Development Cheatsheet!
18K
EleutherAI
@AiEleuther
Nov 20, 2024
EleutherAI is hiring post-doctoral researchers and interns who don't have PhDs. Email contact@ with: - Two papers you thought were interesting and what you would do differently from them - A cool research idea - A resume, if you really want to
24K