Matthew Leavitt (@leavittron) / X

Matthew Leavitt

3,192 posts

Matthew Leavitt

@leavittron

Co-Founder @datologyai. Former: Head of Data Research @MosaicML; FAIR. 🧠 and 🤖 intelligence // views are from nowhere

The Bay

Joined March 2011

Pinned
Matthew Leavitt
@leavittron
Aug 18, 2025
Very excited to announce BeyondWeb, @datologyAI’s synthetic pretraining data generation paradigm. BeyondWeb is a rephrasing-based approach that substantially outperforms existing public synthetic pretraining data baselines, and is a core part of our curation pipeline.
33K
Matthew Leavitt
@leavittron
Jun 24, 2023
As a neuroscientist imma call bullshit on this. All these "mind reading" techniques rely on an fmri scanner: a multimillion dollar, 10000lb+ machine that requires a purpose-built facility and you have to lie perfectly still in it for it to work. Nobody's stealing your thoughts
Enzo Avigo
@0zne
Jun 23, 2023
We’re basically done.
00:00
401K
Matthew Leavitt
@leavittron
May 20, 2023
v excited to finally announce our new work that formalizes one of the most effective practices for training LLMs—something that many industry leaders have conspicuously avoided discussing
167K
Matthew Leavitt
@leavittron
Nov 20, 2023
64K
Matthew Leavitt
@leavittron
Aug 20, 2025
Ali Ghodsi
@alighodsi
Aug 19, 2025
Databricks just signed a Series K term sheet at >$100B valuation to scale two flagship products: 🔥 Lakebase — serverless Postgres with true compute/storage separation 🧠 Agent Bricks — agentic framework with built-in reasoning guardrails for enterprise data
18K
Matthew Leavitt
@leavittron
Oct 20, 2023
There are like 5 people in all of deep learning who have actually looked at the pretraining data that a 7B+ model has been trained on (and three of them went mad)
69K
Matthew Leavitt
@leavittron
May 12, 2023
By now you may have seen some hubbub about @MosaicML’s MPT-7B series of models: MPT-7B base, MPT-7B-Instruct, MPT-7B-Chat, and MPT-7B-StoryWriter-65k+. These models were pretrained on the same 1T token data mix. In this 🧵I break down the decisions behind our pretraining data mix
72K
Matthew Leavitt
@leavittron
Dec 7, 2023
A few (somewhat data-centric) thoughts on the Gemini whitepaper 🧵: Can't be much more direct than this: "We find that data quality is critical to a highly-performing model". It feels especially true cuz they provide next to no information on the training data.
39K
Matthew Leavitt
@leavittron
Jul 31, 2025
post-training researchers
12K
Matthew Leavitt
@leavittron
Nov 14, 2024
🧵We’ve spent the last few months at @datologyai building a state-of-the-art data curation pipeline and I’m SO excited to share our first results: we curated image-text pretraining data and massively improved CLIP model quality, training speed, and inference efficiency 🔥🔥🔥
GIF
76K
Matthew Leavitt
@leavittron
Oct 20, 2023
It seems likely to me that Mistral 7B's quality comes from its data. You know, the thing they provide exactly zero information about. The sliding window attention is a red herring.
Lior Alexander
@LiorOnAI
Oct 12, 2023
Mistral just released the paper behind their impressive LLM: Mistral 7B. The model outperforms Llama2 13B on every benchmark. Architecture: - Uses Grouped-query attention (GQA) for faster inference -Uses Sliding Window Attention (SWA) to handle longer sequences at smaller
37K
Matthew Leavitt
@leavittron
May 20, 2023
Replying to @leavittron
s/o to @danielking36 for the exceptional title. We also considered "Training on the test set is all you need", "The Unreasonable Effectiveness of Training on the Test Set", and "Intriguing Properties of Training on Test Data"
9.4K
Matthew Leavitt
@leavittron
Feb 22, 2024
The next 10x in deep learning efficiency gains are going to come from intelligent intervention on training data. But tools for automated data curation at scale didn’t exist—until now. I’m so excited to announce that I’ve co-founded @DatologyAI, with @arimorcos and @hurrycane
15K
Matthew Leavitt
@leavittron
Jun 19, 2025
"Pretraining is dead" is dead
8K