Goodfire (@GoodfireAI) / X

Goodfire

635 posts

Goodfire

@GoodfireAI

Using interpretability to understand, learn from, and design AI.

San Francisco

Joined August 2024

Pinned
Goodfire
@GoodfireAI
May 7
Neural networks might speak English, but they think in shapes. Understanding their rich *neural geometry* is key to understanding how they work – and to debugging and controlling them with precision. Starting today, we’re releasing a series of posts on this research agenda. 🧵
00:00
3.2M
Goodfire
@GoodfireAI
15h
Cool work applying the idea from our work on RLFR to RL task generation!
Augustine Mavor-Parker
@MavorParker
15h
Training a model to generate RL tasks not too hard, not too easy costs many solver runs per task. PROPEL predicts difficulty via a probe on its activations instead, amortizing cost and speeding up generator optimization. New open-ended RL research from @vmax + @GoodfireAI.
GIF
4.3K
Goodfire
@GoodfireAI
Jun 4
New Goodfire research: using logits to monitor for eval awareness!
Santiago Aranguri
@santiaranguri
Jun 4
Would an LLM tell you if it’s gaming your eval? Often, no. But we can still catch the model thinking about it. New research: we measure how close a model comes to saying it’s being tested. This detects eval awareness with 10× to 100× fewer samples than monitoring model
10K
Goodfire reposted
Salesforce Ventures
@SalesforceVC
Jun 2
The idea that launched @GoodfireAI🔥 When ChatGPT launched, most people were blinded by the possibilities. Eric Ho saw the risk it posed. "I kind of saw the next few years unfold before me, where we were about to get increasingly powerful models ... massive amounts of compute,
00:00
1.7K
Goodfire reposted
Ekdeep Singh Lubana
@EkdeepL
Jun 1
Very excited to have this paper out! We show by having more parameters, larger models see reduced interference between updates. This allows them to retain memories of rarely observed samples of a task, eventually allowing them to learn even the tail-end of the distribution. (1/3)
00:00
Christopher Potts
@ChrisGPotts
Jun 1
We take for granted that larger models are better than smaller ones, but why is this so? Our new paper, led by Jing Huang and @EkdeepL, traces this to a data-induced competition for resources (neurons), using formal analysis, idealized tasks, and real pretraining.
16K
Goodfire
@GoodfireAI
Jun 1
New research from Goodfire and collaborators: why do larger models learn more tasks? (spoiler: it’s bottlenecked by data)
Christopher Potts
@ChrisGPotts
Jun 1
We take for granted that larger models are better than smaller ones, but why is this so? Our new paper, led by Jing Huang and @EkdeepL, traces this to a data-induced competition for resources (neurons), using formal analysis, idealized tasks, and real pretraining.
21K
Goodfire reposted
Can Rager
@can_rager
May 21
The "tiling" perspective explains a lot of the common problems with SAEs
00:21
Goodfire
@GoodfireAI
May 21
The most popular way to interpret AI is missing the bigger picture. Models think in curved shapes. But sparse autoencoders (SAEs) work with straight lines. Can they still capture models’ curved neural geometry? Yes, but not how you might think! (1/7)
16K
Goodfire
@GoodfireAI
May 21
Replying to @GoodfireAI
SAEs remain useful, as long as we’re aware of their limitations. And we have new techniques in the works that recover manifolds more directly, allowing us to understand models better and control them more effectively! Read the full post here:
Can SAEs Capture Neural Geometry?
From goodfire.ai
4.4K
Goodfire
@GoodfireAI
May 21
Replying to @GoodfireAI
So instead of interpreting features in isolation, what if we searched for features that act together? We turned this idea into an unsupervised pipeline to cluster SAE features based on firing patterns. Together, a cluster of features reveals the overall geometry. (6/7)
25K
Goodfire
@GoodfireAI
May 21
Replying to @GoodfireAI
This helps explain why SAEs can feel both illuminating and unsatisfying! Looking at SAE features one-by-one is like trying to understand the proverbial elephant by talking with each of the blind men: each label may be locally accurate, but the global structure is missing. (5/7)
3.1K
Goodfire
@GoodfireAI
May 21
Replying to @GoodfireAI
We now know that models think using curved shapes, not just straight lines. But SAE features can still give us a window into neural geometry. How? We show that related SAE features often “tile” manifolds, pointing to different (but overlapping) regions on the curve. (4/7)
15K
Goodfire
@GoodfireAI
May 21
Replying to @GoodfireAI
SAEs decompose neural representations using linear directions, decoding a model’s inner world. Researchers hoped each direction (feature) would be a single concept, with magnitude corresponding to something like intensity or confidence. (3/7)
4K
Goodfire
@GoodfireAI
May 21
Replying to @GoodfireAI
Consider the parable of the blind men encountering an elephant for the first time. Each touches a different part—the trunk, the tusk, the leg—and comes to a different conclusion about the elephant: one says it's like a tree, another says it’s like a rope, and so on. (2/7)
5.1K