Sally Zhu (@SallyHZhu) / X

Sally Zhu

44 posts

Sally Zhu

@SallyHZhu

@flappyairplanes | Math, CS @stanford!

California

Joined October 2024

Pinned
Sally Zhu
@SallyHZhu
Jan 28
We’re charting something new and incredibly exciting here at @flappyairplanes! Grateful to be working with this brilliant and supportive team. Blue skies ahead ✈️🌁
Flapping Airplanes
@flappyairplanes
Jan 28
Announcing Flapping Airplanes! We’ve raised $180M from GV, Sequoia, and Index to assemble a new guard in AI: one that imagines a world where models can think at human level without ingesting half the internet.
GIF
19K
Sally Zhu
@SallyHZhu
Oct 23, 2025
🔎Did someone steal your language model? We can tell you, as long as you shuffled your training data🔀. All we need is some text from their model! Concretely, suppose Alice trains an open-weight model and Bob uses it to produce text. Can Alice prove Bob used her model?🚨
00:00
472K
Sally Zhu
@SallyHZhu
Oct 23, 2025
Replying to @SallyHZhu
Paper🔗: arxiv.org/abs/2510.19796… Will appear at NeurIPS '25 as a Spotlight🌟! I'm extremely grateful to have worked on this with my amazing, brilliant collaborators and mentors @rckpudi, Jing and advisors @percyliang @ChrisGPotts @Diyi_Yang. Happy provenance-ing!
arxiv.org
Blackbox Model Provenance via Palimpsestic Membership Inference
Suppose Alice trains an open-weight language model and Bob uses a blackbox derivative of Alice's model to produce text. Can Alice prove that Bob is using her model, either by querying Bob's...
12K
Sally Zhu
@SallyHZhu
Oct 23, 2025
Replying to @SallyHZhu
We develop metrics that correlate properties of Bob’s model/text with the ordering of Alice’s training data (e.g., does the text overlap more with later-seen examples), motivated by the *palimpsestic* effects of data ordering on a model’s generations: models more strongly
8.3K
Sally Zhu
@SallyHZhu
Oct 23, 2025
Replying to @SallyHZhu
In the query setting, we run our test on various fine-tunes (SFT, DPO, etc.) of Pythia and OLMo models (which have openly available training data 🩵). We get *extremely* low p-values in almost all cases. We are able to identify cases on HuggingFace where models are mislabeled:
6.4K
Sally Zhu
@SallyHZhu
Oct 23, 2025
Replying to @SallyHZhu
Finally, we have early signs of life applying our tests to the much harder distillation setting: 📌Can Alice test if Bob *distilled* from her model (e.g. trained on text generated from her model)? More to come, and definitely a problem worth studying more 😁!
6.4K
Sally Zhu
@SallyHZhu
Oct 23, 2025
Replying to @SallyHZhu
We study this question in two fully *blackbox* settings — 1) (query) Alice can query Bob’s model behind an API by prompting, and 2) (observational) Alice observes text from Bob.
9K
Sally Zhu
@SallyHZhu
Oct 23, 2025
Replying to @SallyHZhu
In the observational setting, our strongest test (which involves partial retraining of Alice’s model as a one-time expense) is able to reliably attribute Bob’s text to Alice’s model from as little as ~300 tokens. 🤯 This method approaches the effectiveness of inference-time
5.8K
Sally Zhu
@SallyHZhu
Jul 22, 2025
Interesting that you need the teacher + student to share the same base model; reminds me of linear mode connectivity stuff. Have people looked into mode connectivity for a misalignment landscape?
Owain Evans
@OwainEvans_UK
Jul 22, 2025
Replying to @OwainEvans_UK
We think transmission of traits (liking owls, misalignment) does NOT depend on semantic associations in the data b/c: 1. We do rigorous data filtering 2. Transmission fails if data are presented in-context 3. Transmission fails if student and teacher have different base models
2.3K
Sally Zhu
@SallyHZhu
Oct 23, 2025
Replying to @SallyHZhu
We’re also able to obtain low p-values from n-gram overlap statistics between Bob’s text and Alice’s training documents, but using considerably more (~10M) tokens (e.g., to estimate the total fraction of text on a social media platform from Alice’s model).
5.4K
Sally Zhu
@SallyHZhu
Jul 22, 2025
Replying to @OwainEvans_UK
woah... How far do you think you can push this? (e.g. teacher model generates free-form text --> teacher model only generates numbers) What if the teacher model can only give answers to (unrelated) MC questions, might there still be hidden signals?
4.1K
Sally Zhu
@SallyHZhu
Jul 22, 2025
Replying to @sudarshk_
Yeah I guess it makes sense. But of course there are some texts where this works regardless of the base model (e.g. "I like owls"). I wonder what this transfer threshold is and if it would be interpretable to humans at that point
205
Sally Zhu
@SallyHZhu
Jul 19, 2025
Replying to @SherylHsu02
Crazy stuff 🤯 congrats Sheryl!
299