Weijia Shi (@WeijiaShi2) / X

Weijia Shi

1,245 posts

Weijia Shi

@WeijiaShi2

PhD student @uwnlp | Prev @allen_ai @MetaAI @CS_UCLA

Seattle, WA

weijiashi.notion.site/swj

Joined August 2019

Pinned
Weijia Shi
@WeijiaShi2
Jul 9, 2025
Can data owners & LM developers collaborate to build a strong shared model while each retaining data control? Introducing FlexOlmo💪, a mixture-of-experts LM enabling: • Flexible training on your local data without sharing it • Flexible inference to opt in/out your data
00:00
94K
Weijia Shi
@WeijiaShi2
Dec 20, 2024
Introducing 𝐋𝐥𝐚𝐦𝐚𝐅𝐮𝐬𝐢𝐨𝐧: empowering Llama 🦙 with diffusion 🎨 to understand and generate text and images in arbitrary sequences. ✨ Building upon Transfusion, our recipe fully preserves Llama’s language performance while unlocking its multimodal understanding and
161K
Weijia Shi
@WeijiaShi2
Oct 26, 2023
Ever wondered which data black-box LLMs like GPT are pretrained on? 🤔 We build a benchmark WikiMIA and develop Min-K% Prob 🕵️, a method for detecting undisclosed pretraining data from LLMs (relying solely on output probs). Check out our project: swj0419.github.io/detect-pretrai… [1/n]
175K
Weijia Shi
@WeijiaShi2
Oct 17, 2023
Introduce In-Context Pretraining🖇️: train LMs on contexts of related documents. Improving 7B LM by simply reordering pretrain docs 📈In-context learning +8% 📈Faithful +16% 📈Reading comprehension +15% 📈Retrieval augmentation +9% 📈Long-context reason +5% arxiv.org/abs/2301.12652
113K
Weijia Shi
@WeijiaShi2
Dec 20, 2022
🙋‍♀️How to present the same text in diff. tasks/domains as diff. embeddings W/O training? We introduce Instructor👨‍🏫, an instruction-finetuned embedder that can generate text embeddings tailored to any task given the task instruction➡️sota on 7⃣0⃣tasks👇! instructor-embedding.github.io
100K
Weijia Shi
@WeijiaShi2
Sep 30, 2024
Visual Chain-of-Thought with ✏️Sketchpad Happy to share ✏️Visual Sketchpad accepted to #NeurIPS2024. Sketchpad thinks🤔by creating visual reasoning chains for multimodal LMs, enhancing GPT-4o's reasoning on math and vision tasks We’ve open-sourced code: visualsketchpad.github.io
109K
Weijia Shi
@WeijiaShi2
Jan 17, 2024
Happy to share In-Context Pretraining 🖇️ is accepted as an #ICLR2024 spotlight. We study how to pretrain LLMs with improved context understanding ability paper📄: arxiv.org/pdf/2310.10638… code: github.com/swj0419/in-con…
65K
Weijia Shi
@WeijiaShi2
Mar 28, 2024
When augmented with retrieval, LMs sometimes overlook retrieved docs and hallucinate 🤖💭 To make LMs trust evidence more and hallucinate less, we introduce Context-Aware Decoding: a decoding algorithm improving LM's focus on input contexts 📖 arxiv.org/pdf/2305.14739… #NAACL2024
44K
Weijia Shi
@WeijiaShi2
Jul 15, 2024
Can 𝐦𝐚𝐜𝐡𝐢𝐧𝐞 𝐮𝐧𝐥𝐞𝐚𝐫𝐧𝐢𝐧𝐠 make language models forget their training data? We shows Yes but at the cost of privacy and utility. Current unlearning scales poorly with the size of the data to be forgotten and can’t handle sequential unlearning requests. 🔗:
73K
Weijia Shi
@WeijiaShi2
Sep 18, 2025
𝐋𝐥𝐚𝐦𝐚𝐅𝐮𝐬𝐢𝐨𝐧 is also accepted to #NeurIPS2025 🎉
Weijia Shi
@WeijiaShi2
Dec 20, 2024
Introducing 𝐋𝐥𝐚𝐦𝐚𝐅𝐮𝐬𝐢𝐨𝐧: empowering Llama 🦙 with diffusion 🎨 to understand and generate text and images in arbitrary sequences. ✨ Building upon Transfusion, our recipe fully preserves Llama’s language performance while unlocking its multimodal understanding and
28K
Weijia Shi
@WeijiaShi2
Jun 14, 2024
Augmenting GPT-4o with Visual Sketchpad ✏️ We introduce Sketchpad agent, a framework that equips multimodal LLMs with a visual canvas and drawing tools 🎨 . Improving GPT-4o's performance in vision and math tasks 📈 🔗: visualsketchpad.github.io
01:25
Yushi Hu
@huyushi98
Jun 14, 2024
Humans draw to facilitate reasoning and communication. Why not let LLMs do so? 🚀We introduce✏️Sketchpad, which gives multimodal LLMs a sketchpad to draw and facilitate reasoning! arxiv.org/abs/2406.09403 Sketchpad gives GPT-4o great boosts on many vision and math tasks 📈 The
50K
Weijia Shi
@WeijiaShi2
Mar 15, 2024
Happy to share REPLUG🔌 is accepted to #NAACL2024 We introduce a retrieval-augmented LM framework that combines a frozen LM with a frozen/tunable retriever. Improving GPT-3 in language modeling & downstream tasks by prepending retrieved docs to LM inputs. 📄:
22K
Weijia Shi
@WeijiaShi2
Apr 16, 2025
Our previous work showed that 𝐜𝐫𝐞𝐚𝐭𝐢𝐧𝐠 𝐯𝐢𝐬𝐮𝐚𝐥 𝐜𝐡𝐚𝐢𝐧‑𝐨𝐟‑𝐭𝐡𝐨𝐮𝐠𝐡𝐭𝐬 𝐯𝐢𝐚 𝐭𝐨𝐨𝐥 𝐮𝐬𝐞 significantly boosts GPT‑4o’s visual reasoning performance. Excited to see this idea incorporated into OpenAI’s o3 and o4‑mini models (openai.com/index/thinking…).
Weijia Shi
@WeijiaShi2
Sep 30, 2024
Visual Chain-of-Thought with ✏️Sketchpad Happy to share ✏️Visual Sketchpad accepted to #NeurIPS2024. Sketchpad thinks🤔by creating visual reasoning chains for multimodal LMs, enhancing GPT-4o's reasoning on math and vision tasks We’ve open-sourced code: visualsketchpad.github.io
Thinking with images
From openai.com
27K
Weijia Shi
@WeijiaShi2
Nov 22, 2023
Interested in finding out if specific book excerpts have been pre-trained by OpenAI's text-davinci-003 model? 🔍 Check out our demo:
Weijia Shi
@WeijiaShi2
Oct 26, 2023
Ever wondered which data black-box LLMs like GPT are pretrained on? 🤔 We build a benchmark WikiMIA and develop Min-K% Prob 🕵️, a method for detecting undisclosed pretraining data from LLMs (relying solely on output probs). Check out our project: swj0419.github.io/detect-pretrai… [1/n]
Detect Pretraining Data - a Hugging Face Space by swj0419
From huggingface.co
41K