Instructions/reasoning are now everywhere in retrieval - we want embeddings to do it all! 🚀
But... is it even possible? 🤔
Turns out, it's not possible for single-vector models 😱 theoretically and empirically! To make it obvious we OSS a simple eval SoTA models flop on!
🧵
Orion Weller
643 posts
PhD student @jhuclsp Prev Intern @AIatMeta @GoogleDeepMind, @samaya_ai, @allen_ai Research: LLMs, Search, Agents
- LLMs can use complex instructions - why can’t retrieval models? We build FollowIR, a training/test set of real-world human retrieval instructions. Our FollowIR-7B is the best IR model for instruct-following, even beating @cohere @openai retrievers 🤯 📝 arxiv.org/abs/2403.15246
- Ever wonder how test-time compute would do in retrieval? 🤔 introducing ✨rank1✨ rank1 is distilled from R1 & designed for reranking. rank1 is state-of-the-art at complex reranking tasks in reasoning, instruction-following, and general semantics (often 2x RankLlama 🤯) 🧵
- 🚨 We all complain a lot about reviewers/ACs/SACs in the ML/NLP community. But why not look at the data to see what’s going on? I found some crazy statistics about who is doing/not doing this service in the *CL community. 😱 orionweller.github.io/blog/2024/revi… 🧵
- 🤔 Have you ever wondered how good ModernBERT is compared to decoders like Llama? We made an open-data version of ModernBERT and used the same recipe for encoders and decoders. Turns out, our encoder model beat ModernBERT and our decoder model beats Llama 3.2 / SmolLM2 🤯 🧵
- Can we guide LLMs to quote text from their pre-training data using prefixes like "According To ..", improving grounding and reducing hallucination? We discovered that LLMs do have this capability and can increase or decrease quoting on request 🤯 📝:arxiv.org/abs/2305.13252 1/5
- Introducing ✨Promptriever ✨ the first retriever that can be prompted like an LM with free-form prompts! Our secret: query-level instruction training lets you keep the promptability of the base LM! 🚫 keyword-matching ✅ instruction search 📝 arxiv.org/abs/2409.11136
- Using LLMs for query or document expansion in retrieval (e.g. HyDE and Doc2Query) have scores going 📈 But do these approaches work for all IR models and for different types of distribution shifts? Turns out its actually more 📉 🚨 📝 (arxiv soon): orionweller.github.io/assets/pdf/LLM…
- I'm excited to announce that I have been awarded both the NSF GRFP and the DoD NDSEG fellowships!
- Life update: I'll be joining @jhuclsp to start my PhD in the fall! I'm grateful for the many researchers who have helped mentor me to this point and am looking forward to future collaborations on the East Coast!
- Now accepted to ICLR! Excited to see everyone in Singapore ✈️Introducing ✨Promptriever ✨ the first retriever that can be prompted like an LM with free-form prompts! Our secret: query-level instruction training lets you keep the promptability of the base LM! 🚫 keyword-matching ✅ instruction search 📝 arxiv.org/abs/2409.11136
- The Google search summary vs the actual page
- Excited to share our #NAACL2022 work (and my first @jhuclsp) on pretraining, federated learning (FL), and multilingual data! We provide the first study on the impact of multilingual partitioning on FL algorithms, showing that non-IID settings can cause a large drop in performance
- Search is becoming critical to many NLP tasks now - but how can we defend against malicious actors who attack knowledge sources? We introduce a simple and effective method that uses query augmentation and answer redundancy to provide gains of 5-20% EM! arxiv.org/abs/2212.10002














