Ana-Maria Istrate

Interactive feedback changes how code LLMs perform and rank, a new study finds. Led by senior author Valerie Chen, a Carnegie Mellon PhD student and recent visiting researcher at CDS, the project also included Courant PhD student Jane Pan, CDS PhD student Jacob Pfau, and CDS Assistant Professor He He. The team simulated developer interactions to replace static coding benchmarks like HumanEval and MBPP. When models received human-style feedback—code edits, single-sentence prompts, or paragraph-length instructions—they sometimes improved by up to 9.3 percentage points in accuracy and moved as many as four spots in ranking. “Testing on new problems the models had never seen made feedback even more powerful,” said Chen. Read more: https://lnkd.in/eAPTP5NR

5

A single punctuation mark can be enough to reshuffle AI leaderboards. New research from CDS PhD student Jingtong Su and CDS PhD alum Jianyu Zhang shows that changing the delimiter in few-shot prompts can swing benchmark scores by up to 23 percent on MMLU, calling into question how robust model rankings really are. The study finds that this instability persists as models scale, with even large, closed-source systems showing dramatic performance spreads depending only on formatting choices. “One can manipulate model rankings to put any model in the lead by only modifying the single character separating examples,” the authors write. https://lnkd.in/eRB6jmzU

4

🚀 What if RL post-training was easier to experiment with? In Episode 30 of LLMs on the Run, UC Berkeley Professor Joseph Gonzalez introduces NovaSky, a new framework designed to make reinforcement learning (and post-training in general) faster to iterate, easier to compare, and more useful across diverse settings. Joey explains: ✅ Why post-training methods need to generalize across tasks—not just benchmarks ✅ How NovaSky helps explore tradeoffs across RL, SFT, and hybrid techniques ✅ What the Sky RL framework enables for faster, deeper model improvement NovaSky aims to give the community better tools for experimenting with the full spectrum of post-training strategies. Brought to you by RunLLM: https://lnkd.in/gZ3-amPP #LLMsOnTheRun #NovaSky #RLHF #ReinforcementLearning #PostTraining #LLMs #SkyRL #JoeyGonzalez #AIResearch #RunLLM #SFT #AIInfra

4

Excellent work by the Diversity News and AI Lab! The DNA lab is one of four labs hosted by RMIT's Digital Ethnography Research Centre, each tackling key contemporary issues in digital culture, policy and use. https://lnkd.in/gwrbq7hP

4

In 5 years, AI could automate software tasks that take a human a month to complete. Large Language Models (LLMs) are improving, but practically measuring their progress has always been challenging. So, researchers at Berkeley’s METR adopted a different approach. They wanted their measurement practical and repeatable, tied to what programmers do and how long it takes them to do it. Then they used that time as a benchmark for LLMs. The goal was to answer two specific questions: • What tasks can AI models complete with a 50% success rate? • How complex are these tasks becoming as models improve? 50% is not a lot, sure. But that’s not what matters here. The results highlighted the speed at which progress is affecting what tasks LLMs can complete. They noticed that every 7 months, LLMs double the length of tasks they can complete at a 50% success rate. Models are getting better. Improvement is expected, but maybe not so quickly? Projecting this forward to the end of the decade is a bit of a gamble. Things may not keep improving at the same rate. Energy supply will have to keep up with the growing demand. And this research is still limited to software-specific tasks. BUT Humans aren’t built to embrace change so quickly and so often. With AI, it feels like we are still stuck thinking of what we can do faster instead of what could be possible in just a few months. #ai #llms #aiprogress  

6

5 Comments

The Autonomy Stack Showdown: Models vs. Infrastructure! Next week at CMU Lab to Market in San Francisco, we're bringing together leaders who’ve shaped autonomy across research, company-building, and capital: Martial Hebert (Moderator), Dean, CMU School of Computer Science Arvind Gupta, Partner, Mayfield Ryan Oksenhorn, Co-Founder, Zipline Nancy Pollard, Professor, CMU Robotics Institute & SCS Sebastian Scherer, Co-Founder, FieldAI & CMU Professor of Robotics Together, they’ll examine where autonomy systems truly bottleneck today, how technical tradeoffs surface at scale, and what it takes to move from frontier advances to reliable, real-world systems. ✨ San Francisco | Feb 11, 2026 | https://lnkd.in/eyYatuJd #CMULabToMarket #CarnegieMellon Carnegie Mellon University's College of Engineering Carnegie Mellon University School of Computer Science Jesse Thill Meredith Meyer Grelli Namrata Banerjee Matthew Katsaros, M.A. Jenny Belardi Anita Jesionowski Carnegie Mellon University Alumni Association Carnegie Mellon University Robotics Institute

9

The #DOECSGF builds community that fosters important scientific collaborations, such as the new Open Molecules 2025 dataset. Four alumni were critical to today's groundbreaking release: Berkeley Lab's Samuel Blau, Aditi Krishnapriyan (also University of California, Berkeley), and Santiago V.; and AI at Meta's Zachary Ulissi. https://lnkd.in/gnGuY5ui #computationalchemistry #DFT #machinelearning #HPC Harvard University, Stanford University, UCLA, Massachusetts Institute of Technology, Berkeley Lab Computing Sciences

105

2 Comments

By building a taxonomy for compound AI systems, researchers are showing that when agents are optimized using structured evaluation frameworks, downstream task accuracy improves. Better evaluation → better agents. Featuring Melissa Pan and Mert Cemri of University of California, Berkeley / CSGE Berkeley Interviewed by Jessica Dai and SAIL Media Thank you to ‪Lambda‬ for making these interviews possible! 👉 Subscribe to our calendar for priority invites to upcoming events: lu.ma/twill #AI #ArtificialIntelligence #MultiAgentSystems #LLM #LLMAgents #CompoundAI #MachineLearning #AIResearch #BerkeleyAI #CSGEBerkeley #TechEvents #StartupCommunity #SiliconValley #FutureOfAI #AI #ArtificialIntelligence #MultiAgentSystems #LLM #LLMAgents #CompoundAI #MachineLearning #AIResearch #BerkeleyAI #CSGEBerkeley #TechEvents #StartupCommunity

34

2 Comments

JuliaC introduces a new way to build executables and shared libraries directly from Julia code. In this session, Jeff Bezanson and Kristoffer Carlsson explain the new compiler interface and show how JuliaC enables compact binaries and cross-language deployment. https://lnkd.in/g7D76Wwt #JuliaLang #JuliaProgramming #JuliaC #ScientificComputing #TechnicalComputing #HighPerformanceComputing #EmbeddedSystems #SystemsProgramming

7

Stanford Center for Research on Foundational Model's Marin project has released the first fully open model in JAX. It’s an 'open lab' sharing the entire research process - including code, data, and logs, to enable reproducibility and further innovation. Check out the project: https://goo.gle/44AMeLY

41

2 Comments

New paper to be presented at the International Conference on Learning Representations (ICLR): Gradient-Based Diversity Optimization with Differentiable Top-K Objective, by Tianyi Zhou, Sebastian Dalleiger, Ece Calikus & Aristides Gionis. Available here: https://lnkd.in/dmE7DXUH

8

1 Comment

People of ACM: Sewon Min, Assistant Professor at UC Berkeley and Research Scientist at the Allen Institute for AI 🌟 Her groundbreaking work on non-parametric large language models is reshaping how we think about data, attribution, and reliability in AI systems. Her doctoral research earned an Honorable Mention for the ACM Doctoral Dissertation Award — an achievement recognizing deep impact on how large language models (LLMs) learn and reason. 💡 From addressing hallucinations in AI to exploring legally responsible architectures, Sewon’s research pushes the boundaries of factuality, control, and creativity in LLMs. 📖 Read her full interview and insights on the future of language models: https://bit.ly/4lcRGun #ACMWomen #PeopleOfACM #LLMs #AIResearch #WomenInSTEM #ResponsibleAI #DataAttribution #ACMFellows #BerkeleyAI #AIethics #SewonMin University of California, Berkeley Allen Institute ACM, Association for Computing Machinery

10

Announcing Leveling Up with Generative AI, a new tutorial series designed for graduate students, postdocs, and researchers who want practical, hands-on ways to use generative AI in their work!   This hands-on workshop series builds practical skills with LLMs and foundation models. Participants will learn how to apply these tools to real academic workflows, including research assistance, data analysis automation, complex problem-solving, and responsible, technically grounded AI use.   Christopher Stubbs, Francesca Dominici, Mauricio Tec, Claudio Battiloro, Michelle Audirac, Bret Nestor, and Konstantina Yaneva   📅 In person workshops begin February 20 📍Northwest Building in Room B103 🔗 Register and learn more here:  https://lnkd.in/eRa6mpsC    #GenerativeAI #ResearchTools #HarvardDataScience

15