Khush Gupta

A new RL algorithm that fixes a hidden flaw in PPO The authors propose CE-GPPO (“Coordinating Entropy via Gradient-Preserving Policy Optimization”), a variant of PPO that restores gradient contributions from clipped actions in the policy update. They argue that traditional clipping discards useful gradient signals from low-probability tokens, which play an important role in controlling the agent’s entropy during training. By bounding those gradients in a controlled way, CE-GPPO maintains exploration–exploitation balance more stably than prior methods. They provide a theoretical analysis showing that CE-GPPO mitigates entropy instability, and empirically test it on mathematical reasoning benchmarks. Their results indicate consistent improvement over strong baselines across different model scales, demonstrating that preserving clipped gradients can lead to better performance in reinforcement learning for reasoning tasks. https://lnkd.in/gYZsJ-8f

1

Missed Pytorch Conf? Don't worry, you can still catch llm-d contributors Maroon Ayoub (IBM Research) & Cong Liu (Google) presenting "Disaggregated Serving of PyTorch LLMs with llm-d & Kubernetes." They dive deep into llm-d showing how to decouple and optimize inference by splitting prefill (compute-heavy) and decode (latency-sensitive) tasks. The talk covers: ⚡️ Optimizing GPU utilization & QoS alignment. 🌐 Gateway API Inference Extension (GIE) for routing based on load, cache, and affinity. 📊 Real-world benchmarks on heterogeneous K8s hardware. See how the community is pushing open-source inference forward! 👇 🔗 https://lnkd.in/enc-MzHy #llmd #PyTorch #OpenSource #Kubernetes #vLLM #AICommunity

27

How can we effectively use LLMs for relevance modeling? In a recent paper from Pinterest, they leveraged a LLM as backbone in a a cross-encoder architecture to predict the relevance of a Pin to a search query. I tried similar setup in recommendation for the movie rating prediction task. 🔍 In this post, I explore: - The distinction between bi-encoders vs. cross-encoders. - Building an LLM-based cross-encoder for recommendation. - Fine-tuning the LLM using actual rating data. - Role of data enrichment, prompting, and problem formulation. If you’re interested in LLMs for relevance modeling or pushing retrieval models toward reasoning, check it out: 👉 LLM-based Cross-Encoder for Recommendation: https://lnkd.in/gAuGJbwg Would love to hear if others are experimenting in this space too. #LLMs #RecommendationSystems #CrossEncoders #RelevanceModeling #MachineLearning #AIResearch

112

8 Comments

LLMs are great at answering questions. But can they actually reason through multi-step tasks, adapt to feedback, and coordinate with other agents? A new survey from researchers at UIUC, Meta, Amazon, and Google DeepMind tackles exactly this. "Agentic Reasoning for Large Language Models" proposes a structured framework that moves beyond chain-of-thought prompting toward treating LLMs as autonomous agents. The key insight: the authors organize agentic reasoning into three layers: 1. Foundational Agentic Reasoning — core single-agent capabilities like planning, tool use, and search in stable environments. Think of this as the basic "operating system" an agent needs. 2. Self-Evolving Agentic Reasoning — how agents refine their capabilities through feedback, memory, and adaptation. This is where things get interesting: agents that actually learn from experience rather than starting fresh each time. 3. Collective Multi-Agent Reasoning — extending intelligence to collaborative settings with coordination, knowledge sharing, and shared goals across multiple agents. What makes this framework especially useful is how it distinguishes between two optimization modes: in-context reasoning (scaling test-time interaction through structured orchestration) and post-training reasoning (optimizing via reinforcement learning and supervised fine-tuning). The survey also covers real-world applications across science, robotics, healthcare, autonomous research, and mathematics — showing where these ideas are already being applied. If you're building or evaluating AI agent systems, this is a solid reference to bookmark. 📄 Paper: https://lnkd.in/gd4e--FW 💻 GitHub: https://lnkd.in/gB8z_s67 #AgenticAI #LLMs #AIResearch #MachineLearning #DeepLearning

26

A smaller AI model just beat a larger one by 114%. Not by training longer. Not by adding more data. But by refusing to read everything at once. That result comes from a recent paper by Alex Zhang and Omar Khattab on Recursive Language Models (RLMs) — and it challenges one of the most common assumptions in AI today. For years, we believed progress meant one thing: Bigger context windows = smarter models. So we kept increasing them. 128k tokens. 256k tokens. And beyond. Yet accuracy quietly went in the opposite direction. As context length grew, models started missing details, confusing facts, and reasoning poorly. This degradation is now widely known as context rot. More context, less intelligence. What’s actually going wrong Long-context models are asked to process everything at once. On the OOLONG benchmark, which tests reasoning over extremely long inputs: • Performance drops sharply as context increases • Even top-tier models struggle past ~100k tokens • Smaller models degrade even faster The model “sees” more, but understands less. The RLM idea RLMs flip the problem. Instead of stuffing the entire context into a prompt, the data lives outside the model. The model interacts with it selectively — searching, inspecting, and delegating work to smaller recursive calls. The model doesn’t read the book. It uses the book. Why this works (with real numbers) On OOLONG with ~132k tokens: • An RLM using a smaller model outperformed a larger baseline by 34+ points • That’s a ~114% relative improvement on trec_coarse • Average cost per query was lower, despite multiple calls When the context doubled to ~263k tokens: • RLMs still led by 15+ points • Standard models continued to degrade • Context rot was dramatically reduced Scaling to absurd sizes In BrowseComp-Plus, the task expanded to 1,000 documents — potentially 10M+ tokens. • RLM performance stayed perfect • Baselines and retrieval agents degraded steadily • No vector database or retriever was used The model effectively built its own search and indexing strategy on the fly. The bigger takeaway This isn’t just a performance trick. It’s a mindset shift: • From “read everything” • To “navigate information” If this direction holds, the future of AI won’t be defined by how much context a model can consume — but by how intelligently it decides what to look at. Source: https://lnkd.in/gDBPKub3 #AI #LLMs #GenAI #RAG #MachineLearning #AIResearch #LongContext #SystemsThinking #FutureOfAI

3

It had been on my list for a while to read the SID-1 technical report after seeing a post about how using OpenAI-style messages in RL can be surprisingly dangerous in multi-turn settings with many tool calls. The first insight is about the messages abstraction. Converting a raw token stream into messages and then back into tokens is lossy because it changes how the exact byte sequence is tokenized. A concrete example is when the model generates a sequence of bytes that the tokenizer splits into two very common tokens with normal probabilities. After parsing and reformatting through the chat template, those same bytes can be re-serialized in a way that the tokenizer now maps to a single, very rare token that the model almost never produces. The reward is computed on the original tokens the model actually generated, but the log-probs used for the update correspond to this new token that was never sampled. Since this token has an extremely low probability, its log-prob has a large magnitude, and in policy gradient this translates into a disproportionately large gradient. That single artificial token can end up dominating the update, creating a feedback loop that gradually destabilizes training and eventually leads to collapse. There is a second, distinct effect happening at the same time. Malformed tool calls or slightly wrong formatting can get “repaired” by the parser and chat template, so bad rollouts end up looking syntactically valid before they ever reach the trainer. The environment sees a correct tool call, the reward is good, and the trainer also sees a correct sequence. The model never receives signal that it actually produced something malformed. Stability is preserved, but the model does not learn tool correctness because the fixing layer steals the learning signal. Their fix is conceptually simple. Use a strict Tokens-In Tokens-Out pipeline, where the trainer sees exactly the token sequence the model generated. No parsing, no chat templates, no message abstraction in between. Another very interesting part of the report is their analysis of the “length debiasing” proposed in Dr. GRPO and similar works, where the per-token advantage is no longer normalized by the rollout length. This assumes that rollout length and quality are unrelated. In tool-use and reasoning-heavy environments this is not true. Bad rollouts tend to be longer than good ones. In that case, removing the length bias makes the average per-token advantage negative. Over long runs, this slowly pushes down the logits of all sampled tokens while pushing up the logits of tokens the model never uses, until the model starts emitting garbage or out-of-vocabulary tokens and collapses.

7

2 Comments

I don't think much of OpenAI's recent "paper" on Hallucinations : if production LLMs really did hallicinate because of the RLHF, why do base models pre RLHF do the same ? If you let the LLM answer IDK when usually it does know the answer, its gives it the ability to play dumb and pass safety testing easily, which in my mind doesn't lead to more trustworthy AI ? I believe its because of the forward projection of the token prediction system (transformer architecture) and you are best to mitigate via a neurosymbolic approach. I think this is much better research, and the GitHub code is here : https://lnkd.in/eUKDQsTG.

2

There’s a new RL algorithm in town… and it’s already shaking up the LLM research community. 🤯 Group Sequence Policy Optimization (GSPO) was created by the research team at Qwen and is purpose built for operating at the sequence level rather than the token level. This means it evaluates, clips, and optimizes entire response sequences by their overall likelihood. The important upshot is that it directly aligns the optimization process with how rewards are actually assigned in real world scenarios! Some key reasons to be excited! 1. Better stability. GSPO improves training stability and helps avoids model collapse that can happen with algorithms like GRPO. 2. Scalable RL for mixture of experts (MoE) models. GSPO inherently solves stability challenges teams face when scaling RL for MoE architectures. Large scale RL becomes more practically feasible! 3. Simplifies Infra. Sequence-level optimization is inherently less sensitive to small numerical differences between training and inference systems compared to token-level methods, which should lead to more robust RL pipelines! We can already see signs that this has paid off in big ways for the open-source community. Qwen3 models are doing really well on benchmarks, boasting competitive performance with top models like Gemini-2.5 Pro on GPQA, AIME25, and LiveCodeBench! https://lnkd.in/gCZMQvUj P.S. 👀 If you want to run RL on your own Qwen3 model with GRPO/GSPO, keep an eye on our open-source post-building repo 🧡 : https://lnkd.in/gXZ32qkm #AI #LLM #ReinforcementLearning

84

10 Comments

A few weeks ago, we wrote about why LLMs hallucinate. This post is the sequel. After observing thoroughly, one pattern kept showing up across client systems: The model was forgetting. Context windows overflow. Tokens decay. Critical constraints fall out of memory. The model keeps answering anyway, now confidently wrong. Million-token contexts didn’t magically fix this. They just made bad context management more expensive. We broke this down in detail here: https://lnkd.in/gwDjV_du If you’re building agentic systems, long-running workflows, or large RAG pipelines, this will save you debugging hours and cloud bills. #AI #Hallucinations #Context #LLMs

3