Aman Chadha

If you are someone pushing boundaries of LLM interaction, improving prompt engineering strategies, and most importantly building more robust AI application, this paper on agentic context engineering could be a good read for you !! https://lnkd.in/gG97xZGk

39

1 Comment

🚀 Just wrapped up a hands-on project applying Retrieval-Augmented Generation (RAG) to a real-world use case: interpreting LIC’s New Jeevan Shanti policy document using LLMs. The core idea? Combine vector search over regulatory PDFs with GPT-4 to generate grounded, faithful answers to complex policy questions. Key capabilities we built: ✅ LCEL-based RAG architecture using LangChain ✅ Contextual compression retriever for focused document lookup ✅ Modular LLM & embedding selection (OpenAI, HuggingFace, Mistral) ✅ Evaluation using LLM-powered metrics: faithfulness, relevance, correctness 📄 The document used is publicly available here: LIC New Jeevan Shanti Policy PDF 💻 Full code available on GitHub: https://lnkd.in/gS9mEzWA 💡 This architecture brings the best of both worlds — retrieval accuracy and generative flexibility — especially in regulated domains like insurance and financial services. 🔜 Next step: Incorporate Agents that can reason across multiple steps — not just answer a question, but decide which tools to use, when to search, how to validate, and when to stop. That’s where the future of enterprise GenAI lies. Would love to hear how others are pushing RAG, agents, and LLMs in production! #GenAI #LangChain #LLM #RAG #AIForOps #InsuranceTech #LLMEvaluation #LangChainAgents

61

1 Comment

Introducing Generative Adversarial Distillation (GAD): a novel GAN-style formulation and framework that facilitates both on-policy and black-box distillation of large language models (LLMs). GAD is the first technique to enable block-box on-policy distillation from proprietary teachers where internal logits or parameters are inaccessible, or distillation between teacher and student LLMs with incompatible vocabularies. GAD expands our prior work on white-box on-policy distillation (i.e., MiniLLM), pioneering block-box on-policy distillation for LLM training. Specifically, GAD frames the student LLM as a generator and trains a discriminator to distinguish its responses from the teacher LLM’s, creating a minimax game. The discriminator acts as an on-policy reward model that co-evolves with the student, providing stable, adaptive feedback. Experimental results show that GAD consistently surpasses the commonly used sequence-level knowledge distillation.  In particular, Qwen2.5-14B-Instruct (student) trained with GAD becomes comparable to its teacher, GPT-5-Chat, on the LMSYS-Chat automatic evaluation. The results establish GAD as a promising and effective paradigm for black-box LLM distillation. Our team has been conducting fundamental research in knowledge distillation with wide adoptions across the industry. -         MiniLM: We introduced multi-head attention distillation, establishing the most effective distillation method for BERT-style models. The open-source MiniLM models (e.g., 6x384) have become the most widely utilized small encoder models on the Hugging Face. -         MiniLLM: Our proposed Reverse KLD is recognized as one of the most effective, de facto on-policy distillation approaches for modern LLM training, which has been widely used by Thinking Machines, Gemma, and many other teams and models. -         BitDistill: We proposed BitNet Distillation to finetune off-the-shelf full-precision LLMs (e.g., Qwen) into 1.58-bit precision (ternary weights {-1, 0, 1}), achieving performance parity with the full-precision counterparts on specific downstream tasks. -         GAD: The development of Generative Adversarial Distillation (GAD) now allows for black-box on-policy distillation, overcoming two major prior limitations: (1) Distillation from proprietary teachers where internal logits or parameters are inaccessible; (2) Distillation between teacher and student LLMs with incompatible vocabularies. https://lnkd.in/gMaP2c7w

125

2 Comments

What we often chalk up to “expected randomness” in LLMs turns out to be partly a byproduct of infrastructure choices—batch sizes, request load, how attention kernels are implemented. The idea of batch invariance stood out. If your kernel changes reduction order when batch size or KV cache state changes, then your “same prompt” no longer means “same output.” For applications that depend on reproducibility, audit trails, or consistency (e.g. RL training, regulated domains), that’s not just academic. What do you think: for your stack, how big is the gap between “probabilistic slack acceptable” vs “exact repeatability required”? Are we shifting toward expecting determinism even in deployed systems? https://lnkd.in/gbKTt9-g #AIEngineering #Reproducibility #LLMInference

12

💥 ICML-25 Paper Alert! Introducing the JETTS benchmark, Judge Evaluation for Test-Time Scaling ⏲️ LLM-judges specialize in evaluating responses generated by other LLMs. Test-time scaling requires multiple model responses to be evaluated for the best one. So, can LLM-judges help test-time scaling? Check this thread to find the answer and some interesting analysis👇 Repo: https://bit.ly/3YteE6Z X: https://lnkd.in/gXvQsd4C Paper: https://bit.ly/4jtRGVM

105

Another interesting new RL perspective. They propose Reinforced Attention Learning (RAL) to address the difficulty of applying RL techniques that work well for LLMs but tend to struggle more with multimodal LLMs, although I think the idea is more general than that. They argue that this approach is more robust to reward hacking and less prone to diversity collapse. In practice, they add to the standard RL objective an additional term that looks like: E[ A_t · D(p_t^θ || p_t^old) ] Here, A_t is the advantage, while p_t is the attention distribution policy at generation step t, defined from the final attention layer (averaged across heads): the current one (θ) and the one that generated the batch (old). p_t^θ(i) is simply the attention weight from the generated token at position t to a previous token at position i < t, normalized by the sum of attention weights before t. D is the Jensen–Shannon divergence (JSD), a symmetric and bounded divergence defined via the mixture of the two distributions, and less aggressive than Kullback-Leibler divergence (KL). JSD(P || Q) = 1/2 · KL(P || M) + 1/2 · KL(Q || M) With M = 1/2 · (P + Q). What is happening here is that we incentivize attention patterns associated with high advantage and discourage those associated with low advantage. As the authors put it, when A_t < 0 the model is explicitly pushed to explore new reasoning paths. The full objective becomes: L_RL + λ · L_AttnRL. They then introduce an extension that combines RAL with on-policy knowledge distillation and attention distillation. The idea is similar to on-policy distillation: you generate with the student (the current network), compute the teacher outputs on the same generation, and align them using a reverse KL on the logits. In addition, they also align the attention distributions between teacher and student, again using JSD. Unlike RAL, this attention distillation term does not use an advantage, it is pure structural alignment. Takeaway: RL and on-policy distillation do not have to operate only at the token level. Extending the training signal to internal structures like attention can be quite effective, and I expect this direction to be explored much more in the future. Thanks Petru-Daniel (Dan) T. for suggesting the paper!

84

1 Comment

India is not behind on talent. It is behind on risk appetite. Building LLMs or foundational AI infra is not about capability. It is about capital. Not just any capital, but capital that is comfortable with long timelines, unclear outcomes, and massive burn. Most of India’s funding still chases near-term traction. SaaS, fintech, consumer. Sectors where returns are easier to model. But AI infra does not play by those rules. It demands belief before proof. The Middle East launching a sovereign fund to accelerate AI through public private bets is not just money. That is intent. India will not build foundational AI by hoping startups will figure it out alone. It needs conviction at the system level. And a willingness to back uncertainty at scale.

56

10 Comments

Fascinating deep-dive on LLM nondeterminism: Even at temperature 0, models aren't reproducible - not due to GPU concurrency as commonly believed, but because of batch invariance issues. Varying server loads = different outputs for identical requests. This hits home at Altimate AI. Building AI teammates for data teams, we've implemented numerous guardrails for reproducibility - essential when AI analyzes SQL queries or suggests optimizations. Yet despite our best efforts, nondeterminism still creeps in at times. The elegance of their solution - making kernels batch-invariant - shows that what many of us accept as "inevitable randomness in AI" might actually be solvable. This isn't just academic; it's about making AI tools as reliable as the data pipelines they're meant to improve. For those building production AI systems: we're all fighting this battle together. Reproducibility remains one of the hardest challenges in moving from impressive demos to trustworthy production systems that data teams can actually depend on. This research gives hope that true determinism in LLM inference is achievable. https://lnkd.in/gvKRty6Y #llm #DataEngineering #ArtificialIntelligence #MLOps

26

1 Comment

🧠 Model Context Protocol (MCP): The Future of Structured AI Interactions In the fast-evolving world of Generative AI, building truly intelligent systems means more than just prompting a large language model — it requires structured, contextual reasoning. That’s where Model Context Protocol (MCP) comes in. In my latest Medium article, I break down: 🔹 What MCP is and why it’s critical for building multi-agent and tool-augmented LLM systems 🔹 How context is passed between agents, memory, retrievers, and tools 🔹 Real-world use cases like: - AI Telemedicine Assistants - RAG-powered Enterprise Bots - Financial Portfolio Advisors 🔹 With working Python examples to help you implement it yourself This article is written for AI engineers, builders, and anyone exploring advanced GenAI workflows. Would love to hear your thoughts or any feedback from others building in this space 🚀 #GenerativeAI #AI #LLM #LangChain #OpenAI #OCI #MCP #AgenticAI #PromptEngineering #RAG

10

2 Comments