Meng Cao

Get faster and smarter MOE inference straight out of the box. 👇Deep dive on scaling expert parallelism with TensorRT-LLM. LLMs with MOE promise higher model capacity without linearly increasing compute costs - but they introduce new challenges -- more conditional computation, dynamic routing, and non-uniform GPU utilization -- solved by TensorRT-LLM. ✨ New in TensorRT-LLM has native support for expert parallelism—designed for fast, efficient inference with MoE models like Mixtral (Mistral AI) and DeepSeek (DeepSeek AI). This gives you: ✅ Dynamic expert routing: Automatically route tokens to the top-k experts with minimal overhead. ✅ Efficient expert scheduling: Balance expert loads across GPUs using smart sharding and token bucketization. ✅ Memory-aware execution: Maximize hardware utilization while respecting memory budgets. ✅ Drop-in support: Use @HuggingFace models with minimal code changes via TensorRT-LLM's #Python API. 🧠 How it works: MoE models activate only a subset of "experts" for each token. This dynamic nature is powerful—but hard to optimize. It’s all done under the hood using custom #CUDA kernels and NCCL-based communication primitives—giving you low latency, high throughput, and better GPU scaling. ✨ TensorRT-LLM handles: ✅Token-expert mapping using the gating network. ✅Token sorting to batch same-expert tokens together. ✅Expert parallel execution across GPUs. ✅Merging outputs for final predictions. 🛠️ Developer workflow - here is the code to get started. # Clone the repo git clone https://lnkd.in/g-GiDX23 # Use included examples to load and run a Mixtral model cd TensorRT-LLM/examples/mixtral From there, the Python API lets you load the model, convert it with TensorRT, and run expert parallel inference—all with a few lines of code. Results? 📈 Performance at scale. Tests show up to 2.3x faster inference throughput compared to standard tensor parallelism when using 8 GPUs and top-2 experts per token. Even better—TensorRT-LLM keeps efficiency high across increasing batch sizes. Want to see it in action or contribute? 👉 Read the full tech blog: https://lnkd.in/g_7YV3vV 👉 Explore the code on GitHub: https://lnkd.in/gNjQ5W2U 👉 Follow updates in the TensorRT-LLM repo: https://lnkd.in/gqSHYQ4u Share your experiences with us.

78

3 Comments

🧪 A key challenge to solving #InverseProblems, especially in science and engineering settings, is limited data and compute for simulations. 🥧 The #ICLR2025 work of Apivich Hemachandra, Gregory Lau, See-Kiong Ng, and Bryan Kian Hsiang Low, PIED, is an #ExperimentalDesign framework for #PDE inverse problems that efficiently optimizes for the best observation inputs. 📌 PIED utilizes #PINNs to perform forward simulation and to solve the inverse problems, which allows for higher efficiency over numerical simulators and allow for meshless and differentiable simulations. 🤖 Using PINNs along with differentiable observation selection criteria, PIED selects the optimal design parameters for one-shot deployment, while exploitation parallel computation and gradient-based optimization methods. 📈 These benefits allow PIED to outperform existing ED benchmarks in IPs for both finite-dimensional and function-values inverse parameters, as demonstrated in various settings including on real experimental data. Check out our #ICLR2025 Poster #28 on 26 Apr in Poster Session 6 Sat 3pm Hall 3 + Hall 2B. Paper: https://lnkd.in/gDxav2Dn GitHub: https://lnkd.in/gZXDkA5H

31

1 Comment

The most interesting paper I read last week, showing promising work on optical generative models that offload parts of the compute from electrons to photons. UCLA researchers demonstrated generating images by physical light propagation rather than digital computation. The approach maps diffusion model concepts to free-space optics: a shallow digital encoder converts random noise to phase patterns, then a trained diffractive surface processes light to generate images with the optical computation happening in under a nanosecond (overall speed is limited by the SLM refresh rate). This cross-pollination between fields (GenAI → Optics → GenAI) strengthens the entire research ecosystem. Moving from bits to light as the medium opens new possibilities for energy-efficient and significantly faster inference. Paper: https://lnkd.in/gC4By9Vv

39

1 Comment

NVIDIA TensorRT LLM provides a high-level Python LLM API, with its PyTorch-native architecture enabling developers to experiment with the runtime or extend functionality. Learn how these latest TensorRT-LLM optimizations boost reasoning inference performance in our recent blog post. 🖇️ https://lnkd.in/gx52zwTA #PyTorch #OpenSourceAI #AI #Inference #Innovation

94

2 Comments

Coactive AI runs real-time inference and bursty batch multimodal data processing. Their existing setup for image and video processing led to underutilized GPUs and rising costs during peak demand. By moving to Ray clusters on Anyscale with fractional GPU allocation and autoscaling, Coactive could scale up and down more precisely with workload demand. As a result, Coactive achieved 1-day dev-to-prod deployments, 4× lower per-image processing cost, and 75% lower GPU compute spend, all while supporting production-scale multimodal workloads. https://lnkd.in/gkZvs78h

24

Introducing Generative Adversarial Distillation (GAD): a novel GAN-style formulation and framework that facilitates both on-policy and black-box distillation of large language models (LLMs). GAD is the first technique to enable block-box on-policy distillation from proprietary teachers where internal logits or parameters are inaccessible, or distillation between teacher and student LLMs with incompatible vocabularies. GAD expands our prior work on white-box on-policy distillation (i.e., MiniLLM), pioneering block-box on-policy distillation for LLM training. Specifically, GAD frames the student LLM as a generator and trains a discriminator to distinguish its responses from the teacher LLM’s, creating a minimax game. The discriminator acts as an on-policy reward model that co-evolves with the student, providing stable, adaptive feedback. Experimental results show that GAD consistently surpasses the commonly used sequence-level knowledge distillation.  In particular, Qwen2.5-14B-Instruct (student) trained with GAD becomes comparable to its teacher, GPT-5-Chat, on the LMSYS-Chat automatic evaluation. The results establish GAD as a promising and effective paradigm for black-box LLM distillation. Our team has been conducting fundamental research in knowledge distillation with wide adoptions across the industry. -         MiniLM: We introduced multi-head attention distillation, establishing the most effective distillation method for BERT-style models. The open-source MiniLM models (e.g., 6x384) have become the most widely utilized small encoder models on the Hugging Face. -         MiniLLM: Our proposed Reverse KLD is recognized as one of the most effective, de facto on-policy distillation approaches for modern LLM training, which has been widely used by Thinking Machines, Gemma, and many other teams and models. -         BitDistill: We proposed BitNet Distillation to finetune off-the-shelf full-precision LLMs (e.g., Qwen) into 1.58-bit precision (ternary weights {-1, 0, 1}), achieving performance parity with the full-precision counterparts on specific downstream tasks. -         GAD: The development of Generative Adversarial Distillation (GAD) now allows for black-box on-policy distillation, overcoming two major prior limitations: (1) Distillation from proprietary teachers where internal logits or parameters are inaccessible; (2) Distillation between teacher and student LLMs with incompatible vocabularies. https://lnkd.in/gMaP2c7w

125

2 Comments

NeMo RL now supports the Decoupled Clip and Dynamic Sampling Policy Optimization (DAPO) algorithm. DAPO enhances the Generalized Reinforcement Policy Optimization (GRPO) by incorporating several advanced features: - Clip-Higher - Dynamic Sampling - Token-Level Policy Gradient Loss - Overlong Reward Shaping These additions aim to provide more stable and efficient reinforcement learning training. For further details, refer to the DAPO guide. https://lnkd.in/geGgziXy

112

Very excited to present with Haocheng Bian on how we leverage Ray and PyTorch to build two high-performance ML frameworks at Apple: • Aurora — focuses on deep learning and lightweight LLM training, leveraging FSDP and DeepSpeed. • Aether — focuses on massive-scale pre-training and RL, supporting models with hundreds of billions of parameters through PyTorch’s multi-dimensional parallelism. With support for distributed training, advanced checkpointing, mixed-precision training, kernel optimizations, deterministic DataLoader and worker checkpoint/recovery, these frameworks enable reliable and scalable LLM training across thousands of GPUs. Most importantly, the system maximizes flexibility: all framework interfaces are abstracted into a clean, notebook-style workflow—removing infrastructure complexity while giving researchers full freedom to build on new capabilities and iterate rapidly Talk link: https://lnkd.in/enqNv2Fu

178

1 Comment

We had great papers and demos on 12/9 at #ASRU. Here are some of them: 👉 Selection of Layers from Self-supervised Learning Models for Predicting Mean-Opinion-Score of Speech, Xinyu Liang, Fredrik Cumlin, Victor Ungureanu, Chandan K. A. Reddy, Christian Schuldt, Saikat Chatterjee. Investigate which layers of self-supervised speech models are most effective for speech quality assessment. Early layers of self-supervised models are better than final layers for predicting speech quality (MOS), leading to simpler, superior SQA models. 👉Transcribe, Translate, or Transliterate: An Investigation of Intermediate Representations in Spoken Language Models Tolúlopé Ògúnrèmí, Christopher Manning, Dan Jurafsky, Karen Livescu. Analysis shows Whisper-based SLMs form a semantic interlingua, while ASR-only encoders yield phonetic transliterations, revealing different audio-to-text linking methods. 👉Competitive Audio-Language Models with Data-Efficient Single-Stage Training on Public Data, Gokul Karthik Kumar, Rishabh Saraf, Ludovick Lepauloux, Abdul Muneer, Billel Mokeddem, Hakim Hacid. Falcon3-Audio, trained end-to-end in a single stage on fewer than 30K hours of fully public audio. The 7B model matches state-of-the-art open-weight performance on the MMAU benchmark, suggesting that complex training curricula and multi-encoder architectures are not necessary for competitive audio understanding. Also, the conversational AI demo by Tatiana Likhomanenko was amazing!!

31

1 Comment

🆕 The Ultimate Guide to Open-Source RL Libraries for LLMs: 9 Frameworks Compared 🎯 Three primary RL applications for LLMs: 1️⃣ RLHF for human preference alignment 2️⃣ reasoning models for problem-solving 3️⃣ multi-turn agentic interactions [via Philipp Moritz of Anyscale ] https://lnkd.in/gcF2dNGq

57

1 Comment

In this episode, we discuss LLM Post-Training: A Deep Dive into Reasoning Large Language Models by Komal Kumar, Tajamul Ashraf, Omkar Thawakar, Rao Muhammad Anwer, Hisham Cholakkal, Mubarak Shah, Ming-Hsuan Yang, Phillip H. S. Torr, Fahad Shahbaz Khan, Salman Khan. The paper surveys post-training techniques for Large Language Models (LLMs) that enhance performance beyond initial pretraining, focusing on fine-tuning, reinforcement learning, and test-time scaling. It addresses challenges like catastrophic forgetting and reward hacking while exploring model alignment and scalable adaptation. The survey also provides a public repository to track ongoing advancements in post-training methods.

8

1 Comment