Lequn Chen (@abcdabcd987) / X

Lequn Chen

61 posts

Lequn Chen

@abcdabcd987

Faster and cheaper LLM inference.

Seattle, WA

Joined January 2012

Lequn Chen
@abcdabcd987
Apr 19, 2025
It has been such a wonderful year at @perplexity_ai. Keep building 😆
45K
Lequn Chen
@abcdabcd987
Nov 9, 2023
🤔Assuming a large language model application takes 5 GPUs to serve, does it require 50 GPUs to serve 10 different LLM apps? 🌟Out latest research project, Punica, enables serving multiple LoRA finetuned LLMs at the cost of one! github.com/punica-ai/puni…
58K
Lequn Chen
@abcdabcd987
Feb 10, 2025
We are building our in-house LLM inference stack. Join us if this excites you! And, I have a more in-depth tutorial about achieving 3200 Gbps here:
Perplexity
@perplexity_ai
Feb 10, 2025
Using a custom RDMA-based networking library, we've been able to achieve 3200 Gbps GPU memory transfers, bypassing NCCL limits for 97.1% theoretical bandwidth efficiency. Our latest blog shares our journey of building a custom high-performance networking solution on AWS.
le.qun.ch
Harnessing 3200 Gbps Network: A Journey with RDMA, EFA, and libfabric
Earlier this year, I had the fortune of joining Perplexity AI, where I finally got to use servers with the most powerful configuration—AWS p5 instances equipped with 8 NVIDIA H100 GPUs interconnect...
30K
Lequn Chen
@abcdabcd987
Apr 2, 2025
10x faster than PyTorch All-to-All. 2x faster than DeepEP on single node. Although 2x slower than DeepEP on 128 GPUs, our impl is less picky about hardware and software. Make your MoE go brrr
Perplexity Developers
@PPLXDevs
Apr 2, 2025
We've built custom NVSHMEM-based kernels for Mixture-of-Experts (MoE) models that deliver up to 10x faster communication than standard all-to-all operations. Our approach balances performance with adaptability across different hardware configurations.
GitHub - perplexityai/pplx-kernels: Perplexity GPU Kernels
From github.com
11K
Lequn Chen
@abcdabcd987
Jan 2, 2025
Start a new year's work with coffee in a Perplexity mug!
Jeremiah Warren ◡̈
@jeremiahjw
Dec 6, 2024
A few photos I took of the @perplexity_ai mugs and the coffee bag designed by @RypeArts ☕️
3.8K
Lequn Chen
@abcdabcd987
Nov 14, 2023
Just made a demo: use Punica to serve multiple LoRA finetuned LLMs at the cost of one! Previously: x.com/abcdabcd987/st…
00:00
4.5K
Lequn Chen
@abcdabcd987
Apr 18, 2025
Lower latency and Higher throughput -- Get both with multi-node deployment for MoE models like DeepSeek-V3/R1.
Perplexity Developers
@PPLXDevs
Apr 18, 2025
Replying to @PPLXDevs and @perplexity_ai
DeepSeek-V3/R1 contains 671B total parameters but activates only 37B per token. Testing shows EP128 configurations deliver up to 5x higher throughput at equivalent output speeds compared to single-node deployments. Higher EP values assign fewer experts per GPU, reducing memory
5.8K
Lequn Chen
@abcdabcd987
Nov 9, 2023
Replying to @abcdabcd987
🚀Punica is able to deliver 12x throughout compared to state-of-the-art LLM serving systems. 📄Dive into our paper: arxiv.org/abs/2310.18547 💻Explore our code: github.com/punica-ai/puni… 🗨️Join the HackerNews discussion: news.ycombinator.com/item?id=381966… #LLM #LoRA
1.1K
Lequn Chen
@abcdabcd987
Nov 9, 2023
Replying to @abcdabcd987
How? We developed a CUDA kernel, called SGMV, that efficiency runs different LoRA models in a batch. SGMV enables strong batching effect. 🔥Increasing batch size does not increase latency significantly. You can run multiple LoRA models at the cost of one.
1.4K
Lequn Chen
@abcdabcd987
Feb 5, 2024
🚀FlashInfer: Highly optimized Attention kernel for {single, batch} x {prefill, decode, append} x {ragged tensor, paging} x {FP16, FP8, INT4} x {4090, Ada6000, A100, H100} 🔥Python Wheels available! Check it out!
Zihao Ye
@ye_combinator
Feb 5, 2024
(1/4) Announcing FlashInfer, a kernel library that provides state-of-the-art kernel implementations for LLM Inference/Serving. FlashInfer's unique features include: - Comprehensive Attention Kernels: covering prefill/decode/append attention for various KV-Cache formats (Page
1.7K
Lequn Chen
@abcdabcd987
Dec 11, 2023
Really good observation from @tianle_cai and @junrushao . I did a quick sanity check. Delta between Mixtral 8x7B MoE and Mistral 7B is NOT low-rank. SGMV is not applicable here. We need new research :)
Junru Shao
@junrushao
Dec 11, 2023
Workload-wise, the dynamic routing operator per se in MoE is pretty similar to the SGMV kernel in multi-LoRA serving as introduced in Pinuca @abcdabcd987 @ye_combinator :))
1.7K
Lequn Chen
@abcdabcd987
Jan 24, 2021
Replying to @iskyzh
我个人而言就做一个技术博主，至少能为中文互联网贡献一点有价值的内容。至于平台上大家喜不喜欢，有没有人看，我已经无所谓了。偶尔有人私信说帮到了他，我觉得就挺开心的。
Lequn Chen
@abcdabcd987
Apr 3, 2025
Replying to @lmsysorg
Nandor is the GOAT! Kudos to the whole team!
59
Lequn Chen
@abcdabcd987
May 16, 2025
I prefer this UI (win2003 even better) to today's UI. Today's UI feels inconsistent, whitespace is too big, info is hidden in nested menus. Screen and resolution gets bigger and bigger, but information density gets lower and lower.
░ perfectloop ░
@PERFECTL00P
May 15, 2025
🇴 🇻 🇪 🇷 🇱 🇴 🇦 🇩 🪟🪟🪟🫨
GIF
720