Add return routed experts to the completions and chat/completions endpoints by mansoor-s · Pull Request #17434 · sgl-project/sglang

mansoor-s · 2026-01-20T16:42:50Z

Motivation

For RL training, it is highly desirable to capture the routed experts during the rollout. Currently, only the /generate endpoint returns the routed experts data but most agentic harnesses depend on the completions endpoints so it makes sense to return this data if it's requested. The response format is identical to the /generate endpoint.

Modifications

Added return_routed_experts parameter support to /v1/chat/completions and /v1/completions OpenAI-compatible endpoints (previously only available on /generate)
Updated protocol classes (CompletionRequest, ChatCompletionRequest, response classes) with new routed_experts field
Added process_routed_experts_from_ret() utility function for processing routed experts in non-streaming responses
Added streaming support for routed experts in both chat and completion endpoints
Fixed detokenizer to return None instead of empty list when no routed experts
Updated documentation with usage examples for the new parameter
Extended tests to cover all three endpoints (/generate, /v1/chat/completions, /v1/completions)

Testing

Launched server with and without --enable-return-routed-experts

python -m sglang.launch_server --model-path Qwen/Qwen3-30B-A3B --host 0.0.0.0 --port 30000 --enable-return-routed-experts

Verified valid response with:

curl http://localhost:30000/v1/chat/completions   -H "Content-Type: application/json"     -d '{"messages": [{"role": "user", "content": "Hello"}], "return_routed_experts": true}'

Streaming response mode:

curl http://localhost:30000/v1/chat/completions   -H "Content-Type: application/json"     -d '{"messages": [{"role": "user", "content": "Hello"}], "return_routed_experts": true, "stream": true}'

Launched server without --enable-return-routed-experts and ran the verification as above.

Accuracy Tests

NA

Benchmarking and Profiling

NA

Checklist

[x ] Format your code according to the Format code with pre-commit.
[ x] Add unit tests according to the Run and add unit tests.
[ x] Update documentation according to Write documentations.
Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
[x ] Follow the SGLang code style guidance.

Review Process

Ping Merge Oncalls to start the PR flow. See the PR Merge Process.
Get approvals from CODEOWNERS and other reviewers.
Trigger CI tests with comments or contact authorized users to do so.
- /tag-run-ci-label, /rerun-failed-ci, /tag-and-rerun-ci
After green CI and required approvals, ask Merge Oncalls to merge.

gemini-code-assist · 2026-01-20T16:43:14Z

Summary of Changes

Hello @mansoor-s, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request enhances the API by integrating the capability to return routed expert information directly within the OpenAI-compatible chat and completion endpoints. This change is crucial for applications like Reinforcement Learning (RL) training, which benefit from capturing expert routing data during model rollouts. By making this data accessible through the standard completion interfaces, it streamlines the development of agentic harnesses and improves the observability of MoE model behavior.

Highlights

New Feature: Return Routed Experts: Introduced the return_routed_experts parameter to the /v1/chat/completions and /v1/completions OpenAI-compatible endpoints, allowing users to retrieve routed expert data for Mixture-of-Experts (MoE) models, previously only available via the /generate endpoint.
API Protocol Updates: Modified CompletionRequest, ChatCompletionRequest, and their corresponding response classes (CompletionResponseChoice, ChatCompletionResponseChoice, DeltaMessage, etc.) to include the new return_routed_experts field in requests and routed_experts field in responses.
Streaming Support: Implemented full streaming support for routed_experts in both chat and completion endpoints, ensuring that expert routing data can be retrieved incrementally during generation.
Utility and Detokenizer Fix: Added a new utility function process_routed_experts_from_ret() for handling routed experts in non-streaming responses and fixed the detokenizer to return None instead of an empty list when no routed experts are present.
Documentation and Testing: Updated documentation with usage examples for the new parameter and extended the test suite to cover all three relevant endpoints (/generate, /v1/chat/completions, /v1/completions) for returning routed experts.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

The pull request successfully integrates the return_routed_experts parameter across the completions and chat/completions endpoints. This includes updating protocol classes, adding utility functions for processing routed experts, and extending streaming support. The documentation has been updated with usage examples, and comprehensive tests have been added for all three relevant endpoints. The changes are well-structured and maintain code quality.

mansoor-s · 2026-01-22T20:03:34Z

@ishandhanani

…ints same as the /generated endpoint

…old SGLang specific response extensions

ishandhanani · 2026-01-23T03:27:19Z

/tag-and-rerun-ci

ishandhanani · 2026-01-23T17:24:37Z

/rerun-failed-ci

ishandhanani · 2026-01-23T20:12:16Z

Tests are green

…points (sgl-project#17434)

mansoor-s requested review from CatherineSue, JustinTong0323, Ying1123, hnyls2002, ispobock, merrymercy, slin1237 and xiezhq-hermann as code owners January 20, 2026 16:42

github-actions Bot added the documentation Improvements or additions to documentation label Jan 20, 2026

gemini-code-assist Bot reviewed Jan 20, 2026

View reviewed changes

mansoor-s force-pushed the routed_experts_chat_completions branch from 3a5ed56 to 1157034 Compare January 23, 2026 03:21

mansoor-s added 2 commits January 22, 2026 22:22

Return routed experts in v1/chat/completions and v1/completions endpo…

3b7076c

…ints same as the /generated endpoint

Add a sgl_ext struct in completion and chat/completions endpoint to h…

318a1d3

…old SGLang specific response extensions

mansoor-s force-pushed the routed_experts_chat_completions branch from 1157034 to 318a1d3 Compare January 23, 2026 03:22

github-actions Bot added the run-ci label Jan 23, 2026

ishandhanani approved these changes Jan 23, 2026

View reviewed changes

ishandhanani merged commit bdaa3de into sgl-project:main Jan 23, 2026
259 of 275 checks passed

ishandhanani mentioned this pull request Jan 28, 2026

[HiCache] feat: Add detailed cache hit breakdown for HiCache in sglext and Prometheus metrics #17648

Merged

Johnsonms pushed a commit to Johnsonms/sglang that referenced this pull request Feb 14, 2026

Add return routed experts to the completions and chat/completions end…

d400eed

…points (sgl-project#17434)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add return routed experts to the completions and chat/completions endpoints#17434

Add return routed experts to the completions and chat/completions endpoints#17434
ishandhanani merged 2 commits intosgl-project:mainfrom
mansoor-s:routed_experts_chat_completions

mansoor-s commented Jan 20, 2026

Uh oh!

gemini-code-assist Bot commented Jan 20, 2026

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

mansoor-s commented Jan 22, 2026

Uh oh!

ishandhanani commented Jan 23, 2026

Uh oh!

ishandhanani commented Jan 23, 2026

Uh oh!

ishandhanani commented Jan 23, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

mansoor-s commented Jan 20, 2026

Motivation

Modifications

Testing

Accuracy Tests

Benchmarking and Profiling

Checklist

Review Process

Uh oh!

gemini-code-assist Bot commented Jan 20, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

mansoor-s commented Jan 22, 2026

Uh oh!

ishandhanani commented Jan 23, 2026

Uh oh!

ishandhanani commented Jan 23, 2026

Uh oh!

ishandhanani commented Jan 23, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants