Motivation

The native API supports returning hidden states from the model. This PR implements the same feature for OpenAI.

Returning hidden states is important for model verification and diagnostics purposes. Inference providers that use SGLang as a backend route requests that are in the OpenAI format, so if an inference provider would like to do internal diagnostics and verification, it is much more straightforward to simply include return_hidden_states.

This PR also fixes #5761 and adds appropriate test coverage for that bug.

Modifications

Changes were made to protocol.py to support the return_hidden_states flag as well as returning hidden_states on /v1/completions and /v1/chat/completions for both streaming and non-streaming.

If no hidden states are requested, the hidden states property is omitted from the response instead of including a null field. That way, the responses are completely backwards compatible.

The adapter was changed to include hidden states in responses when requested.

The dictionary n_prev_tokens was not being updated in v1_chat_completions for a streaming response, leading the same top logprobs to being repeated for every chunk. This has been fixed as well.

Checklist

• Format your code according to the Code Formatting with Pre-Commit.
• Add unit tests as outlined in the Running Unit Tests.
• Update documentation / docstrings / example tutorials as needed, according to Writing Documentation.
• Provide throughput / latency benchmark results and accuracy evaluation results as needed, according to Benchmark and Profiling and Accuracy Results.
• For reviewers: If you haven't made any contributions to this PR and are only assisting with merging the main branch, please remove yourself as a co-author when merging the PR.
• Please feel free to join our Slack channel at https://slack.sglang.ai to discuss your PR.