Checklist
Describe the bug
I'm encountering an issue when making a request to the v1/chat/completions endpoint with the "logprobs": true parameter. The choices.logprobs field in the response is always null, even though it should be populated with probability values.
Reproduction
Script to launch the SGLang server:
python -m sglang.launch_server \
--model-path Qwen/Qwen2-7B-Instruct \
--port 8000Script to reproduce the request:
curl http://127.0.0.1:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen2-7B-Instruct",
"messages": [{"role": "user", "content": "Hi!"}],
"stream": false,
"temperature": 1,
"max_tokens": 1,
"top_logprobs": 2,
"logprobs": true
}'Response:
{"id":"704abb89338b4892bd63b877764f9593","object":"chat.completion","created":1722369690,"model":"Qwen/Qwen2-7B-Instruct","choices":[{"index":0,"message":{"role":"assistant","content":"Hello"},"logprobs":null,"finish_reason":"FINISH_LENGTH: 1"}],"usage":{"prompt_tokens":21,"total_tokens":22,"completion_tokens":1}}Environment
Python: 3.10.14 (main, May 6 2024, 19:42:50) [GCC 11.2.0]
CUDA available: True
GPU 0: NVIDIA A100-SXM4-80GB
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 12.3, V12.3.107
CUDA Driver Version: 525.105.17
PyTorch: 2.3.1
sglang: 0.2.7
flashinfer: 0.1.2+cu121torch2.3
requests: 2.32.3
tqdm: 4.66.4
numpy: 1.26.4
aiohttp: 3.9.5
fastapi: 0.111.1
hf_transfer: 0.1.8
huggingface_hub: 0.24.3
interegular: 0.3.3
packaging: 24.1
PIL: 10.4.0
psutil: 6.0.0
pydantic: 2.8.2
uvicorn: 0.30.3
uvloop: 0.19.0
zmq: 26.0.3
vllm: 0.5.3.post1
openai: 1.37.1
anthropic: 0.32.0
NVIDIA Topology:
GPU0 NIC0 NIC1 NIC2 NIC3 NIC4 NIC5 NIC6 NIC7 NIC8 NIC9 CPU Affinity NUMA Affinity
GPU0 X SYS SYS SYS SYS SYS SYS NODE NODE PXB PXB 96-127,224-255 3
NIC0 SYS X PIX SYS SYS SYS SYS SYS SYS SYS SYS
NIC1 SYS PIX X SYS SYS SYS SYS SYS SYS SYS SYS
NIC2 SYS SYS SYS X PIX SYS SYS SYS SYS SYS SYS
NIC3 SYS SYS SYS PIX X SYS SYS SYS SYS SYS SYS
NIC4 SYS SYS SYS SYS SYS X PXB SYS SYS SYS SYS
NIC5 SYS SYS SYS SYS SYS PXB X SYS SYS SYS SYS
NIC6 NODE SYS SYS SYS SYS SYS SYS X PIX NODE NODE
NIC7 NODE SYS SYS SYS SYS SYS SYS PIX X NODE NODE
NIC8 PXB SYS SYS SYS SYS SYS SYS NODE NODE X PXB
NIC9 PXB SYS SYS SYS SYS SYS SYS NODE NODE PXB X
Legend:
X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks
NIC Legend:
NIC0: mlx5_0
NIC1: mlx5_1
NIC2: mlx5_2
NIC3: mlx5_3
NIC4: mlx5_4
NIC5: mlx5_5
NIC6: mlx5_6
NIC7: mlx5_7
NIC8: mlx5_8
NIC9: mlx5_9
ulimit soft: 1048576
Checklist
Describe the bug
I'm encountering an issue when making a request to the
v1/chat/completionsendpoint with the"logprobs": trueparameter. Thechoices.logprobsfield in the response is always null, even though it should be populated with probability values.Reproduction
Script to launch the SGLang server:
python -m sglang.launch_server \ --model-path Qwen/Qwen2-7B-Instruct \ --port 8000Script to reproduce the request:
Response:
{"id":"704abb89338b4892bd63b877764f9593","object":"chat.completion","created":1722369690,"model":"Qwen/Qwen2-7B-Instruct","choices":[{"index":0,"message":{"role":"assistant","content":"Hello"},"logprobs":null,"finish_reason":"FINISH_LENGTH: 1"}],"usage":{"prompt_tokens":21,"total_tokens":22,"completion_tokens":1}}Environment
Python: 3.10.14 (main, May 6 2024, 19:42:50) [GCC 11.2.0] CUDA available: True GPU 0: NVIDIA A100-SXM4-80GB CUDA_HOME: /usr/local/cuda NVCC: Cuda compilation tools, release 12.3, V12.3.107 CUDA Driver Version: 525.105.17 PyTorch: 2.3.1 sglang: 0.2.7 flashinfer: 0.1.2+cu121torch2.3 requests: 2.32.3 tqdm: 4.66.4 numpy: 1.26.4 aiohttp: 3.9.5 fastapi: 0.111.1 hf_transfer: 0.1.8 huggingface_hub: 0.24.3 interegular: 0.3.3 packaging: 24.1 PIL: 10.4.0 psutil: 6.0.0 pydantic: 2.8.2 uvicorn: 0.30.3 uvloop: 0.19.0 zmq: 26.0.3 vllm: 0.5.3.post1 openai: 1.37.1 anthropic: 0.32.0 NVIDIA Topology: GPU0 NIC0 NIC1 NIC2 NIC3 NIC4 NIC5 NIC6 NIC7 NIC8 NIC9 CPU Affinity NUMA Affinity GPU0 X SYS SYS SYS SYS SYS SYS NODE NODE PXB PXB 96-127,224-255 3 NIC0 SYS X PIX SYS SYS SYS SYS SYS SYS SYS SYS NIC1 SYS PIX X SYS SYS SYS SYS SYS SYS SYS SYS NIC2 SYS SYS SYS X PIX SYS SYS SYS SYS SYS SYS NIC3 SYS SYS SYS PIX X SYS SYS SYS SYS SYS SYS NIC4 SYS SYS SYS SYS SYS X PXB SYS SYS SYS SYS NIC5 SYS SYS SYS SYS SYS PXB X SYS SYS SYS SYS NIC6 NODE SYS SYS SYS SYS SYS SYS X PIX NODE NODE NIC7 NODE SYS SYS SYS SYS SYS SYS PIX X NODE NODE NIC8 PXB SYS SYS SYS SYS SYS SYS NODE NODE X PXB NIC9 PXB SYS SYS SYS SYS SYS SYS NODE NODE PXB X Legend: X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks NIC Legend: NIC0: mlx5_0 NIC1: mlx5_1 NIC2: mlx5_2 NIC3: mlx5_3 NIC4: mlx5_4 NIC5: mlx5_5 NIC6: mlx5_6 NIC7: mlx5_7 NIC8: mlx5_8 NIC9: mlx5_9 ulimit soft: 1048576