python3 benchmark/gsm8k/bench_sglang.py

Latency: 92.746
Invalid: 0.000
Accuracy: 0.935

python3 -m sglang.bench_serving --backend sglang --num-prompts 5000

============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Successful requests:                     5000
Benchmark duration (s):                  346.41
Total input tokens:                      1224620
Total generated tokens:                  1061203
Total generated tokens (retokenized):    1055493
Request throughput (req/s):              14.43
Input token throughput (tok/s):          3535.22
Output token throughput (tok/s):         3063.47
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   145289.76
Median E2E Latency (ms):                 143439.03
---------------Time to First Token----------------
Mean TTFT (ms):                          60141.13
Median TTFT (ms):                        55335.66
P99 TTFT (ms):                           131151.83
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          742.79
Median TPOT (ms):                        549.99
P99 TPOT (ms):                           4744.62
---------------Inter-token Latency----------------
Mean ITL (ms):                           424.83
Median ITL (ms):                         237.24
P99 ITL (ms):                            1699.88
=================================================