Commit 6aa057c9d7de0b2535cd9d18369b60f9a506a55f Avg latency: 1.4640847804024815 seconds 10% percentile latency: 1.415596424601972 seconds 25% percentile latency: 1.4453958957456052 seconds 50% percentile latency: 1.4736635386943817 seconds 75% percentile latency: 1.49137840885669 seconds 90% percentile latency: 1.5071828443557025 seconds 99% percentile latency: 1.5301525697298348 seconds Avg latency: 1.5124143635233243 seconds 10% percentile latency: 1.484238139167428 seconds 25% percentile latency: 1.4916537082754076 seconds 50% percentile latency: 1.5120969396084547 seconds 75% percentile latency: 1.5286095263436437 seconds 90% percentile latency: 1.5470494726672768 seconds 99% percentile latency: 1.5536995528824626 seconds Avg latency: 1.4919949095075329 seconds 10% percentile latency: 1.4549806559458376 seconds 25% percentile latency: 1.475863394793123 seconds 50% percentile latency: 1.4985982133075595 seconds 75% percentile latency: 1.5087692718952894 seconds 90% percentile latency: 1.5240337627008558 seconds 99% percentile latency: 1.5535430888645352 seconds GSM8K Results for Qwen/Qwen3-Next-80B-A3B-Instruct-FP8: Measured metric: 0.8567 Expected metric: 0.8500 Tolerance: 0.0800 Questions: 1319 Invalid rate: 0.000 Latency: 66.1s QPS: 19.9 ✅ GSM8K test passed for Qwen/Qwen3-Next-80B-A3B-Instruct-FP8 ============ Serving Benchmark Result ============ Successful requests: 200 Failed requests: 0 Request rate configured (RPS): 1.00 Benchmark duration (s): 202.43 Total input tokens: 43560 Total generated tokens: 44697 Request throughput (req/s): 0.99 Output token throughput (tok/s): 220.80 Peak output token throughput (tok/s): 403.00 Peak concurrent requests: 8.00 Total token throughput (tok/s): 435.98 ---------------Time to First Token---------------- Mean TTFT (ms): 62.06 Median TTFT (ms): 56.51 P99 TTFT (ms): 174.60 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 5.35 Median TPOT (ms): 5.14 P99 TPOT (ms): 9.05 ---------------Inter-token Latency---------------- Mean ITL (ms): 11.91 Median ITL (ms): 11.58 P99 ITL (ms): 31.41 ---------------Speculative Decoding--------------- Acceptance rate (%): 66.09 Acceptance length: 2.32 Drafts: 19234 Draft tokens: 38468 Accepted tokens: 25422 Per-position acceptance (%): Position 0: 77.61 Position 1: 54.56 ================================================== ============ Serving Benchmark Result ============ Successful requests: 200 Failed requests: 0 Request rate configured (RPS): 1.00 Benchmark duration (s): 202.34 Total input tokens: 43560 Total generated tokens: 44697 Request throughput (req/s): 0.99 Output token throughput (tok/s): 220.90 Peak output token throughput (tok/s): 408.00 Peak concurrent requests: 8.00 Total token throughput (tok/s): 436.17 ---------------Time to First Token---------------- Mean TTFT (ms): 54.70 Median TTFT (ms): 55.22 P99 TTFT (ms): 132.90 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 5.24 Median TPOT (ms): 5.05 P99 TPOT (ms): 8.09 ---------------Inter-token Latency---------------- Mean ITL (ms): 11.67 Median ITL (ms): 11.39 P99 ITL (ms): 30.71 ---------------Speculative Decoding--------------- Acceptance rate (%): 66.15 Acceptance length: 2.32 Drafts: 19233 Draft tokens: 38466 Accepted tokens: 25446 Per-position acceptance (%): Position 0: 77.61 Position 1: 54.70 ================================================== ============ Serving Benchmark Result ============ Successful requests: 200 Failed requests: 0 Request rate configured (RPS): 1.00 Benchmark duration (s): 202.41 Total input tokens: 43560 Total generated tokens: 44697 Request throughput (req/s): 0.99 Output token throughput (tok/s): 220.82 Peak output token throughput (tok/s): 408.00 Peak concurrent requests: 8.00 Total token throughput (tok/s): 436.03 ---------------Time to First Token---------------- Mean TTFT (ms): 54.79 Median TTFT (ms): 55.56 P99 TTFT (ms): 132.07 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 5.25 Median TPOT (ms): 5.08 P99 TPOT (ms): 8.84 ---------------Inter-token Latency---------------- Mean ITL (ms): 11.63 Median ITL (ms): 11.37 P99 ITL (ms): 30.69 ---------------Speculative Decoding--------------- Acceptance rate (%): 65.82 Acceptance length: 2.32 Drafts: 19280 Draft tokens: 38560 Accepted tokens: 25382 Per-position acceptance (%): Position 0: 77.37 Position 1: 54.28 ================================================== Commit 5f5b33c13899fb489bdd6bcd23f37876e4024ed4 Avg latency: 1.2354058088113864 seconds 10% percentile latency: 1.2097864486277103 seconds 25% percentile latency: 1.2263582372106612 seconds 50% percentile latency: 1.2362298052757978 seconds 75% percentile latency: 1.2548624626360834 seconds 90% percentile latency: 1.264122142829001 seconds 99% percentile latency: 1.2875863956660032 seconds Avg latency: 1.2270676594848433 seconds 10% percentile latency: 1.196521044895053 seconds 25% percentile latency: 1.2093436256982386 seconds 50% percentile latency: 1.2219400526955724 seconds 75% percentile latency: 1.2374051650986075 seconds 90% percentile latency: 1.2688898637890815 seconds 99% percentile latency: 1.286326204352081 seconds Avg latency: 1.2188149772584438 seconds 10% percentile latency: 1.1886682329699396 seconds 25% percentile latency: 1.1987244160845876 seconds 50% percentile latency: 1.2178448056802154 seconds 75% percentile latency: 1.239045416470617 seconds 90% percentile latency: 1.2577871596440673 seconds 99% percentile latency: 1.285305601283908 seconds GSM8K Results for Qwen/Qwen3-Next-80B-A3B-Instruct-FP8: Measured metric: 0.8552 Expected metric: 0.8500 Tolerance: 0.0800 Questions: 1319 Invalid rate: 0.001 Latency: 64.0s QPS: 20.6 ✅ GSM8K test passed for Qwen/Qwen3-Next-80B-A3B-Instruct-FP8 ============ Serving Benchmark Result ============ Successful requests: 200 Failed requests: 0 Request rate configured (RPS): 1.00 Benchmark duration (s): 201.89 Total input tokens: 43560 Total generated tokens: 44697 Request throughput (req/s): 0.99 Output token throughput (tok/s): 221.40 Peak output token throughput (tok/s): 395.00 Peak concurrent requests: 7.00 Total token throughput (tok/s): 437.16 ---------------Time to First Token---------------- Mean TTFT (ms): 108.58 Median TTFT (ms): 52.33 P99 TTFT (ms): 1674.06 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 4.99 Median TPOT (ms): 4.21 P99 TPOT (ms): 29.13 ---------------Inter-token Latency---------------- Mean ITL (ms): 10.81 Median ITL (ms): 9.30 P99 ITL (ms): 29.77 ---------------Speculative Decoding--------------- Acceptance rate (%): 66.05 Acceptance length: 2.32 Drafts: 19236 Draft tokens: 38472 Accepted tokens: 25411 Per-position acceptance (%): Position 0: 77.52 Position 1: 54.59 ================================================== ============ Serving Benchmark Result ============ Successful requests: 200 Failed requests: 0 Request rate configured (RPS): 1.00 Benchmark duration (s): 201.97 Total input tokens: 43560 Total generated tokens: 44697 Request throughput (req/s): 0.99 Output token throughput (tok/s): 221.31 Peak output token throughput (tok/s): 401.00 Peak concurrent requests: 7.00 Total token throughput (tok/s): 436.99 ---------------Time to First Token---------------- Mean TTFT (ms): 53.10 Median TTFT (ms): 50.58 P99 TTFT (ms): 136.39 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 4.33 Median TPOT (ms): 4.20 P99 TPOT (ms): 6.94 ---------------Inter-token Latency---------------- Mean ITL (ms): 9.57 Median ITL (ms): 9.26 P99 ITL (ms): 28.60 ---------------Speculative Decoding--------------- Acceptance rate (%): 66.41 Acceptance length: 2.33 Drafts: 19188 Draft tokens: 38376 Accepted tokens: 25485 Per-position acceptance (%): Position 0: 77.54 Position 1: 55.28 ================================================== ============ Serving Benchmark Result ============ Successful requests: 200 Failed requests: 0 Request rate configured (RPS): 1.00 Benchmark duration (s): 201.96 Total input tokens: 43560 Total generated tokens: 44697 Request throughput (req/s): 0.99 Output token throughput (tok/s): 221.32 Peak output token throughput (tok/s): 387.00 Peak concurrent requests: 7.00 Total token throughput (tok/s): 437.01 ---------------Time to First Token---------------- Mean TTFT (ms): 50.88 Median TTFT (ms): 49.77 P99 TTFT (ms): 138.55 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 4.32 Median TPOT (ms): 4.15 P99 TPOT (ms): 7.14 ---------------Inter-token Latency---------------- Mean ITL (ms): 9.48 Median ITL (ms): 9.03 P99 ITL (ms): 14.80 ---------------Speculative Decoding--------------- Acceptance rate (%): 65.27 Acceptance length: 2.31 Drafts: 19365 Draft tokens: 38730 Accepted tokens: 25280 Per-position acceptance (%): Position 0: 77.11 Position 1: 53.43 ================================================== ============ Serving Benchmark Result ============ Successful requests: 200 Failed requests: 0 Request rate configured (RPS): 1.00 Benchmark duration (s): 202.06 Total input tokens: 43560 Total generated tokens: 44697 Request throughput (req/s): 0.99 Output token throughput (tok/s): 221.21 Peak output token throughput (tok/s): 389.00 Peak concurrent requests: 7.00 Total token throughput (tok/s): 436.79 ---------------Time to First Token---------------- Mean TTFT (ms): 55.20 Median TTFT (ms): 52.91 P99 TTFT (ms): 146.35 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 4.30 Median TPOT (ms): 4.17 P99 TPOT (ms): 7.33 ---------------Inter-token Latency---------------- Mean ITL (ms): 9.56 Median ITL (ms): 9.24 P99 ITL (ms): 16.12 ---------------Speculative Decoding--------------- Acceptance rate (%): 66.78 Acceptance length: 2.34 Drafts: 19123 Draft tokens: 38246 Accepted tokens: 25540 Per-position acceptance (%): Position 0: 77.83 Position 1: 55.73 ==================================================