# server startup
vllm serve amd/DeepSeek-R1-0528-MXFP4 --tensor-parallel-size=8 --gpu-memory-utilization=0.94 --dtype=auto --kv-cache-dtype=fp8 --max-num-batched-tokens=8192 --attention-backend ROCM_AITER_MLA -cc.pass_config.fuse_rope_kvcache_cat_mla=True -cc.use_inductor_graph_partition=True --no-enable-prefix-caching --disable-uvicorn-access-log

# benchmark serving
vllm bench serve --model amd/DeepSeek-R1-0528-MXFP4 --percentile-metrics tpot,itl,e2el --ignore-eos --dataset-name random --random-input-len 1024 --random-output-len 1024 --max-concurrency 32 --num-prompts 320

# benchmark accuracy
lm_eval --model local-completions --model_args model=amd/DeepSeek-R1-0528-MXFP4,base_url=http://0.0.0.0:8000/v1/completions,num_concurrent=256,max_retries=10,max_gen_toks=2048 --batch_size auto --tasks gsm8k --num_fewshot 5 --limit 200 --output_path .

============ Serving Benchmark Result ============
Successful requests:                     320       
Failed requests:                         0         
Maximum request concurrency:             32        
Benchmark duration (s):                  188.52    
Total input tokens:                      327680    
Total generated tokens:                  327680    
Request throughput (req/s):              1.70      
Output token throughput (tok/s):         1738.18   
Peak output token throughput (tok/s):    1952.00   
Peak concurrent requests:                64.00     
Total token throughput (tok/s):          3476.36   
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          17.77     
Median TPOT (ms):                        17.64     
P99 TPOT (ms):                           19.82     
---------------Inter-token Latency----------------
Mean ITL (ms):                           17.77     
Median ITL (ms):                         16.87     
P99 ITL (ms):                            17.86     
----------------End-to-end Latency----------------
Mean E2EL (ms):                          18844.60  
Median E2EL (ms):                        18443.61  
P99 E2EL (ms):                           22806.19  
==================================================

|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value|   |Stderr|
|-----|------:|----------------|-----:|-----------|---|----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  | 0.95|±  |0.0154|
|     |       |strict-match    |     5|exact_match|↑  | 0.95|±  |0.0154|

============ Serving Benchmark Result ============
Successful requests:                     320       
Failed requests:                         0         
Maximum request concurrency:             32        
Benchmark duration (s):                  185.98    
Total input tokens:                      327680    
Total generated tokens:                  327680    
Request throughput (req/s):              1.72      
Output token throughput (tok/s):         1761.94   
Peak output token throughput (tok/s):    1952.00   
Peak concurrent requests:                64.00     
Total token throughput (tok/s):          3523.89   
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          17.45     
Median TPOT (ms):                        17.23     
P99 TPOT (ms):                           21.09     
---------------Inter-token Latency----------------
Mean ITL (ms):                           17.45     
Median ITL (ms):                         16.62     
P99 ITL (ms):                            17.58     
----------------End-to-end Latency----------------
Mean E2EL (ms):                          18594.01  
Median E2EL (ms):                        18189.58  
P99 E2EL (ms):                           22745.50  
==================================================

|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value|   |Stderr|
|-----|------:|----------------|-----:|-----------|---|----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  | 0.97|±  |0.0121|
|     |       |strict-match    |     5|exact_match|↑  | 0.00|±  |0.0000|