Test run: https://github.com/InferenceMAX/InferenceMAX/actions/runs/20662552221

Summary

Note: This PR will be merged 10 days from the time it is opened. This gives ample time for all parties to review.

Warmup is standard practice in production deployments via Kubernetes readiness probes, pre-traffic validation. This is how production systems operate. SGLang, OpenAI, and Together AI all warmup their inference engine before serving traffic. Major frameworks such as vLLM and SGLang benchmarking tools already support warmup and it is widely used in production.

Reference implementations:

• vLLM warmup implementation
• SGLang warmup implementation
• NVIDIA aiperf warmup implementation
• HuggingFace inference-benchmarker warmup implementation
• TRT-LLM warmup implementation
• Llama CPP warmup implementation
• And many other examples in inference benchmarking in open source & internal AI labs tools have warmup as this closely follows real world prod via k8s readiness probes

Our benchmarking tool closely follows the vLLM benchmarking tool as it should reflect real-world production performance, not cold-start artifacts. Adding warmup aligns our methodology with industry standard practice.

This implementation seeks to standardize warmup across all single node configurations. We will now be sending up to 2x concurrency level requests as warmup across the board. Additionally, we will be upstreaming our benchmark suite into InferenceMAX as part of these changes instead of having it be in a separate side repo.

Many months ago we already said we will add warmup & will provide an additional 10 days notice as of sending this document out to yall.

#339

Most importantly, this is standard industry practice. Warmup is supported and the default in the official SGLang and vLLM benchmarking scripts. It is a widely used practice throughout the SGLang/vLLM communities. Warmup is common in real-world production environments during container startup, and used by firms such as OpenAI, TogetherAI, xAI as well as OSS projects such as KServe, NVIDIA aiperf, HuggingFace, Llama.cpp, and more. As always, InferenceMAX aims to benchmark "real-world" scenarios and as such, adding warmup is natural.

Changes

Warmup

We add the --num-warmups n argument to the benchmark_serving.py script, which takes in a positive integer n. When specified, the benchmark_serving.py script invokes this loop within the benchmark function to send the first randomly generated request n times to the server. This ends up being a one-line change in terms of how benchmarks are launched—now we just add the --num-warmups flag.

Before:

python3 benchmark_serving.py \
    --model "$model" \
    --backend "$backend" \
    --base-url "http://0.0.0.0:$port" \
    --dataset-name random \
    --random-input-len "$input_len" \
    --random-output-len "$output_len" \
    --random-range-ratio "$random_range_ratio" \
    --num-prompts "$num_prompts" \
    --max-concurrency "$max_concurrency" \
    --request-rate inf \
    --ignore-eos \
    --save-result \
    --percentile-metrics 'ttft,tpot,itl,e2el' \
    --result-dir "$result_dir" \
    --result-filename "$result_filename.json"

After:

# vllm & sglang benchmark_serving.py also have --num-warmup & inferencemax
# benchmark_serving closely follows vllm script
python3 benchmark_serving.py \
    --model "$model" \
    --backend "$backend" \
    --base-url "http://0.0.0.0:$port" \
    --dataset-name random \
    --random-input-len "$input_len" \
    --random-output-len "$output_len" \
    --random-range-ratio "$random_range_ratio" \
    --num-prompts "$num_prompts" \
    --num-warmups "$num_warmups" \
    --max-concurrency "$max_concurrency" \
    --request-rate inf \
    --ignore-eos \
    --save-result \
    --percentile-metrics 'ttft,tpot,itl,e2el' \
    --result-dir "$result_dir" \
    --result-filename "$result_filename.json"

Each benchmark run will invoke this script up to --num-warmups=(2 * CONC), where CONC is the maximum number of concurrent requests that can be in-flight at any single point in time.

We note that this is the standard implementation as seen in SGLang and vLLM, hence InferenceMAX will be following suit with this implementation. This is important as it is our goal to remain consistent with the broader ML community.

Code Upstream

Currently, we utilize a benchmark suite in a downstream repository, which is a fork of another repository, which is a fork of the original vLLM benchmark serving suite (which has since been refactored). We will be upstreaming the relevant code from this repository into InferenceMAX for maximum transparency.

Note: Multinode runs will be upstreamed at a later day due to the fact that multinode scripts are not fully upstreamed to the InferenceMAX repository at this point.

Related PRs

• chore: upstream bench serving and add warmup #339