Taken over from #20680

Motivation

This PR improves the performance of NixlKVManager by making KV transfer asynchronous and multi-threaded on the prefill node. Previously, add_transfer_request performed each chunk transfer synchronously and the caller (NixlKVSender) had to track and poll all transfer handles. With many decode instances and chunked transfers, this caused the prefill scheduler to block on transfer completion and limited throughput. This change aligns NIXL with the queue-based, multi-worker transfer design.

Performance

We ran Qwen3-32B PD disaggregation with NIXL and observed a clear improvement in transfer latency via NIXL telemetry:

• Mean transfer time: 162,225 μs → 41,225 μs (about 4× lower).
• Distribution: Before, transfer times had high variance with many samples in the 250k–1.2M μs range and a long tail; after the change, the vast majority of samples sit in the 34k–42k μs band with much lower variance and no large outliers.

Async multi-worker transfer removes the synchronous bottleneck on the prefill path: chunks are processed in parallel by worker threads, and decode instances are sharded across queues for better overlap, which explains the lower mean and significantly improved tail (P95/P99) latency.

Modifications

1. Async transfer with queue + worker pool (PREFILL mode)

   • Introduced multiple FastQueue instances (count controlled by SGLANG_DISAGGREGATION_QUEUE_SIZE) and a ThreadPoolExecutor per queue (total worker count from SGLANG_DISAGGREGATION_THREAD_POOL_SIZE).
   • Added a TransferKVChunk dataclass and daemon transfer_worker threads that consume chunks from the queues and execute send_kvcache / send_kvcache_slice, maybe_send_extra, and send_aux in the worker.
   • Default thread pool size: min(max(4, (0.5 * cpu_count) // 8), 12) when the env var is not set; queue size defaults to env (e.g. 4).

2. Non-blocking add_transfer_request

   • add_transfer_request no longer performs transfer inline; it enqueues a TransferKVChunk to transfer_queues[bootstrap_room % len(transfer_queues)] and returns None.
   • Workers update request_status (e.g. Transferring, Success, Failed), so the sender no longer needs to hold or poll transfer handles.

3. NixlKVSender simplifications

   • Removed xfer_handles; poll() now relies on kv_mgr.check_status(bootstrap_room) only.
   • Added clear() to remove bootstrap_room from request_status when appropriate.
   • Last-chunk path no longer deletes request_status in the sender; the worker clears transfer_infos and sets status to Success when the last chunk is done.

4. Scheduler handling of Bootstrapping

   • In prefill.py, requests in KVPoll.Bootstrapping are now treated as undone (together with WaitingForInput and Transferring) so the scheduler does not consider them complete before transfer progress.

Testing

• python3 -m sglang.test.few_shot_gsm8k --num-questions 200 --host 127.0.0.1 --port 8000: Accuracy: 0.945 with Qwen/Qwen3-8B
• TestDisaggregationAccuracy passes with NIXL (score 0.76, throughput 3949 token/s)

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests. (Regression: ran test_disaggregation_basic.py; 7 tests passed.)
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.

Review Process

1. Ping Merge Oncalls to start the PR flow. See the PR Merge Process.
2. Get approvals from CODEOWNERS and other reviewers.
3. Trigger CI tests with comments or contact authorized users to do so. (/tag-run-ci-label, /rerun-failed-ci, /tag-and-rerun-ci)
4. After green CI and required approvals, ask Merge Oncalls to merge.