Motivation

This PR fixes a critical deadlock (hang) issue during the initialization of HiCacheController when running sglang in a multi-node distributed environment (e.g., 2 nodes with 8 GPUs each).

When launching the server with --enable-hierarchical-cache --hicache-storage-backend on multiple nodes, the initialization process hangs at torch.distributed.new_group.

Root Cause Analysis: torch.distributed.new_group is a collective operation. It requires all ranks in the default world group to call it in the same order with the exact same arguments (specifically the ranks list).

In the previous implementation of HiCacheController （PP2 TP8）:

Node 0 (Ranks 0-7) called new_group(ranks=[0...7]).

Node 1 (Ranks 8-15) called new_group(ranks=[8...15]).

Because the arguments differed across ranks, the internal distributed store could not match the creation requests, causing a deadlock where Node 0 waited for Node 1 to acknowledge the creation of Group A, while Node 1 was attempting to create Group B.

reported by: @whybeyoung
CC @stmatengss

Modifications

Accuracy Tests

Benchmarking and Profiling

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.
• Work with maintainers to merge your PR. See the PR Merge Process