Motivation

The RouterManager::select_router_for_request function is on the critical hot path of the gateway, executed for every incoming inference request. The previous implementation suffered from two performance bottlenecks:

1. Per-Request Allocations: The use of collect::<Vec<_>>() and vec![router] triggered heap allocations on every request, leading to system memory allocator overhead and potential lock contention under high concurrency.
2. Map Iteration Locks: Iterating over DashMap entries requires acquiring internal shard-level locks. While efficient for point lookups, this added microsecond-scale latency and jitter during high-throughput routing.

This PR introduces a lock-free snapshot mechanism using the arc_swap crate. By caching the router list in a contiguous, read-optimized Vec that is updated only during registration, we achieve nanosecond-scale routing performance.

Modifications

• Snapshot Architecture: Integrated routers_snapshot: ArcSwap<Vec<Arc<dyn RouterTrait>>> into the RouterManager struct to provide zero-allocation access to candidate routers.

• Atomic Updates: Modified register_router to pre-calculate and store a fresh snapshot whenever a new router joins the registry, shifting overhead from the "hot" request path to the "cold" registration path.

• Zero-Allocation Selection: Refactored select_router_for_request to use lock-free .load() calls.

• Logic Refinement:

  • Simplified model-specific lookups to return matching candidates immediately, bypassing redundant scoring logic when only one valid router exists.

• Dependencies: Added arc_swap = "1.7.1" to Cargo.toml.

Accuracy Tests

Benchmarking and Profiling

benchmarks were conducted ($1\mu s = 1,000ns$). The results show a dramatic reduction in routing overhead.

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.
• Work with maintainers to merge your PR. See the PR Merge Process