Motivation

This PR addresses a key limitation in the SGLang router: the inability to support gRPC workers without predefined tokenizer-path or model-path. Currently, gRPC routers require tokenizer paths at startup, making multi-model serving with dynamic worker registration impossible.

This work implements a TokenizerRegistry system to enable dynamic tokenizer loading when workers join the router, and integrates it into the request pipeline to support tokenizer selection based on the requested model.

Related Router roadmap #13098

Modifications

Core Infrastructure

• TokenizerRegistry (sgl-router/src/tokenizer/registry.rs): Thread-safe registry using DashMap for concurrent tokenizer storage and retrieval

  • load() method with per-key locking to prevent duplicate loads
  • Concurrent-safe: multiple workers with the same model load tokenizer only once
  • Simple API: get(), register(), remove(), contains()
  • Comprehensive unit tests covering basic operations, concurrent loading prevention, multiple models, load failures, and concurrent access patterns

• AppContext Integration (sgl-router/src/app_context.rs): Added tokenizer_registry: Arc field alongside

  • Initialize tokenizer_registry at startup and register the default tokenizer if provided

Request Pipeline Integration

• Worker Registration (sgl-router/src/core/workflow/steps/worker_registration.rs): Enhanced worker registration workflow to automatically load tokenizers

  • Fetch model info from workers using GetModelInfo gRPC call
  • Use extracted model_id as register key
  • Automatically load and register tokenizer in the registry when new workers join
  • Graceful error handling for workers that don't support model info queries

• Request Processing Updated all request processing stages to use tokenizer registry:

  • Regular Router Processor (sgl-router/src/routers/grpc/regular/processor.rs): Look up tokenizer from registry based on requested model
  • Chat Preparation (sgl-router/src/routers/grpc/regular/stages/chat/preparation.rs): Use model-specific tokenizer for chat requests
  • Generate Preparation (sgl-router/src/routers/grpc/regular/stages/generate/preparation.rs): Use model-specific tokenizer for generate requests
  • Streaming Handler (sgl-router/src/routers/grpc/regular/streaming.rs): Use tokenizer registry for streaming response processing

• Router Updates:

  • GrpcRouter (sgl-router/src/routers/grpc/router.rs) and PdRouter (sgl-router/src/routers/grpc/pd_router.rs): Pass tokenizer_registry to request processors
  • Pipeline (sgl-router/src/routers/grpc/pipeline.rs): Updated to work with tokenizer registry

Model ID Resolution for GenerateRequest

• RouterManager (sgl-router/src/routers/router_manager.rs): Added intelligent model ID resolution for requests without explicit model specification
  • New resolve_model_id() helper method: Intelligently resolves model_id when not provided in requests
    • If model_id is provided: use it directly
    • If not provided and only 1 model exists: use it as implicit default (backward compatible for single-model deployments)
    • If not provided and multiple models exist: return clear error listing available models
    • If no models exist: return service unavailable error
  • Updated route_generate() method: Replaced hardcoded "unknown" fallback with intelligent model resolution
  • Ensures GenerateRequest (with optional model field) correctly routes to registered tokenizers
  • Aligns model_id resolution with tokenizer registry keys and worker model_ids

Configuration & Validation

• Config Validation (sgl-router/src/config/validation.rs): Relaxed tokenizer-path requirement for gRPC mode to allow tokenizer-less startup with dynamic loading

Testing Updates

• Updated test infrastructure (tests/common/) to work with tokenizer registry
• Added test coverage for tokenizer registry operations

What's Not Included (Coming in Follow-up PR)

• IGW (Intelligent Gateway) Support: Multi-model routing through IGW will be enabled in a subsequent PR. The current implementation focuses on single-router mode with the tokenizer registry infrastructure in place.

Accuracy Tests

Not applicable - this PR does not affect model outputs or inference results. It only modifies tokenizer loading infrastructure and routing logic to support dynamic tokenizer management.

Benchmarking and Profiling

Performance impact is minimal as the tokenizer registry uses DashMap for lock-free concurrent access. The registry lookup adds negligible overhead to the request path (~microseconds for a concurrent hash map lookup).

Key performance characteristics:

• Tokenizer loading is done once per model during worker registration
• Request routing uses fast concurrent hash map lookups
• No impact on existing single-tokenizer configurations
• Memory usage scales linearly with the number of unique models

Future benchmarking will measure:

• Tokenizer loading time during worker registration
• Memory usage with multiple models
• Request routing latency comparison (should remain unchanged)

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.

Screenshot