Motivation

Motivation
When LoRA and non-LoRA requests are mixed in the same batch, non-LoRA traffic gets severely blocked when LoRA adapter cache misses occur. This is because:

1. Base model (uid=None) was always evicted first from the LoRA memory pool, forcing non-LoRA requests to wait for adapter loading
2. Scheduler stops processing all subsequent requests when encountering a LoRA request that exceeds slot limits, even if non-LoRA requests could still be scheduled

issue code:

Modifications

1. Pin base model slot in LoRA memory pool (mem_pool.py)
   Exclude None (base model) from eviction candidates
   Ensures non-LoRA requests always have a slot available (all non-LoRA requests share this single slot)
2. Skip problematic LoRA requests in scheduler (scheduler.py)
   When a LoRA request would exceed slot limits, use continue instead of break
   Allow non-LoRA requests to continue scheduling instead of blocking the entire queue

Accuracy Tests

Benchmarking and Profiling

Test Setup

Baseline Group (No LoRA):
Benchmark 1 (B1) and Benchmark 2 (B2) command

python3 -m sglang.bench_serving \
  --backend sglang \
  --base-url http://localhost:30001 \
  --dataset-name random \
  --num-prompts 100 \
  --request-rate 2 \
  --random-input-len 2048 \
  --random-output-len 1024 \
  --disable-ignore-eos \
  --disable-tqdm

Run B1 and B2 at the same time

Mixed Traffic Group (16 adapters):
Benchmark 3 (B3) command

python3 -m sglang.bench_serving \
  --backend sglang \
  --base-url http://localhost:30001 \
  --dataset-name random \
  --num-prompts 100 \
  --request-rate 2 \
  --random-input-len 2048 \
  --random-output-len 1024 \
  --disable-ignore-eos \
  --disable-tqdm

Benchmark 4 (B4) command

python3 -m sglang.bench_serving \
  --backend sglang \
  --base-url http://localhost:30001 \
  --dataset-name random \
  --num-prompts 100 \
  --request-rate 2 \
  --random-input-len 2048 \
  --random-output-len 1024 \
  --disable-ignore-eos \
  --disable-tqdm \
  --lora-name \
    adapter1 \
    adapter2 \
    adapter3 \
    adapter4 \
    adapter5 \
    adapter6 \
    adapter7 \
    adapter8 \
    adapter9 \
    adapter10 \
    adapter11 \
    adapter12 \
    adapter13 \
    adapter14 \
    adapter15 \
    adapter16

Run B3 and B4 at the same time. The only difference between B3 and B4 is that B4 has 16 lora adapters to build request.

Before this PR

Non-LoRA Traffic benchmark

LoRA Traffic Benchmark (16 Adapters)

Mean TTFT: 8,666ms
Mean E2E: 15,157ms
Max ITL: 953ms

After this PR

Non-LoRA Traffic benchmark

LoRA Traffic Benchmark (16 Adapters)

Mean TTFT: 3,802ms (median: 306ms) - 2.3x faster
Mean E2E: 10,897ms - 1.4x faster
Max ITL: 327ms - 2.9x faster

Checklist

• [ x] Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• [ x] Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• [ x] Follow the SGLang code style guidance.
• [ x] Work with maintainers to merge your PR. See the PR Merge Process