Motivation

Fix two CUDA stream race conditions that cause illegal memory access (IMA) in FA3 persistent kernel when HiCache write_through policy is active with eviction pressure on Hopper GPUs. The bug is triggered when --hicache-write-policy write_through (the default) causes D2H transfers on write_stream to race with FA3 on forward_stream.

This is a long-standing issue reported across multiple models and configurations (#19737, #14363, #11842, #13912, #18785, #18166, many of them closed due to inactivity but the bug they pointed out remains). As #14363 noted: "it is very difficult to reproduce" since the crash requires sufficient eviction pressure (usually caused by high enough request concurrency) to trigger the race condition. I found that setting --mem-fraction-static 0.40 reliably reproduces the crash within minutes by forcing frequent eviction and D2H transfers.

Together with #20049 (lock_ref leak fix), this PR completes a set of three fixes that address the HiCache + FA3 crash:

Root cause: HiCache's write_stream (D2H transfer) and the scheduler's forward_stream (FA3 kernel) lack bidirectional GPU-side synchronization. This creates two independent race windows:

1. D2H reads before forward finishes writing: write_stream starts reading KV cache slots that FA3 on forward_stream is still writing to. The original code only synchronized with the schedule stream via start_event, missing the forward_stream entirely. Fix: write_stream.wait_stream(forward_stream) before D2H.

2. Forward launches before D2H SM copy finishes: The next batch's FA3 persistent kernel (occupying all 132 SMs) launches on forward_stream while the previous batch's HiCache SM copy kernel (kernel IO backend) is still running on write_stream. The concurrent SM resource conflict corrupts FA3's TMA descriptor uniform registers, causing CUDBG_EXCEPTION_WARP_ILLEGAL_ADDRESS. Fix: forward_stream.wait_event(last_write_finish_event) before forward.

The diagram below shows the three CUDA streams in SGLang's overlap scheduler and where each fix adds synchronization:

SGLang's overlap scheduler uses three CUDA streams: DEFAULT (scheduling/memory ops), FORWARD (FA3 + MLP forward), and WRITE (HiCache D2H transfer). Each iteration, the CPU submits run_batch(N) to FORWARD, then pop_and_process(N-1) triggers D2H on WRITE for the previous batch's KV data. Without synchronization:

• WRITE's D2H may start reading KV data that FORWARD's FA3 hasn't finished writing → write_stream.wait_stream(forward_stream) ensures D2H waits
• FORWARD's next FA3 may launch while WRITE's SM copy kernel is still running → forward_stream.wait_event(last_write_finish_event) ensures FA3 waits

With both fixes, FA3 and D2H SM copy kernel never overlap on the GPU, eliminating the TMA descriptor corruption.

Fixes #19737, fixes #18785, fixes #14363
Related issues: #11842, #13912, #18166
Related PR: #20049 (independent fix for write-through lock_ref leak during decode)

Note: #19737's Workaround 2 (current_stream().wait_event(finish_event)) fixes the same race but blocks the CPU thread synchronously. This PR uses forward_stream.wait_event() instead, which is a GPU-side fence that preserves overlap scheduling performance.

Modifications

cache_controller.py:

• Add producer_stream and last_write_finish_event attributes to HiCacheController
• Add set_producer_stream() method for scheduler to register the forward stream
• In start_writing(): replace start_event.record() + start_event.wait() pattern with explicit write_stream.wait_stream(current_stream) + write_stream.wait_stream(producer_stream), ensuring D2H waits for both schedule and forward streams
• Expose last_write_finish_event after D2H finish event is recorded

scheduler.py:

• In HiRadixCache initialization: call set_producer_stream(self.forward_stream) to register the forward stream with the cache controller
• In overlap path (run_batch): add forward_stream.wait_event(last_write_finish_event) before forward_batch_generation, ensuring the forward kernel doesn't launch until the previous D2H SM copy kernel completes

Debugging with CUDA Coredump Analysis

We collected CUDA coredumps using SGLANG_CUDA_COREDUMP=1 from two crash scenarios on 8xH100 serving MiniMax-M2.5 (456B MoE, TP8 EP8) with --mem-fraction-static 0.40 --enable-hierarchical-cache --hicache-io-backend kernel --attention-backend fa3. All experiments had #20049 (lock_ref fix) applied to prevent OOM during sustained decode. In both cases the crash kernel is FlashAttnFwdSm90 — the FA3 persistent kernel, Grid=(132,1,1) occupying all 132 SMs on H100.

Crash A: with #20049 only (no stream sync), crashed at 2.2 min / 48 waves:

• Crash instruction: LDGSTS.E.BYPASS.LTC128B.128 desc[UR40] in FA3 mainloop (loading KV data via TMA)
• UR40 = 0x0, UR41 = 0x0 (null TMA descriptors)
• Root cause: write_stream reads KV slots before forward_stream (FA3) finishes writing them

Crash B: with #20049 + write_stream.wait_stream(forward_stream), crashed at 20 min / 427 waves:

• Crash instruction: STG.E desc[UR40][R4.64],R3 in FA3 epilogue (writing attention output via TMA)
• UR40 = 0x0, UR41 = 0x0 (null TMA descriptors), R3 = 0xFF800000 (-inf, softmax LSE init value)
• Two independent warps on different SMs (SM 4 and SM 6) both triggered the error — rules out hardware fault, confirms systematic TMA descriptor corruption
• Root cause: HiCache SM copy kernel (on write_stream) and FA3 persistent kernel (on forward_stream) run concurrently, causing SM resource conflict that corrupts FA3's uniform registers

The different crash sites (mainloop LDGSTS vs epilogue STG) with the same symptom (UR40=0x0 TMA descriptor corruption) confirm two independent race conditions. The first fix eliminates the read-before-write race, exposing the SM resource conflict race which crashes at a different instruction.

cuda-gdb -batch \
  -ex "target cudacore cuda_coredump_5ac27cf368dd.175.1773213005" \
  -ex "info cuda kernels" \
  -ex "info cuda threads" \
  -ex "cuda kernel 0" \
  -ex "x/i \$pc" \
  -ex "info registers UR40 UR41 R3 R4 R5"

# 1. Apply PR #20049 first (prevents OOM that masks the stream race)

# 2. Start server with tight cache + coredump collection
SGLANG_CUDA_COREDUMP=1 PYTHONUNBUFFERED=1 python3 -m sglang.launch_server \
  --model-path <any-model> \
  --enable-hierarchical-cache \
  --mem-fraction-static 0.40 \
  --hicache-io-backend kernel \
  --attention-backend fa3 \
  --port 8000

# 3. Send concurrent requests (crash within minutes on Hopper SXM)
python3 -m sglang.bench_serving \
  --backend sglang \
  --port 8000 \
  --dataset-name random \
  --random-input 1024 \
  --random-output 256 \
  --num-prompts 512 \
  --request-rate 10

# 4. Coredump saved to logs/cuda-coredumps/ after crash

Stress Test with High Request Concurrency

Validated on 8xH100 SXM serving MiniMax-M2.5 (456B MoE). All experiments had #20049 (lock_ref fix) applied.

Command

python3 -m sglang.launch_server \
  --model-path MiniMaxAI/MiniMax-M2.5 \
  --tp 8 --ep 8 \
  --mem-fraction-static 0.40 \
  --enable-hierarchical-cache \
  --hicache-ratio 2.0 \
  --hicache-size 64 \
  --hicache-write-policy write_through \
  --hicache-io-backend kernel \
  --attention-backend fa3 \
  --decode-attention-backend flashinfer \
  --chunked-prefill-size 8192 \
  --page-size 64 \
  --enable-cache-report \
  --trust-remote-code \
  --port 8000

Client:

# Sustained stress test: 64 concurrent requests per wave, looping for 120 min
python3 repro_sustained.py single \
  --url http://localhost:8000 \
  --n-requests 64 \
  --output-dir ./results
# Wrapped in a loop externally with 120-min duration target

Scripts: repro_sustained.py (sustained multi-turn stress test), repro-sglang-cuda-crash.py (single-shot crash reproduction).

• --mem-fraction-static 0.40 forces tight GPU memory, keeping token usage at 0.83 with eviction active throughout
• Throughput comparison (experiment B vs C, same workload): ~1,350 req/min in both — zero measurable regression from wait_event
• wait_event is a GPU-side no-op when D2H has already completed before the next forward launch (the common case)

Accuracy Tests

No changes to model forward code or kernel logic. The fix only adds GPU-side stream synchronization barriers.

Benchmarking and Profiling

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.

Review Process

1. Ping Merge Oncalls to start the PR flow. See the PR Merge Process.
2. Get approvals from CODEOWNERS and other reviewers.
3. Trigger CI tests with comments or contact authorized users to do so.
   • /tag-run-ci-label, /rerun-failed-ci, /tag-and-rerun-ci
4. After green CI and required approvals, ask Merge Oncalls to merge.