Motivation

PR 2 of 2 (based on #27445). test/registered/chunked_prefill/test_scripted_core_4gpu.py is flaky in CI (example failure on an unrelated PR): the PP0 scheduler crashes with

File ".../sglang/srt/mem_cache/radix_cache.py", line 512, in cache_unfinished_req
    self.dec_lock_ref(req.last_node)
File ".../sglang/srt/mem_cache/radix_cache.py", line 594, in dec_lock_ref
    node is self.root_node
AssertionError: This request holds the node from another tree

via event_loop_pp -> _pp_process_batch_result -> process_batch_result_prefill -> maybe_cache_unfinished_req, right after a Cache flushed successfully! log, and the whole test file then times out (1200s).

Root cause: is_fully_idle() cannot see in-flight PP microbatches

Under pipeline parallelism, the scheduler keeps in-flight microbatches in self.mbs[slot]; a batch launched at slot i only has its result processed pp_size + pp_async_batch_depth - 1 iterations later. is_fully_idle() checked:

• running_batch / running_mbs — only contains requests after their extend result has been processed and merged for decode;
• chunked_req — cleared as soon as the last chunk is dispatched;
• cur_batch / last_batch — rebound every iteration to the current slot only;
• waiting_queue — the request left it when chunking started.

So after the last prefill chunk of a request is dispatched but before its batch results are processed, every structure is_fully_idle() looks at is empty while 1+ extend microbatches are still in flight. A /flush_cache processed inside that window passes the idle gate and runs tree_cache.reset() + req_to_token_pool.clear() + token_to_kv_pool_allocator.clear(). When the in-flight chunk result is processed afterwards, cache_unfinished_req calls dec_lock_ref(req.last_node) with a node from the pre-flush tree and hits the assertion above.

This is reachable outside tests: any PP + chunked-prefill deployment that calls /flush_cache right after traffic stops (e.g. RL weight-update flows) can land in the window. The same blind spot also gates on_idle() housekeeping (pool-leak / tree invariant checks) and HiCache attach/detach.

The scripted-runtime harness had the same blind spot (_get_all_reqs only scanned the current slot bindings, so ScriptedReqHandle.finished flipped to true spuriously while chunk results were in flight), and its inter-subtest reset drained on that partial view and silently proceeded to flush_cache even when the drain budget was exhausted — which is how CI landed the flush in exactly that window.

Modifications

The commits are ordered test-first so the unfixed failure is reproducible from this PR's own history:

1. ad55152a5dd09b95c560a87dbdcaca7d33892f6b / b565cc65d52a08eb15078d29a98c4eba9a212a3d — add test_pp_flush_cache_during_inflight_chunk_results: the script waits for the exact race window (queues and current slot bindings clear, scheduler.mbs still holding dispatched chunk batches) and posts flush_cache into it.
2. 3fb6204e29827af3938542de9171771f01acb8d2 — Scheduler.is_fully_idle(): also require all in-flight PP microbatches (self.mbs) to be empty, so flush_cache and other destructive idle-gated operations wait for pending microbatch results.
3. b799d93cc0a13b22b6ed886323b6ebd273e2bc60 — scripted runtime harness: _get_all_reqs scans all PP microbatch slots; _reset_engine_state drains on scheduler.is_fully_idle() and raises if the engine does not quiesce instead of silently flushing.
4. 0540cc7edd0577dc89c2d2270a16e3ba62d56861 / c99b622a72e7a99935aa31b78a503cd9438a0844 — extract _pp_microbatches_drained() and drop redundant comments.

Verification

Deterministic A/B on a 4-GPU machine (Qwen3-0.6B, tp=1 pp=4 pp_async_batch_depth=2, chunked_prefill_size=64), same command for both:

cd test/registered/chunked_prefill
python3 test_scripted_core_4gpu.py TestScriptedPpChunkSweep.test_pp_flush_cache_during_inflight_chunk_results

A — test only, fix absent (checkout b565cc65d52a08eb15078d29a98c4eba9a212a3d, the commit right before the fix): the flush posted into the window is accepted by the pre-fix idle gate and the scheduler crashes with the exact CI signature; the harness then hangs until timeout:

[PP0] Cache flushed successfully!            <- script's flush, accepted mid-flight (all 4 ranks)
...
AssertionError: This request holds the node from another tree
...
exit code 124 (timeout)

B — with the fix (checkout c99b622a72e7a99935aa31b78a503cd9438a0844, this PR's HEAD): the same flush is refused purely by the new in-flight-microbatch check (both legacy counters are 0), the request completes, and the whole file passes:

[PP0] Cache not flushed because there are pending requests. #queue-req: 0, #running-req: 0   (x4 ranks)
...
Ran 2 tests in 40.774s
OK

Full scripted-runtime matrix on the chain (all green): test_scripted_core_4gpu.py (sweep + new test, repeated x3), test_scripted_core_1gpu.py, test_scripted_swa_1gpu.py, test_scripted_runtime_core.py (42 tests), unit/scripted_runtime/test_http_server.py, unit/scripted_runtime/test_scheduler_hook.py, unit/scripted_runtime/test_tokenizer_recv_proxy.py.

Accuracy Tests

N/A (scheduler idle-gating and test harness only).

Speed Tests and Profiling

N/A.

CI States

Latest PR Test (Base): 🚫 Run #27088850686
Latest PR Test (Extra): ✅ Run #27088850584