fix: streaming session race condition + some metrics by ishandhanani · Pull Request #21875 · sgl-project/sglang

ishandhanani · 2026-04-01T20:24:27Z

Summary

Stabilize streaming session KV lifecycle: fix multiple memory leaks, add safety for session close during active decoding, and add Prometheus metrics.

Race conditions & lifecycle

Fix open_session waiter race: register the future before sending OpenSessionReqInput, clean up defensively, and tolerate late responses
Deduplicate OpenSessionReqOutput under DP-attention: every rank opens the session, but only the designated scheduler rank returns the response
Defer session close when a request is still decoding: mark for cleanup after the in-flight request finishes, preventing KV corruption

KV memory leak fixes

Overalloc tail trim (common.py): trim speculative/overallocated KV tail before transferring pool ownership to SessionSlot, preventing stranded pages between turns
Abort-skip in match_prefix: when a request is destined for abort (e.g. input too long), skip restore_to_req so the aborted request gets a fresh pool slot instead of overwriting the session's accumulated KV
Abort KV cleanup in cache_finished_req: free the transient 1-token KV from aborted requests without touching the session slot
Release accounting (_resolve_release_state): re-match the tree prefix at release time and intersect with the slot's row to handle tree splits that occurred during the session's lifetime
Owned-prefix cap: cap the release-time rematch to slot.cache_protected_len so it never grows past what the session actually owns
Cleanup flags sync: mark kv_committed_freed and kv_overallocated_freed after SessionAwareCache transfers or frees KV, so busy-time memory checks don't double-count
Session shrink fix: when a client retries with a shorter prompt (e.g. after conversation compaction), free orphaned tail pages before save_from_req overwrites the slot with a smaller committed length

Includes the fix from the merged PR #22273 which added the abort-skip and abort-cleanup paths.

Observability

sglang:num_streaming_sessions and sglang:streaming_session_held_tokens Prometheus gauges, gated behind enable_streaming_session

Test plan

Unit tests for overalloc trim, release rematch, prefix cap, abort-skip, abort accounting, session shrink, and page-aligned shrink (test_streaming_session_unit.py, 7 tests)
Integration test for abort-heavy streaming session workload (test_streaming_session.py)
Cluster validation on sa-b200 with MiniMax M2.5 FP8: leak reduced from 534K tokens to ~4K (page-alignment residual) across 5 input-length aborts

🤖 Generated with Claude Code

gemini-code-assist · 2026-04-01T20:24:32Z

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

ishandhanani · 2026-04-01T20:39:34Z

/tag-and-rerun-ci

gemini-code-assist · 2026-04-01T20:39:39Z

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

ishandhanani · 2026-04-01T21:52:25Z

/tag-and-rerun-ci

Closing a streaming session while a request is actively decoding frees KV pool indices out from under the scheduler, corrupting memory accounting and hanging the engine. - Add `pending_close` flag to Session; `_close()` returns early when an unfinished request exists instead of calling `release_session()` on live memory. - `maybe_reap()` polls pending_close sessions each second and completes the release once all requests finish. - Scheduler rejects new requests for pending_close sessions (treated as "session not found").

Only log "Deferring session close" on the first deferral. Subsequent polls while pending_close is already set are silent. The "Deferred close ready" log still fires when the session actually releases. Eliminates ~90 duplicate log lines per session close during long generations with TP>1.

…better # Conflicts: # test/registered/sessions/test_streaming_session.py

…; reject closing session

hnyls2002 · 2026-04-12T22:24:32Z

/rerun-test test_streaming_session_unit.py test_streaming_session.py test_session_control.py

github-actions · 2026-04-12T22:25:07Z

✅ ubuntu-latest (1 test): View workflow run

cd test/ && python3 registered/unit/mem_cache/test_streaming_session_unit.py

✅ 1-gpu-h100 (2 tests): View workflow run

cd test/ && python3 registered/sessions/test_streaming_session.py
cd test/ && python3 registered/sessions/test_session_control.py

hnyls2002 · 2026-04-12T22:39:59Z

/rerun-test test_streaming_session.py test_session_control.py test_streaming_session_unit.py

github-actions · 2026-04-12T22:40:35Z

✅ 1-gpu-h100 (2 tests): View workflow run

cd test/ && python3 registered/sessions/test_streaming_session.py
cd test/ && python3 registered/sessions/test_session_control.py

✅ ubuntu-latest (1 test): View workflow run

cd test/ && python3 registered/unit/mem_cache/test_streaming_session_unit.py

hnyls2002 · 2026-04-12T22:50:54Z

/rerun-test test_streaming_session.py test_session_control.py test_streaming_session_unit.py

github-actions · 2026-04-12T22:51:27Z

✅ 1-gpu-h100 (2 tests): View workflow run

cd test/ && python3 registered/sessions/test_streaming_session.py
cd test/ && python3 registered/sessions/test_session_control.py

✅ ubuntu-latest (1 test): View workflow run

cd test/ && python3 registered/unit/mem_cache/test_streaming_session_unit.py

hnyls2002 · 2026-04-12T23:24:21Z

/rerun-test test_session_latency.py test_streaming_session.py test_session_control.py test_streaming_session_unit.py

github-actions · 2026-04-12T23:25:01Z

✅ 1-gpu-h100 (3 tests): View workflow run

cd test/ && python3 registered/sessions/test_session_latency.py
cd test/ && python3 registered/sessions/test_streaming_session.py
cd test/ && python3 registered/sessions/test_session_control.py

✅ ubuntu-latest (1 test): View workflow run

cd test/ && python3 registered/unit/mem_cache/test_streaming_session_unit.py

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: hnyls2002 <lsyincs@gmail.com> Co-authored-by: Liangsheng Yin <hnyls2002@gmail.com>

Co-authored-by: hnyls2002 <lsyincs@gmail.com> Co-authored-by: Liangsheng Yin <hnyls2002@gmail.com>

ishandhanani added 2 commits March 25, 2026 19:19

Fix streaming session open race

e4b93c1

Deduplicate session control responses

0b44676

Add minimal streaming session metrics

511c5fc

ishandhanani marked this pull request as ready for review April 1, 2026 20:39

ishandhanani requested review from Ying1123, fzyzcjy, hanming-lu, hnyls2002, hzh0425, ispobock, merrymercy, sufeng-buaa, xiezhq-hermann and yizhang2077 as code owners April 1, 2026 20:39

github-actions Bot added the run-ci label Apr 1, 2026

hnyls2002 assigned hnyls2002 and ishandhanani Apr 1, 2026

ishandhanani added 4 commits April 1, 2026 20:51

Gate streaming session metrics by feature flag

57bf457

Merge branch 'main' into ishan/streaming-sesh-better

6301c1c

Add streaming session smoke test

367884b

Remove manual streaming session smoke test

23af40f

ishandhanani and others added 5 commits April 2, 2026 02:19

lint

fb6cd8a

Merge branch 'main' into ishan/streaming-sesh-better

8523d13

Merge branch 'main' into ishan/streaming-sesh-better

0bd710e

hnyls2002 added 8 commits April 12, 2026 13:19

Merge remote-tracking branch 'origin/main' into ishan/streaming-sesh-…

549dfa6

…better # Conflicts: # test/registered/sessions/test_streaming_session.py

black fmt session_controller

9a58df2

rename _active_ to _alive_ streaming session count

0493385

remove unnecessary hasattr guard

a341830

tiny

bfe48c0

rename pending_close to close_on_finish; reject closing session

ba3a047

rename pending_close to close_on_finish; has_active to has_unfinished…

3f5651f

…; reject closing session

simplify close_on_finish guard; always log

fa20196

hnyls2002 added high priority run-ci and removed run-ci labels Apr 12, 2026

Merge branch 'main' into ishan/streaming-sesh-better

f25b35d

fix shrink: clamp cache_protected_len; skip tree-protected tokens

be492e1

disable _resolve_release_state re-match; early return

3127b70

fix is_streaming_session: check req.session.streaming

3483b01

hnyls2002 merged commit c1ab68b into main Apr 13, 2026
145 of 167 checks passed

hnyls2002 deleted the ishan/streaming-sesh-better branch April 13, 2026 01:05

This was referenced Apr 13, 2026

Flaky test: test_patch_torch.py fails ~7% on novita runners (CUDA IPC race condition) #22648

Open

streaming session: spec v2 bonus accounting + comprehensive test matrix #22651

Merged

Delete dead rematch path in SessionAwareCache.release_session #22735

Merged

yhyang201 pushed a commit to yhyang201/sglang that referenced this pull request Apr 22, 2026

fix: streaming session race condition + some metrics (sgl-project#21875)

cd59d1c

Co-authored-by: hnyls2002 <lsyincs@gmail.com> Co-authored-by: Liangsheng Yin <hnyls2002@gmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: streaming session race condition + some metrics#21875

fix: streaming session race condition + some metrics#21875
hnyls2002 merged 41 commits intomainfrom
ishan/streaming-sesh-better

ishandhanani commented Apr 1, 2026 •

edited

Loading

Uh oh!

gemini-code-assist Bot commented Apr 1, 2026

Uh oh!

ishandhanani commented Apr 1, 2026

Uh oh!

gemini-code-assist Bot commented Apr 1, 2026

Uh oh!

ishandhanani commented Apr 1, 2026

Uh oh!

hnyls2002 commented Apr 12, 2026

Uh oh!

github-actions Bot commented Apr 12, 2026

Uh oh!

hnyls2002 commented Apr 12, 2026

Uh oh!

github-actions Bot commented Apr 12, 2026

Uh oh!

hnyls2002 commented Apr 12, 2026

Uh oh!

github-actions Bot commented Apr 12, 2026

Uh oh!

hnyls2002 commented Apr 12, 2026

Uh oh!

github-actions Bot commented Apr 12, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

ishandhanani commented Apr 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Race conditions & lifecycle

KV memory leak fixes

Observability

Test plan

Uh oh!

gemini-code-assist Bot commented Apr 1, 2026

Uh oh!

ishandhanani commented Apr 1, 2026

Uh oh!

gemini-code-assist Bot commented Apr 1, 2026

Uh oh!

ishandhanani commented Apr 1, 2026

Uh oh!

hnyls2002 commented Apr 12, 2026

Uh oh!

github-actions Bot commented Apr 12, 2026

Uh oh!

hnyls2002 commented Apr 12, 2026

Uh oh!

github-actions Bot commented Apr 12, 2026

Uh oh!

hnyls2002 commented Apr 12, 2026

Uh oh!

github-actions Bot commented Apr 12, 2026

Uh oh!

hnyls2002 commented Apr 12, 2026

Uh oh!

github-actions Bot commented Apr 12, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ishandhanani commented Apr 1, 2026 •

edited

Loading