Skip to content

integrate streaming session into UnifiedRadixCache#23145

Merged
hnyls2002 merged 4 commits intomainfrom
lsyin/unified-streaming-session
Apr 20, 2026
Merged

integrate streaming session into UnifiedRadixCache#23145
hnyls2002 merged 4 commits intomainfrom
lsyin/unified-streaming-session

Conversation

@hnyls2002
Copy link
Copy Markdown
Collaborator

@hnyls2002 hnyls2002 commented Apr 18, 2026 •
edited
Loading

Summary

  • Integrate streaming-session support natively into UnifiedRadixCache via embedded SessionAwareCache
  • SessionAwareCache exposes a uniform try_* contract shared by external-wrapper mode (RadixCache/SWARadixCache/...) and embedded mode (UnifiedRadixCache composes SessionAwareCache(inner=self)) -- zero duplication between the two paths

Changes

BasePrefixCache

  • Add default supports_streaming_session(), release_session(), and session_held_*() no-op methods so callers (scheduler, runtime checker) don't need isinstance(tree_cache, SessionAwareCache) guards

UnifiedRadixCache

  • Compose SessionAwareCache(inner=self) when enable_streaming_session is set; override supports_streaming_session() to return True
  • match_prefix / inc_lock_ref / dec_lock_ref / cache_finished_req pre-check session state via try_*, falling through to the normal radix path otherwise (no recursion -- preconditions prevent re-entry)

Scheduler

  • Skip external SessionAwareCache wrap when tree_cache.supports_streaming_session() is already True
  • Drop isinstance(tree_cache, SessionAwareCache) checks in scheduler_runtime_checker_mixin -- calls now dispatch through the base class

gemini-code-assist[bot]
gemini-code-assist Bot reviewed Apr 18, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors the streaming session implementation by integrating SessionAwareCache directly into UnifiedRadixCache via composition. This change simplifies the cache management logic by allowing session-related operations to be called directly on the cache object, removing the need for explicit type checks in the scheduler and session controller. Additionally, the BasePrefixCache now includes default implementations for streaming session methods. I have no feedback to provide.

Base automatically changed from lsyin/session-subfolder to main April 20, 2026 00:34
@hnyls2002
Merge remote-tracking branch 'origin/main' into lsyin/unified-streami…
4388a73 
…ng-session

# Conflicts:
#	python/sglang/srt/session/session_controller.py
@hnyls2002
Copy link
Copy Markdown
Collaborator Author

hnyls2002 commented Apr 20, 2026

/rerun-test test/registered/unit/mem_cache/test_streaming_session_unit.py test/registered/unit/mem_cache/test_unified_radix_cache_unittest.py test/registered/sessions/test_streaming_session.py test/registered/sessions/test_streaming_session_swa.py test/registered/sessions/test_session_control.py test/registered/sessions/test_session_latency.py

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Apr 20, 2026

✅ ubuntu-latest (1 test): View workflow run

cd test/ && python3 registered/unit/mem_cache/test_streaming_session_unit.py

✅ 1-gpu-5090 (1 test): View workflow run

cd test/ && python3 registered/unit/mem_cache/test_unified_radix_cache_unittest.py

✅ 1-gpu-h100 (4 tests): View workflow run

cd test/ && python3 registered/sessions/test_streaming_session.py
cd test/ && python3 registered/sessions/test_streaming_session_swa.py
cd test/ && python3 registered/sessions/test_session_control.py
cd test/ && python3 registered/sessions/test_session_latency.py

@hnyls2002 hnyls2002 mentioned this pull request Apr 20, 2026
Merged
@hnyls2002 hnyls2002 merged commit a7276b6 into main Apr 20, 2026
108 of 148 checks passed
@hnyls2002 hnyls2002 deleted the lsyin/unified-streaming-session branch April 20, 2026 03:47
@ispobock ispobock mentioned this pull request Apr 21, 2026
Open
23 tasks
zhangying098 pushed a commit to zhangying098/sglang that referenced this pull request Apr 23, 2026
@hnyls2002
integrate streaming session into UnifiedRadixCache (sgl-project#23145)
1a2f284
kyx1999 pushed a commit to KMSorSMS/sglang that referenced this pull request Apr 27, 2026
@hnyls2002 @kyx1999
integrate streaming session into UnifiedRadixCache (sgl-project#23145)
d32c020
@hnyls2002 hnyls2002 mentioned this pull request Apr 29, 2026
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@merrymercy merrymercy Awaiting requested review from merrymercy merrymercy is a code owner

@Ying1123 Ying1123 Awaiting requested review from Ying1123 Ying1123 is a code owner

@xiezhq-hermann xiezhq-hermann Awaiting requested review from xiezhq-hermann xiezhq-hermann is a code owner

@hanming-lu hanming-lu Awaiting requested review from hanming-lu hanming-lu is a code owner

@yizhang2077 yizhang2077 Awaiting requested review from yizhang2077 yizhang2077 is a code owner

@hzh0425 hzh0425 Awaiting requested review from hzh0425 hzh0425 is a code owner

@ispobock ispobock Awaiting requested review from ispobock ispobock is a code owner

1 more reviewer

@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

high priority run-ci

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@hnyls2002