[HiCache] fix: release write-through lock_ref during decode by alphabetc1 · Pull Request #20049 · sgl-project/sglang

alphabetc1 · 2026-03-06T17:17:38Z

Motivation

Root cause: check_hicache_events() was only called in the prefill path (_get_new_batch_prefill_raw). During pure decode loops, write_through operations locked radix-tree nodes via inc_lock_ref(), but the acks (which call dec_lock_ref()) were never processed. This caused all evictable tokens to become non-evictable (protected), eventually leading to 'Out of memory even after retracting all other requests'.

Fix: Add flush_write_through_acks() to BasePrefixCache (no-op default), override in HiRadixCache to call writing_check(), and call it every decode iteration in update_running_batch() before check_decode_mem().

Timeline(before fix)

T1: req1 + req2 arrive, prefill completes.

T2: req1 decode finishes

cache_finished_req → insert → _inc_hit_count
write_backup(node_A)
inc_lock_ref(node_A) [write-through lock acquired]

T3: req2 decode continues

write-through for node_A completes asynchronously
ack is pushed into ack_write_queue
but writing_check() is never called
Result: node_A.lock_ref stays at 1 → node_A is non-evictable

T4: req2 decode finishes

same pattern for node_B (write_backup → inc_lock_ref(node_B))

T5: No new requests → pure decode loop

_get_new_batch_prefill_raw returns early
check_hicache_events is never reached
Result: acks are not processed → lock_ref is never released
Risk: OOM under memory pressure

Timeline(after fix)

T1: req1 + req2 arrive, prefill completes.

T2: req1 decode finishes

write_backup(node_A)
inc_lock_ref(node_A)

T3: Next req2 decode iteration

update_running_batch
flush_write_through_acks
- writing_check
- dec_lock_ref(node_A) [write-through lock released promptly]

T4: req2 decode finishes

write_backup(node_B)
inc_lock_ref(node_B)

T5: Next prefill (or next decode iteration)

writing_check
dec_lock_ref(node_B) [lock released]

Modifications

Accuracy Tests

Benchmarking and Profiling

Checklist

Format your code according to the Format code with pre-commit.
Add unit tests according to the Run and add unit tests.
Update documentation according to Write documentations.
Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
Follow the SGLang code style guidance.

Review Process

Ping Merge Oncalls to start the PR flow. See the PR Merge Process.
Get approvals from CODEOWNERS and other reviewers.
Trigger CI tests with comments or contact authorized users to do so.
- /tag-run-ci-label, /rerun-failed-ci, /tag-and-rerun-ci
After green CI and required approvals, ask Merge Oncalls to merge.

gemini-code-assist · 2026-03-06T17:20:59Z

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request resolves a critical memory management issue within the HiCache system. Previously, write-through operations could lead to a gradual accumulation of locked cache nodes, preventing their eviction and ultimately causing out-of-memory conditions during continuous decode operations. The fix ensures that these locks are properly released as soon as the write-through acknowledgments are processed, significantly improving memory efficiency and system stability.

Highlights

Memory Leak Fix: Addressed a memory leak in HiCache where write-through operation locks (lock_ref) were not being released during pure decode loops, leading to nodes becoming non-evictable and eventual 'Out of memory' errors.
Write-Through Lock Release: Introduced a mechanism to eagerly release write-through locks by calling flush_write_through_acks() in update_running_batch during decode iterations, ensuring that completed write-through nodes become evictable promptly.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

python/sglang/srt/managers/scheduler.py
- Added a conditional call to self.tree_cache.flush_write_through_acks() within the update_running_batch method to eagerly release write-through locks.
- Included a comment explaining the purpose of the new call: to make nodes evictable and improve batch scheduling headroom.
python/sglang/srt/mem_cache/base_prefix_cache.py
- Defined a new abstract method flush_write_through_acks with a default no-op implementation.
- Provided a docstring for flush_write_through_acks explaining its role in releasing lock_ref on radix-tree nodes after write-through completion.
python/sglang/srt/mem_cache/hiradix_cache.py
- Implemented the flush_write_through_acks method to call self.writing_check(), thereby processing write-through acknowledgments.
python/sglang/srt/mem_cache/session_aware_cache.py
- Added a passthrough implementation for the flush_write_through_acks method, delegating the call to the inner cache instance.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request effectively resolves a memory leak in HiCache by ensuring that write-through lock references are released during pure decode loops. The core of the fix is the introduction of flush_write_through_acks(), which is now called in every decode iteration. This prevents tokens from becoming permanently non-evictable and avoids potential out-of-memory errors. The implementation is clean, with the new method correctly added to the BasePrefixCache and its relevant subclasses. The changes are well-targeted and appear to be a robust solution to the identified problem.

_{Note: Security Review did not run due to the size of the PR.}

alphabetc1 · 2026-03-07T11:29:40Z

/tag-and-rerun-ci

alphabetc1 · 2026-03-10T02:52:29Z

TODO: Try to call dec_lock_ref before retract whenever possible.

alphabetc1 · 2026-03-10T12:00:53Z

After several tests, no measurable performance regression was observed from adding flush_write_through_acks to the decode path.

alphabetc1 · 2026-03-10T12:09:29Z

/rerun-failed-ci 11

ispobock · 2026-03-13T15:39:46Z

we may also need to add it for HiMambaRadixCache

alphabetc1 · 2026-03-13T18:31:51Z

we may also need to add it for HiMambaRadixCache

done

alphabetc1 · 2026-03-15T12:21:02Z

/rerun-failed-ci

…ect#20049)

[HiCache] fix: release write-through lock_ref during decode

7e820fe

alphabetc1 requested review from Ying1123, hanming-lu, hnyls2002, merrymercy, xiezhq-hermann and yizhang2077 as code owners March 6, 2026 17:17

gemini-code-assist Bot reviewed Mar 6, 2026

View reviewed changes

github-actions Bot added the run-ci label Mar 7, 2026

alphabetc1 mentioned this pull request Mar 7, 2026

fix: gracefully abort last request in retract_decode on OOM #19881

Merged

5 tasks

xiezhq-hermann added the high priority label Mar 10, 2026

xiezhq-hermann self-assigned this Mar 10, 2026

xiezhq-hermann approved these changes Mar 10, 2026

View reviewed changes

Merge branch 'main' into fix/hicache-write-through-lock-ref-leak

3b27e36

alphabetc1 requested review from hzh0425 and ispobock as code owners March 12, 2026 08:12

alphabetc1 added 3 commits March 12, 2026 19:22

Merge branch 'main' into fix/hicache-write-through-lock-ref-leak

88b0504

Merge branch 'main' into fix/hicache-write-through-lock-ref-leak

b8c6dff

Merge branch 'main' into fix/hicache-write-through-lock-ref-leak

f5a73e2

fix HiMambaRadixCache

5fd450a

Weili-0234 mentioned this pull request Mar 15, 2026

[HiCache] Fix illegal memory access caused by stream race between FA3 forward and D2H transfer #20611

Open

5 tasks

ispobock approved these changes Mar 16, 2026

View reviewed changes

ispobock merged commit 42f18fe into sgl-project:main Mar 16, 2026
246 of 274 checks passed

alphabetc1 deleted the fix/hicache-write-through-lock-ref-leak branch March 16, 2026 06:52

lawrence-harmonic added a commit to lawrence-harmonic/sglang that referenced this pull request Mar 19, 2026

backport sgl-project#20049 (sgl-project#9)

fb30ec9

Wangzheee pushed a commit to Wangzheee/sglang that referenced this pull request Mar 21, 2026

[HiCache] fix: release write-through lock_ref during decode (sgl-proj…

69ba533

…ect#20049)

0-693 pushed a commit to 0-693/sglang that referenced this pull request Mar 25, 2026

[HiCache] fix: release write-through lock_ref during decode (sgl-proj…

bec6049

…ect#20049)

JustinTong0323 pushed a commit to JustinTong0323/sglang that referenced this pull request Apr 7, 2026

[HiCache] fix: release write-through lock_ref during decode (sgl-proj…

c7510f0

…ect#20049)

yhyang201 pushed a commit to yhyang201/sglang that referenced this pull request Apr 22, 2026

[HiCache] fix: release write-through lock_ref during decode (sgl-proj…

3251680

…ect#20049)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[HiCache] fix: release write-through lock_ref during decode#20049

[HiCache] fix: release write-through lock_ref during decode#20049
ispobock merged 6 commits intosgl-project:mainfrom
alphabetc1:fix/hicache-write-through-lock-ref-leak

alphabetc1 commented Mar 6, 2026 •

edited

Loading

Uh oh!

gemini-code-assist Bot commented Mar 6, 2026

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

alphabetc1 commented Mar 7, 2026

Uh oh!

alphabetc1 commented Mar 10, 2026

Uh oh!

alphabetc1 commented Mar 10, 2026

Uh oh!

alphabetc1 commented Mar 10, 2026 •

edited

Loading

Uh oh!

ispobock commented Mar 13, 2026

Uh oh!

alphabetc1 commented Mar 13, 2026

Uh oh!

alphabetc1 commented Mar 15, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

alphabetc1 commented Mar 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Timeline(before fix)

Timeline(after fix)

Modifications

Accuracy Tests

Benchmarking and Profiling

Checklist

Review Process

Uh oh!

gemini-code-assist Bot commented Mar 6, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

alphabetc1 commented Mar 7, 2026

Uh oh!

alphabetc1 commented Mar 10, 2026

Uh oh!

alphabetc1 commented Mar 10, 2026

Uh oh!

alphabetc1 commented Mar 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ispobock commented Mar 13, 2026

Uh oh!

alphabetc1 commented Mar 13, 2026

Uh oh!

alphabetc1 commented Mar 15, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

alphabetc1 commented Mar 6, 2026 •

edited

Loading

alphabetc1 commented Mar 10, 2026 •

edited

Loading