feat(metrics): expose raw KV cache pool token counts as prometheus gauges#22726
Merged
ishandhanani merged 1 commit intomainfrom Apr 14, 2026
Merged
feat(metrics): expose raw KV cache pool token counts as prometheus gauges#22726ishandhanani merged 1 commit intomainfrom
ishandhanani merged 1 commit intomainfrom
Conversation
Contributor
|
Warning You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again! |
Contributor
|
Warning You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again! |
Collaborator
Author
|
/rerun-stage stage-a-test-1-gpu-small |
Collaborator
Author
|
/rerun-stage stage-b-test-1-gpu-small |
Contributor
|
✅ Triggered |
Contributor
|
✅ Triggered |
2eee7f6 to
3edcc0f
Compare
Collaborator
Author
|
/rerun-stage stage-a-test-1-gpu-small |
Collaborator
Author
|
/rerun-stage stage-b-test-1-gpu-small |
Contributor
|
✅ Triggered |
Contributor
|
✅ Triggered |
Collaborator
Author
|
The correct CI passed |
pyc96
pushed a commit
to pyc96/sglang
that referenced
this pull request
Apr 14, 2026
yhyang201
pushed a commit
to yhyang201/sglang
that referenced
this pull request
Apr 22, 2026
ishandhanani
added a commit
to ai-dynamo/dynamo
that referenced
this pull request
May 6, 2026
Closes #8151 (now unblocked: sgl-project/sglang#22726 landed in v0.5.11, which is the new floor after this PR). Adds a "KV Pool Detail" row to deploy/observability/grafana_dashboards/sglang.json with two new panels driven by the gauges added in 0.5.11: * `KV Pool Breakdown (tokens)` — stacked timeseries of `sglang:kv_used_tokens` (locked by running requests), `sglang:kv_evictable_tokens` (radix-cached, reclaimable), and `sglang:kv_available_tokens` (free). The three series sum to <= `sglang:max_total_num_tokens` per the invariant documented in SGLang's metrics_collector.py. * `KV Pool Physical Usage %` — `(1 - kv_available / (kv_available + kv_evictable + kv_used)) * 100`. Captures true pool occupancy including evictable slots, vs. `sglang:token_usage` which excludes them. 90% threshold drawn in red for the "no headroom even after evict" case. The existing `GPU KV Cache Usage %` panel (driven by `sglang:token_usage`) is unchanged — it's still useful as the "bottleneck across full / SWA / mamba pools" view that the new gauges don't replicate. Verified live on a Qwen/Qwen3-0.6B agg worker: all three gauges export at `<system_port>/metrics`, and `kv_available + kv_evictable + kv_used` = `max_total_num_tokens` after a real request.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Expose three Prometheus gauges for the raw KV cache pool token counts:
sglang:kv_available_tokens-- free pool slotssglang:kv_evictable_tokens-- radix-cached, reclaimable slotssglang:kv_used_tokens-- actively pinned slotsMotivation
The existing
sglang:token_usagemetric reports only non-evictable tokens (active requests + pinned sessions). Evictable radix cache nodes are excluded, making the pool appear emptier than it is. This matters for agentic workloads where subagent KV lingers in the radix tree after completion --token_usageshows ~2% while physical GPU memory is 72% consumed.Exposing the raw counts at the most natural granularity lets operators derive any ratio they need in PromQL/Grafana, e.g.:
1 - (kv_available_tokens / (kv_available_tokens + kv_evictable_tokens + kv_used_tokens))kv_evictable_tokens / (kv_available_tokens + kv_evictable_tokens + kv_used_tokens)Changes
metrics_collector.py: Addkv_available_tokens,kv_evictable_tokens,kv_used_tokensfields toSchedulerStats, add Gauges toSchedulerMetricsCollector, log inlog_stats()scheduler_runtime_checker_mixin.py: Plumb raw counts fromPoolStats.update_scheduler_stats()