Motivation

For prefill requests, the engine log only shows #queue-req and #running-req, which is insufficient for understanding load — especially for long-context requests where a single request can have hundreds of thousands of tokens pending prefill across multiple chunks.

This PR adds a #pending-token metric that shows the total number of tokens still waiting to be prefilled. This is useful for:

• Monitoring chunked prefill progress in engine logs
• Balancing load across engines when handling long-context requests via /get_load

Modifications

• PrefillStats (scheduler_metrics_mixin.py): Added num_pending_tokens field, snapshotted at batch-scheduling time for correct reporting under the overlap scheduler.
• _get_num_pending_tokens() (scheduler_metrics_mixin.py): New shared helper that computes pending tokens from the waiting queue plus remaining tokens of the currently chunked request. Accepts a chunk_deduct parameter to handle the timing difference between scheduling time (where prefix_indices has not yet been updated) and query time (get_load, where it has).
• Prefill batch log (scheduler_metrics_mixin.py): Added #pending-token: {N} to the log line.
• GetLoadReqOutput (io_struct.py): Added num_pending_tokens field returned by /get_load.
• get_load() (scheduler_metrics_mixin.py): Populates num_pending_tokens using the shared helper.
• get_new_batch_prefill() (scheduler.py): Snapshots num_pending_tokens into PrefillStats at scheduling time.

Example log output with a 30K-token chunked prefill (chunk size 8192):

Prefill batch, #new-seq: 1, #new-token: 8192, ..., #pending-token: 21825
Prefill batch, #new-seq: 1, #new-token: 8192, ..., #pending-token: 13633
Prefill batch, #new-seq: 1, #new-token: 8192, ..., #pending-token: 5441
Prefill batch, #new-seq: 1, #new-token: 5441, ..., #pending-token: 0

Accuracy Tests

N/A — observability-only change, no model output affected.

Speed Tests and Profiling

N/A — adds a lightweight sum over the waiting queue during logging and load queries.

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.