I think we shouldn't be putting them all into server args, but to have a default value and maybe configure through environment variables or a json file.

Good idea! Putting everything into server args is indeed a bit verbose. I’m going to revise it and push a new commit.

can you elaborate what this loop count is for? it actually feels quite similar to what the queuing time is already doing

can you elaborate what this loop count is for? it actually feels quite similar to what the queuing time is already doing

@xiezhq-hermann This is meant to capture something slightly different from queuing time.

• Queuing time = how long a request waits before prefill starts.
• Prefill loop count = how many rounds a request goes through during prefill.
  The main purpose is to check if our chunking configuration is reasonable. For example, in some deployment scenarios, if most prefill requests go through multiple rounds, it may indicate that the chunked_prefill_size could be increased to reduce prefill rounds.

I am a bit confused, when a certain request got chunked, the next chunk will also come from this request, which means the number of rounds of a request got processed when chunking happens is basically length / chunk size, which can be easily analysed with input length and the chunk size instead of recording a runtime metrics.

@xiezhq-hermann I agree with your point that in theory the chunk count can be derived as length / chunk_size. However, since there’s currently no metric that reflects the actual chunk_size, when I need to compute the request’s chunk count I would have to hardcode the chunk size in the expression. That makes it less flexible and not aligned with the real deployment setup, where the chunk size may vary. Having a runtime metric is therefore more convenient and more direct in practice.

I think we can do the environment variable reading inside the collector.py

I think we can do the environment variable reading inside the collector.py

good point

is this to measure the time taken for eviction to complete? is it primarily for understanding the eviction overhead or other purpose I am not aware here. This overhead will vary a lot based on the tokens required for eviction, simply recording the time probably wouldn't make too much sense

is this to measure the time taken for eviction to complete? is it primarily for understanding the eviction overhead or other purpose I am not aware here. This overhead will vary a lot based on the tokens required for eviction, simply recording the time probably wouldn't make too much sense

@xiezhq-hermann You make a good point.
My initial idea was mainly to have a quick observable signal in cases where the system slows down noticeably, we can immediately see whether eviction or write-back is taking up significant time.
I think we could enhance the story by also recording evicted/write-back tokens and total bytes. With these, we could compute average time per token and effective eviction throughput, which would give a more complete picture of the overhead.
What do you think?

I think recording number of tokens got evicted will be a helpful metrics to understand memory activity. e.g., we can plot the # evicted tokens over time to see the peak and low, etc.

overhead of eviction might not be something as interesting, but I can see that to be useful though. we are pushing for replacing hiradix tree by a cpp version, and then this overhead might be more insignificant.

@xiezhq-hermann I agree — adding the number of evicted tokens is a good idea, I’ll include that metric.
As for the eviction time, while it may become less relevant once the c++ version is in place, I think it’s still useful at this stage. In our case, we may also develop a custom distributed storage backend for the hiradix tree, and having this metric framework in place will make it easier to extend with additional metrics we need later. That said, if you think the eviction time metric is unnecessary for the community, I can drop it.