self reminder: remove this print

There's a critical logic error in the chunk splitting implementation. The current logic resets total_seq_lens to 0 after creating a chunk, but fails to account for the sequence length of the request that triggered the split. This causes the next chunk to potentially include too many requests, exceeding max_prefill_buffer_size and triggering the assertion in build_one_prefill_chunk.

For example, with seq_lens_cpu = [1000, 1000, 1000] and max_prefill_buffer_size = 1500, the first chunk correctly contains only the first request. However, the second chunk would be created with the remaining two requests, having a total length of 2000, which is over the limit.

Here is a corrected implementation that properly handles chunk creation and sequence length accounting:

This print statement appears to be for debugging and should be removed to avoid polluting logs in production environments.

The current implementation of split_prefill_chunks has a bug when handling requests that are larger than max_prefill_buffer_size. It can lead to creating empty chunks and incorrect chunking, which will cause issues downstream. For example, with seq_lens_cpu=torch.tensor([10]), max_prefill_buffer_size=5, and reqs_start=0, the function returns [(0, 0), (0, 1)], where the first chunk is empty.

A corrected implementation should handle this edge case gracefully, typically by placing the large request in its own chunk. It would also be beneficial to add a test case for this scenario.

Here is a suggested implementation that fixes this issue:

def split_prefill_chunks(seq_lens_cpu: torch.Tensor,
                         max_prefill_buffer_size: int,
                         reqs_start: int) -> list[tuple[int, int]]:
    """
    Split the prefill chunks into a list of tuples of (reqs_start, reqs_end)
    such that the total sequence length of each chunk is less than the
    maximum prefill buffer size.

    Args:
        seq_lens_cpu: The sequence lengths of the prefill requests.
        max_prefill_buffer_size: The maximum prefill buffer size.
        reqs_start: The start index of the prefill requests.
    
    Returns:
        A list of tuples of (reqs_start, reqs_end).
    """
    chunk_seq_ids = []
    total_seq_lens = 0
    for i in range(reqs_start, len(seq_lens_cpu)):
        cur_seq_len = seq_lens_cpu[i].item()
        if total_seq_lens > 0 and total_seq_lens + cur_seq_len > max_prefill_buffer_size:
            chunk_seq_ids.append((reqs_start, i))
            reqs_start = i
            total_seq_lens = 0
        total_seq_lens += cur_seq_len
    if total_seq_lens > 0:
        chunk_seq_ids.append((reqs_start, len(seq_lens_cpu)))
    return chunk_seq_ids

@heheda12345 I think the bot has a point; we should probably assert there is no empty chunks

This assertion will fail if a single prefill request has a sequence length greater than or equal to max_prefill_buffer_size. The chunking logic should be able to handle such cases by putting the large request into its own chunk, but this assertion prevents that and will cause a crash.

As you noted in the PR description ("I don't have a trace that total_seq_lens < self.max_prefill_buffer_size"), this is indeed a problematic check. It should be removed to allow processing of single large requests.

Prevent prefill chunks from equaling buffer size

The new split_prefill_chunks only starts a new chunk when the accumulated length is strictly greater than max_prefill_buffer_size. However, build_one_prefill_chunk asserts total_seq_lens < self.max_prefill_buffer_size, so a chunk whose total length is exactly the buffer size slips through the splitter and immediately triggers the assertion. Two prefill requests with length max_model_len each will produce a chunk of exactly 2 * max_model_len (the configured buffer), causing an assertion failure and request crash. The split predicate should use >= or the assertion should accept equality so that valid workloads do not abort.

Useful? React with 👍 / 👎.

Store prefill chunk length as int, not tensor

build_one_prefill_chunk keeps total_seq_lens as the tensor returned by seq_lens_cpu[reqs_start:reqs_end].sum() and places it in DeepseekV32IndexerPrefillChunkMetadata. The downstream indexer (sparse_attn_indexer in deepseek_v2.py) uses this field to size temporary buffers (torch.empty([chunk.total_seq_lens, …])). Because the metadata now contains a 0‑D tensor rather than a Python int, those allocations will raise TypeError: empty(): argument 'size' must be tuple of ints whenever a prefill chunk is processed. Convert total_seq_lens to an int (e.g. .item()) before storing it.

Useful? React with 👍 / 👎.