Motivation

This PR fixes the bench_latency correctness problem.

When the sampler merged into the CUDA graph, the next_token_ids became a tensor with fixed physical address, which can always change mutably.

Modifications

tolist()

Checklist

• Format your code according to the Contributor Guide.
• Add unit tests as outlined in the Contributor Guide.
• Update documentation as needed, including docstrings or example tutorials.