ruff-ecosystem results

Linter (stable)

✅ ecosystem check detected no linter changes.

Linter (preview)

✅ ecosystem check detected no linter changes.

Formatter (stable)

✅ ecosystem check detected no format changes.

Formatter (preview)

✅ ecosystem check detected no format changes.

Merging this PR will not alter performance

✅ 123 untouched benchmarks

_{Comparing micha/view-benchmark-token-capacity (1e4aafa) with main (258ca11)}

ecosystem-analyzer results

No diagnostic changes detected ✅

Full report with detailed diff (timing results)

Memory usage report

Summary

Significant changes

I think another issue here is that the source that's using the approximate the capacity is always the complete source text, even when the start_offset is a non-zero value. This means that the linter will always over-allocation for multiple cases like stringified annotations, specific rule based lexing, etc. Could we try to take start_offset into account when computing the approximate capacity and see if that reduces / fixes the regression? Or, do you think this shouldn't affect it?

Edit: tried it in #25382, it seems to at least fix the linter benchmark

(Oh nice, that's a good catch.)

I updated the heuristic to reduce the instances where we allocate too much storage. I really like codex explanation here, which is why I copy paste some of it:

The optimization preallocates the parser's token buffer using only the remaining source length, without scanning source contents: This is intentionally an underestimate. The goal is not to predict the final token count precisely; it is to skip several small early Vec growth steps while avoiding the memory regressions caused by over-reserving.

Across 28,758 Python files sampled from Ruff ecosystem projects, a len / 8 estimate is at or below the actual stored-token count for approximately 77.5% of files. That makes it a useful conservative estimate: for most files, the buffer still grows if necessary, but starts far enough ahead to avoid several early reallocations.

Why underestimate?
Overestimating can make peak memory worse. A token-sparse file containing long strings, comments, or docstrings may have many source bytes but few tokens. If an arbitrary estimate reserves too much capacity, the parser can allocate a larger buffer than ordinary Vec growth would ever have reached.

Underestimating is cheaper: the buffer simply performs its normal later growth steps. We still receive most of the allocation reduction without turning uncommon sparse files into systematic memory regressions.

Why the minimum capacity cutoff?

For small files, preallocation has limited value. Reserving fewer than 128 tokens only skips small, cheap growth steps, while increasing the number of small files whose initial allocation exceeds their ordinary final capacity.

Using a 128-token cutoff keeps small files on normal Vec growth and concentrates the optimization on files where it saves meaningful work.

Why round down with ilog2()?
The power-of-two rounding aligns the initial capacity with the capacity buckets produced by geometric Vec growth.
For example, a file with 748 tokens normally grows to a capacity of 1024. An arbitrary initial capacity of 660 grows to 1320, increasing peak memory. Rounding the estimate down to 512 instead produces:

normal growth:       ... -> 512 -> 1024
preallocated growth: 512 -> 1024

The allocation starts later, but the final capacity remains in the same bucket as normal growth.