The documentation states that "a sequence of just the space character ' '" is treated as a word, but this is incomplete. The implementation on line 104-108 treats any sequence of consecutive spaces as a single token. The documentation should clarify that multiple consecutive spaces are grouped together as one token.

The documentation mentions "a sequence of alphanumeric characters as determined by char::is_alphanumeric" but the implementation on lines 110-113 also includes underscores (_) as part of alphanumeric words. This is inconsistent with the documentation. Either update the documentation to mention that underscores are included in alphanumeric sequences, or remove the special handling of underscores if they should be treated as separate tokens.

The word_diff API requires callers to pass mutable references to diff_input and diff that are populated by this function. This design is not intuitive and error-prone because:

1. Callers must create and maintain these objects themselves
2. The function's return type is (), making it unclear that it populates the diff parameter
3. The relationship between the parameters is not obvious from the signature

Consider either:

• Returning the Diff as the function's result
• Using a builder pattern with a dedicated WordDiff type
• Adding documentation that clearly explains the expected usage pattern and the relationship between parameters

While copilot is right in principle, I think compete documentation on parameters could explain why they need to be mutable.
Docs are definitely needed here.

The variable name 'd' is too short and unclear. Consider using a more descriptive name like 'word_diff' or 'inner_diff' to make the test more readable and to clarify that this is a separate diff object from the line-level diff.

The variable name 'd' is too short and unclear. Consider using a more descriptive name like 'word_diff' or 'inner_diff' to make the test more readable and to clarify that this is a separate diff object from the line-level diff.

The test variables diff_input and d are declared once outside the algorithm loop but are reused across all algorithm iterations. While word_diff appears to clear and resize the vectors, this pattern could hide bugs if word_diff doesn't properly reset all state. Consider either:

1. Moving these declarations inside the loop to ensure a fresh state for each algorithm
2. Adding an explicit reset/clear call between iterations to make the intent clear

The token estimation heuristic divides string length by 3 (assuming average word length of 3 characters). This could result in poor allocation sizing:

1. For typical English text, average word length is closer to 4-5 characters when including spaces and punctuation
2. For code with long identifiers, the average could be much higher
3. The estimate doesn't account for the fact that each punctuation character becomes its own token

Consider using a more conservative estimate like (self.0.len() / 5) or implementing a sampling approach similar to ByteLines::estimate_tokens() that examines the first portion of the text to calculate the actual average.

I don't know about this one, but it seems that heuristics aren't always right and maybe there is a way to make it configurable?
Maybe this word-diff is also so tuned to Latin text that we might say it in the function, i.e. word_diff to latin_word_diff.

The word_diff method doesn't clear the interner between calls, which could lead to memory accumulation if called repeatedly. According to the documentation of update_before and update_after, "this does not erase any tokens from the interner and might therefore be considered a memory leak."

While the comment mentions clearing the interner for long-running processes, for a public API like word_diff, it would be better to either:

1. Clear the interner at the start of this method
2. Document this behavior clearly in the word_diff documentation
3. Provide a way for callers to manage this explicitly

The test variables diff_input and d are declared once outside the algorithm loop but are reused across all algorithm iterations. While word_diff appears to clear and resize the vectors, this pattern could hide bugs if word_diff doesn't properly reset all state. Consider either:

1. Moving these declarations inside the loop to ensure a fresh state for each algorithm
2. Adding an explicit reset/clear call between iterations to make the intent clear

Maybe it's a feature to let state accumulate without resetting it to ensure the call can handle it.