Add Tokens newtype wrapper, TokenKind iterator#11361
Conversation
|
Tokens newtype wrapperTokens newtype wrapper, TokenKind iterator
|
I might possibly need to maintain a Edit: I'm using binary search to get the start and end index. |
| /// Then, the range `4..10` returns an iterator which yields `Name`, `Lpar`, `Rpar`, and | ||
| /// `Colon` token. But, if the given position doesn't match any of the tokens, an empty | ||
| /// iterator is returned. | ||
| pub fn kinds_within_range<T: Ranged>(&self, ranged: T) -> TokenKindIter { |
There was a problem hiding this comment.
So, this isn't currently being used anywhere. This would basically replace the usages of lex_starts_at but it turns out all of the usages of the function is being done in the AST checker where we don't have access to the token stream.
One solution here would be to store a TokenKinds struct which contains Vec<(TokenKind, TextRange)> on the Checker. This way the rules which utilizes the lex_starts_at or lex function can still get the tokens.
Another would be to club this change with the parser. I'm leaning more towards this.
Summary
Alternative to #11237
This PR adds a new
Tokensstruct which is a newtype wrapper around a vector of lexer output. This allows us to add akindsmethod which returns an iterator over the correspondingTokenKind. This iterator is implemented as a separateTokenKindIterstruct to allow using the type and provide additional methods likepeekdirectly on the iterator.This exposes the linter to access the stream of
TokenKindinstead ofTok.Edit: I've made the necessary downstream changes and plan to merge the entire stack at once.