refactor(estree/tokens): convert token kinds to ESTreeKind#19875
Merged
graphite-app[bot] merged 1 commit intomainfrom Mar 1, 2026
Merged
Conversation
This was referenced Mar 1, 2026
Member
Author
This was referenced Mar 1, 2026
Merging this PR will degrade performance by 9.57%
Performance Changes
Comparing 
Footnotes
|
a0c0e12 to
95e7606
Compare
95e7606 to
fe2d845
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
This PR refactors the raw transfer token processing in oxc_estree_tokens to pre-convert all token kinds to a compact ESTreeKind enum (12 variants) before sending to the JS side, instead of leaving the full Kind enum (169 variants) for JS to handle.
Changes:
- Introduces a new
ESTreeKindenum with 12 variants mirroring ESTree token types, plus aCONVERSION_TABLEmapping all 169Kindvariants to theirESTreeKindequivalents. - Updates
RawContext::advance_toandfinishto eagerly convert skipped tokens' kinds toESTreeKinddiscriminants stored asKindbyte values, and updates all explicit emit methods to useESTreeKindinstead ofKind. - Removes
Kind::JSXIdentifierfrom the lexer'sKindenum (it was only used for ESTree conversion, never produced by the lexer), reducingKind::VARIANTS.len()from 170 to 169.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
crates/oxc_parser/src/lexer/kind.rs |
Removes Kind::JSXIdentifier (no longer needed as a token kind) |
crates/oxc_estree_tokens/src/lib.rs |
Registers the new estree_kind module |
crates/oxc_estree_tokens/src/estree_kind.rs |
New file defining ESTreeKind enum and CONVERSION_TABLE |
crates/oxc_estree_tokens/src/raw_transfer.rs |
Updates token kind handling to use ESTreeKind throughout |
This was referenced Mar 1, 2026
fe2d845 to
cd255b5
Compare
ad5f1b4 to
4efb610
Compare
cd255b5 to
f53737d
Compare
4efb610 to
3edfe90
Compare
Contributor
Merge activity
|
#19856 added an `update_tokens` function to update tokens, ready to be sent over to JS side via raw transfer. Except for tokens where the token `Kind` needs specific modification, this function left token's `Kind`s alone. Unfortunately this means we have to handle all 169 variants of `Kind` on JS side. Instead, convert them to `ESTreeKind`, which has only the 11 variants which ESTree tokens have. This change has negative perf impact on Rust side (-9%), but it improves perf on JS side in return.
3edfe90 to
ee4982b
Compare
f53737d to
449080c
Compare
Base automatically changed from
om/03-01-feat_parser_add_variants_const_to_kind_via_fieldless_enum_macro
to
main
March 1, 2026 17:47
This was referenced Mar 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
#19856 added an
update_tokensfunction to update tokens, ready to be sent over to JS side via raw transfer.Except for tokens where the token
Kindneeds specific modification, this function left token'sKinds alone. Unfortunately this means we have to handle all 169 variants ofKindon JS side.Instead, convert them to
ESTreeKind, which has only the 11 variants which ESTree tokens have.This change has negative perf impact on Rust side (-9%), but it improves perf on JS side in return.