perf(napi/parser): do not generate tokens except in tests#21811
Merged
graphite-app[bot] merged 1 commit intomainfrom Apr 27, 2026
Merged
Conversation
Member
Author
How to use the Graphite Merge QueueAdd either label to this PR to merge it via the merge queue:
You must have a Graphite account in order to use the merge queue. Sign up using this link. An organization admin has enabled the Graphite Merge Queue in this repository. Please do not merge from GitHub as this will restart CI on PRs being processed by the merge queue. This stack of pull requests is managed by Graphite. Learn more about stacking. |
Member
Author
Merge activity
|
Contributor
There was a problem hiding this comment.
Pull request overview
This PR makes token generation in napi/parser opt-in via a Cargo feature to avoid unnecessary runtime work and (more importantly) prevent downstream binary-size bloat caused by compiling the parser twice with different configs.
Changes:
- Introduces an optional Cargo feature (
tokens) that pulls inoxc_estree_tokensonly when enabled. - Gates token-related parsing/configuration and raw-transfer token conversion behind
#[cfg(feature = "tokens")]. - Enables the
tokensfeature only for thebuild-testscript so conformance tests can still validate tokens.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
napi/parser/src/raw_transfer.rs |
Guards token conversion/imports behind the tokens feature; defaults to (0,0) offsets when disabled. |
napi/parser/src/lib.rs |
Uses the default no-tokens parser config unless the tokens feature is enabled. |
napi/parser/package.json |
Builds tests with --features tokens so token snapshots/conformance tests continue to work. |
napi/parser/Cargo.toml |
Makes oxc_estree_tokens optional and adds a tokens feature that enables it. |
#19885 added support for tokens in `oxc-parser`, but only via raw transfer and behind an undocumented `experimentalTokens` option. It's currently only used in tests for the purpose of conformance testing. This slowed down `oxc-parser` a little (very little as the difference in execution time on Rust side is dwarfed by the cost of JSON transfer). The worse effect is that it caused a large binary size increase in Rolldown (rolldown/rolldown#9166) because Rolldown re-exports `napi/parser`'s Rust code, which meant that Rolldown now contains 2 versions of Oxc's parser with different `ParserConfig`s, leading to the entire parser being compiled and included in the binary twice. Fix this by introducing a Cargo feature `tokens` on `napi/parser` crate. Only when `tokens` feature is enabled is a `ParserConfig` which generates tokens used. When disabled (the default), the default `ParserConfig` which does not produce tokens is used. Enable the `tokens` feature only in tests.
727655b to
9fa362e
Compare
camc314
pushed a commit
that referenced
this pull request
Apr 27, 2026
### 💥 BREAKING CHANGES - 502e804 ast: [**BREAKING**] Reduce size of `TSTypePredicateName` (#21711) (overlookmotel) - 5651539 ast: [**BREAKING**] Reduce size of `JSXExpression` (#21710) (overlookmotel) - c44e280 ast: [**BREAKING**] Reduce size of `ArrayExpressionElement` (#21709) (overlookmotel) - c5b3deb syntax: [**BREAKING**] Remove `CommentNodeId` (#21679) (overlookmotel) ### 🚀 Features - b738a39 allocator: Add `Allocator::cursor_ptr` method (#21773) (overlookmotel) - 678767e ast: Generate node_id accessors for AST enum wrappers (#21653) (camc314) - f091d77 minifier: Inline constant spread elements into arrays (#21095) (Armano) ### 🐛 Bug Fixes - 0d608c2 minifier: Preserve raw CR in template literals (#21645) (Dunqing) - a889ea9 minifier: Track pure functions in DCE mode (#21722) (Dunqing) - 674dfac allocator: `Arena` retry allocation when chunk size approaches maximum (#21777) (overlookmotel) - f130cc0 allocator: Fix arithmetic overflow in `Arena::new_chunk_memory_details` (#21745) (overlookmotel) - b9bf239 allocator: Fix UB in `Arena::grow_zeroed` (#21739) (overlookmotel) - d2b9389 allocator: Clippy warning when building without `testing` feature (#21681) (camc314) - 503dc86 codegen: Map sourcemaps from visible output starts (#21662) (Dunqing) - c92bd3b transformer: Use SPAN for synthesized helper calls to prevent comment misattribution (#21578) (Dunqing) - 0d80441 codegen: Add mapping before printing `#` for private ident (#21619) (camc314) ### ⚡ Performance - 9fa362e napi/parser: Do not generate tokens except in tests (#21811) (overlookmotel) - 0044392 allocator: Reduce branches when allocating new chunk (#21776) (overlookmotel) - 7896bd0 allocator: `Allocator::used_bytes` do not use chunk iterator (#21771) (overlookmotel) - a5c562f allocator: Remove check in `Arena::new_chunk_memory_details` (#21750) (overlookmotel) - 35bbe1f allocator: `Arena` use unchecked size round up where guaranteed no overflow (#21743) (overlookmotel) - ffe229b allocator: Remove unnecessary check from `Arena::try_alloc_layout_slow_impl` (#21732) (overlookmotel) - 72fece5 allocator: Use `NonNull::offset_from_unsigned` in `Arena::chunk_capacity` (#21731) (overlookmotel) - cab32ae ast: Add `#[inline(always)]` to `node_id` methods on enums with all variants unboxed (#21707) (overlookmotel) - b179688 parser: Allocate `TriviaBuilder` comments in the arena (#21512) (Boshen) - 2290f31 lexer: Fix perf of `Token::set_*` methods on Rust 1.95.0 (#21659) (overlookmotel) - 1b58029 allocator: Move code into cold path in `Arena::alloc_layout` (#21622) (overlookmotel) - 3cf7cef allocator: Reduce instructions on allocation hot path (#21510) (overlookmotel) ### 📚 Documentation - ce65070 data_structures: Document why `as_ref` and `as_mut` on `NonNullConst` and `NonNullMut` take `self` (#21800) (overlookmotel) - 93b7dbd allocator: Improve doc comments for `ChunkFooter` (#21733) (overlookmotel) - 295db8d transformer: Fix comment (#21717) (overlookmotel) - 5c93af8 ast: Add comments explaining `#[inline(always)]` to `node_id` methods on enums (#21706) (overlookmotel) - e4cea25 transform: Use the `node:` namespace in the example (#19998) (루밀LuMir) ### 🛡️ Security - d8076c9 deps: Update rolldown (#21639) (renovate) Co-authored-by: Boshen <1430279+Boshen@users.noreply.github.com>
Member
Author
|
Seems like this didn't have the desired effect in Rolldown: rolldown/rolldown#9166 (comment) |
Contributor
I think this had a positive effect. but there are still other issues ther. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
#19885 added support for tokens in
oxc-parser, but only via raw transfer and behind an undocumentedexperimentalTokensoption. It's currently only used in tests for the purpose of conformance testing.This slowed down
oxc-parsera little (very little as the difference in execution time on Rust side is dwarfed by the cost of JSON transfer).The worse effect is that it caused a large binary size increase in Rolldown (rolldown/rolldown#9166) because Rolldown re-exports
napi/parser's Rust code, which meant that Rolldown now contains 2 versions of Oxc's parser with differentParserConfigs, leading to the entire parser being compiled and included in the binary twice.Fix this by introducing a Cargo feature
tokensonnapi/parsercrate. Only whentokensfeature is enabled is aParserConfigwhich generates tokens used. When disabled (the default), the defaultParserConfigwhich does not produce tokens is used.Enable the
tokensfeature only in tests.