refactor(semantic): migrate temp semantic state to allocator scratch arena#19134
Closed
Boshen wants to merge 2 commits into
Closed
refactor(semantic): migrate temp semantic state to allocator scratch arena#19134Boshen wants to merge 2 commits into
Boshen wants to merge 2 commits into
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Refactors semantic analysis to allocate temporary, per-build state into a new allocator “scratch” arena, and updates the public construction pattern of SemanticBuilder to require an &Allocator across the repo.
Changes:
- Added scratch-arena constructors to
oxc_allocator::Vecandoxc_allocator::HashMap, and extendedAllocatorwith an internal scratchBumpplus reset behavior. - Migrated semantic temporary state (unresolved refs stack, hoisting vars, module instance cache) to scratch-backed arena containers.
- Updated
SemanticBuilder::newto accept&Allocatorand migrated call sites across crates/apps/napi/tasks; adjusted mangler helpers to borrowselffor allocator reuse.
Reviewed changes
Copilot reviewed 48 out of 49 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| tasks/transform_checker/src/lib.rs | Update SemanticBuilder::new call to pass &Allocator. |
| tasks/track_memory_allocations/src/lib.rs | Pass allocator into semantic build to match new constructor. |
| tasks/minsize/src/lib.rs | Pass allocator into semantic build to match new constructor. |
| tasks/benchmark/benches/transformer.rs | Bench updated for new SemanticBuilder::new(&allocator) API. |
| tasks/benchmark/benches/semantic.rs | Bench updated for new SemanticBuilder::new(&allocator) API. |
| tasks/benchmark/benches/pipeline.rs | Bench updated for new SemanticBuilder::new(&allocator) API. |
| tasks/benchmark/benches/minifier.rs | Bench updated for new SemanticBuilder::new(&allocator) API. |
| tasks/benchmark/benches/linter.rs | Bench updated for new SemanticBuilder::new(&allocator) API. |
| tasks/benchmark/benches/codegen.rs | Bench updated for new SemanticBuilder::new(&allocator) API. |
| napi/transform/src/transformer.rs | N-API transform path updated for new semantic builder constructor. |
| napi/playground/src/lib.rs | Thread allocator through semantic build and lint semantic build. |
| napi/parser/src/raw_transfer.rs | Semantic error reporting path updated for new constructor. |
| napi/parser/src/lib.rs | Semantic error reporting path updated for new constructor. |
| crates/oxc_transformer_plugins/tests/integrations/replace_global_defines.rs | Tests updated for new constructor. |
| crates/oxc_transformer_plugins/tests/integrations/inject_global_variables.rs | Tests updated for new constructor. |
| crates/oxc_transformer_plugins/src/module_runner_transform.rs | Test helper updated for new constructor. |
| crates/oxc_transformer_plugins/examples/define.rs | Example updated for new constructor. |
| crates/oxc_transformer/tests/integrations/main.rs | Integration tests updated for new constructor. |
| crates/oxc_transformer/examples/transformer.rs | Example updated for new constructor. |
| crates/oxc_semantic/tests/main.rs | Tests updated for new constructor. |
| crates/oxc_semantic/tests/integration/util/mod.rs | Test utility updated for new constructor. |
| crates/oxc_semantic/src/unresolved_stack.rs | Migrate unresolved stack storage to scratch-backed arena containers. |
| crates/oxc_semantic/src/scoping.rs | Adjust unresolved reference value type to arena Vec with allocator lifetime. |
| crates/oxc_semantic/src/lib.rs | Internal test helper updated for new constructor. |
| crates/oxc_semantic/src/jsdoc/parser/jsdoc_tag.rs | JSDoc semantic test helper updated for new constructor. |
| crates/oxc_semantic/src/jsdoc/parser/jsdoc.rs | JSDoc semantic test helper updated for new constructor. |
| crates/oxc_semantic/src/jsdoc/builder.rs | JSDoc semantic test helper updated for new constructor. |
| crates/oxc_semantic/src/builder.rs | Store allocator ref; migrate temp maps to scratch; route unresolved refs via new helper. |
| crates/oxc_semantic/src/binder.rs | Update hoisting temp state insertion to scratch-backed bindings. |
| crates/oxc_semantic/examples/semantic.rs | Example updated for new constructor. |
| crates/oxc_semantic/examples/cfg.rs | Example updated for new constructor. |
| crates/oxc_minifier/src/lib.rs | Minifier semantic builds updated for new constructor. |
| crates/oxc_minifier/src/compressor.rs | Compressor semantic builds updated for new constructor. |
| crates/oxc_mangler/src/lib.rs | Use temp allocator for semantic build; borrow self to allow reuse. |
| crates/oxc_mangler/src/keep_names.rs | Tests updated for new constructor. |
| crates/oxc_linter/src/utils/jest.rs | Tests updated for new constructor. |
| crates/oxc_linter/src/utils/comment.rs | Tests updated for new constructor. |
| crates/oxc_linter/src/service/runtime.rs | Runtime semantic build updated for new constructor. |
| crates/oxc_linter/src/disable_directives.rs | Tests updated for new constructor. |
| crates/oxc_linter/examples/linter.rs | Example updated for new constructor. |
| crates/oxc_formatter/src/detect_code_removal/mod.rs | Formatter helper updated for new constructor. |
| crates/oxc_allocator/src/vec.rs | Add Vec::new_in_scratch / with_capacity_in_scratch. |
| crates/oxc_allocator/src/hash_map.rs | Add HashMap::*_in_scratch constructors. |
| crates/oxc_allocator/src/allocator.rs | Add scratch Bump, reset handling, and internal scratch_bump accessor. |
| crates/oxc/src/compiler.rs | Thread allocator into compiler semantic step; rebuild scoping with allocator. |
| crates/oxc/README.md | Docs updated to reflect new SemanticBuilder::new(&allocator) usage. |
| apps/oxlint/src/js_plugins/parse.rs | Oxlint plugin parser updated for new constructor. |
CodSpeed Performance ReportMerging this PR will degrade performance by 3.62%Comparing Summary
Performance Changes
Footnotes
|
Dunqing
pushed a commit
that referenced
this pull request
Jun 29, 2026
…ocation (#23927) ## Summary `SemanticBuilder::hoisting_variables` is a nested map `FxHashMap<ScopeId, IdentHashMap<'a, SymbolId>>`. Each `entry(scope_id).or_default()` on a vacant scope heap-allocates a fresh inner `IdentHashMap` via the **system** allocator, and subsequent inserts grow it. On `antd.js` this is the single largest system-allocation site in semantic analysis (`VariableDeclarator::bind` / `Function::bind` in `binder.rs`). This flattens it to a single `FxHashMap<(ScopeId, Ident<'a>), SymbolId>`: - one insert / one point-probe per hoisted variable (was: outer probe → inner-map allocation → inner probe); - no per-scope inner map, so the per-scope system allocations disappear; - one hash instead of two. Behaviour is identical: `hoisting_variables` is only ever inserted into (`binder.rs`) and point-queried in `check_redeclaration` (`builder.rs`) — it is never iterated, so a flat `(scope, name)` key is a drop-in. ## Results — `cargo allocs` (system allocations) | fixture | phase | before | after | | --- | --- | ---: | ---: | | antd.js | semantic | 1020 | **28** (−97%) | | antd.js | minifier | 3041 | **1040** (−66%, rebuilds semantic internally) | | pdf.mjs | semantic | 25 | 16 | | kitchen-sink.tsx | semantic | 41 | 36 | `allocs_semantic.snap` / `allocs_minifier.snap` are updated accordingly. This is on the hot path for tools that run `SemanticBuilder` repeatedly per module (e.g. Rolldown). ## Relation to #19134 #19134 previously tried to move this temporary state (including `hoisting_variables`) to a scratch arena; CodSpeed reported a ~3.6% regression and it was closed. This PR takes a different, minimal approach: it does **not** introduce an arena, does **not** change `SemanticBuilder::new`'s signature, and touches no call sites. It removes a hash and an allocation per hoisted variable (rather than adding arena threading), so it should be CodSpeed-neutral-or-better — please let CI confirm. ## Tests - `cargo test -p oxc_semantic --lib --tests` — 60 passed (incl. redeclaration / var-hoist tests) - `cargo test -p oxc_minifier --lib --tests` — 538 passed ## AI disclosure Prepared with AI assistance (Claude). I reviewed the change, located the hotspot with a per-call-site allocation profile, verified the reduction with `cargo allocs`, and ran the test suites above.
camc314
pushed a commit
that referenced
this pull request
Jul 3, 2026
…ocation (#23927) ## Summary `SemanticBuilder::hoisting_variables` is a nested map `FxHashMap<ScopeId, IdentHashMap<'a, SymbolId>>`. Each `entry(scope_id).or_default()` on a vacant scope heap-allocates a fresh inner `IdentHashMap` via the **system** allocator, and subsequent inserts grow it. On `antd.js` this is the single largest system-allocation site in semantic analysis (`VariableDeclarator::bind` / `Function::bind` in `binder.rs`). This flattens it to a single `FxHashMap<(ScopeId, Ident<'a>), SymbolId>`: - one insert / one point-probe per hoisted variable (was: outer probe → inner-map allocation → inner probe); - no per-scope inner map, so the per-scope system allocations disappear; - one hash instead of two. Behaviour is identical: `hoisting_variables` is only ever inserted into (`binder.rs`) and point-queried in `check_redeclaration` (`builder.rs`) — it is never iterated, so a flat `(scope, name)` key is a drop-in. ## Results — `cargo allocs` (system allocations) | fixture | phase | before | after | | --- | --- | ---: | ---: | | antd.js | semantic | 1020 | **28** (−97%) | | antd.js | minifier | 3041 | **1040** (−66%, rebuilds semantic internally) | | pdf.mjs | semantic | 25 | 16 | | kitchen-sink.tsx | semantic | 41 | 36 | `allocs_semantic.snap` / `allocs_minifier.snap` are updated accordingly. This is on the hot path for tools that run `SemanticBuilder` repeatedly per module (e.g. Rolldown). ## Relation to #19134 #19134 previously tried to move this temporary state (including `hoisting_variables`) to a scratch arena; CodSpeed reported a ~3.6% regression and it was closed. This PR takes a different, minimal approach: it does **not** introduce an arena, does **not** change `SemanticBuilder::new`'s signature, and touches no call sites. It removes a hash and an allocation per hoisted variable (rather than adding arena threading), so it should be CodSpeed-neutral-or-better — please let CI confirm. ## Tests - `cargo test -p oxc_semantic --lib --tests` — 60 passed (incl. redeclaration / var-hoist tests) - `cargo test -p oxc_minifier --lib --tests` — 538 passed ## AI disclosure Prepared with AI assistance (Claude). I reviewed the change, located the hotspot with a per-call-site allocation profile, verified the reduction with `cargo allocs`, and ran the test suites above.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
oxc_allocator::Vecandoxc_allocator::HashMapAllocatorwith internal scratch bump access and reset behaviormodule_instance_state_cache,hoisting_variables, unresolved-reference stack) to scratch-backed arena containersSemanticBuilder::newto take&Allocatorand migrate all call sites across crates/apps/napi/tasksselfwhen reusing temp allocatorRelated
Validation
cargo fmt --allcargo checkcargo test -p oxc_allocatorcargo test -p oxc_allocator --features track_allocationscargo test -p oxc_allocator --features from_raw_partscargo test -p oxc_semantic --lib --testscargo test -p oxc_mangler --libNote:
cargo test -p oxc_semanticcurrently fails onexamples/cfg.rsdue missingmainfor that example target (pre-existing in this branch state).AI Disclosure
This PR was prepared with AI assistance (Codex/GPT-5). I reviewed the generated changes and ran the validation commands above.