Skip to content

refactor(semantic): migrate temp semantic state to allocator scratch arena#19134

Closed
Boshen wants to merge 2 commits into
mainfrom
chore/semantic-scratch-migration
Closed

refactor(semantic): migrate temp semantic state to allocator scratch arena#19134
Boshen wants to merge 2 commits into
mainfrom
chore/semantic-scratch-migration

Conversation

@Boshen

@Boshen Boshen commented Feb 8, 2026

Copy link
Copy Markdown
Member

Summary

  • add scratch-arena constructors for oxc_allocator::Vec and oxc_allocator::HashMap
  • extend Allocator with internal scratch bump access and reset behavior
  • migrate semantic temporary state (module_instance_state_cache, hoisting_variables, unresolved-reference stack) to scratch-backed arena containers
  • update SemanticBuilder::new to take &Allocator and migrate all call sites across crates/apps/napi/tasks
  • fix mangler semantic build helpers to borrow self when reusing temp allocator

Related

Validation

  • cargo fmt --all
  • cargo check
  • cargo test -p oxc_allocator
  • cargo test -p oxc_allocator --features track_allocations
  • cargo test -p oxc_allocator --features from_raw_parts
  • cargo test -p oxc_semantic --lib --tests
  • cargo test -p oxc_mangler --lib

Note: cargo test -p oxc_semantic currently fails on examples/cfg.rs due missing main for that example target (pre-existing in this branch state).

AI Disclosure

This PR was prepared with AI assistance (Codex/GPT-5). I reviewed the generated changes and ran the validation commands above.

Copilot AI review requested due to automatic review settings February 8, 2026 04:31
@github-actions github-actions Bot added A-linter Area - Linter A-parser Area - Parser A-semantic Area - Semantic A-cli Area - CLI A-minifier Area - Minifier A-transformer Area - Transformer / Transpiler A-formatter Area - Formatter C-cleanup Category - technical debt or refactoring. Solution not expected to change behavior labels Feb 8, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Refactors semantic analysis to allocate temporary, per-build state into a new allocator “scratch” arena, and updates the public construction pattern of SemanticBuilder to require an &Allocator across the repo.

Changes:

  • Added scratch-arena constructors to oxc_allocator::Vec and oxc_allocator::HashMap, and extended Allocator with an internal scratch Bump plus reset behavior.
  • Migrated semantic temporary state (unresolved refs stack, hoisting vars, module instance cache) to scratch-backed arena containers.
  • Updated SemanticBuilder::new to accept &Allocator and migrated call sites across crates/apps/napi/tasks; adjusted mangler helpers to borrow self for allocator reuse.

Reviewed changes

Copilot reviewed 48 out of 49 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
tasks/transform_checker/src/lib.rs Update SemanticBuilder::new call to pass &Allocator.
tasks/track_memory_allocations/src/lib.rs Pass allocator into semantic build to match new constructor.
tasks/minsize/src/lib.rs Pass allocator into semantic build to match new constructor.
tasks/benchmark/benches/transformer.rs Bench updated for new SemanticBuilder::new(&allocator) API.
tasks/benchmark/benches/semantic.rs Bench updated for new SemanticBuilder::new(&allocator) API.
tasks/benchmark/benches/pipeline.rs Bench updated for new SemanticBuilder::new(&allocator) API.
tasks/benchmark/benches/minifier.rs Bench updated for new SemanticBuilder::new(&allocator) API.
tasks/benchmark/benches/linter.rs Bench updated for new SemanticBuilder::new(&allocator) API.
tasks/benchmark/benches/codegen.rs Bench updated for new SemanticBuilder::new(&allocator) API.
napi/transform/src/transformer.rs N-API transform path updated for new semantic builder constructor.
napi/playground/src/lib.rs Thread allocator through semantic build and lint semantic build.
napi/parser/src/raw_transfer.rs Semantic error reporting path updated for new constructor.
napi/parser/src/lib.rs Semantic error reporting path updated for new constructor.
crates/oxc_transformer_plugins/tests/integrations/replace_global_defines.rs Tests updated for new constructor.
crates/oxc_transformer_plugins/tests/integrations/inject_global_variables.rs Tests updated for new constructor.
crates/oxc_transformer_plugins/src/module_runner_transform.rs Test helper updated for new constructor.
crates/oxc_transformer_plugins/examples/define.rs Example updated for new constructor.
crates/oxc_transformer/tests/integrations/main.rs Integration tests updated for new constructor.
crates/oxc_transformer/examples/transformer.rs Example updated for new constructor.
crates/oxc_semantic/tests/main.rs Tests updated for new constructor.
crates/oxc_semantic/tests/integration/util/mod.rs Test utility updated for new constructor.
crates/oxc_semantic/src/unresolved_stack.rs Migrate unresolved stack storage to scratch-backed arena containers.
crates/oxc_semantic/src/scoping.rs Adjust unresolved reference value type to arena Vec with allocator lifetime.
crates/oxc_semantic/src/lib.rs Internal test helper updated for new constructor.
crates/oxc_semantic/src/jsdoc/parser/jsdoc_tag.rs JSDoc semantic test helper updated for new constructor.
crates/oxc_semantic/src/jsdoc/parser/jsdoc.rs JSDoc semantic test helper updated for new constructor.
crates/oxc_semantic/src/jsdoc/builder.rs JSDoc semantic test helper updated for new constructor.
crates/oxc_semantic/src/builder.rs Store allocator ref; migrate temp maps to scratch; route unresolved refs via new helper.
crates/oxc_semantic/src/binder.rs Update hoisting temp state insertion to scratch-backed bindings.
crates/oxc_semantic/examples/semantic.rs Example updated for new constructor.
crates/oxc_semantic/examples/cfg.rs Example updated for new constructor.
crates/oxc_minifier/src/lib.rs Minifier semantic builds updated for new constructor.
crates/oxc_minifier/src/compressor.rs Compressor semantic builds updated for new constructor.
crates/oxc_mangler/src/lib.rs Use temp allocator for semantic build; borrow self to allow reuse.
crates/oxc_mangler/src/keep_names.rs Tests updated for new constructor.
crates/oxc_linter/src/utils/jest.rs Tests updated for new constructor.
crates/oxc_linter/src/utils/comment.rs Tests updated for new constructor.
crates/oxc_linter/src/service/runtime.rs Runtime semantic build updated for new constructor.
crates/oxc_linter/src/disable_directives.rs Tests updated for new constructor.
crates/oxc_linter/examples/linter.rs Example updated for new constructor.
crates/oxc_formatter/src/detect_code_removal/mod.rs Formatter helper updated for new constructor.
crates/oxc_allocator/src/vec.rs Add Vec::new_in_scratch / with_capacity_in_scratch.
crates/oxc_allocator/src/hash_map.rs Add HashMap::*_in_scratch constructors.
crates/oxc_allocator/src/allocator.rs Add scratch Bump, reset handling, and internal scratch_bump accessor.
crates/oxc/src/compiler.rs Thread allocator into compiler semantic step; rebuild scoping with allocator.
crates/oxc/README.md Docs updated to reflect new SemanticBuilder::new(&allocator) usage.
apps/oxlint/src/js_plugins/parse.rs Oxlint plugin parser updated for new constructor.

Comment thread crates/oxc_allocator/src/allocator.rs
Comment thread crates/oxc_allocator/src/vec.rs
Comment thread crates/oxc_allocator/src/hash_map.rs
@codspeed-hq

codspeed-hq Bot commented Feb 8, 2026

Copy link
Copy Markdown

CodSpeed Performance Report

Merging this PR will degrade performance by 3.62%

Comparing chore/semantic-scratch-migration (9cd14c0) with main (7318275)

Summary

❌ 1 regressed benchmark
✅ 46 untouched benchmarks
⏩ 3 skipped benchmarks1

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
Simulation semantic[binder.ts] 3.2 ms 3.3 ms -3.62%

Footnotes

  1. 3 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

@Boshen Boshen closed this Feb 9, 2026
@Boshen Boshen deleted the chore/semantic-scratch-migration branch February 9, 2026 08:05
Dunqing pushed a commit that referenced this pull request Jun 29, 2026
…ocation (#23927)

## Summary

`SemanticBuilder::hoisting_variables` is a nested map
`FxHashMap<ScopeId, IdentHashMap<'a, SymbolId>>`. Each
`entry(scope_id).or_default()`
on a vacant scope heap-allocates a fresh inner `IdentHashMap` via the
**system**
allocator, and subsequent inserts grow it. On `antd.js` this is the
single largest
system-allocation site in semantic analysis (`VariableDeclarator::bind`
/
`Function::bind` in `binder.rs`).

This flattens it to a single `FxHashMap<(ScopeId, Ident<'a>),
SymbolId>`:

- one insert / one point-probe per hoisted variable (was: outer probe →
inner-map
  allocation → inner probe);
- no per-scope inner map, so the per-scope system allocations disappear;
- one hash instead of two.

Behaviour is identical: `hoisting_variables` is only ever inserted into
(`binder.rs`)
and point-queried in `check_redeclaration` (`builder.rs`) — it is never
iterated, so a
flat `(scope, name)` key is a drop-in.

## Results — `cargo allocs` (system allocations)

| fixture | phase | before | after |
| --- | --- | ---: | ---: |
| antd.js | semantic | 1020 | **28** (−97%) |
| antd.js | minifier | 3041 | **1040** (−66%, rebuilds semantic
internally) |
| pdf.mjs | semantic | 25 | 16 |
| kitchen-sink.tsx | semantic | 41 | 36 |

`allocs_semantic.snap` / `allocs_minifier.snap` are updated accordingly.
This is on the
hot path for tools that run `SemanticBuilder` repeatedly per module
(e.g. Rolldown).

## Relation to #19134

#19134 previously tried to move this temporary state (including
`hoisting_variables`)
to a scratch arena; CodSpeed reported a ~3.6% regression and it was
closed. This PR
takes a different, minimal approach: it does **not** introduce an arena,
does **not**
change `SemanticBuilder::new`'s signature, and touches no call sites. It
removes a hash
and an allocation per hoisted variable (rather than adding arena
threading), so it
should be CodSpeed-neutral-or-better — please let CI confirm.

## Tests

- `cargo test -p oxc_semantic --lib --tests` — 60 passed (incl.
redeclaration / var-hoist tests)
- `cargo test -p oxc_minifier --lib --tests` — 538 passed

## AI disclosure

Prepared with AI assistance (Claude). I reviewed the change, located the
hotspot with a
per-call-site allocation profile, verified the reduction with `cargo
allocs`, and ran
the test suites above.
camc314 pushed a commit that referenced this pull request Jul 3, 2026
…ocation (#23927)

## Summary

`SemanticBuilder::hoisting_variables` is a nested map
`FxHashMap<ScopeId, IdentHashMap<'a, SymbolId>>`. Each
`entry(scope_id).or_default()`
on a vacant scope heap-allocates a fresh inner `IdentHashMap` via the
**system**
allocator, and subsequent inserts grow it. On `antd.js` this is the
single largest
system-allocation site in semantic analysis (`VariableDeclarator::bind`
/
`Function::bind` in `binder.rs`).

This flattens it to a single `FxHashMap<(ScopeId, Ident<'a>),
SymbolId>`:

- one insert / one point-probe per hoisted variable (was: outer probe →
inner-map
  allocation → inner probe);
- no per-scope inner map, so the per-scope system allocations disappear;
- one hash instead of two.

Behaviour is identical: `hoisting_variables` is only ever inserted into
(`binder.rs`)
and point-queried in `check_redeclaration` (`builder.rs`) — it is never
iterated, so a
flat `(scope, name)` key is a drop-in.

## Results — `cargo allocs` (system allocations)

| fixture | phase | before | after |
| --- | --- | ---: | ---: |
| antd.js | semantic | 1020 | **28** (−97%) |
| antd.js | minifier | 3041 | **1040** (−66%, rebuilds semantic
internally) |
| pdf.mjs | semantic | 25 | 16 |
| kitchen-sink.tsx | semantic | 41 | 36 |

`allocs_semantic.snap` / `allocs_minifier.snap` are updated accordingly.
This is on the
hot path for tools that run `SemanticBuilder` repeatedly per module
(e.g. Rolldown).

## Relation to #19134

#19134 previously tried to move this temporary state (including
`hoisting_variables`)
to a scratch arena; CodSpeed reported a ~3.6% regression and it was
closed. This PR
takes a different, minimal approach: it does **not** introduce an arena,
does **not**
change `SemanticBuilder::new`'s signature, and touches no call sites. It
removes a hash
and an allocation per hoisted variable (rather than adding arena
threading), so it
should be CodSpeed-neutral-or-better — please let CI confirm.

## Tests

- `cargo test -p oxc_semantic --lib --tests` — 60 passed (incl.
redeclaration / var-hoist tests)
- `cargo test -p oxc_minifier --lib --tests` — 538 passed

## AI disclosure

Prepared with AI assistance (Claude). I reviewed the change, located the
hotspot with a
per-call-site allocation profile, verified the reduction with `cargo
allocs`, and ran
the test suites above.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-cli Area - CLI A-formatter Area - Formatter A-linter Area - Linter A-minifier Area - Minifier A-parser Area - Parser A-semantic Area - Semantic A-transformer Area - Transformer / Transpiler C-cleanup Category - technical debt or refactoring. Solution not expected to change behavior

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants