refactor(allocator, linter/plugins, napi/parser): store raw transfer metadata within Arena chunk#21869
Conversation
How to use the Graphite Merge QueueAdd either label to this PR to merge it via the merge queue:
You must have a Graphite account in order to use the merge queue. Sign up using this link. An organization admin has enabled the Graphite Merge Queue in this repository. Please do not merge from GitHub as this will restart CI on PRs being processed by the merge queue. This stack of pull requests is managed by Graphite. Learn more about stacking. |
Merging this PR will not alter performance
Comparing Footnotes
|
There was a problem hiding this comment.
Pull request overview
Refactors raw-transfer memory layout so RawTransferMetadata and FixedSizeAllocatorMetadata live inside the allocator chunk (with ChunkFooter at the very end of the block), reducing JS-side exposure to allocator internals and simplifying alignment/ownership boundaries.
Changes:
- Recomputed/renamed raw-transfer constants (
BLOCK_*,BUFFER_*,ACTIVE_SIZE, metadata size/align) and regenerated Rust/JS constant outputs accordingly. - Updated Rust raw-transfer writers/readers (allocator pool, linter, NAPI parser, oxlint JS plugins) to use the new in-chunk metadata positions and to pass a full
BLOCK_SIZEview to Rust while exposing a smallerBUFFER_SIZEview to JS. - Updated generated JS deserializers/offsets to match the new
RawTransferMetadataposition.
Reviewed changes
Copilot reviewed 8 out of 23 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| tasks/ast_tools/src/generators/raw_transfer.rs | Recomputes buffer/layout constants and regenerates Rust/JS constants for the new in-chunk metadata layout. |
| napi/parser/src/raw_transfer.rs | Adjusts raw-transfer parsing to treat the input as a BLOCK_SIZE block and write metadata at ACTIVE_SIZE. |
| napi/parser/src/generated/raw_transfer_constants.rs | Regenerated raw-transfer constants reflecting BLOCK_*, BUFFER_*, ACTIVE_SIZE, and metadata align/size. |
| napi/parser/src-js/raw-transfer/common.js | Uses block view for Rust calls while keeping buffer/views sized to BUFFER_SIZE. |
| napi/parser/src-js/generated/deserialize/ts_range_parent.js | Updates hardcoded root pointer index to new metadata position. |
| napi/parser/src-js/generated/deserialize/ts_range.js | Updates hardcoded root pointer index to new metadata position. |
| napi/parser/src-js/generated/deserialize/ts_parent.js | Updates hardcoded root pointer index to new metadata position. |
| napi/parser/src-js/generated/deserialize/ts.js | Updates hardcoded root pointer index to new metadata position. |
| napi/parser/src-js/generated/deserialize/js_range_parent.js | Updates hardcoded root pointer index to new metadata position. |
| napi/parser/src-js/generated/deserialize/js_range.js | Updates hardcoded root pointer index to new metadata position. |
| napi/parser/src-js/generated/deserialize/js_parent.js | Updates hardcoded root pointer index to new metadata position. |
| napi/parser/src-js/generated/deserialize/js.js | Updates hardcoded root pointer index to new metadata position. |
| napi/parser/src-js/generated/constants.js | Regenerated JS constants (introducing BLOCK_SIZE/BLOCK_ALIGN, updated offsets). |
| crates/oxc_linter/src/lib.rs | Writes RawTransferMetadata relative to FixedSizeAllocatorMetadata pointer under the new layout. |
| crates/oxc_allocator/src/pool/fixed_size.rs | Moves allocator metadata inside the chunk, updates cursor/reset logic, and adds const layout assertions. |
| crates/oxc_allocator/src/generated/fixed_size_constants.rs | Regenerated allocator fixed-size constants for the new layout. |
| apps/oxlint/src/js_plugins/parse.rs | Updates JS-plugin raw parsing to use BLOCK_SIZE/BLOCK_ALIGN inputs and write metadata at ACTIVE_SIZE. |
| apps/oxlint/src/js_plugins/external_linter.rs | Clarifies that the JS-visible Uint8Array excludes FixedSizeAllocatorMetadata and ChunkFooter. |
| apps/oxlint/src/generated/raw_transfer_constants.rs | Regenerated raw-transfer constants for oxlint’s JS plugin path. |
| apps/oxlint/src-js/package/parse.ts | Adds separate blockBuffer view for Rust while keeping JS-visible buffer sized to BUFFER_SIZE. |
| apps/oxlint/src-js/generated/deserialize.js | Updates hardcoded root pointer index to new metadata position. |
| apps/oxlint/src-js/generated/constants.ts | Regenerated TS constants (introducing BLOCK_SIZE/BLOCK_ALIGN, updated offsets). |
| apps/oxlint/src-js/bindings.d.ts | Updates docs to refer to BLOCK_* instead of BUFFER_ALIGN, matching new API/constants. |
2774b4a to
0d04ab0
Compare
c2925db to
b86197d
Compare
0d04ab0 to
b83af4a
Compare
b83af4a to
f4ed0be
Compare
f4ed0be to
648e3c0
Compare
Merge activity
|
…metadata within `Arena` chunk (#21869) Large change to how raw transfer stores metadata about the allocations it uses. Previously the two metadata structures `RawTransferMetadata` and `FixedSizeAllocator` after the chunk's `ChunkFooter`. This had a few problems: 1. It was confusing and unwieldy - a recipe for bugs. 2. `ChunkFooter`'s memory was exposed on JS side - a bit unsafe as altering bytes in this region could easily trigger UB. 3. It entwined `Arena` (which is just the thing that allocates) with the details of exactly _what_ raw transfer allocates. 4. Imposed annoying alignment requirements, because `ChunkFooter` must be aligned on 16, and so anything after it must ensure it doesn't break that invariant. Old layout: ``` WHOLE BLOCK - aligned on 4 GiB <-----------------------------------------------------> Allocated block (`BLOCK_SIZE` bytes) ALLOCATOR <-----------------------------------------> `Allocator` chunk (`CHUNK_SIZE` bytes) <----> `ChunkFooter` (aligned on 16) <-----------------------------------> `Allocator` chunk data storage (for AST) (`ACTIVE_SIZE` bytes) METADATA <----> `RawTransferMetadata` <----> `FixedSizeAllocatorMetadata` BUFFER SENT TO JS <-----------------------------------------------> Buffer sent to JS (`BUFFER_SIZE` bytes) ``` This PR moves `RawTransferMetadata` and `FixedSizeAllocatorMetadata` into the chunk itself. New layout: ``` WHOLE BLOCK - size 2 GiB - 16, aligned on 4 GiB <-----------------------------------------------------> Allocated block (`BLOCK_SIZE` bytes) ARENA <-----------------------------------------------------> Chunk (fills whole block) <--------------------------------------> Allocatable region for AST (`ACTIVE_SIZE` bytes) <---> `RawTransferMetadata` <---> `FixedSizeAllocatorMetadata` <---> `ChunkFooter` (aligned on 16, last in block) BUFFER SENT TO JS <-------------------------------------------> Buffer sent to JS (`BUFFER_SIZE` bytes) ``` `FixedSizeAllocatorMetadata` and `ChunkFooter` are no longer in the region which is shared with JS side. As far as `Arena` is concerned, they're now just some data (like any other data) which is allocated in the arena. Also: - Introduce more consistency to the naming of constants which specify the size and position of these various data structures in the arena. - Add more const assertions to ensure everything is laid out and aligned as it should be.
3b67f17 to
a55635d
Compare
648e3c0 to
99eef72
Compare
Remove `Allocator::end_ptr` method. It's no longer required since #21869.

Large change to how raw transfer stores metadata about the allocations it uses.
Previously the two metadata structures
RawTransferMetadataandFixedSizeAllocatorafter the chunk'sChunkFooter.This had a few problems:
ChunkFooter's memory was exposed on JS side - a bit unsafe as altering bytes in this region could easily trigger UB.Arena(which is just the thing that allocates) with the details of exactly what raw transfer allocates.ChunkFootermust be aligned on 16, and so anything after it must ensure it doesn't break that invariant.Old layout:
This PR moves
RawTransferMetadataandFixedSizeAllocatorMetadatainto the chunk itself. New layout:FixedSizeAllocatorMetadataandChunkFooterare no longer in the region which is shared with JS side. As far asArenais concerned, they're now just some data (like any other data) which is allocated in the arena.Also: