Skip to content

refactor(allocator, linter/plugins, napi/parser): store raw transfer metadata within Arena chunk#21869

Merged
graphite-app[bot] merged 1 commit intomainfrom
om/04-27-refactor_allocator_linter_plugins_napi_parser_store_raw_transfer_metadata_within_arena_chunk
Apr 30, 2026
Merged

refactor(allocator, linter/plugins, napi/parser): store raw transfer metadata within Arena chunk#21869
graphite-app[bot] merged 1 commit intomainfrom
om/04-27-refactor_allocator_linter_plugins_napi_parser_store_raw_transfer_metadata_within_arena_chunk

Conversation

@overlookmotel
Copy link
Copy Markdown
Member

@overlookmotel overlookmotel commented Apr 27, 2026

Large change to how raw transfer stores metadata about the allocations it uses.

Previously the two metadata structures RawTransferMetadata and FixedSizeAllocator after the chunk's ChunkFooter.

This had a few problems:

  1. It was confusing and unwieldy - a recipe for bugs.
  2. ChunkFooter's memory was exposed on JS side - a bit unsafe as altering bytes in this region could easily trigger UB.
  3. It entwined Arena (which is just the thing that allocates) with the details of exactly what raw transfer allocates.
  4. Imposed annoying alignment requirements, because ChunkFooter must be aligned on 16, and so anything after it must ensure it doesn't break that invariant.

Old layout:

                                                        WHOLE BLOCK - aligned on 4 GiB
<-----------------------------------------------------> Allocated block (`BLOCK_SIZE` bytes)

                                                        ALLOCATOR
<----------------------------------------->             `Allocator` chunk (`CHUNK_SIZE` bytes)
                                     <---->             `ChunkFooter` (aligned on 16)
<----------------------------------->                   `Allocator` chunk data storage (for AST)
                                                        (`ACTIVE_SIZE` bytes)

                                                        METADATA
                                           <---->       `RawTransferMetadata`
                                                 <----> `FixedSizeAllocatorMetadata`

                                                        BUFFER SENT TO JS
<----------------------------------------------->       Buffer sent to JS (`BUFFER_SIZE` bytes)

This PR moves RawTransferMetadata and FixedSizeAllocatorMetadata into the chunk itself. New layout:

                                                        WHOLE BLOCK - size 2 GiB - 16, aligned on 4 GiB
<-----------------------------------------------------> Allocated block (`BLOCK_SIZE` bytes)

                                                        ARENA
<-----------------------------------------------------> Chunk (fills whole block)
<-------------------------------------->                Allocatable region for AST (`ACTIVE_SIZE` bytes)
                                        <--->           `RawTransferMetadata`
                                             <--->      `FixedSizeAllocatorMetadata`
                                                  <---> `ChunkFooter` (aligned on 16, last in block)

                                                        BUFFER SENT TO JS
<------------------------------------------->           Buffer sent to JS (`BUFFER_SIZE` bytes)

FixedSizeAllocatorMetadata and ChunkFooter are no longer in the region which is shared with JS side. As far as Arena is concerned, they're now just some data (like any other data) which is allocated in the arena.

Also:

  • Introduce more consistency to the naming of constants which specify the size and position of these various data structures in the arena.
  • Add more const assertions to ensure everything is laid out and aligned as it should be.

Copy link
Copy Markdown
Member Author

overlookmotel commented Apr 27, 2026


How to use the Graphite Merge Queue

Add either label to this PR to merge it via the merge queue:

  • 0-merge - adds this PR to the back of the merge queue
  • hotfix - for urgent changes, fast-track this PR to the front of the merge queue

You must have a Graphite account in order to use the merge queue. Sign up using this link.

An organization admin has enabled the Graphite Merge Queue in this repository.

Please do not merge from GitHub as this will restart CI on PRs being processed by the merge queue.

This stack of pull requests is managed by Graphite. Learn more about stacking.

@codspeed-hq
Copy link
Copy Markdown

codspeed-hq Bot commented Apr 27, 2026

Merging this PR will not alter performance

✅ 48 untouched benchmarks
⏩ 3 skipped benchmarks1


Comparing om/04-27-refactor_allocator_linter_plugins_napi_parser_store_raw_transfer_metadata_within_arena_chunk (99eef72) with main (ce62f16)2

Open in CodSpeed

Footnotes

  1. 3 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

  2. No successful run was found on main (99eef72) during the generation of this report, so ce62f16 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report.

@overlookmotel overlookmotel marked this pull request as ready for review April 27, 2026 23:13
@overlookmotel overlookmotel requested a review from camc314 as a code owner April 27, 2026 23:13
Copilot AI review requested due to automatic review settings April 27, 2026 23:13
@overlookmotel overlookmotel added C-cleanup Category - technical debt or refactoring. Solution not expected to change behavior A-allocator Area - Allocator labels Apr 27, 2026
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Refactors raw-transfer memory layout so RawTransferMetadata and FixedSizeAllocatorMetadata live inside the allocator chunk (with ChunkFooter at the very end of the block), reducing JS-side exposure to allocator internals and simplifying alignment/ownership boundaries.

Changes:

  • Recomputed/renamed raw-transfer constants (BLOCK_*, BUFFER_*, ACTIVE_SIZE, metadata size/align) and regenerated Rust/JS constant outputs accordingly.
  • Updated Rust raw-transfer writers/readers (allocator pool, linter, NAPI parser, oxlint JS plugins) to use the new in-chunk metadata positions and to pass a full BLOCK_SIZE view to Rust while exposing a smaller BUFFER_SIZE view to JS.
  • Updated generated JS deserializers/offsets to match the new RawTransferMetadata position.

Reviewed changes

Copilot reviewed 8 out of 23 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
tasks/ast_tools/src/generators/raw_transfer.rs Recomputes buffer/layout constants and regenerates Rust/JS constants for the new in-chunk metadata layout.
napi/parser/src/raw_transfer.rs Adjusts raw-transfer parsing to treat the input as a BLOCK_SIZE block and write metadata at ACTIVE_SIZE.
napi/parser/src/generated/raw_transfer_constants.rs Regenerated raw-transfer constants reflecting BLOCK_*, BUFFER_*, ACTIVE_SIZE, and metadata align/size.
napi/parser/src-js/raw-transfer/common.js Uses block view for Rust calls while keeping buffer/views sized to BUFFER_SIZE.
napi/parser/src-js/generated/deserialize/ts_range_parent.js Updates hardcoded root pointer index to new metadata position.
napi/parser/src-js/generated/deserialize/ts_range.js Updates hardcoded root pointer index to new metadata position.
napi/parser/src-js/generated/deserialize/ts_parent.js Updates hardcoded root pointer index to new metadata position.
napi/parser/src-js/generated/deserialize/ts.js Updates hardcoded root pointer index to new metadata position.
napi/parser/src-js/generated/deserialize/js_range_parent.js Updates hardcoded root pointer index to new metadata position.
napi/parser/src-js/generated/deserialize/js_range.js Updates hardcoded root pointer index to new metadata position.
napi/parser/src-js/generated/deserialize/js_parent.js Updates hardcoded root pointer index to new metadata position.
napi/parser/src-js/generated/deserialize/js.js Updates hardcoded root pointer index to new metadata position.
napi/parser/src-js/generated/constants.js Regenerated JS constants (introducing BLOCK_SIZE/BLOCK_ALIGN, updated offsets).
crates/oxc_linter/src/lib.rs Writes RawTransferMetadata relative to FixedSizeAllocatorMetadata pointer under the new layout.
crates/oxc_allocator/src/pool/fixed_size.rs Moves allocator metadata inside the chunk, updates cursor/reset logic, and adds const layout assertions.
crates/oxc_allocator/src/generated/fixed_size_constants.rs Regenerated allocator fixed-size constants for the new layout.
apps/oxlint/src/js_plugins/parse.rs Updates JS-plugin raw parsing to use BLOCK_SIZE/BLOCK_ALIGN inputs and write metadata at ACTIVE_SIZE.
apps/oxlint/src/js_plugins/external_linter.rs Clarifies that the JS-visible Uint8Array excludes FixedSizeAllocatorMetadata and ChunkFooter.
apps/oxlint/src/generated/raw_transfer_constants.rs Regenerated raw-transfer constants for oxlint’s JS plugin path.
apps/oxlint/src-js/package/parse.ts Adds separate blockBuffer view for Rust while keeping JS-visible buffer sized to BUFFER_SIZE.
apps/oxlint/src-js/generated/deserialize.js Updates hardcoded root pointer index to new metadata position.
apps/oxlint/src-js/generated/constants.ts Regenerated TS constants (introducing BLOCK_SIZE/BLOCK_ALIGN, updated offsets).
apps/oxlint/src-js/bindings.d.ts Updates docs to refer to BLOCK_* instead of BUFFER_ALIGN, matching new API/constants.

Comment thread apps/oxlint/src/js_plugins/parse.rs
@graphite-app graphite-app Bot changed the base branch from om/04-27-refactor_allocator_fixed_size_allocators_store_backing_allocation_pointer_in_chunkfooter_ to graphite-base/21869 April 27, 2026 23:34
@graphite-app graphite-app Bot force-pushed the om/04-27-refactor_allocator_linter_plugins_napi_parser_store_raw_transfer_metadata_within_arena_chunk branch from 2774b4a to 0d04ab0 Compare April 27, 2026 23:42
@graphite-app graphite-app Bot force-pushed the graphite-base/21869 branch from c2925db to b86197d Compare April 27, 2026 23:42
@graphite-app graphite-app Bot changed the base branch from graphite-base/21869 to main April 27, 2026 23:43
@graphite-app graphite-app Bot force-pushed the om/04-27-refactor_allocator_linter_plugins_napi_parser_store_raw_transfer_metadata_within_arena_chunk branch from 0d04ab0 to b83af4a Compare April 27, 2026 23:43
@overlookmotel overlookmotel changed the base branch from main to graphite-base/21869 April 30, 2026 01:05
@overlookmotel overlookmotel force-pushed the om/04-27-refactor_allocator_linter_plugins_napi_parser_store_raw_transfer_metadata_within_arena_chunk branch from b83af4a to f4ed0be Compare April 30, 2026 01:05
@overlookmotel overlookmotel changed the base branch from graphite-base/21869 to om/04-30-refactor_linter_do_not_compile_linter_convert_and_call_external_linter_on_32-bit_or_big_endian April 30, 2026 01:05
@overlookmotel overlookmotel force-pushed the om/04-27-refactor_allocator_linter_plugins_napi_parser_store_raw_transfer_metadata_within_arena_chunk branch from f4ed0be to 648e3c0 Compare April 30, 2026 01:18
@graphite-app
Copy link
Copy Markdown
Contributor

graphite-app Bot commented Apr 30, 2026

Merge activity

…metadata within `Arena` chunk (#21869)

Large change to how raw transfer stores metadata about the allocations it uses.

Previously the two metadata structures `RawTransferMetadata` and `FixedSizeAllocator` after the chunk's `ChunkFooter`.

This had a few problems:

1. It was confusing and unwieldy - a recipe for bugs.
2. `ChunkFooter`'s memory was exposed on JS side - a bit unsafe as altering bytes in this region could easily trigger UB.
3. It entwined `Arena` (which is just the thing that allocates) with the details of exactly _what_ raw transfer allocates.
4. Imposed annoying alignment requirements, because `ChunkFooter` must be aligned on 16, and so anything after it must ensure it doesn't break that invariant.

Old layout:

```
                                                        WHOLE BLOCK - aligned on 4 GiB
<-----------------------------------------------------> Allocated block (`BLOCK_SIZE` bytes)

                                                        ALLOCATOR
<----------------------------------------->             `Allocator` chunk (`CHUNK_SIZE` bytes)
                                     <---->             `ChunkFooter` (aligned on 16)
<----------------------------------->                   `Allocator` chunk data storage (for AST)
                                                        (`ACTIVE_SIZE` bytes)

                                                        METADATA
                                           <---->       `RawTransferMetadata`
                                                 <----> `FixedSizeAllocatorMetadata`

                                                        BUFFER SENT TO JS
<----------------------------------------------->       Buffer sent to JS (`BUFFER_SIZE` bytes)
```

This PR moves `RawTransferMetadata` and `FixedSizeAllocatorMetadata` into the chunk itself. New layout:

```
                                                        WHOLE BLOCK - size 2 GiB - 16, aligned on 4 GiB
<-----------------------------------------------------> Allocated block (`BLOCK_SIZE` bytes)

                                                        ARENA
<-----------------------------------------------------> Chunk (fills whole block)
<-------------------------------------->                Allocatable region for AST (`ACTIVE_SIZE` bytes)
                                        <--->           `RawTransferMetadata`
                                             <--->      `FixedSizeAllocatorMetadata`
                                                  <---> `ChunkFooter` (aligned on 16, last in block)

                                                        BUFFER SENT TO JS
<------------------------------------------->           Buffer sent to JS (`BUFFER_SIZE` bytes)
```

`FixedSizeAllocatorMetadata` and `ChunkFooter` are no longer in the region which is shared with JS side. As far as `Arena` is concerned, they're now just some data (like any other data) which is allocated in the arena.

Also:

- Introduce more consistency to the naming of constants which specify the size and position of these various data structures in the arena.
- Add more const assertions to ensure everything is laid out and aligned as it should be.
@graphite-app graphite-app Bot force-pushed the om/04-30-refactor_linter_do_not_compile_linter_convert_and_call_external_linter_on_32-bit_or_big_endian branch from 3b67f17 to a55635d Compare April 30, 2026 01:27
@graphite-app graphite-app Bot force-pushed the om/04-27-refactor_allocator_linter_plugins_napi_parser_store_raw_transfer_metadata_within_arena_chunk branch from 648e3c0 to 99eef72 Compare April 30, 2026 01:28
graphite-app Bot pushed a commit that referenced this pull request Apr 30, 2026
Remove `Allocator::end_ptr` method. It's no longer required since #21869.
Base automatically changed from om/04-30-refactor_linter_do_not_compile_linter_convert_and_call_external_linter_on_32-bit_or_big_endian to main April 30, 2026 01:31
@graphite-app graphite-app Bot merged commit 99eef72 into main Apr 30, 2026
36 of 37 checks passed
@graphite-app graphite-app Bot deleted the om/04-27-refactor_allocator_linter_plugins_napi_parser_store_raw_transfer_metadata_within_arena_chunk branch April 30, 2026 01:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-allocator Area - Allocator C-cleanup Category - technical debt or refactoring. Solution not expected to change behavior

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants