refactor(allocator): per-platform implementation of freeing fixed-size chunks#22122
Conversation
Merging this PR will not alter performance
Comparing Footnotes
|
There was a problem hiding this comment.
Pull request overview
Refactors fixed-size arena chunk deallocation to route through per-platform implementations, in preparation for a future Windows-specific change (VirtualFree) while keeping the allocator drop path consistent with the fixed-size pool’s explicit free logic.
Changes:
- Adds
dealloc_fixed_size_arena_chunkper platform (linux/macos/windows) and re-exports it fromarena::fixed_size. - Updates the fixed-size allocator pool to call
dealloc_fixed_size_arena_chunkwhen freeing a fixed-size allocator. - Refactors arena drop deallocation to delegate fixed-size chunk frees to the platform function.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| crates/oxc_allocator/src/pool/fixed_size.rs | Switches fixed-size pool deallocation to the new platform-specific dealloc entrypoint. |
| crates/oxc_allocator/src/arena/mod.rs | Makes arena::fixed_size module accessible within the crate to support the new call sites. |
| crates/oxc_allocator/src/arena/drop.rs | Refactors chunk deallocation, delegating fixed-size frees to arena::fixed_size. |
| crates/oxc_allocator/src/arena/fixed_size/mod.rs | Re-exports dealloc_fixed_size_arena_chunk from the active platform module. |
| crates/oxc_allocator/src/arena/fixed_size/linux.rs | Adds Linux implementation of dealloc_fixed_size_arena_chunk. |
| crates/oxc_allocator/src/arena/fixed_size/macos.rs | Adds macOS implementation of dealloc_fixed_size_arena_chunk. |
| crates/oxc_allocator/src/arena/fixed_size/windows.rs | Adds Windows implementation of dealloc_fixed_size_arena_chunk. |
8ad4c56 to
a654d30
Compare
c66882c to
d14a4f2
Compare
b6c9c93 to
d5b9c75
Compare
d14a4f2 to
b710538
Compare
d5b9c75 to
f4cbe3d
Compare
b710538 to
6fe6d15
Compare
Merge activity
|
…e chunks (#22122) Add per-platform implementations of `dealloc_fixed_size_arena_chunk`, and delegate to them when dropping an `Arena` which has fixed-size chunks. Currently, all the implementations are identical - they just deallocate the memory with `System` allocator. But this prepares the way for #22124, which alters the Windows impl to use `VirtualFree` instead.
6fe6d15 to
f5ed139
Compare
f4cbe3d to
5086ddc
Compare
Add a private method `Arena::grow_fixed_size_chunk` which attempts to grow a fixed-size `Arena` chunk in place. Similar to #22122, there's multiple implementations for different platforms, but all are currently the same - all return `None` to indicate "could not grow". This is preparatory work for #22124, which adds a Windows-specific implementation.
…22124) Fixes #19395 - OOM errors in Oxlint JS plugins on Windows. ### The problem Previously, on all platforms each fixed-size (raw transfer-enabled) allocator obtained a large chunk of memory from the `System` allocator (6 GiB in the case of Windows). This is fine on MacOS and most Linux distros, as they use overcommit allocation accounting, so these allocations consume only virtual memory (address space), not actual physical memory. The available address space is enormous (~128 TiB), so it's hard to exhaust it. Physical memory is only consumed when pages of the allocation are actually touched, which the vast majority never are. Windows behaves differently. Allocations obtained from `System` allocator immediately count towards the memory limit. So, on memory-constrained machines, allocating 6 GiB can immediately fail - OOM just from linting the first file. ### This PR [#22088](#22088), #22122, and #22123 introduced platform-specific code for handling allocation and deallocation of fixed-size `Arena`s, which are used in raw transfer/Oxlint JS plugins. This PR switches to a different allocation method on Windows for fixed-size `Arena`s. It switches to using [VirtualAlloc](https://learn.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-virtualalloc). This Windows-only API allows reserving address space and committing pages of that reservation as separate operations (similar to `mmap`). It works like this: - Fixed-size allocators now still reserve 6 GiB, but that only consumes address space, and does not count towards max memory limit. - Of that 6 GiB reservation, a 2 GiB "container", aligned on a 4 GiB boundary, will be used as the active part of the memory for storing the AST and other data. - Initially only the last 16 KiB of the container is committed (made into usable memory, with physical pages backing it). - If AST does not fit in the committed region, the region grows downwards, committing further pages to accomodate allocation requests, up to the maximum of 2 GiB. Essentially, this replicates something similar to Linux-style allocation, but on Windows. The only difference is that on Windows it's necessary to explicitly commit pages before touching them, whereas Linux/MacOS commit on demand automatically. ### Implementation details and testing This PR uses the battle-tested [windows-sys](https://crates.io/crates/windows-sys) crate for `VirtualAlloc` and `VirtualFree`. The allocation implementation is written from scratch, but while I was researching to validate that I'd not missed any edge cases, I discovered that it's almost identical to [wasmtime's implementation](https://github.com/bytecodealliance/wasmtime/blob/386a3280dee61f5c4120ce7cde621c1039e383d5/crates/wasmtime/src/runtime/vm/sys/windows/mmap.rs) which uses the same APIs. Comments in the code go over in some detail considerations of Rust's memory model and pointer provenance. As far as I can see, the implementation is fairly watertight. There are tests for the essential functionality and edge cases, and #22121 made these tests run under Miri on Windows. However, I don't have access to a Windows machine to fully test this by running Oxlint on a massive repo. I am hoping members of the community who do can put this through its paces a bit before we merge. ### What's missing In enormous projects, when linting with both JS plugins and `import` plugin, it's possible to end up with a huge number of ASTs in memory simultaneously. With each AST consuming 6 GiB of address space, it's possible to exhaust the entire 128 TiB usable address space, and get OOM on _virtual_ memory. At present we have a workaround to prevent too many fixed-size allocators existing concurrently, so avoiding OOM, but the workaround is a drag on performance. Future work will aim to address this.

Add per-platform implementations of
dealloc_fixed_size_arena_chunk, and delegate to them when dropping anArenawhich has fixed-size chunks.Currently, all the implementations are identical - they just deallocate the memory with
Systemallocator. But this prepares the way for #22124, which alters the Windows impl to useVirtualFreeinstead.