Skip to content

Conversation

@runarorama
Copy link
Contributor

@runarorama runarorama commented Aug 12, 2025

Overview

This change adds PinnedByteArray support to Unison, enabling efficient pinned memory management for low-level I/O operations and system calls. Previously, Unison only had MutableByteArray for mutable byte operations, but lacked the ability to create pinned arrays that prevent the garbage collector from moving memory during critical operations.

Old behavior: Users could only work with regular mutable byte arrays that could be moved by the garbage collector, making them unsuitable for system calls, certain I/O operations, or FFI that require stable memory addresses.

New behavior: Users can now create pinned byte arrays using e.g. IO.pinnedByteArray ensuring memory remains at a fixed address during critical operations. They can be cast to mutable arrays using PinnedByteArray.cast, so wherever a MutableByteArray can be used, a PinnedByteArray can also be used after casting. Operations that require pinned memory use PinnedByteArray directly. Crucially, a MutableByteArray cannot be cast to a PinnedByteArray, which otherwise would cause segfaults.

Example usage:

-- Create a pinned byte array
pinned = IO.pinnedByteArray 1024

-- Cast to mutable array for read/write operations
mutable = PinnedByteArray.cast pinned

-- Write data (memory stays pinned)
write8 mutable 0 42
write16be mutable 1 12345

-- Read data back
value1 = read8 mutable 0  -- Returns 42
value2 = read16be mutable 1  -- Returns 12345

This is particularly useful for:

  • Network operations that require stable buffer addresses
  • System calls that need pinned memory
  • Future FFI operations using external C libraries
  • Performance-critical I/O where memory movement is unacceptable

Fixed buffer I/O

This PR also adds Socket and Handle I/O operations that use pinned memory:

  • IO.socketSendBuf - Sends a number of bytes from a PinnedByteArray to a Socket.
  • IO.socketReceiveBuf - Blocking read of some number of bytes from a Socket into a PinnedByteArray.
  • IO.putBuf - Writes a number of bytes from a PinnedByteArray to a Handle.
  • IO.getBufSome - Reads whatever bytes are available on a Handle to a PinnedByteArray. Waits only if no bytes are available.
  • IO.fillBuf - Waits until enough bytes are available on a Handle to fill a PinnedByteArray, then reads them into the array.

Test coverage

Operations on pinned arrays are exercised via transcripts.

@runarorama runarorama requested a review from Copilot August 12, 2025 20:10
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Copy link
Contributor

@dolio dolio left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code looks fine to me.

My only question has to do with e.g. fillBuf. For the non-socket versions, they don't take a size, and just use the whole buffer. Should there be sized versions, too? I guess the normal situation with a buffer is that you just use it for uniform chunking, and don't reuse a larger buffer than necessary for multiple chunk sizes. But I thought I'd bring it up.

Copy link
Member

@pchiusano pchiusano left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool.

For fillBuf, I kind of like the idea of it taking a Nat to control how many bytes it's waiting for. Even if 99% of the time you just pass in the buffer size, it seems more flexible.

@runarorama
Copy link
Contributor Author

I decided to just fill the buffer so we wouldn't have to do a range check, since in the most common case you're just filling the buffer. Taking a Nat is more flexible, but it means that if you're reading in a loop then you're checking the range on every iteration.

@runarorama
Copy link
Contributor Author

But then I guess the Socket read version should also not take a size.

@runarorama
Copy link
Contributor Author

We can just provide both, I think that's the move.

@runarorama runarorama merged commit 336d0e0 into trunk Aug 13, 2025
44 of 45 checks passed
@runarorama runarorama deleted the runarorama/pinnedArrays branch August 13, 2025 19:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants