feat(pass-style,marshal): ByteArray, a binary Passable#2414
Closed
feat(pass-style,marshal): ByteArray, a binary Passable#2414
Conversation
eed07e2 to
1df60e9
Compare
835d70e to
74007b3
Compare
4dd9bf8 to
d19e509
Compare
74007b3 to
6af60d7
Compare
d19e509 to
585042f
Compare
6af60d7 to
affa6b3
Compare
Contributor
Author
|
Converting to draft until more of the stubbed out cases are implemented, and until more of both are tested. |
8dce401 to
06e15fe
Compare
53ec91b to
ce0fd14
Compare
erights
added a commit
that referenced
this pull request
Sep 4, 2024
Staged on #2419 Closes: #XXXX Refs: #2414 #2418 #2419 ## Description #2414 by itself does not work on Node 18 and Node 20 because - those platforms do not have `Array.prototype.transfer`, so #2414 must use `structuredClone` instead - `structuredClone` does exist on Node >= 18, so it should be on supported platforms (though see #2418 ). However, `structuredClone` itself is dangerous and so must not be added to the shared intrinsics. As a result, in #2414 , when `@endo/pass-style` is initialized in a created compartment, it fails to find either `Array.prototype.transfer` and `structuredClone To solve this, @kriskowal suggested that we also shim `Array.prototype.transfer` if needed during `lockdown`, along with other repairs. We are avoiding similarly shimming `Array.prototype.transferToImmutable` because it is not yet standard. But `Array.prototype.transfer` is standard, and so `lockdown` can globally shim it before hardening the shared intrinsics. This PR implements @kriskowal 's suggestion. ### Security Considerations none ### Scaling Considerations by itself, none ### Documentation Considerations nothing signicant. ### Testing Considerations See #2418 . Aside from that, none ### Compatibility and Upgrade Considerations On platforms with neither `Array.prototype.transfer` nor a global `structuredClone`, the ses-shim will *currently* not install an emulation of `Array.prototype.transfer`. However, once we verify that endo is not intended to support platforms without both, we may change lockdown to throw, failing to lock down.
ce0fd14 to
066637e
Compare
066637e to
e696f45
Compare
erights
commented
Oct 14, 2024
| // then according to their lengths. | ||
| // Thus, if the data of ByteArray X is a prefix of | ||
| // the data of ByteArray Y, then X is smaller than Y. | ||
| return comparator(left.byteLength, right.byteLength); |
Contributor
Author
There was a problem hiding this comment.
Should switch to shortlex. See https://en.wikipedia.org/wiki/Shortlex_order
That way, encodePassable could encode as prefix followed by the unescaped content.
erights
commented
Oct 14, 2024
| // Thus, if the data of ByteArray X is a prefix of | ||
| // the data of ByteArray Y, then X is smaller than Y. | ||
| // @ts-expect-error narrowed | ||
| return compareRank(left.byteLength, right.byteLength); |
Contributor
Author
There was a problem hiding this comment.
Should delegate the entire compare to rankOrder, which, btw, will switch to shortlex order.
e696f45 to
6312dd7
Compare
6312dd7 to
566c4c8
Compare
erights
added a commit
that referenced
this pull request
Jan 1, 2025
Closes: #XXXX Refs: - ocapn/ocapn#125 - https://github.com/ocapn/ocapn/blob/28af626441da888c4a520309222e18266dd2f1f2/draft-specifications/Model.md - #2452 - Agoric/agoric-sdk#10084 - https://github.com/tc39/proposal-immutable-arraybuffer - #2414 ## Description Revise Smallcaps cheatsheet to track OCapn as of ocapn/ocapn#125 ### Security Considerations none ### Scaling Considerations none ### Documentation Considerations the point. The cheatsheet was becoming too stale. ### Testing Considerations none ### Compatibility Considerations none ### Upgrade Considerations none
566c4c8 to
a89c4e2
Compare
Contributor
Author
|
Closing in favor of #1538 , since we no longer need to use the ponyfill and avoid the shim. |
3 tasks
erights
added a commit
that referenced
this pull request
May 1, 2025
Closes: #1331 Refs: ocapn/ocapn#5 (comment) #2414 ## Description Recognize Immutable ArrayBuffers, when also frozen, as the new pass-style `byteArray`. Add a branch for each of the dispatches on pass-style that must now understand byte-arrays as well. As of this PR, many of those additional branches are stubbed out with "not yet implemented" errors as placeholders. ### Security Considerations The underlying shim implementation, if we're still using that, uses a normal mutable ArrayBuffer internally, but safely encapsulates it so nothing outside the shim can mutate it. ### Scaling Considerations The underlying shim implementation helps us handle bulk binary data in a largely no-copy or minimal-copy manner. ### Documentation Considerations Will add a new pass-style, so everywhere we enumerate all the pass-styles, we'll need to add byte-array. ### Testing Considerations - [ ] TODO tests. For each stubbed out case, there instead should be a `test.failing` test. ### Compatibility Considerations Old code that dispatched only on the pass-styles it knew about will not correctly handle passables that contain byte-arrays. The shim cannot reasonably support TypedArrays of DataViews backed by Immutable ArrayBuffers. ### Upgrade Considerations - [ ] TODO BREAKING.md - [ ] TODO NEWS.md
boneskull
pushed a commit
that referenced
this pull request
Jun 4, 2025
Closes: #1331 Refs: ocapn/ocapn#5 (comment) #2414 ## Description Recognize Immutable ArrayBuffers, when also frozen, as the new pass-style `byteArray`. Add a branch for each of the dispatches on pass-style that must now understand byte-arrays as well. As of this PR, many of those additional branches are stubbed out with "not yet implemented" errors as placeholders. ### Security Considerations The underlying shim implementation, if we're still using that, uses a normal mutable ArrayBuffer internally, but safely encapsulates it so nothing outside the shim can mutate it. ### Scaling Considerations The underlying shim implementation helps us handle bulk binary data in a largely no-copy or minimal-copy manner. ### Documentation Considerations Will add a new pass-style, so everywhere we enumerate all the pass-styles, we'll need to add byte-array. ### Testing Considerations - [ ] TODO tests. For each stubbed out case, there instead should be a `test.failing` test. ### Compatibility Considerations Old code that dispatched only on the pass-styles it knew about will not correctly handle passables that contain byte-arrays. The shim cannot reasonably support TypedArrays of DataViews backed by Immutable ArrayBuffers. ### Upgrade Considerations - [ ] TODO BREAKING.md - [ ] TODO NEWS.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes: #1331
Refs: #1538 ocapn/ocapn#5 (comment)
Description
Use the
@endo/immutable-arraybufferponyfill directly, rather than the shim@endo/immutable-arraybuffer/shim.js, to provide objects that act like the immutable ArrayBuffers that would result from thetransferToImmutableproposal.Recognize these objects, when also frozen, as the new pass-style
byteArray. Add a branch for each of the dispatches on pass-style that must now understand byte-arrays as well. As of this PR, many of those additional branches are stubbed out with "not yet implemented" errors as placeholders.Security Considerations
Does correctly enforce immutability. But by depending only on the ponyfill rather than the shim, this PR inherits the eval twin problem of the ponyfill. Each instantiation of
passStyleOfwill only recognize the byte-arrays from the instantiation of@endo/pass-stylethat made that byte-array. For environment that, for example, use thepassStyleOfendowed to them by liveslots, liveslots will also need to endow them with the corresponding byte-array maker.In theory, this is not much worse than the need to co-endow
passStyleOfandProxy, so thatpassStyleOfcan reject apparent copy structures that are also proxies.Scaling Considerations
The underlying ponyfill implementation helps us handle bulk binary data in a largely no-copy or minimal-copy manner.
Documentation Considerations
Will add a new pass-style, so everywhere we enumerate all the pass-styles, we'll need to add byte-array.
Testing Considerations
test.failingtest.Compatibility Considerations
Old code that dispatched only on the pass-styles it knew about will not correctly handle passables that contain byte-arrays.
Upgrade Considerations