Skip to content

feat(marshal): passable to quasi-quoted Justin expr#2799

Merged
erights merged 2 commits intomasterfrom
markm-qp-quote-passable
May 15, 2025
Merged

feat(marshal): passable to quasi-quoted Justin expr#2799
erights merged 2 commits intomasterfrom
markm-qp-quote-passable

Conversation

@erights
Copy link
Copy Markdown
Contributor

@erights erights commented May 12, 2025

Closes: #XXXX
Refs: Agoric/agoric-sdk#5401

Description

We often need to see the deep contents of a Passable. Indeed, this was kicked off by a slack thread about wanting to quote

M.splitRecord(
    harden({
      assetKind: M.or('nat', 'set', 'copySet', 'copyBag'),
      decimalPlaces: M.and(M.gte(-100), M.lte(100)),
    }),
  )

This PR provides a qp function meaning "quote passable". You can use it where you'd use q. It produces a quasi-quoted Justin expression of the form in which the passable would be passed. For example, the qp of the above pattern is

`makeTagged("match:splitRecord",  [
  {
    assetKind: makeTagged("match:or", [
      "nat",
      "set",
      "copySet",
      "copyBag",
    ]),
    decimalPlaces: makeTagged("match:and", [
      makeTagged("match:gte", -100),
      makeTagged("match:lte", 100),
    ]),
  },
])`

Security Considerations

Unlike q, qp does not lazily postpone rendering the expression into Justin. q can do this easily because it is inside the 'ses' assert.js module and can use the rights amplification available only there. qp depends on '@endo/marshal' which is several levels up from 'ses'. Rather than expose these rights amplification powers (or several other kludges that failed), I just made qp non-lazy. Thus there are no security implications.

Scaling Considerations

Because qp is not lazy, simply constructing an error with Fail or makeError(X...) will pay all the cost of rendering to Justin, whether or not that error is ever used.

Documentation Considerations

Because Justin is a subset of HardenedJS, the Justin expressions produced by qp can be used in documentation if needed. Though see Compatibility Considerations below.

Testing Considerations

  • Only minimal tests written for qp. We should also write a test that loops through all the jsonJustinPairs, just as other Justin tests do.

Compatibility Considerations

Using Justin with symbols exposes in the rendered string how Justin currently represents passable symbols. That changes in #2793. Whichever of these gets merged first, the other will have to reconcile merge conflicts. And any saved Justin strings containing that old Justin representation will need to be updated or marked stale.

Upgrade Considerations

Other than the Compatibility Considerations above, none.

  • Update NEWS.md for user-facing changes.

@erights erights self-assigned this May 12, 2025
@erights erights marked this pull request as ready for review May 12, 2025 04:28
@erights erights force-pushed the markm-qp-quote-passable branch from c06a938 to 165a775 Compare May 12, 2025 04:30
@erights erights requested review from dckc and kriskowal May 12, 2025 04:33
@erights erights force-pushed the markm-qp-quote-passable branch from 165a775 to abbd85b Compare May 12, 2025 05:02
Copy link
Copy Markdown
Contributor

@dckc dckc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

some notes on docs

The code looks plausible. I'd have to study it (and the surrounding context) quite a bit more to be confident that it's correct.

Copy link
Copy Markdown
Contributor

@dckc dckc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm satisfied.

I leave it to you to judge whether to wait for a review from @kriskowal

@erights erights force-pushed the markm-qp-quote-passable branch from 032533a to e515fc5 Compare May 15, 2025 16:59
@erights erights merged commit f6c8b74 into master May 15, 2025
16 checks passed
@erights erights deleted the markm-qp-quote-passable branch May 15, 2025 20:54
erights added a commit that referenced this pull request May 15, 2025
erights added a commit that referenced this pull request May 15, 2025
// The example below is the `patt1` test case from `qp-on-pattern.test.js`.
// Please co-maintain the following doc-comment and that test module.
/**
* `qp` for quote passable as a quasi-quoted Justin expression.
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that this landed, I'd expect it to appear in the list of function on https://endojs.github.io/endo/modules/_endo_marshal.html . But I don't see it. Anybody know why not?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do not.

@mhofman
Copy link
Copy Markdown
Contributor

mhofman commented May 15, 2025

Security Considerations

Unlike q, qp does not lazily postpone rendering the expression into Justin. q can do this easily because it is inside the 'ses' assert.js module and can use the rights amplification available only there. qp depends on '@endo/marshal' which is several levels up from 'ses'. Rather than expose these rights amplification powers (or several other kludges that failed), I just made qp non-lazy. Thus there are no security implications.

Scaling Considerations

Because qp is not lazy, simply constructing an error with Fail or makeError(X...) will pay all the cost of rendering to Justin, whether or not that error is ever used.

My understanding is that ses's quote does not attempt to prevent trapping by the quoted object when lazily evaluated.

Looking at bestEffortStringify is there a reason it couldn't look for an own toString property and invoke that if it exists?

In that case, qp could simply create an object that does what this PR does when its toString method is invoked, to recover the lazy aspect.

@erights
Copy link
Copy Markdown
Contributor Author

erights commented May 15, 2025

That's a good idea. I actually tried to do something similar and gave up because, for some reason, it did not occur to me to modify bestEffortStringify itself. Without mod, I noticed that it looked for a Symbol.toString-named property, and I tried to shoehorn the behavior into a getter for that property. Doing this with a simple non-accessor toString seems good.

If I do it, it'll be in PR separate from #2801 . Or, if anyone else wants to try, I'll be happy to review.

@mhofman
Copy link
Copy Markdown
Contributor

mhofman commented May 15, 2025

Without mod, I noticed that it looked for a Symbol.toString-named property, and I tried to shoehorn the behavior into a getter for that property.

Yeah a {get [Symbol.toStringTag]() { return qp(payload) }} would work too, but be a little more hacky.

erights added a commit that referenced this pull request May 15, 2025
Closes: #XXXX
Refs: #2799 (comment)

## Description

At #2799 (comment) @dckc
suggested
>  I suggest including it in the comment.

I intended to. But somehow I resolved it before actually doing this,
causing me to miss it when evaluating if I should just merge-and-squash
#2799. This PR corrects that omission.

Minor internal comment only. It should not have any other implications.
boneskull pushed a commit that referenced this pull request Jun 4, 2025
Closes: #XXXX
Refs: Agoric/agoric-sdk#5401

## Description

We often need to see the deep contents of a Passable. Indeed, this was
kicked off by a slack thread about wanting to quote
```js
M.splitRecord(
    harden({
      assetKind: M.or('nat', 'set', 'copySet', 'copyBag'),
      decimalPlaces: M.and(M.gte(-100), M.lte(100)),
    }),
  )
```

This PR provides a `qp` function meaning "quote passable". You can use
it where you'd use `q`. It produces a quasi-quoted Justin expression of
the form in which the passable would be passed. For example, the `qp` of
the above pattern is

```js
`makeTagged("match:splitRecord",  [
  {
    assetKind: makeTagged("match:or", [
      "nat",
      "set",
      "copySet",
      "copyBag",
    ]),
    decimalPlaces: makeTagged("match:and", [
      makeTagged("match:gte", -100),
      makeTagged("match:lte", 100),
    ]),
  },
])`
```

### Security Considerations

Unlike `q`, `qp` does not lazily postpone rendering the expression into
Justin. `q` can do this easily because it is inside the `'ses'`
assert.js module and can use the rights amplification available only
there. `qp` depends on `'@endo/marshal'` which is several levels up from
`'ses'`. Rather than expose these rights amplification powers (or
several other kludges that failed), I just made `qp` non-lazy. Thus
there are no security implications.

### Scaling Considerations

Because `qp` is not lazy, simply constructing an error with `Fail` or
`makeError(X...)` will pay all the cost of rendering to Justin, whether
or not that error is ever used.

### Documentation Considerations

Because Justin is a subset of HardenedJS, the Justin expressions
produced by `qp` can be used in documentation if needed. Though see
Compatibility Considerations below.

### Testing Considerations

- [ ] Only minimal tests written for `qp`. We should also write a test
that loops through all the `jsonJustinPairs`, just as other Justin tests
do.

### Compatibility Considerations

Using Justin with symbols exposes in the rendered string how Justin
currently represents passable symbols. That changes in #2793. Whichever
of these gets merged first, the other will have to reconcile merge
conflicts. And any saved Justin strings containing that old Justin
representation will need to be updated or marked stale.

### Upgrade Considerations

Other than the Compatibility Considerations above, none.

- [x] Update `NEWS.md` for user-facing changes.
boneskull pushed a commit that referenced this pull request Jun 4, 2025
Closes: #XXXX
Refs: #2799 (comment)

## Description

At #2799 (comment) @dckc
suggested
>  I suggest including it in the comment.

I intended to. But somehow I resolved it before actually doing this,
causing me to miss it when evaluating if I should just merge-and-squash
#2799. This PR corrects that omission.

Minor internal comment only. It should not have any other implications.
erights added a commit that referenced this pull request Jun 26, 2025
Closes: #XXXX
Refs: #2799 

## Description

#2799 added a new `q`-like quoter named `qp`, for "quote passable", that
renders any value into Justin source text. This was specifically
motivated for displaying patterns within error messages in a readable
and informative manner.

The `@endo/patterns` package itself formulates many error messages to
explain why a specimen did not match a pattern. Some of these message
include quoted forms of nested patterns. But those were still quotes
with `q`, leading @dckc to report still seeing error messages like

```
Error {
    message: '... - Must match one of ["[match:splitRecord]","[match:splitRecord]"]',
  }
```

This PR changes the quoting of nested errors to use `qp` instead, so a
`[match:splitRecord]` might instead render as
```js
`makeTagged("match:splitRecord", [
  {
    assetKind: makeTagged("match:or", [
      "nat",
      "set",
      "copySet",
      "copyBag",
    ]),
    decimalPlaces: makeTagged("match:and", [
      makeTagged("match:gte", -100),
      makeTagged("match:lte", 100),
    ]),
  },
])`
```

This is more verbose. But error messages are designed to carry the
critical clues to help find a bug. For nested patterns, this extra
information is worth the verbosity.

### Security Considerations

We were already quoting nested patterns with `q`, unredacting them,
under the assumption that specimens may carry secrets worth protecting
from callers, but patterns typically do not. Switching these from `q` to
`qp` is consistent with that assumption, but actually exposes info that
`q` happened not to expose. When unredacting happened not to expose
secrets it said were ok to expose, it is easy for developers not to
notice the secrets that could have been shown but weren't. After this
PR, these unredacted secrets will be exposed to callers.

### Scaling Considerations

Where `q` is lazy, `qp` is eager, rendering its argument up front into
Justin even if the containing error is never created or thrown. Thus, we
must take care to only use `qp` when we're already committed to creating
and throwing the error. I believe all such changes in this PR do so, and
so do not add overhead to the happy path.

### Documentation Considerations

I don't know what documentation we provide explaining the interpretation
of error messages. If we have any explain pattern match error messages,
this PR might need us to change those explanations.

### Testing Considerations

Using `qp` to formulate the error messages definitely makes it more
awkward to test is the error message is as expected. You can see this in
the tests this PR needed to change. Fortunately, as shown in these error
message tests, you can use `${q(qp(...))}` to formulate the golden in
the test.

### Compatibility Considerations

Because this PR changes the error messages, as pointed out above, golden
tests of error message text may no longer match, ***causing old working
tests to start failing***. However, this is not ***technically*** a
compat break, because we take the stance that the content of error
messages is not normative, so the error messages can become more
informative over time with a version bump. Thus, I use `:` rather than
`!` to recommend that this PR not cause a version bump. (See
#2868 (review)
below)

### Upgrade Considerations

None.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants