Skip to content

engine: add Rest-SSZ spec#793

Draft
MariusVanDerWijden wants to merge 6 commits into
ethereum:mainfrom
MariusVanDerWijden:rest-ssz
Draft

engine: add Rest-SSZ spec#793
MariusVanDerWijden wants to merge 6 commits into
ethereum:mainfrom
MariusVanDerWijden:rest-ssz

Conversation

@MariusVanDerWijden

@MariusVanDerWijden MariusVanDerWijden commented May 8, 2026

Copy link
Copy Markdown
Member

There have been multiple attempts at this already.
Moving away from JSON-RPC to REST-SSZ.
However most kept the engine api as is.
I think we have a good shot at refactoring the engine api with this change.

This Draft does that; the move and the refactoring.

Happy for any feedback I can get!

The core of the change is:

Old method New endpoint Notes
engine_newPayloadV{1..5} POST /{fork}/payloads parentBeaconBlockRoot and executionRequests folded into the SSZ envelope; expectedBlobVersionedHashes removed; INVALID_BLOCK_HASH removed from the status enum
engine_forkchoiceUpdatedV{1..4} POST /{fork}/forkchoice one atomic call; carries forkchoice state, optional payload_attributes, and (Amsterdam+) optional custody_columns
engine_getPayloadV{1..6} GET /{fork}/payloads/{id} poll-style, same semantics as today
engine_getPayloadBodiesByHashV{1,2} POST /{fork}/bodies/hash {fork} selects the response schema (not the era of requested blocks); POST because hash lists are too large for URLs
engine_getPayloadBodiesByRangeV{1,2} GET /{fork}/bodies?from=...&count=... {fork} selects the response schema
engine_getBlobsV1 POST /blobs/v1 independently versioned; legacy version numbers carry forward
engine_getBlobsV2 POST /blobs/v2 all-or-nothing cell proofs
engine_getBlobsV3 POST /blobs/v3 partial-response cell proofs
engine_getBlobsV4 POST /blobs/v4 cell-range selection
engine_getClientVersionV1 GET /identity + X-Engine-Client-Version request header unscoped
engine_exchangeCapabilities GET /capabilities unscoped
engine_exchangeTransitionConfigurationV1 removed already deprecated since Cancun

@arnetheduck

arnetheduck commented May 14, 2026

Copy link
Copy Markdown
Contributor

One thing that I think would make sense would be to reuse the base structure of the beacon api (https://github.com/ethereum/beacon-APIs/) - this includes several things:

  • primitives and their json/ssz encoding - ie in general, where possible, reuse the types from https://github.com/ethereum/beacon-APIs/blob/master/types/primitive.yaml so that we don't end up with pointless minor differences in how for example a number is string-encoded (0x0 vs 0x and the like)
  • execution payloads and other (consensus) spec types - the "shape" of objects in the beacon api generally follows the shape of things as they travel on the gossip network and their SSZ encoding - by reusing these types, we would reduce the maintenance overhead of having to pointlessly reorder and rename the exact same fields from the beacon api/consensus spec just to send the same info to the execution api in a slightly different shape
  • use of the canonical ssz/json encodings specified here: https://github.com/ethereum/consensus-specs/blob/master/ssz/simple-serialize.md#json-mapping - this aids debugging and removes the need to double-specify things
  • explicit encoding of fork in the http headers -> we can then upgrade to an new hard fork "automatically" without having to come up with V2, V3 etc

@arnetheduck

Copy link
Copy Markdown
Contributor

The core of the change is:

For top-up sync we also need "current block number", similar to eth_blockNumber but limited to latest and with a well-defined behavior for when the EL does not have a state.

@developeruche

Copy link
Copy Markdown

I opened #773 a few weeks ago with a narrower scope: adding a single new endpoint (POST /new-payload-with-witness) that combines engine_newPayload and debug_executionWitness into one call and returns the witness SSZ-encoded over HTTP. The motivation was to unblock zkVM provers and zkAttestors from having to follow the chain one block behind.

Since #793 is now doing a full Engine API refactor with the same REST+SSZ foundation, I think the witness endpoint fits naturally into this design. A few thoughts:

The witness endpoint should be added to this new spec. The existing two-call flow (engine_newPayloaddebug_executionWitness) has real-world latency problems at a ~500 MB witness, the JSON-RPC + JSON approach takes ~8s just to return the witness. With HTTP + SSZ and the EL pipeline optimizations I profiled (moving trie writes off the critical path, parallelizing storage trie updates), this drops to ~932ms total EL time. That's comfortably within the 8s newPayload timeout even for worst-case blocks.

Suggested endpoint: POST /{fork}/payloads/with-witness (or folded directly into POST /{fork}/payloads as an optional response field when requested via a query param or Accept header). The response would carry the PayloadStatus + ExecutionWitness SSZ-encoded, consistent with the rest of the new spec.

Benchmark data (ethrex, 203 txs, 36 Mgas, ~502 MB SSZ witness):

Approach EL Total Wire Size
JSON-RPC + JSON 8,131 ms ~502 MB
HTTP + SSZ 1232 ms 502 MB

Happy to close #773 in support of this PR I think it's a better, more seamless flow. Would love to discuss where the witness endpoint fits best in the new endpoint table.

cc: @MariusVanDerWijden

Comment thread src/engine/refactor.md

| Old method | New endpoint | Notes |
| - | - | - |
| `engine_newPayloadV{1..5}` | `POST /{fork}/payloads` | `parentBeaconBlockRoot` and `executionRequests` folded into the SSZ envelope; `expectedBlobVersionedHashes` removed; `INVALID_BLOCK_HASH` removed from the status enum |

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

dropping /engine/v2 prefix misleads a bit

Comment thread src/engine/refactor.md

| Resource | Endpoint | Purpose |
| - | - | - |
| Payload | `POST /engine/v2/{fork}/payloads` | Submit a payload received from the CL gossip network for the EL to validate / import. Replaces `engine_newPayload`. |

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does v2 mean? Was there v1? Why is fork necessary in the URL? Fork name seems to be a way to describe minor API version, but on other side blobs endpoints have it after resource name, not before and it's a number

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also if payload did not change across forks, does it mean we will need same endpoint under different urls? Just single /vN/ looked simpler

Comment thread src/engine/refactor.md

#### Transport

- **HTTP/2 required**, h2c (cleartext) for both TCP and IPC. No

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the consensus REST spec does not have this requirement - we use 1.1 throughout and going to 2.0 would not be viable short-term for Nimbus.

syjn99 added a commit to syjn99/prysm that referenced this pull request Jun 9, 2026
Adds the EnableEngineSSZHTTP feature flag (off by default), the gate for
the REST + SSZ Engine API v2 transport (ethereum/execution-apis#793). No
behavior yet; JSON-RPC engine_* stays the default transport. Wires the
flag into ConfigureBeaconChain and covers it with a test.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
syjn99 added a commit to syjn99/prysm that referenced this pull request Jun 9, 2026
Lays out all eight REST + SSZ Engine API v2 endpoint operations
(ethereum/execution-apis#793) as methods on enginehttp.Client: NewPayload,
ForkchoiceUpdated, GetPayload, GetPayloadBodiesBy{Hash,Range}, GetBlobs,
Capabilities, Identity. Each is a stub returning errNotImplemented with a
per-endpoint TODO(ssz-over-http) comment, plus a skipped test pinning the
intended call shape, so the Phase 4 empty spots are easy to find. No behavior.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
syjn99 added a commit to OffchainLabs/prysm that referenced this pull request Jun 10, 2026
Adds the EnableEngineSSZHTTP feature flag (off by default), the gate for
the REST + SSZ Engine API v2 transport (ethereum/execution-apis#793). No
behavior yet; JSON-RPC engine_* stays the default transport. Wires the
flag into ConfigureBeaconChain and covers it with a test.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
syjn99 added a commit to OffchainLabs/prysm that referenced this pull request Jun 10, 2026
Lays out all eight REST + SSZ Engine API v2 endpoint operations
(ethereum/execution-apis#793) as methods on enginehttp.Client: NewPayload,
ForkchoiceUpdated, GetPayload, GetPayloadBodiesBy{Hash,Range}, GetBlobs,
Capabilities, Identity. Each is a stub returning errNotImplemented with a
per-endpoint TODO(ssz-over-http) comment, plus a skipped test pinning the
intended call shape, so the Phase 4 empty spots are easy to find. No behavior.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@yperbasis

Copy link
Copy Markdown
Member

Implementer feedback: there are now three EL implementations of this spec —
Erigon (erigontech/erigon#21729),
Nethermind (NethermindEth/nethermind#11887) and
ethrex (lambdaclass/ethrex#6770).
Comparing them at the byte level surfaced five places where the draft gets read
differently. Pinning these would stop the implementations from diverging further.

  1. Bare list vs single-field container for top-level bodies. refactor.md says
    requests/responses are "SSZ-encoded List[...]", but the refactor-ssz.md sketch
    wraps them in containers (BodiesByHashRequest, BodiesResponse, BlobsV1Request, …),
    which adds a 4-byte offset prefix on the wire. Erigon and ethrex went bare;
    Nethermind wraps. Affects every bodies/blobs request and response.

  2. validation_error encoding. The sketch pins Optional[String] =
    List[List[byte, MAX_ERROR_BYTES], 1] (a 4-byte inner offset precedes the text when
    present), and Erigon/ethrex implement that — but it's subtle enough that Nethermind
    shipped a plain List[byte, 1024]. A worked byte example with an error present
    (like the existing 41-byte PayloadStatus example) would remove all doubt.

  3. BuiltPayload for pre-Amsterdam forks is undefined. Two concrete splits:
    field order — Erigon/ethrex follow the spec's …, execution_requests, should_override_builder; Nethermind kept the legacy JSON-RPC order with
    execution_requests last — and the Paris shape (bare ExecutionPayload vs
    {payload, block_value}). A per-fork catalogue for BuiltPayload (and the
    envelope / ForkchoiceUpdate), like the one that exists for ExecutionPayload /
    PayloadAttributes / ExecutionPayloadBody, would settle this.

  4. Bodies range queries past the latest block. The text says past-head block
    numbers come back available=false, which implies the response is padded to
    count entries (Erigon does this); Nethermind and ethrex instead truncate at
    head, carrying over the legacy "no trailing nulls" rule. Please state the
    expected response length explicitly.

  5. Fork-era scoping of /bodies. "{fork} selects both the response schema and
    the era" is normative in refactor.md, and Erigon/ethrex filter accordingly, but
    one implementation only uses the segment for schema selection. Worth a MUST.

syjn99 added a commit to syjn99/prysm that referenced this pull request Jun 11, 2026
Implement enginehttp, an HTTP/2 (h2c) transport client for the REST + SSZ
Engine API v2 (ethereum/execution-apis#793). It round-trips arbitrary SSZ
bodies under /engine/v2/{fork}/... with per-request JWT bearer auth,
generic over the fastssz Marshaler/Unmarshaler interfaces, and decodes
RFC 7807 problem+json errors (branching on HTTP status, not the type URI).
Transport-only: not yet wired to the execution service or a feature flag.

Reuses network JWT signing via a new network.NewJWTRoundTripper, and
promotes golang.org/x/net to a direct dependency for x/net/http2.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
syjn99 added a commit to syjn99/prysm that referenced this pull request Jun 11, 2026
Adds the EnableEngineSSZHTTP feature flag (off by default), the gate for
the REST + SSZ Engine API v2 transport (ethereum/execution-apis#793). No
behavior yet; JSON-RPC engine_* stays the default transport. Wires the
flag into ConfigureBeaconChain and covers it with a test.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
syjn99 added a commit to syjn99/prysm that referenced this pull request Jun 11, 2026
Lays out all eight REST + SSZ Engine API v2 endpoint operations
(ethereum/execution-apis#793) as methods on enginehttp.Client: NewPayload,
ForkchoiceUpdated, GetPayload, GetPayloadBodiesBy{Hash,Range}, GetBlobs,
Capabilities, Identity. Each is a stub returning errNotImplemented with a
per-endpoint TODO(ssz-over-http) comment, plus a skipped test pinning the
intended call shape, so the Phase 4 empty spots are easy to find. No behavior.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@LukaszRozmej

Copy link
Copy Markdown

Implementer feedback from working through this on the Nethermind side (NethermindEth/nethermind#11887). All four are spec-internal issues that surfaced while reconciling the SSZ container sketches with EIP-7594, the consensus-specs PeerDAS document, and the running EL implementation against c-kzg-4844.

1. BYTES_PER_CELL derivation is wrong (refactor-ssz.md MAX_* table)

| `BYTES_PER_CELL` | `BYTES_PER_BLOB / CELLS_PER_EXT_BLOB` (1,024) | derived |

The derivation cites EIP-7594 but the formula contradicts it. EIP-7594 defines (see polynomial-commitments-sampling.md):

  • FIELD_ELEMENTS_PER_CELL = 64
  • BYTES_PER_FIELD_ELEMENT = 32 (EIP-4844)
  • BYTES_PER_CELL = FIELD_ELEMENTS_PER_CELL * BYTES_PER_FIELD_ELEMENT = 2048

Cells span the extended blob (FIELD_ELEMENTS_PER_EXT_BLOB = 2 * FIELD_ELEMENTS_PER_BLOB = 8192), so 8192 / 64 = 128 cells × 2048 bytes = 262144 bytes total — twice BYTES_PER_BLOB. The "BYTES_PER_BLOB / CELLS_PER_EXT_BLOB" formula collapses the original-blob byte count over the extended-blob cell count, which is geometrically wrong.

Concrete consequence for implementers: c-kzg-4844's compute_cells writes CELLS_PER_EXT_BLOB * 2048 bytes. Encoding into a ByteVector[1024] slot either throws on length-mismatch or silently truncates half the cell payload, which then fails any KZG cell-proof verification on the CL side. The only viable implementation is BYTES_PER_CELL = 2048.

Proposed change:

-| `BYTES_PER_CELL` | `BYTES_PER_BLOB / CELLS_PER_EXT_BLOB` (1,024) | derived |
+| `BYTES_PER_CELL` | `FIELD_ELEMENTS_PER_CELL * BYTES_PER_FIELD_ELEMENT` (2,048) | [EIP-7594](https://eips.ethereum.org/EIPS/eip-7594) |

And the matching footnote near the /blobs/v4 section:

-`BYTES_PER_CELL` = `BYTES_PER_BLOB / CELLS_PER_EXT_BLOB` = `1024`
-(EIP-7594).
+`BYTES_PER_CELL` = `FIELD_ELEMENTS_PER_CELL * BYTES_PER_FIELD_ELEMENT` = `2048`
+([EIP-7594](https://eips.ethereum.org/EIPS/eip-7594)).

2. bodies.max_count mismatch between table and capabilities example

refactor-ssz.md MAX_* table:

| `MAX_BODIES_REQUEST` | `2**5` (32) | Shanghai |

refactor.md GET /engine/v2/capabilities example body:

"limits": {
  "bodies.max_count": 128,
  ...
}

128 only appears for blob requests (MAX_BLOBS_REQUEST = MAX_VERSIONED_HASHES_PER_REQUEST = 128). The bodies-side normative value is MAX_BODIES_REQUEST = 32 — inherited from Shanghai's engine_getPayloadBodiesByHashV1 which pins the count at 32. The example value of 128 in refactor.md looks like a copy-paste from the blob row.

Suggested fix:

-    "bodies.max_count":           128,
+    "bodies.max_count":           32,

3. payload.max_bytes has no normative constant backing the example

The MAX_* table in refactor-ssz.md doesn't define a MAX_REQUEST_BODY_SIZE (or similar), but the refactor.md capabilities example pins it at 67108864 (64 MiB). Implementers reading downstream of "limits.payload.max_bytes" have no upper-bound name to point at. It would help to either:

  • Add the constant to the SSZ MAX_* table (e.g. MAX_REQUEST_BODY_SIZE = 2**26 = 67108864), then reference it from refactor.md's capabilities example, or
  • Leave it unfixed and note explicitly that the value is operator-chosen and only advertised, not normative.

This came up while reading the Nethermind PR's MaxBodySize constant — its inline comment claimed it matched a spec constant that didn't actually exist.

4. BlobV3EntryWire vs BlobV2EntryWire (minor — sketch cleanup)

refactor-ssz.md §/blobs/v3:

BlobsV3Response {
    entries: List[BlobV2Entry, MAX_BLOBS_REQUEST]
}

So /blobs/v3 is meant to reuse BlobV2Entry verbatim. The text earlier in the same file ("BlobV3Entry = BlobV2Entry { available: Boolean; contents: BlobAndProofV2 } per spec" in the implementer-side comment) reads like there's a separate BlobV3Entry type. There isn't — the two are wire-identical and the sketch should just say "uses BlobV2Entry" without a separate name to avoid confusion in implementations (Nethermind currently has both as distinct types).


Happy to help with the spec wording if useful; the SSZ EL implementation is now aligned with the #1 (2048-byte cells) and #3 (64 MiB request limit, 32 bodies/req) interpretations above.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants