[Merged by Bors] - Retrospective invalidation of exec. payloads for opt. sync by paulhauner · Pull Request #2837 · sigp/lighthouse

paulhauner · 2021-11-30T04:58:41Z

Issue Addressed

NA

Proposed Changes

Adds the functionality to allow blocks to be validated/invalidated after their import as per the optimistic sync spec. This means:

Updating ProtoArray to allow flipping the execution_status of ancestors/descendants based on payload validity updates.
Creating separation between execution_layer and the beacon_chain by creating a PayloadStatus struct.
Refactoring how the execution_layer selects a PayloadStatus from the multiple statuses returned from multiple EEs.
Adding testing framework for optimistic imports.
Add ExecutionBlockHash(Hash256) new-type struct to avoid confusion between beacon block roots and execution payload hashes.
Add merge to FORKS in the Makefile to ensure we test the beacon chain with merge settings.
- Fix some tests here that were failing due to a missing execution layer.

TODO

Balance tests

paulhauner · 2022-02-26T03:43:07Z

Thanks @realbigsean and @michaelsproul! That was a great review, thanks for the improvements and bug fixes 😬

I have addressed all your comments and am waiting for CI to pass ☺️

I can't see this [FORKS] change, maybe you've got it lurking locally?

I must have lost this one between my desktop and laptop, thanks! Added here 2ae6d78.

michaelsproul

Looking good! I'm satisfied with the latest updates, and the test coverage

Just a few docs that I think we should update, then we can merge

consensus/proto_array/src/proto_array.rs

michaelsproul · 2022-02-27T23:12:00Z

consensus/proto_array/src/proto_array.rs

+                0_i64
+                    .checked_sub(node.weight as i64)
+                    .ok_or(Error::InvalidExecutionDeltaOverflow(node_index))?


consensus/proto_array/src/proto_array.rs

Co-authored-by: Michael Sproul <micsproul@gmail.com>

paulhauner · 2022-02-28T00:05:12Z

Thanks for the approval @michaelsproul 🙏

I'll wait for approval/comments from @realbigsean before merging ☺️

Start adding execution_status tests Complete test 01 Add tests 2, 3 Appease clippy

paulhauner · 2022-02-28T03:23:05Z

I added some non-substantive tests for the proto_array crate in 1ab1c43.

I was testing to check balance propagation with and without proposer boost. I didn't find any bugs.

realbigsean

Ok changes look good to me!

paulhauner · 2022-02-28T22:07:32Z

bors r+

🎉

## Issue Addressed NA ## Proposed Changes Adds the functionality to allow blocks to be validated/invalidated after their import as per the [optimistic sync spec](https://github.com/ethereum/consensus-specs/blob/dev/sync/optimistic.md#how-to-optimistically-import-blocks). This means: - Updating `ProtoArray` to allow flipping the `execution_status` of ancestors/descendants based on payload validity updates. - Creating separation between `execution_layer` and the `beacon_chain` by creating a `PayloadStatus` struct. - Refactoring how the `execution_layer` selects a `PayloadStatus` from the multiple statuses returned from multiple EEs. - Adding testing framework for optimistic imports. - Add `ExecutionBlockHash(Hash256)` new-type struct to avoid confusion between *beacon block roots* and *execution payload hashes*. - Add `merge` to [`FORKS`](https://github.com/sigp/lighthouse/blob/c3a793fd73a3b11b130b82032904d39c952869e4/Makefile#L17) in the `Makefile` to ensure we test the beacon chain with merge settings. - Fix some tests here that were failing due to a missing execution layer. ## TODO - [ ] Balance tests Co-authored-by: Mark Mackey <mark@sigmaprime.io>

bors · 2022-03-01T00:29:39Z

Pull request successfully merged into unstable.

Build succeeded:

## Issue Addressed There has been an [`engine_exchangetransitionconfigurationv1`](https://github.com/ethereum/execution-apis/blob/main/src/engine/specification.md#engine_exchangetransitionconfigurationv1) method added to the execution API specs. The `engine_exchangetransitionconfigurationv1` will be polled every 60s as per this PR: ethereum/execution-apis#189. If that PR is merged as-is, then we will be matching the spec. If that PR *is not* merged, we are still fully compatible with the spec, but just doing more than we are required. ## Additional Info - [x] ~~Blocked on #2837~~ - [x] Add method to EE integration tests

## Issue Addressed Resolves #2936 ## Proposed Changes Adds functionality for calling [`validator/prepare_beacon_proposer`](https://ethereum.github.io/beacon-APIs/?urls.primaryName=dev#/Validator/prepareBeaconProposer) in advance. There is a `BeaconChain::prepare_beacon_proposer` method which, which called, computes the proposer for the next slot. If that proposer has been registered via the `validator/prepare_beacon_proposer` API method, then the `beacon_chain.execution_layer` will be provided the `PayloadAttributes` for us in all future forkchoiceUpdated calls. An artificial forkchoiceUpdated call will be created 4s before each slot, when the head updates and when a validator updates their information. Additionally, I added strict ordering for calls from the `BeaconChain` to the `ExecutionLayer`. I'm not certain the `ExecutionLayer` will always maintain this ordering, but it's a good start to have consistency from the `BeaconChain`. There are some deadlock opportunities introduced, they are documented in the code. ## Additional Info - ~~Blocked on #2837~~ Co-authored-by: realbigsean <seananderson33@GMAIL.com>

paulhauner added blocked work-in-progress PR is a work-in-progress labels Nov 30, 2021

paulhauner force-pushed the opt-sync branch from a6ce95d to 0005db3 Compare December 2, 2021 00:15

paulhauner force-pushed the kintsugi branch from 2d8650b to f3c237c Compare December 2, 2021 03:32

paulhauner force-pushed the opt-sync branch from d5fae8f to dfa7d0b Compare December 2, 2021 05:23

paulhauner deleted the branch sigp:unstable December 2, 2021 05:52

paulhauner closed this Dec 2, 2021

paulhauner reopened this Dec 2, 2021

paulhauner changed the base branch from kintsugi to unstable December 2, 2021 05:59

paulhauner force-pushed the opt-sync branch from dfa7d0b to 017070a Compare December 21, 2021 23:27

paulhauner added the bellatrix Required to support the Bellatrix Upgrade label Jan 28, 2022

paulhauner mentioned this pull request Feb 1, 2022

Retrospective transition block checking #2983

Closed

paulhauner self-assigned this Feb 2, 2022

paulhauner added 12 commits February 8, 2022 10:27

Add invalid propagation

074c5eb

Adjust deltas for invalid blocks

a75d7b8

Avoid invalid blocks when finding the head

5add013

Invalidate descendants of invalid blocks too

4a22482

Pass latest_valid_hash from API

bf8bc8c

Add invalidation of payloads

e372e06

Pass root to execute_payload

0922b17

Add FixedPayloadResponse

1fe6376

Add non-passing invalid payload tests

9b28389

Fix failing test

06eebc2

Add simple invalidation test

e32d95e

Fix errors after rebase

9aad031

paulhauner force-pushed the opt-sync branch from 017070a to 9aad031 Compare February 7, 2022 23:28

paulhauner mentioned this pull request Feb 7, 2022

Update to execution-apis v1.0.0-alpha.6 #2985

Closed

Unify invalid payload processing

1e86e22

paulhauner removed the blocked label Feb 8, 2022

paulhauner changed the title ~~Retrospective invalidation of executon payloads for optimistic sync~~ Retrospective invalidation of exec. payloads for opt. sync Feb 8, 2022

paulhauner added 7 commits February 26, 2022 12:04

Add test for just/fin block getter

ae29684

Log errors whilst processing payload status

7eed122

Remove unused struct

8da2668

Simplify best child/desc code

c7d0c6c

Adjust weights

4e3a078

Add error for irrelevant descendant

b3be437

Invalidate more payloads

e9905aa

paulhauner added ready-for-review The code is ready for review and removed waiting-on-author The reviewer has suggested changes and awaits thier implementation. labels Feb 26, 2022

This was referenced Feb 27, 2022

[Merged by Bors] - Prepare proposer #3043

Closed

Add the execution_optimistic flag to HTTP responses #3031

Closed

michaelsproul approved these changes Feb 27, 2022

View reviewed changes

paulhauner and others added 2 commits February 28, 2022 10:53

Update consensus/proto_array/src/proto_array.rs

031ab86

Co-authored-by: Michael Sproul <micsproul@gmail.com>

Update fork choice docs

6dee54c

Add execution_status FC tests

1ab1c43

Start adding execution_status tests Complete test 01 Add tests 2, 3 Appease clippy

realbigsean approved these changes Feb 28, 2022

View reviewed changes

paulhauner added ready-for-merge This PR is ready to merge. and removed ready-for-review The code is ready for review labels Feb 28, 2022

paulhauner mentioned this pull request Feb 28, 2022

[Merged by Bors] - Poll the engine_exchangeTransitionConfigurationV1 endpoint #3047

Closed

2 tasks

bors bot changed the title ~~Retrospective invalidation of exec. payloads for opt. sync~~ [Merged by Bors] - Retrospective invalidation of exec. payloads for opt. sync Mar 1, 2022

bors bot closed this Mar 1, 2022

paulhauner mentioned this pull request Mar 2, 2022

[Merged by Bors] - Rename random to prev_randao #3040

Closed

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Merged by Bors] - Retrospective invalidation of exec. payloads for opt. sync#2837

[Merged by Bors] - Retrospective invalidation of exec. payloads for opt. sync#2837
paulhauner wants to merge 101 commits intosigp:unstablefrom
paulhauner:opt-sync

paulhauner commented Nov 30, 2021 •

edited

Loading

Uh oh!

paulhauner commented Feb 26, 2022

Uh oh!

michaelsproul left a comment

Uh oh!

Uh oh!

michaelsproul Feb 27, 2022

Uh oh!

Uh oh!

paulhauner commented Feb 28, 2022

Uh oh!

paulhauner commented Feb 28, 2022

Uh oh!

realbigsean left a comment

Uh oh!

paulhauner commented Feb 28, 2022

Uh oh!

bors bot commented Mar 1, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

paulhauner commented Nov 30, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Issue Addressed

Proposed Changes

TODO

Uh oh!

paulhauner commented Feb 26, 2022

Uh oh!

michaelsproul left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

michaelsproul Feb 27, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

paulhauner commented Feb 28, 2022

Uh oh!

paulhauner commented Feb 28, 2022

Uh oh!

realbigsean left a comment

Choose a reason for hiding this comment

Uh oh!

paulhauner commented Feb 28, 2022

Uh oh!

bors bot commented Mar 1, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

paulhauner commented Nov 30, 2021 •

edited

Loading