[Merged by Bors] - Add Holesky by paulhauner · Pull Request #4653 · sigp/lighthouse

paulhauner · 2023-08-23T06:22:41Z

Issue Addressed

NA

Proposed Changes

Add the Holesky network config as per https://github.com/eth-clients/holesky/tree/36e4ff2d5138dcb2eb614f0f60fdb060b2adc1e2/custom_config_data.

Since the genesis state is ~190MB, I've opted to not include it in the binary and instead download it at runtime (see #4564 for context). To download this file we have:

A hard-coded URL for a SigP-hosted S3 bucket with the Holesky genesis state. Assuming this download works correctly, users will be none the wiser that the state wasn't included in the binary (apart from some additional logs)
If the user provides a --checkpoint-sync-url flag, then LH will download the genesis state from that server rather than our S3 bucket.
If the user provides a --genesis-state-url flag, then LH will download the genesis state from that server regardless of the S3 bucket or --checkpoint-sync-url flag.
Whenever a genesis state is downloaded it is checked against a checksum baked into the binary.
A genesis state will never be downloaded if it's already included in the binary.
There is a --genesis-state-url-timeout flag to tweak the timeout for downloading the genesis state file.

Log Output

Example of log output when a state is downloaded:

Aug 23 05:40:13.424 INFO Logging to file                         path: "/Users/paul/.lighthouse/holesky/beacon/logs/beacon.log"
Aug 23 05:40:13.425 INFO Lighthouse started                      version: Lighthouse/v4.3.0-bd9931f+
Aug 23 05:40:13.425 INFO Configured for network                  name: holesky
Aug 23 05:40:13.426 INFO Data directory initialised              datadir: /Users/paul/.lighthouse/holesky
Aug 23 05:40:13.427 INFO Deposit contract                        address: 0x4242424242424242424242424242424242424242, deploy_block: 0
Aug 23 05:40:13.427 INFO Downloading genesis state               info: this may take some time on testnets with large validator counts, timeout: 60s, server: https://sigp-public-genesis-states.s3.ap-southeast-2.amazonaws.com/
Aug 23 05:40:29.895 INFO Starting from known genesis state       service: beacon

Example of log output when there are no URLs specified:

Aug 23 06:29:51.645 INFO Logging to file                         path: "/Users/paul/.lighthouse/goerli/beacon/logs/beacon.log"
Aug 23 06:29:51.646 INFO Lighthouse started                      version: Lighthouse/v4.3.0-666a39c+
Aug 23 06:29:51.646 INFO Configured for network                  name: goerli
Aug 23 06:29:51.647 INFO Data directory initialised              datadir: /Users/paul/.lighthouse/goerli
Aug 23 06:29:51.647 INFO Deposit contract                        address: 0xff50ed3d0ec03ac01d4c79aad74928bff48a7b2b, deploy_block: 4367322
The genesis state is not present in the binary and there are no known download URLs. Please use --checkpoint-sync-url or --genesis-state-url.

Additional Info

I tested the --genesis-state-url flag with all 9 Goerli checkpoint sync servers on https://eth-clients.github.io/checkpoint-sync-endpoints/ and they all worked 🎉

My IDE eagerly formatted some Cargo.toml. I've disabled it but I don't see the value in spending time reverting the changes that are already there.

I also added the GenesisStateBytes enum to avoid an unnecessary clone on the genesis state bytes baked into the binary. This is not a huge deal on Mainnet, but will become more relevant when testing with big genesis states.

When we do a fresh checkpoint sync we're downloading the genesis state to check the genesis_validators_root against the finalised state we receive. This is not entirely pointless, since we verify the checksum when we download the genesis state so we are actually guaranteeing that the finalised state is on the same network. There might be a smarter/less-download-y way to go about this, but I've run out of cycles to figure that out. Perhaps we can grab it in the next release?

realbigsean · 2023-08-24T14:34:23Z

FYI there was discussion on the CL call about serving this state snappy compressed in checkpointz servers

common/eth2_network_config/built_in_network_configs/holesky/boot_enr.yaml

paulhauner · 2023-08-24T22:22:20Z

FYI there was discussion on the CL call about serving this state snappy compressed in checkpointz servers

Thanks, I'm currently lobbying against that: https://discord.com/channels/595666850260713488/745077610685661265/1144388115230887996

common/eth2_network_config/built_in_network_configs/holesky/config.yaml

paulhauner · 2023-08-25T06:27:36Z

I've updated our ENR to be Capella-256-ready. I also added Teku's ENRs.

See a944a7e

paulhauner · 2023-08-28T00:09:52Z

Alrighty, this one is now ready to go.

michaelsproul

Looks good to me! Just a few small nitpicks, then I think we're ready to merge 🚀

account_manager/src/validator/slashing_protection.rs

common/eth2_network_config/built_in_network_configs/holesky/config.yaml

lighthouse/src/main.rs

common/eth2_network_config/built_in_network_configs/holesky/boot_enr.yaml

…ot_enr.yaml Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>

…nfig.yaml Co-authored-by: Michael Sproul <micsproul@gmail.com>

michaelsproul

Happy to merge once merge conflicts are resolved!

paulhauner · 2023-08-28T02:17:19Z

Happy to merge once merge conflicts are resolved!

Thanks, I'll see how CI goes and then merge at will. Thanks for the review @jimmygchen and @michaelsproul 🙏

paulhauner · 2023-08-28T05:34:11Z

bors r+

## Issue Addressed NA ## Proposed Changes Add the Holesky network config as per https://github.com/eth-clients/holesky/tree/36e4ff2d5138dcb2eb614f0f60fdb060b2adc1e2/custom_config_data. Since the genesis state is ~190MB, I've opted to *not* include it in the binary and instead download it at runtime (see #4564 for context). To download this file we have: - A hard-coded URL for a SigP-hosted S3 bucket with the Holesky genesis state. Assuming this download works correctly, users will be none the wiser that the state wasn't included in the binary (apart from some additional logs) - If the user provides a `--checkpoint-sync-url` flag, then LH will download the genesis state from that server rather than our S3 bucket. - If the user provides a `--genesis-state-url` flag, then LH will download the genesis state from that server regardless of the S3 bucket or `--checkpoint-sync-url` flag. - Whenever a genesis state is downloaded it is checked against a checksum baked into the binary. - A genesis state will never be downloaded if it's already included in the binary. - There is a `--genesis-state-url-timeout` flag to tweak the timeout for downloading the genesis state file. ## Log Output Example of log output when a state is downloaded: ```bash Aug 23 05:40:13.424 INFO Logging to file path: "/Users/paul/.lighthouse/holesky/beacon/logs/beacon.log" Aug 23 05:40:13.425 INFO Lighthouse started version: Lighthouse/v4.3.0-bd9931f+ Aug 23 05:40:13.425 INFO Configured for network name: holesky Aug 23 05:40:13.426 INFO Data directory initialised datadir: /Users/paul/.lighthouse/holesky Aug 23 05:40:13.427 INFO Deposit contract address: 0x4242424242424242424242424242424242424242, deploy_block: 0 Aug 23 05:40:13.427 INFO Downloading genesis state info: this may take some time on testnets with large validator counts, timeout: 60s, server: https://sigp-public-genesis-states.s3.ap-southeast-2.amazonaws.com/ Aug 23 05:40:29.895 INFO Starting from known genesis state service: beacon ``` Example of log output when there are no URLs specified: ``` Aug 23 06:29:51.645 INFO Logging to file path: "/Users/paul/.lighthouse/goerli/beacon/logs/beacon.log" Aug 23 06:29:51.646 INFO Lighthouse started version: Lighthouse/v4.3.0-666a39c+ Aug 23 06:29:51.646 INFO Configured for network name: goerli Aug 23 06:29:51.647 INFO Data directory initialised datadir: /Users/paul/.lighthouse/goerli Aug 23 06:29:51.647 INFO Deposit contract address: 0xff50ed3d0ec03ac01d4c79aad74928bff48a7b2b, deploy_block: 4367322 The genesis state is not present in the binary and there are no known download URLs. Please use --checkpoint-sync-url or --genesis-state-url. ``` ## Additional Info I tested the `--genesis-state-url` flag with all 9 Goerli checkpoint sync servers on https://eth-clients.github.io/checkpoint-sync-endpoints/ and they all worked 🎉 My IDE eagerly formatted some `Cargo.toml`. I've disabled it but I don't see the value in spending time reverting the changes that are already there. I also added the `GenesisStateBytes` enum to avoid an unnecessary clone on the genesis state bytes baked into the binary. This is not a huge deal on Mainnet, but will become more relevant when testing with big genesis states. When we do a fresh checkpoint sync we're downloading the genesis state to check the `genesis_validators_root` against the finalised state we receive. This is not *entirely* pointless, since we verify the checksum when we download the genesis state so we are actually guaranteeing that the finalised state is on the same network. There might be a smarter/less-download-y way to go about this, but I've run out of cycles to figure that out. Perhaps we can grab it in the next release?

bors · 2023-08-28T06:26:58Z

Pull request successfully merged into unstable.

Build succeeded!

The publicly hosted instance of bors-ng is deprecated and will go away soon.

If you want to self-host your own instance, instructions are here.
For more help, visit the forum.

If you want to switch to GitHub's built-in merge queue, visit their help page.

## Issue Addressed NA ## Proposed Changes Add the Holesky network config as per https://github.com/eth-clients/holesky/tree/36e4ff2d5138dcb2eb614f0f60fdb060b2adc1e2/custom_config_data. Since the genesis state is ~190MB, I've opted to *not* include it in the binary and instead download it at runtime (see sigp#4564 for context). To download this file we have: - A hard-coded URL for a SigP-hosted S3 bucket with the Holesky genesis state. Assuming this download works correctly, users will be none the wiser that the state wasn't included in the binary (apart from some additional logs) - If the user provides a `--checkpoint-sync-url` flag, then LH will download the genesis state from that server rather than our S3 bucket. - If the user provides a `--genesis-state-url` flag, then LH will download the genesis state from that server regardless of the S3 bucket or `--checkpoint-sync-url` flag. - Whenever a genesis state is downloaded it is checked against a checksum baked into the binary. - A genesis state will never be downloaded if it's already included in the binary. - There is a `--genesis-state-url-timeout` flag to tweak the timeout for downloading the genesis state file. ## Log Output Example of log output when a state is downloaded: ```bash Aug 23 05:40:13.424 INFO Logging to file path: "/Users/paul/.lighthouse/holesky/beacon/logs/beacon.log" Aug 23 05:40:13.425 INFO Lighthouse started version: Lighthouse/v4.3.0-bd9931f+ Aug 23 05:40:13.425 INFO Configured for network name: holesky Aug 23 05:40:13.426 INFO Data directory initialised datadir: /Users/paul/.lighthouse/holesky Aug 23 05:40:13.427 INFO Deposit contract address: 0x4242424242424242424242424242424242424242, deploy_block: 0 Aug 23 05:40:13.427 INFO Downloading genesis state info: this may take some time on testnets with large validator counts, timeout: 60s, server: https://sigp-public-genesis-states.s3.ap-southeast-2.amazonaws.com/ Aug 23 05:40:29.895 INFO Starting from known genesis state service: beacon ``` Example of log output when there are no URLs specified: ``` Aug 23 06:29:51.645 INFO Logging to file path: "/Users/paul/.lighthouse/goerli/beacon/logs/beacon.log" Aug 23 06:29:51.646 INFO Lighthouse started version: Lighthouse/v4.3.0-666a39c+ Aug 23 06:29:51.646 INFO Configured for network name: goerli Aug 23 06:29:51.647 INFO Data directory initialised datadir: /Users/paul/.lighthouse/goerli Aug 23 06:29:51.647 INFO Deposit contract address: 0xff50ed3d0ec03ac01d4c79aad74928bff48a7b2b, deploy_block: 4367322 The genesis state is not present in the binary and there are no known download URLs. Please use --checkpoint-sync-url or --genesis-state-url. ``` ## Additional Info I tested the `--genesis-state-url` flag with all 9 Goerli checkpoint sync servers on https://eth-clients.github.io/checkpoint-sync-endpoints/ and they all worked 🎉 My IDE eagerly formatted some `Cargo.toml`. I've disabled it but I don't see the value in spending time reverting the changes that are already there. I also added the `GenesisStateBytes` enum to avoid an unnecessary clone on the genesis state bytes baked into the binary. This is not a huge deal on Mainnet, but will become more relevant when testing with big genesis states. When we do a fresh checkpoint sync we're downloading the genesis state to check the `genesis_validators_root` against the finalised state we receive. This is not *entirely* pointless, since we verify the checksum when we download the genesis state so we are actually guaranteeing that the finalised state is on the same network. There might be a smarter/less-download-y way to go about this, but I've run out of cycles to figure that out. Perhaps we can grab it in the next release?

NA Add the Holesky network config as per https://github.com/eth-clients/holesky/tree/36e4ff2d5138dcb2eb614f0f60fdb060b2adc1e2/custom_config_data. Since the genesis state is ~190MB, I've opted to *not* include it in the binary and instead download it at runtime (see sigp#4564 for context). To download this file we have: - A hard-coded URL for a SigP-hosted S3 bucket with the Holesky genesis state. Assuming this download works correctly, users will be none the wiser that the state wasn't included in the binary (apart from some additional logs) - If the user provides a `--checkpoint-sync-url` flag, then LH will download the genesis state from that server rather than our S3 bucket. - If the user provides a `--genesis-state-url` flag, then LH will download the genesis state from that server regardless of the S3 bucket or `--checkpoint-sync-url` flag. - Whenever a genesis state is downloaded it is checked against a checksum baked into the binary. - A genesis state will never be downloaded if it's already included in the binary. - There is a `--genesis-state-url-timeout` flag to tweak the timeout for downloading the genesis state file. Example of log output when a state is downloaded: ```bash Aug 23 05:40:13.424 INFO Logging to file path: "/Users/paul/.lighthouse/holesky/beacon/logs/beacon.log" Aug 23 05:40:13.425 INFO Lighthouse started version: Lighthouse/v4.3.0-bd9931f+ Aug 23 05:40:13.425 INFO Configured for network name: holesky Aug 23 05:40:13.426 INFO Data directory initialised datadir: /Users/paul/.lighthouse/holesky Aug 23 05:40:13.427 INFO Deposit contract address: 0x4242424242424242424242424242424242424242, deploy_block: 0 Aug 23 05:40:13.427 INFO Downloading genesis state info: this may take some time on testnets with large validator counts, timeout: 60s, server: https://sigp-public-genesis-states.s3.ap-southeast-2.amazonaws.com/ Aug 23 05:40:29.895 INFO Starting from known genesis state service: beacon ``` Example of log output when there are no URLs specified: ``` Aug 23 06:29:51.645 INFO Logging to file path: "/Users/paul/.lighthouse/goerli/beacon/logs/beacon.log" Aug 23 06:29:51.646 INFO Lighthouse started version: Lighthouse/v4.3.0-666a39c+ Aug 23 06:29:51.646 INFO Configured for network name: goerli Aug 23 06:29:51.647 INFO Data directory initialised datadir: /Users/paul/.lighthouse/goerli Aug 23 06:29:51.647 INFO Deposit contract address: 0xff50ed3d0ec03ac01d4c79aad74928bff48a7b2b, deploy_block: 4367322 The genesis state is not present in the binary and there are no known download URLs. Please use --checkpoint-sync-url or --genesis-state-url. ``` I tested the `--genesis-state-url` flag with all 9 Goerli checkpoint sync servers on https://eth-clients.github.io/checkpoint-sync-endpoints/ and they all worked 🎉 My IDE eagerly formatted some `Cargo.toml`. I've disabled it but I don't see the value in spending time reverting the changes that are already there. I also added the `GenesisStateBytes` enum to avoid an unnecessary clone on the genesis state bytes baked into the binary. This is not a huge deal on Mainnet, but will become more relevant when testing with big genesis states. When we do a fresh checkpoint sync we're downloading the genesis state to check the `genesis_validators_root` against the finalised state we receive. This is not *entirely* pointless, since we verify the checksum when we download the genesis state so we are actually guaranteeing that the finalised state is on the same network. There might be a smarter/less-download-y way to go about this, but I've run out of cycles to figure that out. Perhaps we can grab it in the next release?

paulhauner added 10 commits August 23, 2023 12:59

Add holesky via Github download

98e2f7a

Add checksum

4f9f8c3

Simplify checksum deserialization

3e8207f

Support using the checkpoint sync URL

ca721c3

Always assume checkpointz server

09101fb

Tidy, fix bugs

36dc36a

Pass logger into download function

bd9931f

Add timeout flag

1f5a8fe

Fix preferences for state url

34e8f9c

Fix clippy lints

666a39c

paulhauner added work-in-progress PR is a work-in-progress v4.4.1 ETA August 2023 labels Aug 23, 2023

paulhauner mentioned this pull request Aug 23, 2023

fetch genesis from remote url #4644

Closed

Fix failing test and error message

410c873

paulhauner mentioned this pull request Aug 24, 2023

Reduce binary size by compressing genesis.ssz #4564

Open

paulhauner added 3 commits August 24, 2023 11:53

Load genesis state on demand

6ea4c2a

Avoid cloning genesis state bytes

bbf1aa6

Only fetch eth2 config when needed

2ac56d4

paulhauner mentioned this pull request Aug 24, 2023

Attestation service error whilst waiting for genesis #4654

Closed

paulhauner added 4 commits August 24, 2023 12:48

Improve error message

75fc721

Merge branch 'unstable' into add-holesky

1e10bbb

Tidy

e40b6d5

Fix clippy lints

48c03dd

paulhauner added ready-for-review The code is ready for review and removed work-in-progress PR is a work-in-progress labels Aug 24, 2023

antondlr reviewed Aug 24, 2023

View reviewed changes

common/eth2_network_config/built_in_network_configs/holesky/boot_enr.yaml Show resolved Hide resolved

jimmygchen marked this pull request as ready for review August 25, 2023 00:33

jimmygchen reviewed Aug 25, 2023

View reviewed changes

common/eth2_network_config/built_in_network_configs/holesky/config.yaml Outdated Show resolved Hide resolved

Update SigP boot ENR and add Teku

a944a7e

paulhauner removed the blocked label Aug 25, 2023

michaelsproul mentioned this pull request Aug 25, 2023

Add Ephemery support #4664

Open

12 tasks

Remove duplicate ENR

1540e78

michaelsproul approved these changes Aug 28, 2023

View reviewed changes

account_manager/src/validator/slashing_protection.rs Outdated Show resolved Hide resolved

common/eth2_network_config/built_in_network_configs/holesky/config.yaml Outdated Show resolved Hide resolved

lighthouse/src/main.rs Show resolved Hide resolved

jimmygchen reviewed Aug 28, 2023

View reviewed changes

common/eth2_network_config/built_in_network_configs/holesky/boot_enr.yaml Outdated Show resolved Hide resolved

paulhauner and others added 5 commits August 28, 2023 11:22

Update common/eth2_network_config/built_in_network_configs/holesky/bo…

f65c405

…ot_enr.yaml Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>

Update common/eth2_network_config/built_in_network_configs/holesky/co…

0b05d08

…nfig.yaml Co-authored-by: Michael Sproul <micsproul@gmail.com>

Add tests for new flags

9bdc243

Add GVR to testnet config

add8bd6

Fix comment

726d7ed

michaelsproul approved these changes Aug 28, 2023

View reviewed changes

Merge branch 'unstable' into add-holesky

1e809c7

Run cargo fmt

5d25579

paulhauner added ready-for-merge This PR is ready to merge. and removed ready-for-review The code is ready for review labels Aug 28, 2023

paulhauner mentioned this pull request Aug 28, 2023

[Merged by Bors] - Release v4.4.0 #4673

Closed

bors bot changed the title ~~Add Holesky~~ [Merged by Bors] - Add Holesky Aug 28, 2023

bors bot closed this Aug 28, 2023

paulhauner mentioned this pull request Aug 29, 2023

[Merged by Bors] - feat: add chiado #4530

Closed

michaelsproul mentioned this pull request Sep 15, 2023

Genesis state download sometimes panics #4738

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Merged by Bors] - Add Holesky#4653

[Merged by Bors] - Add Holesky#4653
paulhauner wants to merge 35 commits intosigp:unstablefrom
paulhauner:add-holesky

paulhauner commented Aug 23, 2023 •

edited

Loading

Uh oh!

realbigsean commented Aug 24, 2023

Uh oh!

Uh oh!

paulhauner commented Aug 24, 2023

Uh oh!

Uh oh!

paulhauner commented Aug 25, 2023

Uh oh!

paulhauner commented Aug 28, 2023

Uh oh!

michaelsproul left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

michaelsproul left a comment

Uh oh!

paulhauner commented Aug 28, 2023

Uh oh!

paulhauner commented Aug 28, 2023

Uh oh!

bors bot commented Aug 28, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

paulhauner commented Aug 23, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Issue Addressed

Proposed Changes

Log Output

Additional Info

Uh oh!

realbigsean commented Aug 24, 2023

Uh oh!

Uh oh!

paulhauner commented Aug 24, 2023

Uh oh!

Uh oh!

paulhauner commented Aug 25, 2023

Uh oh!

paulhauner commented Aug 28, 2023

Uh oh!

michaelsproul left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

michaelsproul left a comment

Choose a reason for hiding this comment

Uh oh!

paulhauner commented Aug 28, 2023

Uh oh!

paulhauner commented Aug 28, 2023

Uh oh!

bors bot commented Aug 28, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

paulhauner commented Aug 23, 2023 •

edited

Loading