Skip to content

Regression in minimal preset - Lighthouse fails to start #1173

@praetoriansentry

Description

@praetoriansentry

This command is failing:

kurtosis run github.com/ethpandaops/ethereum-package@0e1548a638fde573fa77840441c4a2cf149ef3d6 '{"network_params":{"preset":"minimal"}}'

A similar command but tied to an older commit seems to be working fine:

kurtosis run github.com/ethpandaops/ethereum-package@82e5a7178138d892c0c31c3839c89d53ffd42d9a '{"network_params":{"preset":"minimal"}}'

These are the error logs in the current commit:

Adding service with name 'cl-1-lighthouse-geth' and image 'ethpandaops/lighthouse:stable'
There was an error executing Starlark code 
An error occurred executing instruction (number 28) at github.com/ethpandaops/ethereum-package/src/cl/lighthouse/lighthouse_launcher.star[89:38]:
  add_service(name="cl-1-lighthouse-geth", config=ServiceConfig(image="ethpandaops/lighthouse:stable", ports={"http": PortSpec(number=4000, transport_protocol="TCP", application_protocol="http"), "metrics": PortSpec(number=5054, transport_protocol="TCP", application_protocol="http"), "quic-discovery": PortSpec(number=9001, transport_protocol="UDP", application_protocol=""), "tcp-discovery": PortSpec(number=9000, transport_protocol="TCP", application_protocol=""), "udp-discovery": PortSpec(number=9000, transport_protocol="UDP", application_protocol="")}, public_ports={}, files={"/jwt": "jwt_file", "/network-configs": "el_cl_genesis_data"}, entrypoint=["sh", "-c"], cmd=["exec lighthouse beacon_node --debug-level=info --datadir=/data/lighthouse/beacon-data --listen-address=0.0.0.0 --port=9000 --http --http-address=0.0.0.0 --http-port=4000 --disable-packet-filter --execution-endpoints=http://{{kurtosis:0c01170c73564d73a41e73c2eb6a0d79:ip_address.runtime_value}}:8551 --jwt-secrets=/jwt/jwtsecret --suggested-fee-recipient=0x8943545177806ED17B9F23F0a21ee5948eCaa776 --disable-enr-auto-update --enr-address=KURTOSIS_IP_ADDR_PLACEHOLDER --enr-tcp-port=9000 --enr-udp-port=9000 --enr-quic-port=9001 --quic-port=9001 --metrics --metrics-address=0.0.0.0 --metrics-allow-origin=* --metrics-port=5054 --enable-private-discovery --testnet-dir=/network-configs --target-peers=0"], env_vars={"RUST_BACKTRACE": "full"}, private_ip_address_placeholder="KURTOSIS_IP_ADDR_PLACEHOLDER", ready_conditions=ReadyCondition(recipe=GetHttpRequestRecipe(port_id="http", endpoint="/eth/v1/node/health"), field="code", assertion="IN", target_value=[200, 206], timeout="30m"), labels={"ethereum-package.client": "lighthouse", "ethereum-package.client-image": "ethpandaops-lighthouse_stable", "ethereum-package.client-language": "rust", "ethereum-package.client-type": "beacon", "ethereum-package.connected-client": "geth", "ethereum-package.node-index": "1", "ethereum-package.sha256": ""}, tolerations=[], node_selectors={}))
  Caused by: Unexpected error occurred starting service 'cl-1-lighthouse-geth'
  Caused by: An error occurred waiting for all TCP and UDP ports to be open for service 'cl-1-lighthouse-geth' with private IP '172.16.0.11'; this is usually due to a misconfiguration in the service itself, so here are the logs:
  == SERVICE 'cl-1-lighthouse-geth' LOGS ===================================
  Sep 02 22:33:21.333 INFO  Lighthouse started                            version: "Lighthouse/v7.1.0-cfb1f73" 
  Sep 02 22:33:21.333 INFO  Configured network                            network_name: "custom (/network-configs)" 
  Sep 02 22:33:21.338 INFO  Data directory initialised                    datadir: /data/lighthouse/beacon-data 
  Sep 02 22:33:21.338 WARN  Discv5 packet filter is disabled             
  Sep 02 22:33:21.342 INFO  Deposit contract                              deploy_block: 0, address: 0x00000000219ab540356cbb839cbe05303d7705fa 
  Sep 02 22:33:21.363 INFO  Blob DB initialized                           path: "/data/lighthouse/beacon-data/beacon/blobs_db", oldest_blob_slot: Some(Slot(0)), oldest_data_column_slot: None 
  Sep 02 22:33:23.009 INFO  Starting from known genesis state            
  Sep 02 22:33:23.012 INFO  Block production enabled                     
  Sep 02 22:33:23.014 CRIT  Failed to start beacon node                   reason: "Failed to build beacon chain: Head block not found in store" 
  Sep 02 22:33:23.014 INFO  Internal shutdown received                   
  Sep 02 22:33:23.014 INFO  Shutting down..                               reason: Failure("Failed to start beacon node") 
  Failed to start beacon node
  
  == FINISHED SERVICE 'cl-1-lighthouse-geth' LOGS ===================================
  Caused by: An error occurred while waiting for all TCP and UDP ports to be open
  Caused by: Unsuccessful ports check for IP '172.16.0.11' and port spec '{privatePortSpec:0xc002602ae0}', even after '240' retries with '500' milliseconds in between retries. Timeout '2m0s' has been reached
  Caused by: An error occurred while calling network address '172.16.0.11:4000' with port protocol 'TCP' and using time out '200ms'
  Caused by: dial tcp 172.16.0.11:4000: i/o timeout

Error encountered running Starlark code.

It seems like there may be some regression in this minimal preset for lighthouse and the consensus layer.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions