Skip to content

The fast node reports triedb layer & Head state missing when stoppping & starting #2829

@galaio

Description

@galaio

System information

Geth version: v1.5.2-alpha & v1.4.15
OS & Version: Windows/Linux/OSX
Commit hash : 3a6ef6a & ec318b9

Expected behaviour

The fast node restarts as normal.

Actual behaviour

image

Steps to reproduce the behaviour

enable the fast node feature with --tries-verify-mode none, and restart the node.

Backtrace

t=2024-12-26T10:13:02+0000 lvl=error msg="Bsc extension Handshake failed" peer=4aae98d0 err=EOF
t=2024-12-26T10:13:03+0000 lvl=info msg="Imported new chain segment" number=45208967 hash=0xfeead671252756c4545f21fa3a5094eac70510728a7df8ed33fa0bb6cfd76497 miner=0xF8de5e61322302b2c6e0a525cC842F10332811bf blocks=1 txs=161 blobs=0 mgas=33.202115 elapsed=119.459ms mgasps=277.9350712873532 snapdiffs="4.06 MiB" triedirty="0.00 B" trieimutabledirty="0.00 B"
t=2024-12-26T10:13:03+0000 lvl=info msg="Got interrupt, shutting down..."
t=2024-12-26T10:13:03+0000 lvl=info msg="HTTP server stopped" endpoint=[::]:8545
t=2024-12-26T10:13:03+0000 lvl=info msg="IPC endpoint closed" url=/server/data-seed/geth.ipc
t=2024-12-26T10:13:04+0000 lvl=info msg="Ethereum protocol stopped"
t=2024-12-26T10:13:04+0000 lvl=info msg="Transaction pool stopped"
t=2024-12-26T10:13:04+0000 lvl=info msg="Failed to journal in-memory trie nodes" err="triedb layer [0xf08e7cf30ae1b3c439ed4068e4ac16259213eb563b64868520511832ee804bcd] missing"
t=2024-12-26T10:13:04+0000 lvl=info msg="Blockchain stopped"
t=2024-12-26T10:13:23+0000 lvl=warn msg="Using deprecated resource file, please move this file to the 'geth' subdirectory of datadir." file=/server/data-seed/nodekey
t=2024-12-26T10:13:23+0000 lvl=warn msg="The light server has been deprecated, please remove this flag" flag=light.serve
t=2024-12-26T10:13:23+0000 lvl=info msg="Automatically disables snap protocol due to verify mode" mode=none
t=2024-12-26T10:13:23+0000 lvl=warn msg="Only local TriesVerifyMode can support snap sync, resetting to full sync" mode=none
t=2024-12-26T10:13:23+0000 lvl=info msg="Set global gas cap" cap=50000000
t=2024-12-26T10:13:23+0000 lvl=info msg="Initializing the KZG library" backend=gokzg
t=2024-12-26T10:13:23+0000 lvl=info msg="Using pebble as the backing database"
t=2024-12-26T10:13:23+0000 lvl=info msg="Allocated cache and file handles" database=/server/data-seed/geth/chaindata cache="1.95 GiB" handles=5000 "memory table"="500.00 MiB"
t=2024-12-26T10:13:24+0000 lvl=info msg="Opened ancient database" database=/server/data-seed/geth/chaindata/ancient/chain readonly=false frozen=45118956
t=2024-12-26T10:13:24+0000 lvl=info msg="State scheme set to already existing disk db" scheme=path
t=2024-12-26T10:13:24+0000 lvl=info msg="Capped dirty cache size" provided="1.22 GiB" adjusted="256.00 MiB"
t=2024-12-26T10:13:24+0000 lvl=info msg="Clean cache size" provided="750.00 MiB" adjusted="1.70 GiB"
t=2024-12-26T10:13:24+0000 lvl=info msg="Allocated memory caches" state_scheme=path trie_clean_cache="1.70 GiB" trie_dirty_cache="256.00 MiB" snapshot_cache="1000.00 MiB"
t=2024-12-26T10:13:24+0000 lvl=info msg="Unprotected transactions allowed"
t=2024-12-26T10:13:24+0000 lvl=info msg=Parlia chainConfig="{ChainID: 56 Homestead: 0 DAO: <nil> DAOSupport: false EIP150: 0 EIP155: 0 EIP158: 0 Byzantium: 0 Constantinople: 0 Petersburg: 0 Istanbul: 0, Muir Glacier: 0, Ramanujan: 0, Niels: 0, MirrorSync: 5184000, Bruno: 13082000, Berlin: 31302048, YOLO v3: <nil>, CatalystBlock: <nil>, London: 31302048, ArrowGlacier: <nil>, MergeFork:<nil>, Euler: 18907621, Gibbs: 23846001, Nano: 21962149, Moran: 22107423, Planck: 27281024,Luban: 29020050, Plato: 30720096, Hertz: 31302048, Hertzfix: 34140700, ShanghaiTime: 1705996800, KeplerTime: 1705996800, FeynmanTime: 1713419340, FeynmanFixTime: 1713419340, CancunTime: 1718863500, HaberTime: 1718863500, HaberFixTime: 1727316120, BohrTime: 1727317200, Engine: parlia}"
t=2024-12-26T10:13:25+0000 lvl=info msg="Initialising Ethereum protocol" network=56 dbversion=8
t=2024-12-26T10:13:25+0000 lvl=error msg="Zero trie root hash!"
t=2024-12-26T10:13:25+0000 lvl=info msg="New journal reader for journal kv"
t=2024-12-26T10:13:25+0000 lvl=info msg="Failed to load journal, discard it" err="unmatched journal want 51a6371b7ba6bf72f5e2dac9c1caa42e3957cda534f93bcde53487050af55cb2 got 56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421"
t=2024-12-26T10:13:25+0000 lvl=info msg="New async node buffer" limit="256.00 MiB" layers=0
t=2024-12-26T10:13:25+0000 lvl=info msg="Initialised chain configuration" config="{ChainID: 56 Homestead: 0 DAO: <nil> DAOSupport: false EIP150: 0 EIP155: 0 EIP158: 0 Byzantium: 0 Constantinople: 0 Petersburg: 0 Istanbul: 0, Muir Glacier: 0, Ramanujan: 0, Niels: 0, MirrorSync: 5184000, Bruno: 13082000, Berlin: 31302048, YOLO v3: <nil>, CatalystBlock: <nil>, London: 31302048, ArrowGlacier: <nil>, MergeFork:<nil>, Euler: 18907621, Gibbs: 23846001, Nano: 21962149, Moran: 22107423, Planck: 27281024,Luban: 29020050, Plato: 30720096, Hertz: 31302048, Hertzfix: 34140700, ShanghaiTime: 1705996800, KeplerTime: 1705996800, FeynmanTime: 1713419340, FeynmanFixTime: 1713419340, CancunTime: 1718863500, HaberTime: 1718863500, HaberFixTime: 1727316120, BohrTime: 1727317200, Engine: parlia}"
t=2024-12-26T10:13:25+0000 lvl=info msg="Loaded most recent local block" number=45208967 hash=0xfeead671252756c4545f21fa3a5094eac70510728a7df8ed33fa0bb6cfd76497 root=0xf08e7cf30ae1b3c439ed4068e4ac16259213eb563b64868520511832ee804bcd td=89840182 age=22s
t=2024-12-26T10:13:25+0000 lvl=info msg="Loaded most recent local finalized block" number=45208965 hash=0xc0321316d4f8a7109afdcff773d0d501841363f14d5bbbbd60bf0c66659e3bc0 root=0x613237e08b1a2294e41621aacfe9b62cc81bd4bec1c599f042292996df5a42fe td=89840178 age=28s
t=2024-12-26T10:13:25+0000 lvl=info msg="Loaded last snap-sync pivot marker" number=6920578
t=2024-12-26T10:13:25+0000 lvl=warn msg="Head state missing, repairing" number=45208967 hash=0xfeead671252756c4545f21fa3a5094eac70510728a7df8ed33fa0bb6cfd76497 diskRoot=0x6922d0fd3f8e6126c8db5eb8ae829de018815ddca27a381e185e24661c34997c
t=2024-12-26T10:13:25+0000 lvl=info msg="Snapshot loaded" diskRoot=0x6922d0fd3f8e6126c8db5eb8ae829de018815ddca27a381e185e24661c34997c root=0xf08e7cf30ae1b3c439ed4068e4ac16259213eb563b64868520511832ee804bcd
t=2024-12-26T10:13:25+0000 lvl=info msg="Rewound to block with state" number=45208879 hash=0x488bb3e5bf8b8c9c61a1c4686dfd85455e0e85a3f48ffce22f19aa473aff23b7
t=2024-12-26T10:13:25+0000 lvl=info msg="Loaded most recent local header" number=45208967 hash=0xfeead671252756c4545f21fa3a5094eac70510728a7df8ed33fa0bb6cfd76497 hash=0xf08e7cf30ae1b3c439ed4068e4ac16259213eb563b64868520511832ee804bcd td=89840182 age=22s
t=2024-12-26T10:13:25+0000 lvl=info msg="Loaded most recent local block" number=45208879 hash=0x488bb3e5bf8b8c9c61a1c4686dfd85455e0e85a3f48ffce22f19aa473aff23b7 root=0x6922d0fd3f8e6126c8db5eb8ae829de018815ddca27a381e185e24661c34997c td=89840008 age=4m50s
t=2024-12-26T10:13:25+0000 lvl=info msg="Loaded most recent local snap block" number=45208967 hash=0xfeead671252756c4545f21fa3a5094eac70510728a7df8ed33fa0bb6cfd76497 root=0xf08e7cf30ae1b3c439ed4068e4ac16259213eb563b64868520511832ee804bcd td=89840182 age=22s
t=2024-12-26T10:13:25+0000 lvl=info msg="Loaded most recent local finalized block" number=45208965 hash=0xc0321316d4f8a7109afdcff773d0d501841363f14d5bbbbd60bf0c66659e3bc0 root=0x613237e08b1a2294e41621aacfe9b62cc81bd4bec1c599f042292996df5a42fe td=89840178 age=28s
t=2024-12-26T10:13:25+0000 lvl=info msg="Loaded last snap-sync pivot marker" number=692057
t=2024-12-26T10:13:25+0000 lvl=info msg="Loaded most recent local finalized block" number=45208965 hash=0xc0321316d4f8a7109afdcff773d0d501841363f14d5bbbbd60bf0c66659e3bc0 root=0x613237e08b1a2294e41621aacfe9b62cc81bd4bec1c599f042292996df5a42fe td=89840178 age=28s
t=2024-12-26T10:13:25+0000 lvl=info msg="Loaded last snap-sync pivot marker" number=6920578
t=2024-12-26T10:13:25+0000 lvl=warn msg="Enabling snapshot recovery" chainhead=45208879 diskbase=45208879
t=2024-12-26T10:13:25+0000 lvl=warn msg="Snapshot is not continuous with chain" snaproot=0xf08e7cf30ae1b3c439ed4068e4ac16259213eb563b64868520511832ee804bcd chainroot=0x6922d0fd3f8e6126c8db5eb8ae829de018815ddca27a381e185e24661c34997c
t=2024-12-26T10:13:25+0000 lvl=info msg="Snapshot loaded" diskRoot=0x6922d0fd3f8e6126c8db5eb8ae829de018815ddca27a381e185e24661c34997c root=0x6922d0fd3f8e6126c8db5eb8ae829de018815ddca27a381e185e24661c34997c
t=2024-12-26T10:13:25+0000 lvl=info msg="Initialized transaction indexer" range="entire chain"
t=2024-12-26T10:13:25+0000 lvl=warn msg="Sanitizing invalid txpool reannounce time" provided=0s updated=1m0s
t=2024-12-26T10:13:25+0000 lvl=info msg="Create votePool successfully"
t=2024-12-26T10:13:25+0000 lvl=warn msg="Sanitizing invalid gasprice oracle price cap" provided=<nil> updated=100000000000
t=2024-12-26T10:13:25+0000 lvl=warn msg="Sanitizing invalid gasprice oracle ignore price" provided=<nil> updated=4
t=2024-12-26T10:13:25+0000 lvl=warn msg="Sanitizing invalid gasprice oracle max header history" provided=0 updated=1
t=2024-12-26T10:13:25+0000 lvl=warn msg="Sanitizing invalid gasprice oracle max block history" provided=0 updated=1
t=2024-12-26T10:13:25+0000 lvl=warn msg="Old unclean shutdowns found" count=12
t=2024-12-26T10:13:25+0000 lvl=warn msg="Unclean shutdown detected" booted=2021-05-13T03:22:26+0000 age=3y8mo3d
t=2024-12-26T10:13:25+0000 lvl=warn msg="Unclean shutdown detected" booted=2021-05-13T03:23:12+0000 age=3y8mo3d
t=2024-12-26T10:13:25+0000 lvl=warn msg="Unclean shutdown detected" booted=2021-05-13T03:24:03+0000 age=3y8mo3d
t=2024-12-26T10:13:25+0000 lvl=warn msg="Unclean shutdown detected" booted=2021-11-17T07:15:13+0000 age=3y1mo3w
t=2024-12-26T10:13:25+0000 lvl=warn msg="Unclean shutdown detected" booted=2023-11-21T08:10:29+0000 age=1y1mo1w
t=2024-12-26T10:13:25+0000 lvl=warn msg="Unclean shutdown detected" booted=2024-07-11T07:12:38+0000 age=5mo2w4d
t=2024-12-26T10:13:25+0000 lvl=warn msg="Unclean shutdown detected" booted=2024-07-17T17:02:42+0000 age=5mo1w4d
t=2024-12-26T10:13:25+0000 lvl=warn msg="Unclean shutdown detected" booted=2024-07-27T06:04:06+0000 age=5mo2d4h
t=2024-12-26T10:13:25+0000 lvl=warn msg="Unclean shutdown detected" booted=2024-08-06T10:39:38+0000 age=4mo3w23h
t=2024-12-26T10:13:25+0000 lvl=warn msg="Unclean shutdown detected" booted=2024-08-26T15:20:25+0000 age=4mo1d18h
t=2024-12-26T10:13:25+0000 lvl=info msg="Enabling metrics collection"
t=2024-12-26T10:13:25+0000 lvl=info msg="Starting peer-to-peer node" instance=Geth/v1.4.15-ec318b9c-20240919/linux-amd64/go1.21.13
t=2024-12-26T10:13:26+0000 lvl=info msg="New local node record" seq=1708962647676 id=1b104cdb92cb45e58e9255ddf5abc68fe176c6aeb5f0ac9e6e3c3432fef19271 ip=54.162.32.1 udp=30311 tcp=30311
t=2024-12-26T10:13:26+0000 lvl=info msg="Started P2P networking" self=enode://accbc0a5af0af03e1ec3b5e80544bdceea48011a6928cd82d2c1a9c38b65fd48ec970ba17bd8c0b0ec21a28faec9efe1d1ce55134784b9207146e2f62d8932ba@54.162.32.1:30311
t=2024-12-26T10:13:26+0000 lvl=info msg="IPC endpoint opened" url=/server/data-seed/geth.ipc
t=2024-12-26T10:13:26+0000 lvl=info msg="WebSocket enabled" url=ws://[::]:8545
t=2024-12-26T10:13:26+0000 lvl=info msg="HTTP server started" endpoint=[::]:8545 auth=false prefix="" cors=* vhosts=*
t=2024-12-26T10:13:26+0000 lvl=info msg="Block synchronisation started"
t=2024-12-26T10:13:26+0000 lvl=error msg="Bsc extension registration failed" peer=3e108e39 err="peer connected on bsc without compatible eth support" name=Geth/v1.2.2-33e6f840...
t=2024-12-26T10:13:26+0000 lvl=info msg="Extend chain" add=87 number=45208966 hash=0x13823bafbf84af6315a2b440f92e1a19550b4dd9e3342fd8425de8361c49218a
t=2024-12-26T10:13:26+0000 lvl=error msg="Bsc extension registration failed" peer=f97a78ef err="peer connected on bsc without compatible eth support" name=Geth/v1.2.5-9f574b0f...
t=2024-12-26T10:13:27+0000 lvl=warn msg="Inserted known block" number=45208967 hash=0xfeead671252756c4545f21fa3a5094eac70510728a7df8ed33fa0bb6cfd76497 uncles=0 txs=161 gas=33202115 root=0xf08e7cf30ae1b3c439ed4068e4ac16259213eb563b64868520511832ee804bcd
t=2024-12-26T10:13:27+0000 lvl=error msg="Please file an issue, skip known block execution without receipt" hash=0xfeead671252756c4545f21fa3a5094eac70510728a7df8ed33fa0bb6cfd76497 number=45208967
t=2024-12-26T10:13:27+0000 lvl=error msg="Bsc extension registration failed" peer=d48055d3 err="peer connected on bsc without compatible eth support" name=Geth/v1.2.5-53fef9e8...
t=2024-12-26T10:13:27+0000 lvl=error msg="Bsc extension registration failed" peer=52a8e876 err="peer connected on bsc without compatible eth support" name=Geth/v1.2.10-2b83693...
t=2024-12-26T10:13:28+0000 lvl=info msg="Imported new chain segment" number=45208974 hash=0x741dfad345f0dc187d777a5b26a036a3e1c5a830669d50e612d5f5713eb17fbf miner=0x75B851a27D7101438F45fce31816501193239A83 blocks=7 txs=922 blobs=0 mgas=87.910439 elapsed=953.537ms mgasps=92.19404588319853 snapdiffs="3.19 MiB" triedirty="0.00 B" trieimutabledirty="0.00 B"

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions