Skip to content

Occasional panic on restart (Pebble/PathDB journal), followed by BAD BLOCK; requires ~1000-block rewind to recover #3349

Closed
Closed
Occasional panic on restart (Pebble/PathDB journal), followed by BAD BLOCK; requires ~1000-block rewind to recover#3349
Assignees
buddh0
@MockY-B

Description

@MockY-B
MockY-B
opened on Sep 10, 2025

System information

BSC/geth: v1.5.18-0.20250721032634-651069fd5104+dirty (from logs)
Go: go1.24.3
Arch/OS: linux/amd64

Description

After a clean-looking shutdown that journals PathDB diff layers and persists dirty state, restarting the node occasionally panics in Pebble while loading the PathDB journal. If we retry restart 1–2 more times, the panic disappears but the node then hits a BAD BLOCK (nonce too low) and refuses to sync until we rewind roughly 1000 blocks, after which syncing succeeds again.

Log

INFO [09-09|16:10:59.291] ApplyDefaultMinerConfig                  DelayLeftOver=20ms
INFO [09-09|16:10:59.291] ApplyDefaultMinerConfig                  MaxWaitProposalInSecs=45
INFO [09-09|16:10:59.291] ApplyDefaultMinerConfig                  Mev.BidSimulationLeftOver=30ms
INFO [09-09|16:10:59.291] ApplyDefaultMinerConfig                  Mev.NoInterruptLeftOver=170ms
INFO [09-09|16:10:59.291] ApplyDefaultMinerConfig                  Mev.MaxBidsPerBuilder=2
INFO [09-09|16:10:59.292] Maximum peer count                       ETH=50 total=50
INFO [09-09|16:10:59.293] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
panic: runtime error: slice bounds out of range [-2112287638:]

goroutine 1 [running]:
github.com/cockroachdb/pebble/sstable.(*blockIter).SeekGE(0xc0105d6660, {0x5054130, 0xb, 0xb}, 0x20?)
        github.com/cockroachdb/pebble@v1.1.2/sstable/block.go:710 +0x6dd
github.com/cockroachdb/pebble/sstable.(*singleLevelIterator).seekGEHelper(0xc0105d6488, {0x5054130, 0xb, 0xb}, 0x5054130?, 0x0)
        github.com/cockroachdb/pebble@v1.1.2/sstable/reader_iter_single_lvl.go:735 +0x254
github.com/cockroachdb/pebble/sstable.(*singleLevelIterator).seekPrefixGE(0xc0105d6488?, {0x5054130?, 0xb?, 0xb?}, {0x5054130?, 0xb?, 0xb?}, 0x0?, 0x0?)
        github.com/cockroachdb/pebble@v1.1.2/sstable/reader_iter_single_lvl.go:828 +0x33f
github.com/cockroachdb/pebble/sstable.(*singleLevelIterator).SeekPrefixGE(0xc01daff080?, {0x5054130?, 0x1?, 0xb?}, {0x5054130?, 0x38?, 0x2953200?}, 0x1?)
        github.com/cockroachdb/pebble@v1.1.2/sstable/reader_iter_single_lvl.go:765 +0xd4
github.com/cockroachdb/pebble.(*levelIter).SeekPrefixGE(0xc01daff080, {0x5054130, 0xb, 0xb}, {0x5054130, 0xb, 0xb}, 0x0)
        github.com/cockroachdb/pebble@v1.1.2/level_iter.go:770 +0x116
github.com/cockroachdb/pebble.(*getIter).Next(0xc01daff018)
        github.com/cockroachdb/pebble@v1.1.2/get_iter.go:207 +0xc12
github.com/cockroachdb/pebble.(*getIter).First(0x0?)
        github.com/cockroachdb/pebble@v1.1.2/get_iter.go:65 +0x13
github.com/cockroachdb/pebble.(*Iterator).iterFirstWithinBounds(0xc01dafea88)
        github.com/cockroachdb/pebble@v1.1.2/iterator.go:2024 +0xcf
github.com/cockroachdb/pebble.(*Iterator).First(0xc01dafea88)
        github.com/cockroachdb/pebble@v1.1.2/iterator.go:1622 +0xe5
github.com/cockroachdb/pebble.(*DB).getInternal(0xc001167688, {0x5054130, 0xb, 0xb}, 0x0, 0x0)
        github.com/cockroachdb/pebble@v1.1.2/db.go:592 +0x3ad
github.com/cockroachdb/pebble.(*DB).Get(...)
        github.com/cockroachdb/pebble@v1.1.2/db.go:520
github.com/ethereum/go-ethereum/ethdb/pebble.(*Database).Get(0x4c69ca?, {0x5054130?, 0xc01048d570?, 0x44197f?})
        github.com/ethereum/go-ethereum/ethdb/pebble/pebble.go:401 +0xfa
github.com/ethereum/go-ethereum/core/rawdb.ReadTrieJournal(...)
        github.com/ethereum/go-ethereum/core/rawdb/accessors_state.go:143
github.com/ethereum/go-ethereum/triedb/pathdb.(*Database).DetermineJournalTypeForReader(0xc01da9a600)
        github.com/ethereum/go-ethereum/triedb/pathdb/database.go:617 +0x56
github.com/ethereum/go-ethereum/triedb/pathdb.(*Database).loadJournal(0xc01da9a600, {0x27, 0xbe, 0x11, 0xc9, 0xca, 0x46, 0xa3, 0xdb, 0x82, ...})
        github.com/ethereum/go-ethereum/triedb/pathdb/journal.go:178 +0x70
github.com/ethereum/go-ethereum/triedb/pathdb.(*Database).loadLayers(0xc01da9a600)
        github.com/ethereum/go-ethereum/triedb/pathdb/journal.go:231 +0x152
github.com/ethereum/go-ethereum/triedb/pathdb.New({0x3834360, 0xc01cbc1f20}, 0xc01d8c1d80?, 0x0)
        github.com/ethereum/go-ethereum/triedb/pathdb/database.go:247 +0x188
github.com/ethereum/go-ethereum/triedb.NewDatabase({0x3834480, 0xc01cbcf128}, 0xc01da9fb90)
        github.com/ethereum/go-ethereum/triedb/database.go:147 +0x2b9
github.com/ethereum/go-ethereum/core.NewBlockChain({0x3834480, 0xc01cbcf128}, 0xc01048dcf8?, 0x0, 0xc016b85e38, {0x381eed0, 0xc01dafe008}, {0x0, 0x0, 0x0, ...}, ...)
        github.com/ethereum/go-ethereum/core/blockchain.go:352 +0x345
github.com/ethereum/go-ethereum/eth.New(0xc000000460, 0xc00019ed08)
        github.com/ethereum/go-ethereum/eth/backend.go:349 +0x1c54
github.com/ethereum/go-ethereum/cmd/utils.RegisterEthService(0xc000000460, 0x2b711da?)
        github.com/ethereum/go-ethereum/cmd/utils/flags.go:2339 +0x25
main.makeFullNode(0xc000b13440)
        github.com/ethereum/go-ethereum/cmd/geth/config.go:254 +0x5d6
main.geth(0xc000b13440)
        github.com/ethereum/go-ethereum/cmd/geth/main.go:382 +0x125
github.com/urfave/cli/v2.(*Command).Run(0xc0001f4dc0, 0xc000b13440, {0xc000050308, 0x2d, 0x2f})
        github.com/urfave/cli/v2@v2.27.1/command.go:279 +0x7be
github.com/urfave/cli/v2.(*App).RunContext(0xc000127200, {0x3800ae8, 0x5487aa0}, {0xc000050308, 0x2d, 0x2f})
        github.com/urfave/cli/v2@v2.27.1/app.go:337 +0x58b
github.com/urfave/cli/v2.(*App).Run(...)
        github.com/urfave/cli/v2@v2.27.1/app.go:311
main.main()
        github.com/ethereum/go-ethereum/cmd/geth/main.go:329 +0x45
bad block log
t=09-09|16:32:21.914 lvl=info msg="Importing heavy sidechain segment" blocks=111 start=3180537 end=3180647
t=09-09|16:32:21.930 lvl=info msg="Skip duplicated bad block" number=3180537 hash=0x3b71d9f5b1f785db6b3416b6e7cf8e00fe9c8fea1ad27ed6e706013d55a453b2
t=09-09|16:32:21.930 lvl=error msg="\n########## BAD BLOCK #########\nBlock: 3180537 (0x3b71d9f5b1f785db6b3416b6e7cf8e00fe9c8fea1ad27ed6e706013d55a453b2)\nMiner: 0x5e2A531A825d8B61BCc305a35a7433e9A8920f0f\nError: could not apply tx 0 [0xe2d77bb89b297aac54a3bc50ef203e27b89c89936b1e6c7479d8235d23a9b373]: nonce too low: address 0x75825dd9bf3665f4A3519D7fa51867F6761d484F, tx: 8 state: 9\nPlatform: geth v1.5.18-0.20250721032634-651069fd5104+dirty go1.24.3 amd64 linux\nVCS: 651069fd-20250721\nChain config: &params.ChainConfig{ChainID:714, HomesteadBlock:0, DAOForkBlock:<nil>, DAOForkSupport:false, EIP150Block:0, EIP155Block:0, EIP158Block:0, ByzantiumBlock:0, ConstantinopleBlock:0, PetersburgBlock:0, IstanbulBlock:0, MuirGlacierBlock:0, BerlinBlock:8, YoloV3Block:<nil>, CatalystBlock:<nil>, LondonBlock:8, ArrowGlacierBlock:<nil>, GrayGlacierBlock:<nil>, MergeNetsplitBlock:<nil>, ShanghaiTime:(*uint64)(0xc010d51ea0), KeplerTime:(*uint64)(0xc010d51ea0), FeynmanTime:(*uint64)(0xc010d51ea0), FeynmanFixTime:(*uint64)(0xc010d51ea0), CancunTime:(*uint64)(0xc010d51ea0), HaberTime:(*uint64)(0xc010d51ea0), HaberFixTime:(*uint64)(0xc010d51ea0), BohrTime:(*uint64)(0xc010d51ea0), PascalTime:(*uint64)(0xc010d51ea0), PragueTime:(*uint64)(0xc010d51ea0), OsakaTime:(*uint64)(nil), LorentzTime:(*uint64)(0xc010d51ea8), MaxwellTime:(*uint64)(0xc010d51ed0), FermiTime:(*uint64)(nil), VerkleTime:(*uint64)(nil), TerminalTotalDifficulty:<nil>, TerminalTotalDifficultyPassed:false, DepositContractAddress:0x0000000000000000000000000000000000000000, EnableVerkleAtGenesis:false, RamanujanBlock:0, NielsBlock:0, MirrorSyncBlock:1, BrunoBlock:1, EulerBlock:2, GibbsBlock:4, NanoBlock:3, MoranBlock:3, PlanckBlock:5, LubanBlock:6, PlatoBlock:7, HertzBlock:8, HertzfixBlock:8, Ethash:(*params.EthashConfig)(nil), Clique:(*params.CliqueConfig)(nil), Parlia:(*params.ParliaConfig)(0x5487aa0), BlobScheduleConfig:(*params.BlobScheduleConfig)(0x544dae0)}\nReceipts: \n##############################\n"
t=09-09|16:32:21.931 lvl=warn msg="Synchronisation failed, dropping peer" peer=fefe0044d84fa6179c329087968e62bb26f04d2b317344de221e379cf4220ecc name=Geth/v1.5.18-651069f... td=6358372 err="retrieved hash chain is invalid: could not apply tx 0 [0xe2d77bb89b297aac54a3bc50ef203e27b89c89936b1e6c7479d8235d23a9b373]: nonce too low: address 0x75825dd9bf3665f4A3519D7fa51867F6761d484F, tx: 8 state: 9"
t=09-09|16:32:26.902 lvl=info msg="enable EVN features" total=0 whiteListPeerCnt=0 onchainValidatorPeerCnt=0
t=09-09|16:32:36.902 lvl=info msg="enable EVN features" total=0 whiteListPeerCnt=0 onchainValidatorPeerCnt=0

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions