Panic in blockstm After Chain Rewind Triggered by Milestone Hash Mismatch #1631
Description
System information
Bor client version: 2.2.8
Heimdall client version: 0.2.9
OS & Version: Linux / Ubuntu 24.04
Environment: Polygon Mainnet
Type of node: Validator
Overview of the problem
Bor encountered a panic in blockstm.UpdateDeps after a chain rewind triggered by a milestone endblock hash mismatch. The code attempted to access an element that was out of range.
Reproduction Steps
Difficult to reproduce deliberately.
Logs / Traces / Output / Error Messages
Jul 14 03:56:39 bor[1044358]: INFO [07-14|03:56:39.777] Commit new sealing work number=73,936,067 sealhash=c3f155..1bae44 txs=150 gas=27,093,478 fees=4.908526128 elapsed=826.039ms
Jul 14 03:56:39 bor[1044358]: INFO [07-14|03:56:39.777] Waiting for slot to sign and propagate number=73,936,067 hash=0x1a9cde0 delay-in-sec=32,222,315,000 delay=32.222s
Jul 14 03:56:40 bor[1044358]: INFO [07-14|03:56:40.784] Chain reorg detected number=73,936,065 hash=b62c02..ccce49 drop=1 dropfrom=6041ad..8f950e add=1 addfrom=46199a..be0cbe
Jul 14 03:56:40 bor[1044358]: INFO [07-14|03:56:40.813] Imported new chain segment number=73,936,066 hash=46199a..be0cbe blocks=1 txs=0 mgas=0.000 elapsed=35.571ms mgasps=0.000 snapdiffs=6.
21MiB triediffs=208.52MiB triedirty=255.75MiB
Jul 14 03:56:40 bor[1044358]: WARN [07-14|03:56:40.908] End block hash mismatch while whitelisting milestone expected=46199a5fd3debfc131b5a250352c5e74f64555d96889afc0a0dfdbee33be0cbe got=6041adadfd723e2397a229044fdb3
dd504e7a2d3cc09706a6e743738708f950e
Jul 14 03:56:40 bor[1044358]: INFO [07-14|03:56:40.908] Rewinding chain due to milestone endblock hash mismatch number=73,936,065
Jul 14 03:56:40 bor[1044358]: WARN [07-14|03:56:40.909] Rewinding blockchain to block target=73,936,065
Jul 14 03:56:40 bor[1044358]: INFO [07-14|03:56:40.909] Rewound to block with state number=73,936,065 hash=b62c02..ccce49
Jul 14 03:56:40 bor[1044358]: INFO [07-14|03:56:40.923] Loaded most recent local block number=73,936,065 hash=b62c02..ccce49 td=1,420,797,717 age=6s
Jul 14 03:56:40 bor[1044358]: INFO [07-14|03:56:40.923] Inserting canonical chain from=73,936,066 hash=6041ad..8f950e to=73,936,066 hash=6041ad..8f950e
Jul 14 03:56:40 bor[1044358]: WARN [07-14|03:56:40.925] blockstm - env.tcount > len(env.depsMVFullWriteList) env.tcount=19 len(depsMVFullWriteList)=4
Jul 14 03:56:40 bor[1044358]: panic: runtime error: index out of range [4] with length 4
Jul 14 03:56:40 bor[1044358]: goroutine 4988299 [running]:
Jul 14 03:56:40 bor[1044358]: github.com/ethereum/go-ethereum/core/blockstm.UpdateDeps(0xc0374a4c30, {0x12, {0xc05dcbb900, 0x27, 0x27}, {0xc0251480c0, 0x4, 0x4}})
Jul 14 03:56:40 bor[1044358]: /home/ubuntu/bor/core/blockstm/dag.go:100 +0x450
Jul 14 03:56:40 bor[1044358]: github.com/ethereum/go-ethereum/miner.(*worker).commitTransactions.func2(0xc03778bab0)
Jul 14 03:56:40 bor[1044358]: /home/ubuntu/bor/miner/worker.go:948 +0xff
Jul 14 03:56:40 bor[1044358]: created by github.com/ethereum/go-ethereum/miner.(*worker).commitTransactions in goroutine 2423
Jul 14 03:56:40 bor[1044358]: /home/ubuntu/bor/miner/worker.go:946 +0x329
Jul 14 03:56:43 systemd[1]: bor.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Another occurrence:
Jul 14 01:34:54 bor[1022028]: INFO [07-14|01:34:54.349] Rewinding chain due to milestone endblock hash mismatch number=73,932,417
Jul 14 01:34:54 bor[1022028]: WARN [07-14|01:34:54.350] Rewinding blockchain to block target=73,932,417
Jul 14 01:34:54 bor[1022028]: INFO [07-14|01:34:54.350] Rewound to block with state number=73,932,417 hash=a76f3f..b26498
Jul 14 01:34:54 bor[1022028]: INFO [07-14|01:34:54.359] Loaded most recent local block number=73,932,417 hash=a76f3f..b26498 td=1,420,732,884 age=6s
Jul 14 01:34:54 bor[1022028]: INFO [07-14|01:34:54.359] Inserting canonical chain from=73,932,418 hash=f98230..3c1b4e to=73,932,418 hash=f98230..3c
1b4e
Jul 14 01:34:54 bor[1022028]: WARN [07-14|01:34:54.359] blockstm - env.tcount > len(env.depsMVFullWriteList) env.tcount=327 len(depsMVFullWriteList)=321
Jul 14 01:34:54 bor[1022028]: panic: runtime error: index out of range [321] with length 321
Jul 14 01:34:54 bor[1022028]: goroutine 3159113 [running]:
Jul 14 01:34:54 bor[1022028]: github.com/ethereum/go-ethereum/core/blockstm.UpdateDeps(0xc075eef860, {0x146, {0xc000ae1500, 0x21, 0x21}, {0xc06e976008, 0x141, 0x155}})
Jul 14 01:34:54 bor[1022028]: /home/ubuntu/bor/core/blockstm/dag.go:100 +0x450
Jul 14 01:34:54 bor[1022028]: github.com/ethereum/go-ethereum/miner.(*worker).commitTransactions.func2(0xc042134000)
Jul 14 01:34:54 bor[1022028]: /home/ubuntu/bor/miner/worker.go:948 +0xff
Jul 14 01:34:54 bor[1022028]: created by github.com/ethereum/go-ethereum/miner.(*worker).commitTransactions in goroutine 999
Jul 14 01:34:54 bor[1022028]: /home/ubuntu/bor/miner/worker.go:946 +0x329
Jul 14 01:34:56 systemd[1]: bor.service: Main process exited, code=exited, status=2/INVALIDARGUMENT