-
Notifications
You must be signed in to change notification settings - Fork 182
Closed
Labels
Type: BugSomething isn't workingSomething isn't working
Description
Describe the bug
Reported on Slack by @parthshah1
When Lotus Miner + Lotus are unable to get the drand beacon they fail with:
[ 42.161] [ service_lotus-1] [err] 2024-12-06T02:46:13.407Z WARN rpc go-jsonrpc@v0.7.0/handler.go:421 error in RPC call to 'Filecoin.StateGetBeaconEntry': drand failed Get request:
[ 42.161] [ service_lotus-1] [err] github.com/filecoin-project/lotus/chain/beacon/drand.(*DrandBeacon).Entry.func1
[ 42.161] [ service_lotus-1] [err] /lotus/chain/beacon/drand/drand.go:166
[ 42.161] [ service_lotus-1] [err] - no valid clients - decoding response: EOF - decoding response: EOF - decoding response: EOF
[ 42.162] [service_lotus-miner-1] [err] 2024-12-06T02:46:13.408Z ERROR miner miner/miner.go:274 failed getting beacon entry: drand failed Get request: no valid clients - decoding response: EOF - decoding response: EOF - decoding response: EOF
[ 42.162] [service_lotus-miner-1] [err] 2024-12-06T02:46:13.409Z INFO storageminer storage/winning_prover.go:70 Computing WinningPoSt ;[{SealProof:5 SectorNumber:0 SectorKey:<nil> SealedCID:bagboea4b5abcaxqsx6tl25pbxyq3uyyqnwghmdmdrkyy5odehf4eotl6is2yp3zu}]; [79 244 33 56 182 139 242 46 79 79 12 85 255 6 75 126 190 6 129 110 37 10 97 226 86 195 112 190 76 143 86 111]
[ 42.162] [service_lotus-miner-1] [err] 2024-12-06T02:46:13.409Z INFO advmgr sealer/manager_post.go:24 GenerateWinningPoSt run at lotus-miner
[ 42.164] [ service_lotus-1] [err] 2024-12-06T02:46:13.410Z WARN fullnode full/mpool.go:202 Push from ID address (t0100), adjusting to t3w4jtmrcx3mcvfdmjbbkrkyod7bgqupkjl4lrvej7zlgfkabekkirdlsuqzqm6fugfauvnhwjq6k2skrjvxja
[ 42.166] [service_lotus-miner-1] [err] 2024-12-06T02:46:13.412Z INFO main lotus-miner/init.go:625 Waiting for message: bafy2bzacebki4eylo4ruswp6einmq2y223fglaiaceldjafbpyyzw7flwi2ei
[ 42.341] [service_lotus-miner-1] [err] 2024-12-06T02:46:13.587Z INFO storageminer storage/winning_prover.go:77 GenerateWinningPoSt took 178.620219ms
[ 42.341] [service_lotus-miner-1] [err] 2024-12-06T02:46:13.587Z INFO miner miner/warmup.go:87 winning PoSt warmup successful {"took": 0.181291282}
[ 43.164] [ service_lotus-1] [err] 2024-12-06T02:46:14.410Z WARN rpc go-jsonrpc@v0.7.0/handler.go:421 error in RPC call to 'Filecoin.StateGetBeaconEntry': drand failed Get request:
[ 43.164] [ service_lotus-1] [err] github.com/filecoin-project/lotus/chain/beacon/drand.(*DrandBeacon).Entry.func1
After retry, they manage fine and the miner starts mining blocks. Forest during startup isn’t able to catch the drand beacon, it never syncs and just keeps on throwing “validation errors”.
[ 1149.766] [ service_forest] [inf] 2024-12-06T03:04:41.012644Z WARN forest_filecoin::chain_sync::tipset_syncer: Validating block [CID = bafy2bzaceab7cz4luun43frmnssroiwnprsvrfunepsht3mmtcy4ez7ce7twc] in EPOCH = 2 failed: Validation error: Validation error: Consensus error: Failed to validate blocks random beacon values: Error validating data: beacon entry was invalid
[ 1149.766] [ service_forest] [inf] 2024-12-06T03:04:41.012690Z ERROR forest_filecoin::chain_sync::tipset_syncer: Sync messages check state failed for tipset range
[ 1149.766] [ service_forest] [inf] 2024-12-06T03:04:41.012726Z ERROR forest_filecoin::chain_sync::chain_muxer: Bootstrapping failed, re-evaluating the network head to retry the bootstrap. Error = TipsetRangeSyncer(Validation("Validation error: Consensus error: Failed to validate blocks random beacon values: Error validating data: beacon entry was invalid"))
To reproduce
It's likely reproducible by cutting internet access from the relevant services and then restoring it. Either that, or other means or injecting drand fault.
Log output
events_with_drand.log
events_3.log
Expected behaviour
Forest is able to follow the chain, as Lotus does.
Screenshots
Environment (please complete the following information):
- OS: Linux
- Forest commit: dd120ad
- Lotus:
lotus version 1.31.0+2k+git.198ee01.dirty
Other information and links
The setup is in https://github.com/FilecoinFoundationWeb/Filecoin-Antithesis, which is similar to our devnet compose but with custom drand service injected. Ping for access.
Metadata
Metadata
Assignees
Labels
Type: BugSomething isn't workingSomething isn't working
Type
Projects
Status
Done