Skip to content

Synchronization backwards cannot be restored #6042

@beck-8

Description

@beck-8

Describe the bug

After running for a while it will be noticed that he continues to lag behind, restarting will resume, but it will continue to happen.

To reproduce

  1. start service

Log output

Log Output
2025-09-05T16:40:07.713489Z  INFO forest::state_manager: Evaluating tipset: EPOCH=5292784, blocks=5, tsk=[bafy2bzacedw54rftnsj3nujurjmxjxucvrq7u3twnpmwin55pjsxriovmsvpm, bafy2bzaceaaw5pp3ec4bvhdypjvg6y53atamg74zkiddhbywx23n6uxgk2tva, bafy2bzacecmmo32ctpihkqlxhqjor7tzas3ydht4qarcanitznkyqqgijm2gw, bafy2bzacedjdhebd56chuo3mbzjcgxyxfymvg4juck36xryuy5nttpfspm56w, bafy2bzacea4tcz7wbk4zmgynlfnraavycssy3tz7jnyzwnuz7aabbpt3rpeeu]
2025-09-05T16:40:10.851310Z  INFO forest::state_manager: Evaluating tipset: EPOCH=5292785, blocks=3, tsk=[bafy2bzacedr22ob62fddjjywoxw6y6qimfmhyvj3rakp7cprlgy2bfd3t2lp6, bafy2bzaceawh6cszx2d5vmjc2keoqaxmsptenmrr4bhylwcil2lv6wmlwifjy, bafy2bzacedpcfw4ixsxphacmvq2dyal4bmzygzdkaech7judvonxol6qsv4i2]
2025-09-05T16:40:11.062683Z  INFO forest::state_manager: Evaluating tipset: EPOCH=5292786, blocks=4, tsk=[bafy2bzacea5enkrfl7fg6lujyjfb4dw3ows6vnqy5yixlaa4nynplbq4gvu3q, bafy2bzacediofmx7pzjmotirgrboye4nws3fzmg6ecjr5ms3t7fb6dksrnqiq, bafy2bzacec4q2xvrhv3uoixho3jtsgljnov6vlwffiot4uciw7ma52mkax3ua, bafy2bzacebog52aglkfo6yg2ic5liog766pltsiuf77xzc3sxs2jtzfvhuzdg]
2025-09-05T16:40:11.583899Z  INFO forest::state_manager: Evaluating tipset: EPOCH=5292787, blocks=3, tsk=[bafy2bzaceaizjkoyfvlqtl76gog2taxwpqm52gmwvxleaodvjjjwjxtlhlygm, bafy2bzacedlv42qedlmm3sb7alquct7jjrd2estlw7szzbxlsxp73di4zwrnc, bafy2bzaceag6cyk5ibrrlwahrns7fpwijij7jtfkiqsrjla73ejfi4tcaynyw]
2025-09-05T16:40:11.879495Z  INFO forest::state_manager: Evaluating tipset: EPOCH=5292788, blocks=5, tsk=[bafy2bzaceacisrlajgyma3qzssu23do3zkych5pp7hrkrjotefo74252enn2w, bafy2bzacec5regrw4qit2wknb7qo3ltbnnd5djdcydjnyj2udieon2b74bmcg, bafy2bzacecu6k6xmhanohcyx2kmusvl323u3gigssmqwzkgqbgfnspkvzbntg, bafy2bzacedsmw7utuzcgtxntyw6fw6zqermxkjatccyyfix3g5y4snu76mgjy, bafy2bzacebj3avzlitnbrm37jcr3y2eslp7gjcbnwicw6paw7ou4bm7it5zbe]
2025-09-05T16:40:12.131585Z  INFO forest::state_manager: Evaluating tipset: EPOCH=5292789, blocks=5, tsk=[bafy2bzacebwdty3qhaqpt5kxjdplr7fqb6mnqyzi2ne4vuhsmuhykmppl6hlc, bafy2bzacebwpi67s22f4ars5igyuf2nvajmpba36oc53pbwnzpqt5gc7zpuwa, bafy2bzacedyr3dxuq3whl5piuechvm3p75ffhxanrufjavz6onfmkviv23goo, bafy2bzaceax6xni7ows2qgmkot3cd7wxknplvgqnasgulw4ioj5ggbc33n3xq, bafy2bzacecph5ny2xft6mqsh5tl3getzfepd7rii5vzgm6euo7xeyzwvjqzvu]
2025-09-05T16:40:12.465512Z  INFO forest::state_manager: Evaluating tipset: EPOCH=5292790, blocks=11, tsk=[bafy2bzaced5vikpxo3ok7gx7u4fswnuknvsuj5ouwb5s6wb7mvif2sco64deo, bafy2bzacedn63yijitchwmjce625setlwdizjtaa7aiu65lkzlrstcalbf3rq, bafy2bzacebwybaa3yawhq3x2po37bpqzntjtoca7y54qp6rm7bc3vugdsz2nm, bafy2bzacedpjju4gvew2i4cpf277lyyh6lopdnxrxsrry74tlgt2qa36fghls, bafy2bzaceck2lgn4ycs4noch57mabqizlqzxbjknouf2xw6pjr7yhnqqvmam2, bafy2bzacebn45k2wuosb2ue4vcicc3shtsswgfukiwim4vnkhvgmkslypoxzc, bafy2bzacea5sibzp5lttdjrdkk2p5oqwqblr2jsdjjz3dgkco7abmns5cphoa, bafy2bzacecs5yfo4vnw7qc2q2hys52ung4jozylp3xhi6pyjkfvbczucbqwoc, bafy2bzacecdemd3ni63wmuo2wjp5y5etfe5aalgyuwaamzfrhdpgecsiwfpwo, bafy2bzaceaweqbi2ijsdxnir5rsoeultlknxjimb6bnlcvcefdkx4ctpwrb4i, bafy2bzaceacgl2qherq6hulewmxmvdiehwuulrt3bsgtnv5o7hsy7qujm6xri]
2025-09-05T16:40:12.729789Z  INFO forest::state_manager: Evaluating tipset: EPOCH=5292791, blocks=8, tsk=[bafy2bzacecghqcvzo5dqvmf6jepcfx5kvqfuekf23is4zejb7lw3s4bvcdzcg, bafy2bzacecb42wbtux7cu56fz6kprowlyp24mflx253gh7dilfzg3acawnclk, bafy2bzacecliapmzhriqpghiyzoz4pyyc2ostxjs7vijaaoffbmmeosnjjznw, bafy2bzaceavixwlejs5eza5cdeamf3e3teawuitz7k6zpyh3l3kmdwm5xd6ly, bafy2bzacecdlj6nlf2icngynpkr2dnf4oxwfqww47iuhwy32hm6s23hz2isxq, bafy2bzacea6tucn72z3k74ajm3rnyomwkafz6t62zqnkk4xtefsuuucwr55r6, bafy2bzaceco5kqaeiiyblxzkhj6xmeebhhpd563cmnkye55tqix5alilwiyay, bafy2bzacecfddgabcerkgni6wzdvmg4z5q3mna57hyw3s4bewj2en4aizd6o4]
2025-09-05T16:40:13.084789Z  INFO forest::state_manager: Evaluating tipset: EPOCH=5292792, blocks=6, tsk=[bafy2bzacedahbn4rg63ua6ttf2cxd2b75l2ti5b6die7hcwnzgkyz3un4qd6i, bafy2bzacea5ghm65bygeyuyne4tb22df4rslvfokjv4rzcztndhotmevmjdoy, bafy2bzacebq7to6cbisf4zkrca2c6h2gcbhucys3nr5f5xf525gmchuprh5bs, bafy2bzaceb4vylp2aiibeg5rhmw67i4bat5ljpoqkqosdvezvolj7xlvthhpa, bafy2bzacebhafshxbti7aehd7ecrmwnnsybrbxozrvkpvpb7sb37vfnjempey, bafy2bzacebvcwptl5sbaa2u36qo2r4roowk2s7xrwzz6w6l4b2zaeuiyyuzls]
2025-09-05T16:40:13.420626Z  INFO forest::state_manager: Evaluating tipset: EPOCH=5292793, blocks=4, tsk=[bafy2bzacedw7imluhkbjjt2ftljh2wje43xe56oxqzsmyq72hi7avjunlwqj2, bafy2bzacecgeednxwywwafl4ap6eavqjkvuzlexhihismvfqumnx774bx2ob4, bafy2bzacecjzzsq3rxak6cmpboof5g5oq2nzksodiu7lw3gtf43dda2avat3q, bafy2bzaceaqrryzvcuvdqwoayltdi5ziaqififo5k2mtre54g3u6gi6ylhcgq]
2025-09-05T16:40:13.695677Z  INFO forest::state_manager: Evaluating tipset: EPOCH=5292794, blocks=5, tsk=[bafy2bzaceduw5gnahgft6dazorwgan3s4jrmymawu2zgpefykt77qh5k7exzu, bafy2bzacedp3kzgnvezwehccku2l5ywhverjh6bueypd4hl5hyw5pk5t4sbx4, bafy2bzacebuqp2sjmm2xzjo5f2me4ml6ldxpldkkvvhmbgjfpcba6dclzeqtk, bafy2bzaceawaj6pvcdosjpp6fhxg2kag5xha5joprtpgzxsuhcfta72bw34nq, bafy2bzaceb2hrjrcz65egk4w5rg2vfdkewjbcdcumg2vu52uelqdthq2dh7w2]
2025-09-05T16:40:13.930048Z  INFO forest::state_manager: Evaluating tipset: EPOCH=5292795, blocks=5, tsk=[bafy2bzacec4xghyawiuizq3be6sd5aqxvr7zkvmnoozfrdahb7stxjklub5am, bafy2bzacedxsx57x4o54tn2hr3axzek3yo5xcejvk332eau2ulltvbwjlxga4, bafy2bzacebghyawzfiabahr7yr6dgp6e4gsm47lohd5o3pgesgxm7dq46sw6k, bafy2bzaceb2rc7npuue2ubqwysoiaaaqijitnhlyl2tvhkprwbt3hoyybzmak, bafy2bzaceaf5okwncesoj3zntipl7y2qj5ehz7txkalhnpxn6ydn5t2xi2swe]
2025-09-05T16:40:14.249020Z  INFO forest::state_manager: Evaluating tipset: EPOCH=5292796, blocks=4, tsk=[bafy2bzaced576l5bjazu45i5chngrqv57rxnancdvvpxcyea5xxcvghd4tr7g, bafy2bzacebbxjetxidrrbikdnqb7t24ymkexdbulycxk5fm5swl5xyr6g2574, bafy2bzacebu7re2jifnqhz3224p6ce7pwhqztifunq6g2qkxrckqz5wrxhwfi, bafy2bzacean66kyugrbfwxohmowvccwio3l66ulhbaxhu6xko5exa46e37gs6]
2025-09-05T16:40:14.567632Z  INFO forest::state_manager: Evaluating tipset: EPOCH=5292797, blocks=5, tsk=[bafy2bzaceakdnickam6lbikt3ievmfyxhndiotzcdtx2qfra55srjbskg36yu, bafy2bzaceau5mtwxj4t24khrde4s7ckpvzbmgahegynahhuyf7tede4ttk6hk, bafy2bzaceckkvmzuq4f6vgg2omewloptqb5jpf5qkev7cyf3vp57f6qmptqzi, bafy2bzacebvxs2znw35w5wm5x7rcs2uzfyg6xo7mzbrsoot27toxb26ffq3sg, bafy2bzacebd5wetnjnjczwotgboig23d7w6xsecoc6sbperktkjre5howm75c]
2025-09-05T16:40:14.895227Z  INFO forest::state_manager: Evaluating tipset: EPOCH=5292798, blocks=6, tsk=[bafy2bzaced5ctf5fdcnnfbaqpigu6u5krbzz2i6xni6eofcrw7h4nrs7mpny6, bafy2bzacec2hh73x5pjea4nutr67sov6u6xdfo3j5pgs5zaimldruwmmzsaks, bafy2bzacebduxnuaq5wdcxlhegvnziw5bi2gbffrgrrrv4q6ll2vmfxolu346, bafy2bzacedsyrsf4kmxqdnmvxszbyepztt45qea7r6s6eqv3hh2g7ez6s4rpo, bafy2bzaceaknsoxbio23mqny3kufzle23paa2tobjwmh4745i432djvzrkrby, bafy2bzaceavvs5crjvlmaoudcfibzpyfn5gatmq57sbo7vpabwvdydy7d4y66]
2025-09-05T16:40:15.432572Z  INFO forest::state_manager: Evaluating tipset: EPOCH=5292799, blocks=3, tsk=[bafy2bzacecpg323pk5gijeyk6zboslqn27vit72r2uwrk5xiprgos5756fv3c, bafy2bzaceago6rhijzc5ctqjpldkwvbgrkzl4zsc5u36q7tqlvyfyupe6ye24, bafy2bzacecnpz3xpzilaymm7pl525a3aqtykrbgxpo4nkhz4luhop7nhyke3g]
2025-09-06T00:40:28.026+0800    INFO    canonical-log   swarm/swarm_dial.go:620 CANONICAL_PEER_STATUS: peer=12D3KooWHVYg9uhYQ3bARXUwzuNBxNkct66trDPEX5egppJbfLo2 addr=/ip4/120.71.76.122/udp/48072/quic-v1 sample_rate=100 connection_status="established" dir="outbound"
2025-09-05T16:40:28.767867Z  WARN forest::chain_sync::tipset_syncer: Validating block [CID = bafy2bzaceco3w4hvtzgsofvbk2qky3zp7ravk7b7btgp5mxul7dhudej3o4ww] in EPOCH = 5292801 failed: Block received from the future: now = 1757090428, block = 1757090430
2025-09-05T16:40:28.767890Z  WARN forest::chain_sync::chain_follower: Error validating tipset: Block received from the future: now = 1757090428, block = 1757090430
2025-09-05T16:40:32.988190Z  INFO forest::state_manager: Evaluating tipset: EPOCH=5292800, blocks=3, tsk=[bafy2bzaceb7po2vjlyunyrbk2hdozsblzkemfzsgvng3p34q3sri35n2jxciy, bafy2bzacedqas6zaua2x566wjeyvkohj2p3lwz4wafs2lfgbkcegcbmu6z3ak, bafy2bzacec5dwgi2y6yvixseft5kpchjswtx37qicgpdbw2hial2aw67ytymu]
2025-09-05T16:40:56.315717Z  INFO forest::libp2p::service: Re-dialing to bootstrap peer at /dns/bootstrap-venus.mainnet.filincubator.com/tcp/8888/p2p/QmQu8C6deXwKvJP2D8B6QGyhngc3ZiDnFzEHBDx8yeBXST
2025-09-06T00:40:56.561+0800    WARN    f3/ohshitstore  powerstore/powerstore.go:252    Uh-oh, F3 has fallen behind EC by more than 1350 epochs! Engaging the OhShitStore™.     {"ecHead": 5292801, "f3Base": 5025323}
2025-09-05T16:41:56.315819Z  INFO forest::libp2p::service: Re-dialing to bootstrap peer at /dns/bootstrap-venus.mainnet.filincubator.com/tcp/8888/p2p/QmQu8C6deXwKvJP2D8B6QGyhngc3ZiDnFzEHBDx8yeBXST
2025-09-06T00:41:59.908+0800    ERROR   f3      go-f3@v0.8.10/host.go:222       error when receiving alarm: failed fetching chain for instance 82908: computing powertable CID for base: getting power table to compute CID: target epoch 5024973 before the latest F3 finalized base 5026047, move on already
2025-09-06T00:42:55.413+0800    ERROR   f3/ohshitstore  powerstore/powerstore.go:266    failed to store power tables    {"ecHead": 5292801, "f3Base": 5025323, "last-stored": 5025323, "error": "getting power table: getting power table: RPC error (-32603): Can't create a valid state tree from the given root. This error may indicate unsupported version. state_root_cid=bafy2bzacear7yhykw3jhq65hdttlxmzqq4wwmqu5gdulh5btdyw4gr43o7yro, state_root_version=unknown"}



2025-09-05T16:43:56.315775Z  INFO forest::libp2p::service: Re-dialing to bootstrap peer at /dns/bootstrap-venus.mainnet.filincubator.com/tcp/8888/p2p/QmQu8C6deXwKvJP2D8B6QGyhngc3ZiDnFzEHBDx8yeBXST
2025-09-06T00:43:58.693+0800    ERROR   f3      go-f3@v0.8.10/host.go:222       error when receiving alarm: failed fetching chain for instance 83904: computing powertable CID for base: getting power table to compute CID: target epoch 5026057 before the latest F3 finalized base 5027371, move on already
2025-09-06T00:44:53.877+0800    ERROR   f3/ohshitstore  powerstore/powerstore.go:266    failed to store power tables    {"ecHead": 5292801, "f3Base": 5026679, "last-stored": 5025323, "error": "getting power table: getting power table: RPC error (-32603): Can't create a valid state tree from the given root. This error may indicate unsupported version. state_root_cid=bafy2bzacear7yhykw3jhq65hdttlxmzqq4wwmqu5gdulh5btdyw4gr43o7yro, state_root_version=unknown"}
2025-09-05T16:44:56.315784Z  INFO forest::libp2p::service: Re-dialing to bootstrap peer at /dns/bootstrap-venus.mainnet.filincubator.com/tcp/8888/p2p/QmQu8C6deXwKvJP2D8B6QGyhngc3ZiDnFzEHBDx8yeBXST
2025-09-05T16:45:06.345698Z  INFO forest::chain_sync::chain_follower: Downloading 9 tipsets
2025-09-06T00:45:59.377+0800    ERROR   f3      go-f3@v0.8.10/host.go:222       error when receiving alarm: failed fetching chain for instance 85195: computing powertable CID for base: getting power table to compute CID: target epoch 5027393 before the latest F3 finalized base 5028732, move on already
2025-09-05T16:46:06.351859Z  INFO forest::chain_sync::chain_follower: Catching up to HEAD: 5292801[bafy2bzacecbsxxv7zqn6fj4uhdhst6szhlvilkffqg2vpwexl7btbroumbwu2] -> 5292812
2025-09-05T16:46:16.352842Z  INFO forest::chain_sync::chain_follower: Catching up to HEAD: 5292801[bafy2bzacecbsxxv7zqn6fj4uhdhst6szhlvilkffqg2vpwexl7btbroumbwu2] -> 5292812
2025-09-05T16:46:26.353877Z  INFO forest::chain_sync::chain_follower: Catching up to HEAD: 5292801[bafy2bzacecbsxxv7zqn6fj4uhdhst6szhlvilkffqg2vpwexl7btbroumbwu2] -> 5292812
2025-09-05T16:46:36.354824Z  INFO forest::chain_sync::chain_follower: Catching up to HEAD: 5292801[bafy2bzacecbsxxv7zqn6fj4uhdhst6szhlvilkffqg2vpwexl7btbroumbwu2] -> 5292813
2025-09-05T16:46:46.355917Z  INFO forest::chain_sync::chain_follower: Catching up to HEAD: 5292801[bafy2bzacecbsxxv7zqn6fj4uhdhst6szhlvilkffqg2vpwexl7btbroumbwu2] -> 5292813
2025-09-05T16:46:56.315790Z  INFO forest::libp2p::service: Re-dialing to bootstrap peer at /dns/bootstrap-venus.mainnet.filincubator.com/tcp/8888/p2p/QmQu8C6deXwKvJP2D8B6QGyhngc3ZiDnFzEHBDx8yeBXST
2025-09-05T16:46:56.356712Z  INFO forest::chain_sync::chain_follower: Catching up to HEAD: 5292801[bafy2bzacecbsxxv7zqn6fj4uhdhst6szhlvilkffqg2vpwexl7btbroumbwu2] -> 5292813
2025-09-06T00:46:57.747+0800    ERROR   f3/ohshitstore  powerstore/powerstore.go:266    failed to store power tables    {"ecHead": 5292801, "f3Base": 5027988, "last-stored": 5025323, "error": "getting power table: getting power table: RPC error (-32603): Can't create a valid state tree from the given root. This error may indicate unsupported version. state_root_cid=bafy2bzacear7yhykw3jhq65hdttlxmzqq4wwmqu5gdulh5btdyw4gr43o7yro, state_root_version=unknown"}
2025-09-05T16:47:06.357734Z  INFO forest::chain_sync::chain_follower: Catching up to HEAD: 5292801[bafy2bzacecbsxxv7zqn6fj4uhdhst6szhlvilkffqg2vpwexl7btbroumbwu2] -> 5292814
2025-09-05T16:47:16.358742Z  INFO forest::chain_sync::chain_follower: Catching up to HEAD: 5292801[bafy2bzacecbsxxv7zqn6fj4uhdhst6szhlvilkffqg2vpwexl7btbroumbwu2] -> 5292814
2025-09-05T16:47:26.359677Z  INFO forest::chain_sync::chain_follower: Catching up to HEAD: 5292801[bafy2bzacecbsxxv7zqn6fj4uhdhst6szhlvilkffqg2vpwexl7btbroumbwu2] -> 5292814
2025-09-05T16:47:36.360762Z  INFO forest::chain_sync::chain_follower: Catching up to HEAD: 5292801[bafy2bzacecbsxxv7zqn6fj4uhdhst6szhlvilkffqg2vpwexl7btbroumbwu2] -> 5292815
2025-09-05T16:47:46.361954Z  INFO forest::chain_sync::chain_follower: Catching up to HEAD: 5292801[bafy2bzacecbsxxv7zqn6fj4uhdhst6szhlvilkffqg2vpwexl7btbroumbwu2] -> 5292815
2025-09-05T16:47:56.362718Z  INFO forest::chain_sync::chain_follower: Catching up to HEAD: 5292801[bafy2bzacecbsxxv7zqn6fj4uhdhst6szhlvilkffqg2vpwexl7btbroumbwu2] -> 5292815
2025-09-06T00:47:57.974+0800    INFO    canonical-log   swarm/swarm_dial.go:620 CANONICAL_PEER_STATUS: peer=12D3KooWKKkCZbcigsWTEu1cgNetNbZJqeNtysRtFpq7DTqw3eqH addr=/ip4/148.113.16.168/tcp/34000 sample_rate=100 connection_status="established" dir="outbound"
2025-09-06T00:48:03.824+0800    ERROR   f3      go-f3@v0.8.10/host.go:222       error when receiving alarm: failed fetching chain for instance 86504: computing powertable CID for base: getting power table to compute CID: target epoch 5028748 before the latest F3 finalized base 5030138, move on already
2025-09-05T16:48:06.363866Z  INFO forest::chain_sync::chain_follower: Catching up to HEAD: 5292801[bafy2bzacecbsxxv7zqn6fj4uhdhst6szhlvilkffqg2vpwexl7btbroumbwu2] -> 5292816
2025-09-05T16:48:16.364673Z  INFO forest::chain_sync::chain_follower: Catching up to HEAD: 5292801[bafy2bzacecbsxxv7zqn6fj4uhdhst6szhlvilkffqg2vpwexl7btbroumbwu2] -> 5292816
2025-09-05T16:48:26.365822Z  INFO forest::chain_sync::chain_follower: Catching up to HEAD: 5292801[bafy2bzacecbsxxv7zqn6fj4uhdhst6szhlvilkffqg2vpwexl7btbroumbwu2] -> 5292816
2025-09-05T16:48:36.366778Z  INFO forest::chain_sync::chain_follower: Catching up to HEAD: 5292801[bafy2bzacecbsxxv7zqn6fj4uhdhst6szhlvilkffqg2vpwexl7btbroumbwu2] -> 5292817
2025-09-05T16:48:46.367876Z  INFO forest::chain_sync::chain_follower: Catching up to HEAD: 5292801[bafy2bzacecbsxxv7zqn6fj4uhdhst6szhlvilkffqg2vpwexl7btbroumbwu2] -> 5292817
2025-09-05T16:48:56.368710Z  INFO forest::chain_sync::chain_follower: Catching up to HEAD: 5292801[bafy2bzacecbsxxv7zqn6fj4uhdhst6szhlvilkffqg2vpwexl7btbroumbwu2] -> 5292817
2025-09-06T00:49:05.930+0800    ERROR   f3/ohshitstore  powerstore/powerstore.go:266    failed to store power tables    {"ecHead": 5292801, "f3Base": 5029366, "last-stored": 5025323, "error": "getting power table: getting power table: RPC error (-32603): Can't create a valid state tree from the given root. This error may indicate unsupported version. state_root_cid=bafy2bzacear7yhykw3jhq65hdttlxmzqq4wwmqu5gdulh5btdyw4gr43o7yro, state_root_version=unknown"}
2025-09-05T16:49:06.369854Z  INFO forest::chain_sync::chain_follower: Catching up to HEAD: 5292801[bafy2bzacecbsxxv7zqn6fj4uhdhst6szhlvilkffqg2vpwexl7btbroumbwu2] -> 5292818
2025-09-05T16:49:16.370717Z  INFO forest::chain_sync::chain_follower: Catching up to HEAD: 5292801[bafy2bzacecbsxxv7zqn6fj4uhdhst6szhlvilkffqg2vpwexl7btbroumbwu2] -> 5292818
2025-09-05T16:49:26.371879Z  INFO forest::chain_sync::chain_follower: Catching up to HEAD: 5292801[bafy2bzacecbsxxv7zqn6fj4uhdhst6szhlvilkffqg2vpwexl7btbroumbwu2] -> 5292818
2025-09-05T16:49:36.372729Z  INFO forest::chain_sync::chain_follower: Catching up to HEAD: 5292801[bafy2bzacecbsxxv7zqn6fj4uhdhst6szhlvilkffqg2vpwexl7btbroumbwu2] -> 5292819
2025-09-05T16:49:46.373877Z  INFO forest::chain_sync::chain_follower: Catching up to HEAD: 5292801[bafy2bzacecbsxxv7zqn6fj4uhdhst6szhlvilkffqg2vpwexl7btbroumbwu2] -> 5292819
2025-09-05T16:49:56.315845Z  INFO forest::libp2p::service: Re-dialing to bootstrap peer at /dns/bootstrap-venus.mainnet.filincubator.com/tcp/8888/p2p/QmQu8C6deXwKvJP2D8B6QGyhngc3ZiDnFzEHBDx8yeBXST
2025-09-05T16:49:56.374733Z  INFO forest::chain_sync::chain_follower: Catching up to HEAD: 5292801[bafy2bzacecbsxxv7zqn6fj4uhdhst6szhlvilkffqg2vpwexl7btbroumbwu2] -> 5292819
2025-09-05T16:50:06.375717Z  INFO forest::chain_sync::chain_follower: Catching up to HEAD: 5292801[bafy2bzacecbsxxv7zqn6fj4uhdhst6szhlvilkffqg2vpwexl7btbroumbwu2] -> 5292820, downloading 19 tipsets
2025-09-05T16:50:16.376747Z  INFO forest::chain_sync::chain_follower: Catching up to HEAD: 5292801[bafy2bzacecbsxxv7zqn6fj4uhdhst6szhlvilkffqg2vpwexl7btbroumbwu2] -> 5292820, downloading 19 tipsets
2025-09-05T16:50:26.377718Z  INFO forest::chain_sync::chain_follower: Catching up to HEAD: 5292801[bafy2bzacecbsxxv7zqn6fj4uhdhst6szhlvilkffqg2vpwexl7btbroumbwu2] -> 5292820
2025-09-06T00:50:26.704+0800    ERROR   f3      go-f3@v0.8.10/host.go:222       error when receiving alarm: failed fetching chain for instance 87944: computing powertable CID for base: getting power table to compute CID: target epoch 5030279 before the latest F3 finalized base 5031741, move on already
2025-09-06T00:50:28.199+0800    INFO    canonical-log   swarm/swarm_dial.go:620 CANONICAL_PEER_STATUS: peer=12D3KooWNmmKpMywY4e2RpmNuHt81AecfyV7nMMxV1NdnXeBr6yf addr=/ip4/154.42.3.138/tcp/13001 sample_rate=100 connection_status="established" dir="outbound"
2025-09-06T00:50:33.412+0800    INFO    canonical-log   swarm/swarm_dial.go:620 CANONICAL_PEER_STATUS: peer=12D3KooWGnTQ3HGg7mpfRdpXhJVEzPstYf5PejTUSRuKoxWQuaHh addr=/ip4/36.103.210.210/tcp/2381 sample_rate=100 connection_status="established" dir="outbound"
2025-09-05T16:50:36.378757Z  INFO forest::chain_sync::chain_follower: Catching up to HEAD: 5292801[bafy2bzacecbsxxv7zqn6fj4uhdhst6szhlvilkffqg2vpwexl7btbroumbwu2] -> 5292821
2025-09-05T16:50:46.379847Z  INFO forest::chain_sync::chain_follower: Catching up to HEAD: 5292801[bafy2bzacecbsxxv7zqn6fj4uhdhst6szhlvilkffqg2vpwexl7btbroumbwu2] -> 5292821
2025-09-05T16:50:56.315696Z  INFO forest::libp2p::service: Re-dialing to bootstrap peer at /dns/bootstrap.filecoin.chain.love/tcp/1235/p2p/12D3KooWBF8cpp65hp2u9LK5mh19x67ftAam84z9LsfaquTDSBpt
2025-09-05T16:50:56.315759Z  INFO forest::libp2p::service: Re-dialing to bootstrap peer at /dns/bootstrap-venus.mainnet.filincubator.com/tcp/8888/p2p/QmQu8C6deXwKvJP2D8B6QGyhngc3ZiDnFzEHBDx8yeBXST
2025-09-05T16:50:56.380803Z  INFO forest::chain_sync::chain_follower: Catching up to HEAD: 5292801[bafy2bzacecbsxxv7zqn6fj4uhdhst6szhlvilkffqg2vpwexl7btbroumbwu2] -> 5292821
2025-09-06T00:51:02.580+0800    INFO    canonical-log   swarm/swarm_dial.go:620 CANONICAL_PEER_STATUS: peer=QmQu8C6deXwKvJP2D8B6QGyhngc3ZiDnFzEHBDx8yeBXST addr=/ip4/47.242.158.66/tcp/8888 sample_rate=100 connection_status="established" dir="outbound"
2025-09-05T16:51:06.381824Z  INFO forest::chain_sync::chain_follower: Catching up to HEAD: 5292801[bafy2bzacecbsxxv7zqn6fj4uhdhst6szhlvilkffqg2vpwexl7btbroumbwu2] -> 5292822
2025-09-06T00:51:12.541+0800    ERROR   f3/ohshitstore  powerstore/powerstore.go:266    failed to store power tables    {"ecHead": 5292801, "f3Base": 5030829, "last-stored": 5025323, "error": "getting power table: getting power table: RPC error (-32603): Can't create a valid state tree from the given root. This error may indicate unsupported version. state_root_cid=bafy2bzacear7yhykw3jhq65hdttlxmzqq4wwmqu5gdulh5btdyw4gr43o7yro, state_root_version=unknown"}
2025-09-05T16:51:16.382682Z  INFO forest::chain_sync::chain_follower: Catching up to HEAD: 5292801[bafy2bzacecbsxxv7zqn6fj4uhdhst6szhlvilkffqg2vpwexl7btbroumbwu2] -> 5292822
2025-09-05T16:51:26.383692Z  INFO forest::chain_sync::chain_follower: Catching up to HEAD: 5292801[bafy2bzacecbsxxv7zqn6fj4uhdhst6szhlvilkffqg2vpwexl7btbroumbwu2] -> 5292822
2025-09-05T16:51:36.384848Z  INFO forest::chain_sync::chain_follower: Catching up to HEAD: 5292801[bafy2bzacecbsxxv7zqn6fj4uhdhst6szhlvilkffqg2vpwexl7btbroumbwu2] -> 5292823
2025-09-05T16:51:46.385746Z  INFO forest::chain_sync::chain_follower: Catching up to HEAD: 5292801[bafy2bzacecbsxxv7zqn6fj4uhdhst6szhlvilkffqg2vpwexl7btbroumbwu2] -> 5292823

Expected behaviour

Screenshots

Environment (please complete the following information):

  • OS:
  • Branch/commit
    root@DESKTOP-4VQSOHB:/mnt/data/forest# forest -V
    forest-filecoin 0.29.0+git.17678386
  • Hardware:

Other information and links

https://filecoinproject.slack.com/archives/CT2GQTQHZ/p1757090884485189

Metadata

Metadata

Assignees

Labels

Type: BugSomething isn't working

Type

No type

Projects

Status

Done

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions