Skip to content

checkpoint state error, unable to get in sync (main branch) #21272

@mriccobene

Description

@mriccobene

The sync-from-scratch minimal node test has started failing on the main branch since 14 May.
Erigon was unable to synchronize within the test timeframe.

Relevant warning:
[WARN] [05-14|10:28:53.350] error executing clstage app=caplin stage=ForkChoice err="failed to compute and notify services of new fork choice: failed to get head: getCheckpointState: baseState not found in graph"

Test run: https://github.com/erigontech/erigon/actions/runs/25835601066/job/75909910633#step:6:1315

Log fragment:

[...]
[INFO] [05-14|09:53:36.108] [4/6 Execution] DONE                     in=1m0.200744813s block=25069999
[WARN] [05-14|09:53:39.626] sentry.StatusDataProvider: The canonical chain current header not found in the database. Check the database consistency. Using latest available snapshot data. 
[INFO] [05-14|09:53:39.848] [BlockCollector] Inserted blocks         progress=25070999
[...]
[WARN] [05-14|10:28:14.845] BuildFilesInBackground: lastInDB beyond execStep lastInDB=9001 execStep=8985 txNum=3509765625
[...]
[INFO] [05-14|10:28:43.482] head updated                             hash=0xa6775a6f6d427b97c4d23986bffd172e098c507a673e7faba5e2b2460b5cc76c number=25092610 txnum=3516202118 age=33m20s execution=0s alloc=16.0GB sys=45.9GB
[INFO] [05-14|10:28:43.643] Timings: Forkchoice                      flush=156ms commit=4ms alloc=16.0GB sys=45.9GB
[INFO] [05-14|10:28:47.280] [Caplin] Forward Sync                    app=caplin stage=ForwardSync progress=14327501 distance-from-chain-tip=7m36s estimated-time-remaining=5s
[INFO] [05-14|10:28:52.274] [BlockCollector] Inserting blocks        from=25092531 to=25092777
[INFO] [05-14|10:28:52.338] [BlockCollector] Inserted blocks         progress=25092777
[INFO] [05-14|10:28:52.400] [3/6 Senders] Started                    from=25092610 to=25092777
[INFO] [05-14|10:28:52.508] [4/6 Execution] serial starting          from=25092611 to=25092777 initialTxNum=3516202118 initialBlockTxOffset=0 lastFrozenStep=8984 initialCycle=false isForkValidation=false
[WARN] [05-14|10:28:53.350] error executing clstage                  app=caplin stage=ForkChoice err="failed to compute and notify services of new fork choice: failed to get head: getCheckpointState: baseState not found in graph"
[...]
[WARN] [05-14|10:29:54.388] error executing clstage                  app=caplin stage=ForkChoice err="failed to compute and notify services of new fork choice: failed to get head: getCheckpointState: baseState not found in graph"
[...]
[INFO] [05-14|15:30:23.111] [Caplin] Forward Sync                    app=caplin stage=ForwardSync progress=14328956 distance-from-chain-tip=13m36s estimated-time-remaining=999h0m0s
[INFO] [05-14|15:30:26.745] [Caplin] Forward sync: no progress, handing off to ChainTipSync app=caplin stage=ForwardSync progress=14328956 targetSlot=14329024 stale=5m1s
[WARN] [05-14|15:30:26.746] error executing clstage                  app=caplin stage=ForwardSync err="forward sync stale"

It is still failing, see the test run of 19 May: https://github.com/erigontech/erigon/actions/runs/26069986510/job/76649143894#step:6:4014.

Also, the sync-from-scratch archive node test has failed on mainnet and hoodi,
see the test run here

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No fields configured for Bug.

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions