Migrated from a private QA tracker (erigontech/erigon-qa#401). Originally reported by @mriccobene on 2026-03-31.

Original report

with lighthouse https://github.com/erigontech/erigon/actions/runs/23704884846/job/69054997147#step:7:24095

with prism https://github.com/erigontech/erigon/actions/runs/23704884846/job/69054997103#step:7:15472

mainnet branch

also release/3.4, look here https://github.com/erigontech/erigon/actions/runs/24060915769/job/70196526427#step:7:13990

Triage notes (from original thread)

@taratorio — 2026-04-08:

"MDBX_MAP_FULL(-30792)The allocated database storage size limit has been reached"

[WARN] [03-30|11:07:29.604] Cannot update chain head                 hash=0x5223d9a4cc7b76722b21ab25fd02b18033bb63add34ffeb1589e84cf96536b68 err="updateForkChoice: updateForkChoice: flush sd after hasMore: loadIntoTable CommitmentVals: mdbx_cursor_del: MDBX_MAP_FULL(-30792)The allocated database storage size limit has been reached. You can try remove the database files (e.g., by running rm -rf /path/to/db)"

@mriccobene — 2026-04-13:

@taratorio please take a look at this run, I can't see any MDBX_MAP_FULL errors, and the error occurs soon after the 'Snapshots download stage completed' step:

2T02:08:04.8952205Z --> [2026-04-12 04:08:04] [INFO] SyncSentinel: node syncing, currentBlock=0, highestBlock=0
2026-04-12T02:08:06.3019222Z [INFO] [04-12|04:08:06.301] [snapshots:download] Stat                blocks=24.83M indices=24.83M alloc=6.4GB sys=7.1GB
2026-04-12T02:08:06.3105656Z [INFO] [04-12|04:08:06.310] [snapshots:history] Stat                 blocks=1 step2blockNum="8192=24.09M,8704=24.74M,8768=24.81M,8784=24.83M,8786=24.83M,8787=0" txs=3.43G first_history_idx_in_db=0 alloc=6.4GB sys=7.1GB
2026-04-12T02:08:06.3106518Z [INFO] [04-12|04:08:06.310] [1/6 OtterSync] DONE                     in=58m11.878364301s block=24837999
2026-04-12T02:08:06.3107080Z --> [2026-04-12 04:08:06] [INFO] Snapshots download stage completed
2026-04-12T02:08:08.9047359Z [INFO] [04-12|04:08:08.904] [NewPayload] Handling new payload        height=24860215 hash=0xdc0c93cac18049e4f024d8764fbe64401af81f91b4fe53a8330e7a3817dfa977
2026-04-12T02:08:08.9055500Z [INFO] [04-12|04:08:08.904] [EngineBlockDownloader] processing backward download of blocks hash=0x3e7966b0a1f5b394e1f0a8dc5a6775160e6688ec7c5a2a15d208d8e69eda24e3 trigger=NewPayloadTrigger chainTipNum=24860215 chainTipHash=0xdc0c93cac18049e4f024d8764fbe64401af81f91b4fe53a8330e7a3817dfa977
2026-04-12T02:08:08.9056989Z [INFO] [04-12|04:08:08.905] [backward-block-downloader] downloading initial header from all peers hash=0x3e7966b0a1f5b394e1f0a8dc5a6775160e6688ec7c5a2a15d208d8e69eda24e3
2026-04-12T02:08:09.9080451Z [INFO] [04-12|04:08:09.907] [backward-block-downloader] downloading header chain backward from initial header num=24860214 hash=0x3e7966b0a1f5b394e1f0a8dc5a6775160e6688ec7c5a2a15d208d8e69eda24e3
2026-04-12T02:08:10.8224797Z [WARN] [04-12|04:08:10.822] [EngineBlockDownloader] could not process backward download request hash=0x3e7966b0a1f5b394e1f0a8dc5a6775160e6688ec7c5a2a15d208d8e69eda24e3 trigger=NewPayloadTrigger chainTipNum=24860215 chainTipHash=0xdc0c93cac18049e4f024d8764fbe64401af81f91b4fe53a8330e7a3817dfa977 err="could not process backward download of blocks: chain length exceeds limit: num=24860214, hash=0x3e7966b0a1f5b394e1f0a8dc5a6775160e6688ec7c5a2a15d208d8e69eda24e3, len=501, limit=96"

I think the issue here is that "chain length exceeds limit", as the log reports.

@taratorio — 2026-04-14:

@mriccobene that's not an error log, it is expected to happen based on the current logic. NewPayload requests for chains that take more than 96 blocks (MAX_REORG_DEPTH) to connect to the current tip are not processed. Those can be processed by a ForkChoiceUpdated request. The expectation is that the CL will send a FCU eventually, however it looks like in this case Prysm never sent one. Will need to check why.

You can see the type of request that triggered the backward block download by looking at `trigger=NewPayloadTrigger` or `trigger=FcuTrigger` (I only see the former in this case).

@taratorio — 2026-04-15:

Will take care of this one as part of #20419 — I have a few ideas to try out. You can ignore for now until I get to it.

@mriccobene — 2026-04-15:

Not present with other CL clients.