Skip to content

execution: remove legacy headerdownload, bodydownload and dataflow packages#21505

Merged
yperbasis merged 7 commits into
mainfrom
remove_headers_bodies
May 30, 2026
Merged

execution: remove legacy headerdownload, bodydownload and dataflow packages#21505
yperbasis merged 7 commits into
mainfrom
remove_headers_bodies

Conversation

@taratorio

Copy link
Copy Markdown
Member

Why

headerdownload, bodydownload and dataflow implement the legacy devp2p header/body
download used by the staged-sync Headers/Bodies stages. Post-merge they are dead in
production: blocks arrive from the consensus layer via the engine API / execution module
(PipelineStages), which omits the Headers/Bodies stages entirely. The path was already
disabled — sentryMcDisableBlockDownload = true made MultiClient.Hd/Bd empty stubs, the
eth/66 download handlers early-returned, and StageLoop's only caller was the dead
pre-merge-PoW branch in Ethereum.Start().

What

  • Delete the three packages (~4.7k lines).
  • Headers/Bodies stage forward funcs become no-ops; their unwinds are kept but
    decoupled from headerdownload.
  • Remove StageLoop + StageLoopIteration + ProcessFrozenBlocks and the legacy
    non-PoS / non-Bor branch in Ethereum.Start().
  • Strip Hd/Bd from sentry_multi_client: the four download message handlers (now a
    single no-op dispatch case), SendHeaderRequest/SendBodyRequest/Penalize, and the
    dead broadcast.go (BroadcastNewBlock/PropagateNewBlockHashes).
  • Remove the dead params/plumbing this exposed: Sync.posTransition, the execute stage's
    unused hd interface + the dead POSSync()ReportBadHeaderPoS branches,
    disableBlockDownload, blockBufferSize/DefaultBlockBufferSize, maxBlockBroadcastPeers.
  • Rework the MockSentry test fixture to drop the vestigial downloader — tests already
    insert blocks via the PoS engine-API path (InsertChainUpdateForkChoice).

Behavior note: a bare pre-merge PoW chain (not PoS, not Bor) no longer starts a sync
loop — consistent with StageLoop being unused in production. PoS and Polygon are unaffected.

Verification

go build ./..., make erigon integration, and make lint are clean; targeted tests pass
(stagedsync, sentry_multi_client, execmoduletester, execution/tests, polygon/bor).
Full make test-all (on ramdisk) is running.

taratorio added 2 commits May 29, 2026 15:36
…iplePeers

Both fields became write-only after the header/body download handlers were
removed (their only readers were newBlock66 and the header-request path).
Pre-existing dead code: the method had no callers.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR removes Erigon’s legacy devp2p-based header/body download pipeline (headerdownload, bodydownload, dataflow) and the associated staged-sync loop plumbing that was already effectively dead in production post-merge. The remaining staged-sync “Headers/Bodies” stages are turned into forward no-ops (unwinds retained), and the sentry multi-client is simplified to drop the internal downloaders and related message handling/broadcasting.

Changes:

  • Delete legacy header/body downloader and dataflow packages; remove StageLoop-based syncing and related backend plumbing.
  • Simplify sentry_multi_client (drop Hd/Bd, disable download handlers, remove broadcast helpers, reduce constructor deps).
  • Adjust staged-sync stage configs/APIs (headers/bodies forward no-ops; remove downloader hooks from senders/execute) and update affected tests/integration tooling.

Reviewed changes

Copilot reviewed 33 out of 33 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
polygon/bor/bor_test.go Removes now-deleted WithBlockBufferSize option usage in Bor tests.
p2p/sentry/sentry_multi_client/sentry_multi_client.go Drops embedded Hd/Bd downloaders and turns eth/66 download inbound messages into no-ops.
p2p/sentry/sentry_multi_client/sentry_api.go Removes legacy request/penalty APIs tied to header/body downloading.
p2p/sentry/sentry_multi_client/broadcast.go Deletes legacy devp2p new-block/new-hash broadcast helpers.
node/eth/backend.go Removes legacy StageLoop startup/shutdown and sentry multi-client downloader plumbing; PoW-only branch no longer starts sync loop.
node/components/sentry/provider.go Simplifies MultiClientDeps and BuildMultiClient call (removes sync cfg/buffer/broadcast/download flags).
execution/tests/blockchain_test.go Removes now-deleted WithBlockBufferSize option usage in execution tests.
execution/stagedsync/sync.go Removes posTransition gating from break-after-stage logic.
execution/stagedsync/stageloop/stageloop.go Removes StageLoop/ProcessFrozenBlocks/StageLoopIteration and updates stage construction to no longer depend on Hd/Bd.
execution/stagedsync/stage_witness.go Updates Execute stage cfg construction for witness rewind (but currently has a signature/order mismatch—see comment).
execution/stagedsync/stage_senders.go Removes hd-based PoS bad-header reporting hooks from Senders stage.
execution/stagedsync/stage_senders_test.go Updates senders stage cfg call signature.
execution/stagedsync/stage_headers.go Replaces headers stage cfg with a minimal cfg; keeps unwind logic but removes downloader interactions.
execution/stagedsync/stage_execute.go Removes header-downloader interface hook from execute stage configuration.
execution/stagedsync/stage_bodies.go Removes bodies forward download logic; keeps unwind via block writer canonical markers.
execution/stagedsync/exec3.go Removes bad-header reporting hook from wrong-trie-root path; adjusts helper signature accordingly.
execution/stagedsync/exec3_serial.go Removes PoS bad-header reporting hook on serial execution invalid block.
execution/stagedsync/exec3_parallel.go Removes PoS bad-header reporting hook on parallel execution invalid block; updates helper signature usage.
execution/stagedsync/default_stages.go Makes Headers/Bodies forward functions no-ops while retaining unwind hooks.
execution/stagedsync/dataflow/states.go Deletes legacy download state tracking utility.
execution/stagedsync/headerdownload/header_data_struct.go Deletes legacy header download implementation.
execution/stagedsync/headerdownload/header_algos.go Deletes legacy header download implementation.
execution/stagedsync/headerdownload/header_algo_test.go Deletes legacy header download tests.
execution/stagedsync/bodydownload/prefetched_blocks.go Deletes legacy body prefetch cache used by downloader.
execution/stagedsync/bodydownload/body_test.go Deletes legacy body downloader tests.
execution/stagedsync/bodydownload/body_data_struct.go Deletes legacy body download implementation.
execution/stagedsync/bodydownload/body_algos.go Deletes legacy body download implementation.
execution/stagedsync/bodydownload/block_propagator.go Deletes legacy block propagator type used by downloader.
execution/execmodule/execmoduletester/from0_genesis_internal_test.go Removes passing Hd into execute cfg builder.
execution/execmodule/execmoduletester/exec_module_tester.go Removes WithBlockBufferSize, drops internal downloader wiring from mock sentry setup and stage construction.
db/snapshotsync/freezeblocks/dump_test.go Removes now-deleted WithBlockBufferSize option usage.
cmd/integration/commands/state_stages.go Updates execute stage cfg calls for signature change (removes hd arg).
cmd/integration/commands/stages.go Updates bodies/senders/exec integration commands for new stage cfg signatures and NewDefaultStages signature change.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread execution/stagedsync/stage_witness.go Outdated
The inline comments on the StageExecuteBlocksCfg call were off-by-one (stale
from an older signature); relabel them to match the current parameters.
@taratorio taratorio added this pull request to the merge queue May 29, 2026
@taratorio taratorio removed this pull request from the merge queue due to a manual request May 29, 2026
@taratorio taratorio enabled auto-merge May 29, 2026 14:30
@taratorio taratorio added this pull request to the merge queue May 29, 2026
@github-merge-queue github-merge-queue Bot removed this pull request from the merge queue due to failed status checks May 29, 2026
@yperbasis yperbasis enabled auto-merge May 30, 2026 08:14
@yperbasis yperbasis added this pull request to the merge queue May 30, 2026
Merged via the queue into main with commit d0caf67 May 30, 2026
89 checks passed
@yperbasis yperbasis deleted the remove_headers_bodies branch May 30, 2026 09:25
JkLondon pushed a commit that referenced this pull request May 31, 2026
…es/otterprunefixes

Conflict resolution:
- execution/execmodule/forkchoice.go: main renamed agg.CollateAndPruneIfNeeded
  -> CollateAndPrune; kept this PR's DB-backed prune lifecycle (IsInitialCycle
  + captured pruneInitialCycle in the timings log) on top of the renamed call.
- execution/stagedsync/stageloop/stageloop.go: main's legacy-headerdownload
  removal (#21505) deleted the StageLoop / ProcessFrozenBlocks / StageLoopIteration
  free functions this PR had modified. Took main's version — the initial-cycle
  lifecycle (known-tip hint, IsInitialCycle, UpdateTipReached) is now carried by
  the execmodule path (PipelineExecutor.ProcessFrozenBlocks + forkchoice), with
  the known tip coming from the Caplin hint + FCU head instead of hd.Progress().
bloxster added a commit that referenced this pull request Jun 1, 2026
Addresses the CHANGES_REQUESTED review, verified against `main`:

database.md
- Receipts ARE stored: ReceiptDomain (required) + RCacheDomain (optional).
  Reword the "not stored / re-computed" claim to: compact per-txn receipt
  metadata persisted in a required domain, full receipts optionally cached,
  logs re-derived by re-execution only when not cached.
- snapshots/domain holds 6 domains (4 state + 2 receipt), not 4.
- Soften "rm -rf chaindata/": recoverable, but re-derives state from snapshots
  and resyncs the tip from the CL over the Engine API (devp2p block download
  removed in #21505) — not a quick rebuild.

architecture.md
- prune `full` keeps a rolling ~262k-block window (DefaultPruneDistance), not
  all post-Merge blocks; add the `blocks` prune mode to the table.
- Pipeline: Snapshots is stage #1; no separate Commitment stage (it runs inside
  Execution); add Senders.
- Mermaid: Downloader (BitTorrent) is independent of Sentry (devp2p); blocks
  arrive from Caplin via the Engine API.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sahil-4555 pushed a commit to Sahil-4555/erigon that referenced this pull request Jun 2, 2026
…rigontech#21560)

## Summary
An **empty** NewBlockHashes packet (RLP `0xc0`) decodes to a zero-length
slice **without error**, so the surviving `execution/p2p` inbound path
delivers it to observers unpenalized. The `polygon/sync` tip-events
observer then indexed `blockHashes[0]` with no length check and
panicked; observers run in a bare `go observer(event)` with no
`recover()`, so the panic aborted the whole process — a remote,
unauthenticated **crash DoS on polygon/Bor nodes**.

Fix: ignore announcements with no entries before any `blockHashes[0]`
access.

Closes erigontech/security#72

## Scope
Only the polygon/Bor astrid path (`polygon/sync` tip-events) is affected
— it is the live consumer of inbound NewBlockHashes. On post-Merge
Ethereum the eth multi-client drops these messages (erigontech#21505), so this is
polygon-specific. `peer_tracker`'s observer ranges over the slice and is
already safe on empty; `NewBlock` decodes into a struct, so an empty
packet is rejected at decode rather than reaching an observer.

## Test plan
- [x] `TestTipEventsEmptyNewBlockHashesDoesNotPanic` — empty packet no
longer panics the observer (red→green; the red failure was `index out of
range [0] with length 0` at `tip_events.go:213`)
- [x] `TestTipEventsNewBlockHashesEmitsEvent` — non-empty announcements
still emit a `NewBlockHashes` event
- [x] `go test -race ./polygon/sync/` (new tests, `-count=5`)
- [x] `make lint` clean; `make erigon integration`

## Note
A `recover()` in `common/event` `Observers.Notify` would be
complementary defense-in-depth — it would catch any future observer
panic, not just this one — but per the issue it should not replace this
targeted length guard.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: milen <94537774+taratorio@users.noreply.github.com>
Sahil-4555 pushed a commit to Sahil-4555/erigon that referenced this pull request Jun 2, 2026
…ntech#21557)

## Summary
Harden the per-peer inbound loop (`runPeer`) against abusive
`NewBlockHashes`:
- **Oversize (bytes):** reject a packet whose wire size exceeds
`maxNewBlockHashesBytes` **before the payload is buffered** — a multi-MB
announcement is dropped without being allocated.
- **Entry cap:** after the byte-bounded read, walk the RLP list (no
allocation) and reject packets with more than `maxBlockHashesPerMsg`
(4096) entries. The byte ceiling alone doesn't bound the entry count
(RLP entries are variable-length), so this enforces the hard cap.
- **Flood:** a per-peer token-bucket limiter (10/s, burst 30) kicks
peers flooding announcements; it is local to `runPeer`, so its state is
bounded and freed on disconnect.

The peer is disconnected on any of these, and nothing is forwarded to
subscribers (eth multi-client or the polygon `execution/p2p` observer).
The size and rate gates run before the `hasSubscribers` short-circuit,
so abusive peers are kicked regardless of sync state.

The handler-level peek (closed PR erigontech#21476) was obviated by erigontech#21505, which
removed the eth block-download handler; this places the protection at
the framing layer where it also covers the surviving `execution/p2p`
observer path.

## Scope
The live consumer of inbound NewBlockHashes is **polygon/Bor**
(`polygon/sync` tip-events via `execution/p2p`), which still gossips
blocks over devp2p; post-Merge Ethereum doesn't use these messages — the
eth multi-client drops them (erigontech#21505) — so there it's generic
defense-in-depth against peers sending unsolicited announcements. The
thresholds sit well above honest Bor rates (~1 announcement per block),
so legitimate peers are unaffected.

Closes erigontech/security#69
Closes erigontech/security#68

## Test plan
- [x] `TestRunPeer_OversizedNewBlockHashesKicksPeer` — oversized (bytes)
packet kicks peer (`PeerErrorMessageSizeLimit`) and is not forwarded to
subscribers
- [x] `TestRunPeer_TooManyNewBlockHashesEntriesKicksPeer` — a packet
under the byte cap but over the 4096-entry cap kicks the peer
- [x] `TestRunPeer_NewBlockHashesFloodKicksPeer` — flood past burst
kicks peer (`PeerErrorInvalidMessage`)
- [x] `TestRunPeer_NormalNewBlockHashesForwarded` — compliant traffic is
forwarded and not penalized
- [x] `go test -race ./p2p/sentry/` (new tests, `-count=5`)
- [x] `make lint` clean; `make erigon integration`

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: milen <94537774+taratorio@users.noreply.github.com>
chris-mercer pushed a commit to white-b0x/erigon that referenced this pull request Jun 3, 2026
## Summary

A sweep of the repo's **unconditionally-disabled tests** — `t.Skip(...)`
not gated on `testing.Short()`. This removes **24** such skips, either
by fixing the underlying behavior or by converting permanent skips into
clean precondition gates (live node, snapshot file, env var, RAM) that
run when the resource is present and skip cleanly when it isn't. What's
intentionally left skipped is listed below.

(Out of scope: `testing.Short()` skips, Windows/race-guarded skips,
runtime precondition guards, and commented-out skips.)

## t8n / `cmd/evm` (the meaty part)

Un-skipping `TestT8n` surfaced **5 stacked bugs**, all fixed here:

1. `getTransaction` dropped the JSON `v/r/s`, so signed txs got a zero
signature → `ErrInvalidSig`. Now preserves V/R/S (new
`TestGetTransactionPreservesSignature`).
2. **E3 state plumbing** — `MakePreState` read base state from `tx`
while writing to `SharedDomains` (execution saw balance 0), and
`CalculateStateRoot` used a fresh `SharedDomains` (→ empty-trie root).
Now reads/commits on the same `sd` that executed, then flushes and
records the block→txNum so the alloc dumper sees the post-state.
3. `EphemeralExecResult.Difficulty` now serializes as hex
(`*math.HexOrDecimal256`), matching the spec. (Only non-test change; the
field is marshaled solely by t8n.)
4. A blockhash requested but missing from the env now yields exit code 4
instead of a silent tx rejection.
5. Empty receipts serialize as `null`, matching the reference output.

## Deliberately left skipped

- **`node/app/component` gomock tests** (`TestComponentLifecycle`,
`TestDomainLifecycle`, `TestAddRemoveDeps`) — left skipped; this PR
doesn't touch them. The whole package is disabled by a pre-existing
`TestMain` `os.Exit(0)`; re-enabling it exposes a data race (fixed in
erigontech#21551) plus a flaky test and a goroutine-leak deadlock, tracked in
erigontech#21552.
- **HexPatriciaHashed traps** — `Test_ModeUpdate_SiblingConsistency`
(erigontech#20961) and `Test_HexPatriciaHashed_StateRestoreAndContinue`
(concurrent map write): consensus-critical, left as regression markers.
- **Feature-gated** — `execmodule` background-commit and
`component/fuzz_test` wait on features not yet built.
- **`node.TestRegisterProtocols`** — empty `// TODO` stub for p2p
registration that moved to sentry; recommend deleting.
- **Likely real bugs, left for owner judgment** —
`execution/state.TestCreateOnExistingStorage` (Erigon keeps pre-existing
storage on CREATE-over; go-ethereum clears it) and
`db/test.Test_AggregatorV3_RestartOnDatadir_{WithoutDB,WithoutAnything}`
(restart-from-files-without-DB fails `commitmentdb` freshness).

## Testing

`make lint` clean, `make erigon integration` builds, and every touched
test passes (resource-gated ones skip cleanly without their resources).
Latest `main` is merged in, which brings erigontech#21505's removal of the legacy
`headerdownload` / `bodydownload` / `dataflow` packages.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: info@weblogix.biz <admin@10gbps.weblogix.it>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants