execution: remove legacy headerdownload, bodydownload and dataflow packages#21505
Merged
Conversation
…iplePeers Both fields became write-only after the header/body download handlers were removed (their only readers were newBlock66 and the header-request path).
Pre-existing dead code: the method had no callers.
Contributor
There was a problem hiding this comment.
Pull request overview
This PR removes Erigon’s legacy devp2p-based header/body download pipeline (headerdownload, bodydownload, dataflow) and the associated staged-sync loop plumbing that was already effectively dead in production post-merge. The remaining staged-sync “Headers/Bodies” stages are turned into forward no-ops (unwinds retained), and the sentry multi-client is simplified to drop the internal downloaders and related message handling/broadcasting.
Changes:
- Delete legacy header/body downloader and dataflow packages; remove
StageLoop-based syncing and related backend plumbing. - Simplify
sentry_multi_client(drop Hd/Bd, disable download handlers, remove broadcast helpers, reduce constructor deps). - Adjust staged-sync stage configs/APIs (headers/bodies forward no-ops; remove downloader hooks from senders/execute) and update affected tests/integration tooling.
Reviewed changes
Copilot reviewed 33 out of 33 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| polygon/bor/bor_test.go | Removes now-deleted WithBlockBufferSize option usage in Bor tests. |
| p2p/sentry/sentry_multi_client/sentry_multi_client.go | Drops embedded Hd/Bd downloaders and turns eth/66 download inbound messages into no-ops. |
| p2p/sentry/sentry_multi_client/sentry_api.go | Removes legacy request/penalty APIs tied to header/body downloading. |
| p2p/sentry/sentry_multi_client/broadcast.go | Deletes legacy devp2p new-block/new-hash broadcast helpers. |
| node/eth/backend.go | Removes legacy StageLoop startup/shutdown and sentry multi-client downloader plumbing; PoW-only branch no longer starts sync loop. |
| node/components/sentry/provider.go | Simplifies MultiClientDeps and BuildMultiClient call (removes sync cfg/buffer/broadcast/download flags). |
| execution/tests/blockchain_test.go | Removes now-deleted WithBlockBufferSize option usage in execution tests. |
| execution/stagedsync/sync.go | Removes posTransition gating from break-after-stage logic. |
| execution/stagedsync/stageloop/stageloop.go | Removes StageLoop/ProcessFrozenBlocks/StageLoopIteration and updates stage construction to no longer depend on Hd/Bd. |
| execution/stagedsync/stage_witness.go | Updates Execute stage cfg construction for witness rewind (but currently has a signature/order mismatch—see comment). |
| execution/stagedsync/stage_senders.go | Removes hd-based PoS bad-header reporting hooks from Senders stage. |
| execution/stagedsync/stage_senders_test.go | Updates senders stage cfg call signature. |
| execution/stagedsync/stage_headers.go | Replaces headers stage cfg with a minimal cfg; keeps unwind logic but removes downloader interactions. |
| execution/stagedsync/stage_execute.go | Removes header-downloader interface hook from execute stage configuration. |
| execution/stagedsync/stage_bodies.go | Removes bodies forward download logic; keeps unwind via block writer canonical markers. |
| execution/stagedsync/exec3.go | Removes bad-header reporting hook from wrong-trie-root path; adjusts helper signature accordingly. |
| execution/stagedsync/exec3_serial.go | Removes PoS bad-header reporting hook on serial execution invalid block. |
| execution/stagedsync/exec3_parallel.go | Removes PoS bad-header reporting hook on parallel execution invalid block; updates helper signature usage. |
| execution/stagedsync/default_stages.go | Makes Headers/Bodies forward functions no-ops while retaining unwind hooks. |
| execution/stagedsync/dataflow/states.go | Deletes legacy download state tracking utility. |
| execution/stagedsync/headerdownload/header_data_struct.go | Deletes legacy header download implementation. |
| execution/stagedsync/headerdownload/header_algos.go | Deletes legacy header download implementation. |
| execution/stagedsync/headerdownload/header_algo_test.go | Deletes legacy header download tests. |
| execution/stagedsync/bodydownload/prefetched_blocks.go | Deletes legacy body prefetch cache used by downloader. |
| execution/stagedsync/bodydownload/body_test.go | Deletes legacy body downloader tests. |
| execution/stagedsync/bodydownload/body_data_struct.go | Deletes legacy body download implementation. |
| execution/stagedsync/bodydownload/body_algos.go | Deletes legacy body download implementation. |
| execution/stagedsync/bodydownload/block_propagator.go | Deletes legacy block propagator type used by downloader. |
| execution/execmodule/execmoduletester/from0_genesis_internal_test.go | Removes passing Hd into execute cfg builder. |
| execution/execmodule/execmoduletester/exec_module_tester.go | Removes WithBlockBufferSize, drops internal downloader wiring from mock sentry setup and stage construction. |
| db/snapshotsync/freezeblocks/dump_test.go | Removes now-deleted WithBlockBufferSize option usage. |
| cmd/integration/commands/state_stages.go | Updates execute stage cfg calls for signature change (removes hd arg). |
| cmd/integration/commands/stages.go | Updates bodies/senders/exec integration commands for new stage cfg signatures and NewDefaultStages signature change. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
The inline comments on the StageExecuteBlocksCfg call were off-by-one (stale from an older signature); relabel them to match the current parameters.
yperbasis
approved these changes
May 29, 2026
JkLondon
pushed a commit
that referenced
this pull request
May 31, 2026
…es/otterprunefixes Conflict resolution: - execution/execmodule/forkchoice.go: main renamed agg.CollateAndPruneIfNeeded -> CollateAndPrune; kept this PR's DB-backed prune lifecycle (IsInitialCycle + captured pruneInitialCycle in the timings log) on top of the renamed call. - execution/stagedsync/stageloop/stageloop.go: main's legacy-headerdownload removal (#21505) deleted the StageLoop / ProcessFrozenBlocks / StageLoopIteration free functions this PR had modified. Took main's version — the initial-cycle lifecycle (known-tip hint, IsInitialCycle, UpdateTipReached) is now carried by the execmodule path (PipelineExecutor.ProcessFrozenBlocks + forkchoice), with the known tip coming from the Caplin hint + FCU head instead of hd.Progress().
This was referenced Jun 1, 2026
bloxster
added a commit
that referenced
this pull request
Jun 1, 2026
Addresses the CHANGES_REQUESTED review, verified against `main`: database.md - Receipts ARE stored: ReceiptDomain (required) + RCacheDomain (optional). Reword the "not stored / re-computed" claim to: compact per-txn receipt metadata persisted in a required domain, full receipts optionally cached, logs re-derived by re-execution only when not cached. - snapshots/domain holds 6 domains (4 state + 2 receipt), not 4. - Soften "rm -rf chaindata/": recoverable, but re-derives state from snapshots and resyncs the tip from the CL over the Engine API (devp2p block download removed in #21505) — not a quick rebuild. architecture.md - prune `full` keeps a rolling ~262k-block window (DefaultPruneDistance), not all post-Merge blocks; add the `blocks` prune mode to the table. - Pipeline: Snapshots is stage #1; no separate Commitment stage (it runs inside Execution); add Senders. - Mermaid: Downloader (BitTorrent) is independent of Sentry (devp2p); blocks arrive from Caplin via the Engine API. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sahil-4555
pushed a commit
to Sahil-4555/erigon
that referenced
this pull request
Jun 2, 2026
…rigontech#21560) ## Summary An **empty** NewBlockHashes packet (RLP `0xc0`) decodes to a zero-length slice **without error**, so the surviving `execution/p2p` inbound path delivers it to observers unpenalized. The `polygon/sync` tip-events observer then indexed `blockHashes[0]` with no length check and panicked; observers run in a bare `go observer(event)` with no `recover()`, so the panic aborted the whole process — a remote, unauthenticated **crash DoS on polygon/Bor nodes**. Fix: ignore announcements with no entries before any `blockHashes[0]` access. Closes erigontech/security#72 ## Scope Only the polygon/Bor astrid path (`polygon/sync` tip-events) is affected — it is the live consumer of inbound NewBlockHashes. On post-Merge Ethereum the eth multi-client drops these messages (erigontech#21505), so this is polygon-specific. `peer_tracker`'s observer ranges over the slice and is already safe on empty; `NewBlock` decodes into a struct, so an empty packet is rejected at decode rather than reaching an observer. ## Test plan - [x] `TestTipEventsEmptyNewBlockHashesDoesNotPanic` — empty packet no longer panics the observer (red→green; the red failure was `index out of range [0] with length 0` at `tip_events.go:213`) - [x] `TestTipEventsNewBlockHashesEmitsEvent` — non-empty announcements still emit a `NewBlockHashes` event - [x] `go test -race ./polygon/sync/` (new tests, `-count=5`) - [x] `make lint` clean; `make erigon integration` ## Note A `recover()` in `common/event` `Observers.Notify` would be complementary defense-in-depth — it would catch any future observer panic, not just this one — but per the issue it should not replace this targeted length guard. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: milen <94537774+taratorio@users.noreply.github.com>
Sahil-4555
pushed a commit
to Sahil-4555/erigon
that referenced
this pull request
Jun 2, 2026
…ntech#21557) ## Summary Harden the per-peer inbound loop (`runPeer`) against abusive `NewBlockHashes`: - **Oversize (bytes):** reject a packet whose wire size exceeds `maxNewBlockHashesBytes` **before the payload is buffered** — a multi-MB announcement is dropped without being allocated. - **Entry cap:** after the byte-bounded read, walk the RLP list (no allocation) and reject packets with more than `maxBlockHashesPerMsg` (4096) entries. The byte ceiling alone doesn't bound the entry count (RLP entries are variable-length), so this enforces the hard cap. - **Flood:** a per-peer token-bucket limiter (10/s, burst 30) kicks peers flooding announcements; it is local to `runPeer`, so its state is bounded and freed on disconnect. The peer is disconnected on any of these, and nothing is forwarded to subscribers (eth multi-client or the polygon `execution/p2p` observer). The size and rate gates run before the `hasSubscribers` short-circuit, so abusive peers are kicked regardless of sync state. The handler-level peek (closed PR erigontech#21476) was obviated by erigontech#21505, which removed the eth block-download handler; this places the protection at the framing layer where it also covers the surviving `execution/p2p` observer path. ## Scope The live consumer of inbound NewBlockHashes is **polygon/Bor** (`polygon/sync` tip-events via `execution/p2p`), which still gossips blocks over devp2p; post-Merge Ethereum doesn't use these messages — the eth multi-client drops them (erigontech#21505) — so there it's generic defense-in-depth against peers sending unsolicited announcements. The thresholds sit well above honest Bor rates (~1 announcement per block), so legitimate peers are unaffected. Closes erigontech/security#69 Closes erigontech/security#68 ## Test plan - [x] `TestRunPeer_OversizedNewBlockHashesKicksPeer` — oversized (bytes) packet kicks peer (`PeerErrorMessageSizeLimit`) and is not forwarded to subscribers - [x] `TestRunPeer_TooManyNewBlockHashesEntriesKicksPeer` — a packet under the byte cap but over the 4096-entry cap kicks the peer - [x] `TestRunPeer_NewBlockHashesFloodKicksPeer` — flood past burst kicks peer (`PeerErrorInvalidMessage`) - [x] `TestRunPeer_NormalNewBlockHashesForwarded` — compliant traffic is forwarded and not penalized - [x] `go test -race ./p2p/sentry/` (new tests, `-count=5`) - [x] `make lint` clean; `make erigon integration` 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: milen <94537774+taratorio@users.noreply.github.com>
chris-mercer
pushed a commit
to white-b0x/erigon
that referenced
this pull request
Jun 3, 2026
## Summary A sweep of the repo's **unconditionally-disabled tests** — `t.Skip(...)` not gated on `testing.Short()`. This removes **24** such skips, either by fixing the underlying behavior or by converting permanent skips into clean precondition gates (live node, snapshot file, env var, RAM) that run when the resource is present and skip cleanly when it isn't. What's intentionally left skipped is listed below. (Out of scope: `testing.Short()` skips, Windows/race-guarded skips, runtime precondition guards, and commented-out skips.) ## t8n / `cmd/evm` (the meaty part) Un-skipping `TestT8n` surfaced **5 stacked bugs**, all fixed here: 1. `getTransaction` dropped the JSON `v/r/s`, so signed txs got a zero signature → `ErrInvalidSig`. Now preserves V/R/S (new `TestGetTransactionPreservesSignature`). 2. **E3 state plumbing** — `MakePreState` read base state from `tx` while writing to `SharedDomains` (execution saw balance 0), and `CalculateStateRoot` used a fresh `SharedDomains` (→ empty-trie root). Now reads/commits on the same `sd` that executed, then flushes and records the block→txNum so the alloc dumper sees the post-state. 3. `EphemeralExecResult.Difficulty` now serializes as hex (`*math.HexOrDecimal256`), matching the spec. (Only non-test change; the field is marshaled solely by t8n.) 4. A blockhash requested but missing from the env now yields exit code 4 instead of a silent tx rejection. 5. Empty receipts serialize as `null`, matching the reference output. ## Deliberately left skipped - **`node/app/component` gomock tests** (`TestComponentLifecycle`, `TestDomainLifecycle`, `TestAddRemoveDeps`) — left skipped; this PR doesn't touch them. The whole package is disabled by a pre-existing `TestMain` `os.Exit(0)`; re-enabling it exposes a data race (fixed in erigontech#21551) plus a flaky test and a goroutine-leak deadlock, tracked in erigontech#21552. - **HexPatriciaHashed traps** — `Test_ModeUpdate_SiblingConsistency` (erigontech#20961) and `Test_HexPatriciaHashed_StateRestoreAndContinue` (concurrent map write): consensus-critical, left as regression markers. - **Feature-gated** — `execmodule` background-commit and `component/fuzz_test` wait on features not yet built. - **`node.TestRegisterProtocols`** — empty `// TODO` stub for p2p registration that moved to sentry; recommend deleting. - **Likely real bugs, left for owner judgment** — `execution/state.TestCreateOnExistingStorage` (Erigon keeps pre-existing storage on CREATE-over; go-ethereum clears it) and `db/test.Test_AggregatorV3_RestartOnDatadir_{WithoutDB,WithoutAnything}` (restart-from-files-without-DB fails `commitmentdb` freshness). ## Testing `make lint` clean, `make erigon integration` builds, and every touched test passes (resource-gated ones skip cleanly without their resources). Latest `main` is merged in, which brings erigontech#21505's removal of the legacy `headerdownload` / `bodydownload` / `dataflow` packages. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: info@weblogix.biz <admin@10gbps.weblogix.it>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
headerdownload,bodydownloadanddataflowimplement the legacy devp2p header/bodydownload used by the staged-sync Headers/Bodies stages. Post-merge they are dead in
production: blocks arrive from the consensus layer via the engine API / execution module
(
PipelineStages), which omits the Headers/Bodies stages entirely. The path was alreadydisabled —
sentryMcDisableBlockDownload = truemadeMultiClient.Hd/Bdempty stubs, theeth/66 download handlers early-returned, and
StageLoop's only caller was the deadpre-merge-PoW branch in
Ethereum.Start().What
decoupled from
headerdownload.StageLoop+StageLoopIteration+ProcessFrozenBlocksand the legacynon-PoS / non-Bor branch in
Ethereum.Start().Hd/Bdfromsentry_multi_client: the four download message handlers (now asingle no-op dispatch case),
SendHeaderRequest/SendBodyRequest/Penalize, and thedead
broadcast.go(BroadcastNewBlock/PropagateNewBlockHashes).Sync.posTransition, the execute stage'sunused
hdinterface + the deadPOSSync()→ReportBadHeaderPoSbranches,disableBlockDownload,blockBufferSize/DefaultBlockBufferSize,maxBlockBroadcastPeers.MockSentrytest fixture to drop the vestigial downloader — tests alreadyinsert blocks via the PoS engine-API path (
InsertChain→UpdateForkChoice).Behavior note: a bare pre-merge PoW chain (not PoS, not Bor) no longer starts a sync
loop — consistent with
StageLoopbeing unused in production. PoS and Polygon are unaffected.Verification
go build ./...,make erigon integration, andmake lintare clean; targeted tests pass(
stagedsync,sentry_multi_client,execmoduletester,execution/tests,polygon/bor).Full
make test-all(on ramdisk) is running.