cherry-pick: from release/3.4 to main#20658
Conversation
Cherry-pick of #20609 to release/3.4 Co-authored-by: bendertherobert <bendertherobert@gmail.com>
``` [INFO] [04-15|05:15:50.145] [integrity] StateRootVerifyByHistory blks/s=0.5 checked=3.72k/2.48k windows=1860/2488 blkRange=1-24.87M ``` `checked` overflow
we need it because we're already doing out-of-order prune of domains. So need this protection. --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Alex Sharov <AskAlexSharov@gmail.com>
Replace real TCP connections with in-memory net.Pipe-based listener for
the HTTP cancel test, wrapped in synctest.Test for deterministic
fake-time execution. The test now runs in ~0s instead of timing out on
slow CI runners.
WebSocket cancel test keeps real TCP due to complex goroutine shutdown
dependencies incompatible with synctest's bubble requirements.
err:
```
[WARN] [04-07|01:51:47.496] Cannot register RPC callback [invalidRets1] - error must the last return value
[WARN] [04-07|01:51:47.496] Cannot register RPC callback [invalidRets2] - error must the last return value
[WARN] [04-07|01:51:47.497] Cannot register RPC callback [invalidRets3] - maximum 2 return values are allowed, got 3
[WARN] [04-07|01:51:47.507] Cannot register RPC callback [invalidRets1] - error must the last return value
[WARN] [04-07|01:51:47.507] Cannot register RPC callback [invalidRets2] - error must the last return value
[WARN] [04-07|01:51:47.507] Cannot register RPC callback [invalidRets3] - maximum 2 return values are allowed, got 3
[WARN] [04-07|01:51:47.510] Cannot register RPC callback [invalidRets1] - error must the last return value
[WARN] [04-07|01:51:47.510] Cannot register RPC callback [invalidRets2] - error must the last return value
[WARN] [04-07|01:51:47.510] Cannot register RPC callback [invalidRets3] - maximum 2 return values are allowed, got 3
[WARN] [04-07|01:51:47.512] [rpc] served method=test_returnError reqid=1 err=testError errdata="testError data"
[WARN] [04-07|01:51:47.513] Cannot register RPC callback [invalidRets1] - error must the last return value
[WARN] [04-07|01:51:47.513] Cannot register RPC callback [invalidRets2] - error must the last return value
[WARN] [04-07|01:51:47.513] Cannot register RPC callback [invalidRets3] - maximum 2 return values are allowed, got 3
[WARN] [04-07|01:51:47.515] [rpc] served method=no_such_method reqid=3 err="the method no_such_method does not exist/is not available"
[WARN] [04-07|01:51:47.516] Cannot register RPC callback [invalidRets1] - error must the last return value
[WARN] [04-07|01:51:47.516] Cannot register RPC callback [invalidRets2] - error must the last return value
[WARN] [04-07|01:51:47.516] Cannot register RPC callback [invalidRets3] - maximum 2 return values are allowed, got 3
[WARN] [04-07|01:51:47.517] Cannot register RPC callback [invalidRets1] - error must the last return value
[WARN] [04-07|01:51:47.517] Cannot register RPC callback [invalidRets2] - error must the last return value
[WARN] [04-07|01:51:47.518] Cannot register RPC callback [invalidRets3] - maximum 2 return values are allowed, got 3
[WARN] [04-07|01:51:47.519] [rpc] served method=eth_subscribe reqid=1 err="no \"foo_bar\" subscription in eth namespace"
[WARN] [04-07|01:51:47.519] [rpc] served method=eth_subscribe reqid=2 err="no \"foo_bar\" subscription in eth namespace"
[WARN] [04-07|01:51:47.520] Cannot register RPC callback [invalidRets1] - error must the last return value
[WARN] [04-07|01:51:47.520] Cannot register RPC callback [invalidRets2] - error must the last return value
[WARN] [04-07|01:51:47.520] Cannot register RPC callback [invalidRets3] - maximum 2 return values are allowed, got 3
[WARN] [04-07|01:51:47.523] Cannot register RPC callback [invalidRets1] - error must the last return value
[WARN] [04-07|01:51:47.523] Cannot register RPC callback [invalidRets2] - error must the last return value
[WARN] [04-07|01:51:47.524] Cannot register RPC callback [invalidRets3] - maximum 2 return values are allowed, got 3
[WARN] [04-07|01:51:47.527] Cannot register RPC callback [invalidRets1] - error must the last return value
[WARN] [04-07|01:51:47.527] Cannot register RPC callback [invalidRets2] - error must the last return value
[WARN] [04-07|01:51:47.527] Cannot register RPC callback [invalidRets3] - maximum 2 return values are allowed, got 3
[WARN] [04-07|01:51:47.563] Cannot register RPC callback [invalidRets1] - error must the last return value
[WARN] [04-07|01:51:47.563] Cannot register RPC callback [invalidRets2] - error must the last return value
[WARN] [04-07|01:51:47.563] Cannot register RPC callback [invalidRets3] - maximum 2 return values are allowed, got 3
[WARN] [04-07|01:51:56.015] Cannot register RPC callback [invalidRets1] - error must the last return value
[WARN] [04-07|01:51:56.015] Cannot register RPC callback [invalidRets2] - error must the last return value
[WARN] [04-07|01:51:56.015] Cannot register RPC callback [invalidRets3] - maximum 2 return values are allowed, got 3
[WARN] [04-07|01:51:56.019] Cannot register RPC callback [invalidRets1] - error must the last return value
[WARN] [04-07|01:51:56.019] Cannot register RPC callback [invalidRets2] - error must the last return value
[WARN] [04-07|01:51:56.019] Cannot register RPC callback [invalidRets3] - maximum 2 return values are allowed, got 3
[WARN] [04-07|01:51:56.103] Cannot register RPC callback [invalidRets1] - error must the last return value
[WARN] [04-07|01:51:56.103] Cannot register RPC callback [invalidRets2] - error must the last return value
[WARN] [04-07|01:51:56.103] Cannot register RPC callback [invalidRets3] - maximum 2 return values are allowed, got 3
[WARN] [04-07|01:51:58.107] Cannot register RPC callback [invalidRets1] - error must the last return value
[WARN] [04-07|01:51:58.107] Cannot register RPC callback [invalidRets2] - error must the last return value
[WARN] [04-07|01:51:58.107] Cannot register RPC callback [invalidRets3] - maximum 2 return values are allowed, got 3
[WARN] [04-07|01:52:32.735] Cannot register RPC callback [invalidRets1] - error must the last return value
[WARN] [04-07|01:52:32.735] Cannot register RPC callback [invalidRets2] - error must the last return value
[WARN] [04-07|01:52:32.735] Cannot register RPC callback [invalidRets3] - maximum 2 return values are allowed, got 3
[WARN] [04-07|01:52:32.737] Cannot register RPC callback [invalidRets1] - error must the last return value
[WARN] [04-07|01:52:32.737] Cannot register RPC callback [invalidRets2] - error must the last return value
[WARN] [04-07|01:52:32.737] Cannot register RPC callback [invalidRets3] - maximum 2 return values are allowed, got 3
[WARN] [04-07|01:52:32.737] Cannot register RPC callback [invalidRets1] - error must the last return value
[WARN] [04-07|01:52:32.737] Cannot register RPC callback [invalidRets2] - error must the last return value
[WARN] [04-07|01:52:32.737] Cannot register RPC callback [invalidRets3] - maximum 2 return values are allowed, got 3
[WARN] [04-07|01:52:32.738] Cannot register RPC callback [invalidRets1] - error must the last return value
[WARN] [04-07|01:52:32.738] Cannot register RPC callback [invalidRets2] - error must the last return value
[WARN] [04-07|01:52:32.738] Cannot register RPC callback [invalidRets3] - maximum 2 return values are allowed, got 3
[WARN] [04-07|01:52:32.740] [rpc] served method=unknown_method reqid=2 err="the method unknown_method does not exist/is not available"
[WARN] [04-07|01:52:32.741] Cannot register RPC callback [invalidRets1] - error must the last return value
[WARN] [04-07|01:52:32.741] Cannot register RPC callback [invalidRets2] - error must the last return value
[WARN] [04-07|01:52:32.741] Cannot register RPC callback [invalidRets3] - maximum 2 return values are allowed, got 3
[WARN] [04-07|01:52:32.743] Cannot register RPC callback [invalidRets1] - error must the last return value
[WARN] [04-07|01:52:32.743] Cannot register RPC callback [invalidRets2] - error must the last return value
[WARN] [04-07|01:52:32.743] Cannot register RPC callback [invalidRets3] - maximum 2 return values are allowed, got 3
[WARN] [04-07|01:52:32.744] Cannot register RPC callback [invalidRets1] - error must the last return value
[WARN] [04-07|01:52:32.744] Cannot register RPC callback [invalidRets2] - error must the last return value
[WARN] [04-07|01:52:32.744] Cannot register RPC callback [invalidRets3] - maximum 2 return values are allowed, got 3
[WARN] [04-07|01:52:32.745] Cannot register RPC callback [invalidRets1] - error must the last return value
[WARN] [04-07|01:52:32.745] Cannot register RPC callback [invalidRets2] - error must the last return value
[WARN] [04-07|01:52:32.745] Cannot register RPC callback [invalidRets3] - maximum 2 return values are allowed, got 3
[WARN] [04-07|01:52:32.746] [rpc] served method=test_echo reqid=2 err="missing value for required argument 0"
[WARN] [04-07|01:52:32.746] Cannot register RPC callback [invalidRets1] - error must the last return value
[WARN] [04-07|01:52:32.747] Cannot register RPC callback [invalidRets2] - error must the last return value
[WARN] [04-07|01:52:32.747] Cannot register RPC callback [invalidRets3] - maximum 2 return values are allowed, got 3
[WARN] [04-07|01:52:32.748] [rpc] served method=test_echo reqid=2 err="missing value for required argument 0"
[WARN] [04-07|01:52:32.748] [rpc] served method=test_echo reqid=2 err="missing value for required argument 1"
[WARN] [04-07|01:52:32.750] Cannot register RPC callback [invalidRets1] - error must the last return value
[WARN] [04-07|01:52:32.750] Cannot register RPC callback [invalidRets2] - error must the last return value
[WARN] [04-07|01:52:32.750] Cannot register RPC callback [invalidRets3] - maximum 2 return values are allowed, got 3
[WARN] [04-07|01:52:32.751] [rpc] served method=test_echo reqid=3 err="non-array args"
[WARN] [04-07|01:52:32.751] Cannot register RPC callback [invalidRets1] - error must the last return value
[WARN] [04-07|01:52:32.751] Cannot register RPC callback [invalidRets2] - error must the last return value
[WARN] [04-07|01:52:32.751] Cannot register RPC callback [invalidRets3] - maximum 2 return values are allowed, got 3
[WARN] [04-07|01:52:32.752] Cannot register RPC callback [invalidRets1] - error must the last return value
[WARN] [04-07|01:52:32.752] Cannot register RPC callback [invalidRets2] - error must the last return value
[WARN] [04-07|01:52:32.752] Cannot register RPC callback [invalidRets3] - maximum 2 return values are allowed, got 3
[WARN] [04-07|01:52:32.753] [rpc] served method=invalid_method reqid=2 err="the method invalid_method does not exist/is not available"
[WARN] [04-07|01:52:32.754] Cannot register RPC callback [invalidRets1] - error must the last return value
[WARN] [04-07|01:52:32.754] Cannot register RPC callback [invalidRets2] - error must the last return value
[WARN] [04-07|01:52:32.754] Cannot register RPC callback [invalidRets3] - maximum 2 return values are allowed, got 3
[WARN] [04-07|01:52:32.754] Cannot register RPC callback [invalidRets1] - error must the last return value
[WARN] [04-07|01:52:32.754] Cannot register RPC callback [invalidRets2] - error must the last return value
[WARN] [04-07|01:52:32.755] Cannot register RPC callback [invalidRets3] - maximum 2 return values are allowed, got 3
[WARN] [04-07|01:52:32.755] Cannot register RPC callback [invalidRets1] - error must the last return value
[WARN] [04-07|01:52:32.755] Cannot register RPC callback [invalidRets2] - error must the last return value
[WARN] [04-07|01:52:32.755] Cannot register RPC callback [invalidRets3] - maximum 2 return values are allowed, got 3
[WARN] [04-07|01:52:32.757] Cannot register RPC callback [invalidRets1] - error must the last return value
[WARN] [04-07|01:52:32.757] Cannot register RPC callback [invalidRets2] - error must the last return value
[WARN] [04-07|01:52:32.757] Cannot register RPC callback [invalidRets3] - maximum 2 return values are allowed, got 3
[WARN] [04-07|01:52:32.758] Cannot register RPC callback [invalidRets1] - error must the last return value
[WARN] [04-07|01:52:32.758] Cannot register RPC callback [invalidRets2] - error must the last return value
[WARN] [04-07|01:52:32.758] Cannot register RPC callback [invalidRets3] - maximum 2 return values are allowed, got 3
[WARN] [04-07|01:52:32.760] Cannot register RPC callback [invalidRets1] - error must the last return value
[WARN] [04-07|01:52:32.760] Cannot register RPC callback [invalidRets2] - error must the last return value
[WARN] [04-07|01:52:32.760] Cannot register RPC callback [invalidRets3] - maximum 2 return values are allowed, got 3
[WARN] [04-07|01:52:32.771] Cannot register RPC callback [invalidRets1] - error must the last return value
[WARN] [04-07|01:52:32.771] Cannot register RPC callback [invalidRets2] - error must the last return value
[WARN] [04-07|01:52:32.771] Cannot register RPC callback [invalidRets3] - maximum 2 return values are allowed, got 3
[WARN] [04-07|01:52:44.363] Cannot register RPC callback [invalidRets1] - error must the last return value
[WARN] [04-07|01:52:44.363] Cannot register RPC callback [invalidRets2] - error must the last return value
[WARN] [04-07|01:52:44.363] Cannot register RPC callback [invalidRets1] - error must the last return value
[WARN] [04-07|01:52:44.363] Cannot register RPC callback [invalidRets3] - maximum 2 return values are allowed, got 3
[WARN] [04-07|01:52:44.363] Cannot register RPC callback [invalidRets2] - error must the last return value
[WARN] [04-07|01:52:44.363] Cannot register RPC callback [invalidRets3] - maximum 2 return values are allowed, got 3
[WARN] [04-07|01:52:44.363] Cannot register RPC callback [invalidRets1] - error must the last return value
[WARN] [04-07|01:52:44.363] Cannot register RPC callback [invalidRets2] - error must the last return value
[WARN] [04-07|01:52:44.363] Cannot register RPC callback [invalidRets3] - maximum 2 return values are allowed, got 3
--- FAIL: TestClientCancelWebsocket (1.00s)
panic: read tcp 127.0.0.1:59236->127.0.0.1:33899: read: connection reset by peer [recovered, repanicked]
goroutine 66 [running]:
testing.tRunner.func1.2({0x117cd40, 0xc000059720})
/opt/hostedtoolcache/go/1.25.8/x64/src/testing/testing.go:1872 +0x419
testing.tRunner.func1()
/opt/hostedtoolcache/go/1.25.8/x64/src/testing/testing.go:1875 +0x683
panic({0x117cd40?, 0xc000059720?})
/opt/hostedtoolcache/go/1.25.8/x64/src/runtime/panic.go:783 +0x132
github.com/erigontech/erigon/rpc.httpTestClient(0xc000324180, {0x11fcef1, 0x2}, 0xc000059120)
/home/runner/work/erigon/erigon/rpc/client_test.go:619 +0x6b9
github.com/erigontech/erigon/rpc.testClientCancel({0x11fcef1, 0x2}, 0xc000102e00, {0x136dd10, 0xc000300c60})
/home/runner/work/erigon/erigon/rpc/client_test.go:206 +0x256
github.com/erigontech/erigon/rpc.TestClientCancelWebsocket(0xc000102e00)
/home/runner/work/erigon/erigon/rpc/client_test.go:166 +0x58
testing.tRunner(0xc000102e00, 0x1241000)
/opt/hostedtoolcache/go/1.25.8/x64/src/testing/testing.go:1934 +0x21d
created by testing.(*T).Run in goroutine 1
/opt/hostedtoolcache/go/1.25.8/x64/src/testing/testing.go:1997 +0x9d3
goroutine 1 [chan receive]:
testing.tRunner.func1()
/opt/hostedtoolcache/go/1.25.8/x64/src/testing/testing.go:1891 +0x9ad
testing.tRunner(0xc00037e1c0, 0xc000053ae0)
/opt/hostedtoolcache/go/1.25.8/x64/src/testing/testing.go:1940 +0x256
testing.runTests(0xc00035c3c0, {0x198f6a0, 0x28, 0x28}, {0x0?, 0x4bba45?, 0x1999960?})
/opt/hostedtoolcache/go/1.25.8/x64/src/testing/testing.go:2475 +0x96d
testing.(*M).Run(0xc00037a140)
/opt/hostedtoolcache/go/1.25.8/x64/src/testing/testing.go:2337 +0xed5
main.main()
_testmain.go:129 +0x165
goroutine 37 [chan receive]:
```
…mem history reads (#20245) (#20262) Cherry-pick of #20245 to `release/3.4`. **Conflict resolution**: `execution/execmodule/executor.go` doesn't exist in `release/3.4` (the `PipelineExecutor` abstraction was introduced in `main` after the branch point). The equivalent change — adding the `ERIGON_IN_MEM_HISTORY` env flag and wiring it to `SetInMemHistoryReads` — was applied to `execution/stagedsync/stageloop/stageloop.go` which is the relevant call site in `release/3.4`.
4739a96 to
9431a69
Compare
yperbasis
left a comment
There was a problem hiding this comment.
🔴 Regressions — should block merge
- agents.md — 43 lines of project guidance deleted
The diff shows only deletions (0 +, 43 -). What's being removed exists on current main (verified git show origin/main:agents.md):
- Conventions (commit message prefix convention)
- Pull Requests & Workflows (workflow-dispatch commenting rule)
- Pre-push / Lint Notes (ruleguard/prealloc/unslice/newDeref/etc. recipes)
- Workflows (cross-platform shell rule)
- Go Test Caching (git restore-mtime pattern guidance — tied to .github/actions/setup-erigon/action.yml)
These are not stale — they reference infra that's still live. Since CLAUDE.md is a symlink to agents.md, this silently removes the content AI agents and humans rely on. Origin is the
#20517 cherry-pick commit 6bf62ea: on main (339133d) the same PR kept those sections; on this branch the rebase dropped them.
Fix: restore the deleted sections before merge.
- rpc/client_test.go — defer client.Close() silently removed
- defer server.Stop()
defer hs.Close() - defer client.Close() // ← gone
- defer server.Stop()
Effect on current main:
- Before: client.Close(); hs.Close(); server.Stop() (LIFO)
- After: server.Stop(); hs.Close() — client is no longer explicitly closed, and server stops before hs.
PR #20368 has already been cherry-picked via batch #20405 (commit 8ceaedc on main adds testClientCancelSynctest with all three defers). Cherry-picking it again from a different
base yields this divergent result.
Fix: drop the rpc/client_test.go hunk from this PR — it's a no-op goal with negative side effects.
🟡 Worth questioning
common/estimate/esitmated_ram.go — IndexSnapshot 4GB → 1GB
//elias-fano index building is single-threaded
// when set it to 3GB - observed OOM-kil at server with 128Gb ram and 32CPU
IndexSnapshot = EstimatedRamPerWorker(1 * datasize.GB)
Workers() = min(CPUs, RAM/2 / estimate) — lower estimate means more concurrent workers. The existing comment says 3 GB already caused OOM; reducing to 1 GB would allow ~3× more
workers on the same box, which naïvely increases OOM risk. Either the comment is stale and should be updated, or the change is wrong. Please clarify in the PR what changed about
elias-fano memory use (e.g. parallel-building now streams, so per-worker RAM dropped).
Cherry-pick hygiene
Many listed PRs (#20262, #20571, #20576, #20518, #20329, #20368) are already on main via earlier batches. The diff matches that — most show as tiny residuals (e.g.
execution/execmodule/exec_module.go is just a blank-line removal). Including them in the stated scope is confusing and, as shown above, can silently overwrite follow-up fixes.
Consider narrowing the description to the PRs that actually produce net changes (primarily #20635, #20565, #20639, #20576) and dropping the duplicates.
Comment tightening in db/integrity/commitment_integrity.go:1011
// Each goroutine needs its own Sampler — the RNG is not goroutine-safe.
sampler := sc.NewSampler()
is replaced with
// Each goroutine needs its own Sampler
sampler := sc.NewWindowSampler(windowStart)
The why ("RNG not goroutine-safe") was the valuable part. Keep it.
Summary
┌───────────────────────┬───────────────────────────────────────────────────────────────────────────────────────────────────┐
│ │ │
├───────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Must fix before merge │ agents.md restoration; drop rpc/client_test.go hunk │
├───────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Clarify │ IndexSnapshot 4→1 GB rationale │
├───────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Nit │ restore "RNG not goroutine-safe" comment; trim scope list in description │
├───────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Land as-is │ MaxStep fix, aggregator presets, NewWindowSampler, SaveHeapProfile relocation, unwind-log cleanup │
└───────────────────────┴───────────────────────────────────────────────────────────────────────────────────────────────────┘
The MaxStep fix alone justifies getting a corrected version of this merged quickly; I'd suggest splitting it out if addressing the regressions will delay this PR.
…shot comment, restore RNG comment
AskAlexSharov
left a comment
There was a problem hiding this comment.
Addressed all points in ab0ab68:
- agents.md: restored all 43 deleted lines (Conventions, Pull Requests & Workflows, Pre-push, Lint Notes, Workflows, Go Test Caching)
- rpc/client_test.go: restored
defer client.Close()and original LIFO order (server.Stop→hs.Close→client.Close) - common/estimate/esitmated_ram.go: updated comment to say "4GB" (the actual value that caused OOM), keeping
IndexSnapshot = 1 * datasize.GB - db/integrity/commitment_integrity.go: restored "— the RNG is not goroutine-safe." to the Sampler comment
Cherry-pick from
release/3.4tomain:dbg.SaveHeapProfileNearOOMPeriodically()call in all sub-commands #20639 dbg.SaveHeapProfileNearOOMPeriodically() call in all sub-commandsmerge workersto 1. because all components already support own parallel-bulding #20565 merge: set merge workers to 1Unwindbeyond data in snapshots not allowed #20517 teach: Unwind beyond data in snapshots not allowed