Skip to content

cherry-pick: from release/3.4 to main#20658

Merged
AskAlexSharov merged 12 commits into
mainfrom
alex/cp_r34_fixes_35
Apr 20, 2026
Merged

cherry-pick: from release/3.4 to main#20658
AskAlexSharov merged 12 commits into
mainfrom
alex/cp_r34_fixes_35

Conversation

@AskAlexSharov

@AskAlexSharov AskAlexSharov commented Apr 18, 2026

Copy link
Copy Markdown
Collaborator

Cherry-pick from release/3.4 to main:

AskAlexSharov and others added 9 commits April 18, 2026 20:42
Cherry-pick of #20609 to release/3.4

Co-authored-by: bendertherobert <bendertherobert@gmail.com>
```
 [INFO] [04-15|05:15:50.145] [integrity] StateRootVerifyByHistory     blks/s=0.5 checked=3.72k/2.48k windows=1860/2488                 
  blkRange=1-24.87M                                                                                                                     

  ```  
`checked` overflow
…rt own parallel-bulding (#20565)

compression and accessors: support parallel-building means we don't need
multiple `merge_workers` usually

closing: #20560
we need it because we're already doing out-of-order prune of domains. So
need this protection.

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Alex Sharov <AskAlexSharov@gmail.com>
Replace real TCP connections with in-memory net.Pipe-based listener for
the HTTP cancel test, wrapped in synctest.Test for deterministic
fake-time execution. The test now runs in ~0s instead of timing out on
slow CI runners.

WebSocket cancel test keeps real TCP due to complex goroutine shutdown
dependencies incompatible with synctest's bubble requirements.

err:
```
[WARN] [04-07|01:51:47.496] Cannot register RPC callback [invalidRets1] - error must the last return value
[WARN] [04-07|01:51:47.496] Cannot register RPC callback [invalidRets2] - error must the last return value
[WARN] [04-07|01:51:47.497] Cannot register RPC callback [invalidRets3] - maximum 2 return values are allowed, got 3
[WARN] [04-07|01:51:47.507] Cannot register RPC callback [invalidRets1] - error must the last return value
[WARN] [04-07|01:51:47.507] Cannot register RPC callback [invalidRets2] - error must the last return value
[WARN] [04-07|01:51:47.507] Cannot register RPC callback [invalidRets3] - maximum 2 return values are allowed, got 3
[WARN] [04-07|01:51:47.510] Cannot register RPC callback [invalidRets1] - error must the last return value
[WARN] [04-07|01:51:47.510] Cannot register RPC callback [invalidRets2] - error must the last return value
[WARN] [04-07|01:51:47.510] Cannot register RPC callback [invalidRets3] - maximum 2 return values are allowed, got 3
[WARN] [04-07|01:51:47.512] [rpc] served                             method=test_returnError reqid=1 err=testError errdata="testError data"
[WARN] [04-07|01:51:47.513] Cannot register RPC callback [invalidRets1] - error must the last return value
[WARN] [04-07|01:51:47.513] Cannot register RPC callback [invalidRets2] - error must the last return value
[WARN] [04-07|01:51:47.513] Cannot register RPC callback [invalidRets3] - maximum 2 return values are allowed, got 3
[WARN] [04-07|01:51:47.515] [rpc] served                             method=no_such_method reqid=3 err="the method no_such_method does not exist/is not available"
[WARN] [04-07|01:51:47.516] Cannot register RPC callback [invalidRets1] - error must the last return value
[WARN] [04-07|01:51:47.516] Cannot register RPC callback [invalidRets2] - error must the last return value
[WARN] [04-07|01:51:47.516] Cannot register RPC callback [invalidRets3] - maximum 2 return values are allowed, got 3
[WARN] [04-07|01:51:47.517] Cannot register RPC callback [invalidRets1] - error must the last return value
[WARN] [04-07|01:51:47.517] Cannot register RPC callback [invalidRets2] - error must the last return value
[WARN] [04-07|01:51:47.518] Cannot register RPC callback [invalidRets3] - maximum 2 return values are allowed, got 3
[WARN] [04-07|01:51:47.519] [rpc] served                             method=eth_subscribe reqid=1 err="no \"foo_bar\" subscription in eth namespace"
[WARN] [04-07|01:51:47.519] [rpc] served                             method=eth_subscribe reqid=2 err="no \"foo_bar\" subscription in eth namespace"
[WARN] [04-07|01:51:47.520] Cannot register RPC callback [invalidRets1] - error must the last return value
[WARN] [04-07|01:51:47.520] Cannot register RPC callback [invalidRets2] - error must the last return value
[WARN] [04-07|01:51:47.520] Cannot register RPC callback [invalidRets3] - maximum 2 return values are allowed, got 3
[WARN] [04-07|01:51:47.523] Cannot register RPC callback [invalidRets1] - error must the last return value
[WARN] [04-07|01:51:47.523] Cannot register RPC callback [invalidRets2] - error must the last return value
[WARN] [04-07|01:51:47.524] Cannot register RPC callback [invalidRets3] - maximum 2 return values are allowed, got 3
[WARN] [04-07|01:51:47.527] Cannot register RPC callback [invalidRets1] - error must the last return value
[WARN] [04-07|01:51:47.527] Cannot register RPC callback [invalidRets2] - error must the last return value
[WARN] [04-07|01:51:47.527] Cannot register RPC callback [invalidRets3] - maximum 2 return values are allowed, got 3
[WARN] [04-07|01:51:47.563] Cannot register RPC callback [invalidRets1] - error must the last return value
[WARN] [04-07|01:51:47.563] Cannot register RPC callback [invalidRets2] - error must the last return value
[WARN] [04-07|01:51:47.563] Cannot register RPC callback [invalidRets3] - maximum 2 return values are allowed, got 3
[WARN] [04-07|01:51:56.015] Cannot register RPC callback [invalidRets1] - error must the last return value
[WARN] [04-07|01:51:56.015] Cannot register RPC callback [invalidRets2] - error must the last return value
[WARN] [04-07|01:51:56.015] Cannot register RPC callback [invalidRets3] - maximum 2 return values are allowed, got 3
[WARN] [04-07|01:51:56.019] Cannot register RPC callback [invalidRets1] - error must the last return value
[WARN] [04-07|01:51:56.019] Cannot register RPC callback [invalidRets2] - error must the last return value
[WARN] [04-07|01:51:56.019] Cannot register RPC callback [invalidRets3] - maximum 2 return values are allowed, got 3
[WARN] [04-07|01:51:56.103] Cannot register RPC callback [invalidRets1] - error must the last return value
[WARN] [04-07|01:51:56.103] Cannot register RPC callback [invalidRets2] - error must the last return value
[WARN] [04-07|01:51:56.103] Cannot register RPC callback [invalidRets3] - maximum 2 return values are allowed, got 3
[WARN] [04-07|01:51:58.107] Cannot register RPC callback [invalidRets1] - error must the last return value
[WARN] [04-07|01:51:58.107] Cannot register RPC callback [invalidRets2] - error must the last return value
[WARN] [04-07|01:51:58.107] Cannot register RPC callback [invalidRets3] - maximum 2 return values are allowed, got 3
[WARN] [04-07|01:52:32.735] Cannot register RPC callback [invalidRets1] - error must the last return value
[WARN] [04-07|01:52:32.735] Cannot register RPC callback [invalidRets2] - error must the last return value
[WARN] [04-07|01:52:32.735] Cannot register RPC callback [invalidRets3] - maximum 2 return values are allowed, got 3
[WARN] [04-07|01:52:32.737] Cannot register RPC callback [invalidRets1] - error must the last return value
[WARN] [04-07|01:52:32.737] Cannot register RPC callback [invalidRets2] - error must the last return value
[WARN] [04-07|01:52:32.737] Cannot register RPC callback [invalidRets3] - maximum 2 return values are allowed, got 3
[WARN] [04-07|01:52:32.737] Cannot register RPC callback [invalidRets1] - error must the last return value
[WARN] [04-07|01:52:32.737] Cannot register RPC callback [invalidRets2] - error must the last return value
[WARN] [04-07|01:52:32.737] Cannot register RPC callback [invalidRets3] - maximum 2 return values are allowed, got 3
[WARN] [04-07|01:52:32.738] Cannot register RPC callback [invalidRets1] - error must the last return value
[WARN] [04-07|01:52:32.738] Cannot register RPC callback [invalidRets2] - error must the last return value
[WARN] [04-07|01:52:32.738] Cannot register RPC callback [invalidRets3] - maximum 2 return values are allowed, got 3
[WARN] [04-07|01:52:32.740] [rpc] served                             method=unknown_method reqid=2 err="the method unknown_method does not exist/is not available"
[WARN] [04-07|01:52:32.741] Cannot register RPC callback [invalidRets1] - error must the last return value
[WARN] [04-07|01:52:32.741] Cannot register RPC callback [invalidRets2] - error must the last return value
[WARN] [04-07|01:52:32.741] Cannot register RPC callback [invalidRets3] - maximum 2 return values are allowed, got 3
[WARN] [04-07|01:52:32.743] Cannot register RPC callback [invalidRets1] - error must the last return value
[WARN] [04-07|01:52:32.743] Cannot register RPC callback [invalidRets2] - error must the last return value
[WARN] [04-07|01:52:32.743] Cannot register RPC callback [invalidRets3] - maximum 2 return values are allowed, got 3
[WARN] [04-07|01:52:32.744] Cannot register RPC callback [invalidRets1] - error must the last return value
[WARN] [04-07|01:52:32.744] Cannot register RPC callback [invalidRets2] - error must the last return value
[WARN] [04-07|01:52:32.744] Cannot register RPC callback [invalidRets3] - maximum 2 return values are allowed, got 3
[WARN] [04-07|01:52:32.745] Cannot register RPC callback [invalidRets1] - error must the last return value
[WARN] [04-07|01:52:32.745] Cannot register RPC callback [invalidRets2] - error must the last return value
[WARN] [04-07|01:52:32.745] Cannot register RPC callback [invalidRets3] - maximum 2 return values are allowed, got 3
[WARN] [04-07|01:52:32.746] [rpc] served                             method=test_echo reqid=2 err="missing value for required argument 0"
[WARN] [04-07|01:52:32.746] Cannot register RPC callback [invalidRets1] - error must the last return value
[WARN] [04-07|01:52:32.747] Cannot register RPC callback [invalidRets2] - error must the last return value
[WARN] [04-07|01:52:32.747] Cannot register RPC callback [invalidRets3] - maximum 2 return values are allowed, got 3
[WARN] [04-07|01:52:32.748] [rpc] served                             method=test_echo reqid=2 err="missing value for required argument 0"
[WARN] [04-07|01:52:32.748] [rpc] served                             method=test_echo reqid=2 err="missing value for required argument 1"
[WARN] [04-07|01:52:32.750] Cannot register RPC callback [invalidRets1] - error must the last return value
[WARN] [04-07|01:52:32.750] Cannot register RPC callback [invalidRets2] - error must the last return value
[WARN] [04-07|01:52:32.750] Cannot register RPC callback [invalidRets3] - maximum 2 return values are allowed, got 3
[WARN] [04-07|01:52:32.751] [rpc] served                             method=test_echo reqid=3 err="non-array args"
[WARN] [04-07|01:52:32.751] Cannot register RPC callback [invalidRets1] - error must the last return value
[WARN] [04-07|01:52:32.751] Cannot register RPC callback [invalidRets2] - error must the last return value
[WARN] [04-07|01:52:32.751] Cannot register RPC callback [invalidRets3] - maximum 2 return values are allowed, got 3
[WARN] [04-07|01:52:32.752] Cannot register RPC callback [invalidRets1] - error must the last return value
[WARN] [04-07|01:52:32.752] Cannot register RPC callback [invalidRets2] - error must the last return value
[WARN] [04-07|01:52:32.752] Cannot register RPC callback [invalidRets3] - maximum 2 return values are allowed, got 3
[WARN] [04-07|01:52:32.753] [rpc] served                             method=invalid_method reqid=2 err="the method invalid_method does not exist/is not available"
[WARN] [04-07|01:52:32.754] Cannot register RPC callback [invalidRets1] - error must the last return value
[WARN] [04-07|01:52:32.754] Cannot register RPC callback [invalidRets2] - error must the last return value
[WARN] [04-07|01:52:32.754] Cannot register RPC callback [invalidRets3] - maximum 2 return values are allowed, got 3
[WARN] [04-07|01:52:32.754] Cannot register RPC callback [invalidRets1] - error must the last return value
[WARN] [04-07|01:52:32.754] Cannot register RPC callback [invalidRets2] - error must the last return value
[WARN] [04-07|01:52:32.755] Cannot register RPC callback [invalidRets3] - maximum 2 return values are allowed, got 3
[WARN] [04-07|01:52:32.755] Cannot register RPC callback [invalidRets1] - error must the last return value
[WARN] [04-07|01:52:32.755] Cannot register RPC callback [invalidRets2] - error must the last return value
[WARN] [04-07|01:52:32.755] Cannot register RPC callback [invalidRets3] - maximum 2 return values are allowed, got 3
[WARN] [04-07|01:52:32.757] Cannot register RPC callback [invalidRets1] - error must the last return value
[WARN] [04-07|01:52:32.757] Cannot register RPC callback [invalidRets2] - error must the last return value
[WARN] [04-07|01:52:32.757] Cannot register RPC callback [invalidRets3] - maximum 2 return values are allowed, got 3
[WARN] [04-07|01:52:32.758] Cannot register RPC callback [invalidRets1] - error must the last return value
[WARN] [04-07|01:52:32.758] Cannot register RPC callback [invalidRets2] - error must the last return value
[WARN] [04-07|01:52:32.758] Cannot register RPC callback [invalidRets3] - maximum 2 return values are allowed, got 3
[WARN] [04-07|01:52:32.760] Cannot register RPC callback [invalidRets1] - error must the last return value
[WARN] [04-07|01:52:32.760] Cannot register RPC callback [invalidRets2] - error must the last return value
[WARN] [04-07|01:52:32.760] Cannot register RPC callback [invalidRets3] - maximum 2 return values are allowed, got 3
[WARN] [04-07|01:52:32.771] Cannot register RPC callback [invalidRets1] - error must the last return value
[WARN] [04-07|01:52:32.771] Cannot register RPC callback [invalidRets2] - error must the last return value
[WARN] [04-07|01:52:32.771] Cannot register RPC callback [invalidRets3] - maximum 2 return values are allowed, got 3
[WARN] [04-07|01:52:44.363] Cannot register RPC callback [invalidRets1] - error must the last return value
[WARN] [04-07|01:52:44.363] Cannot register RPC callback [invalidRets2] - error must the last return value
[WARN] [04-07|01:52:44.363] Cannot register RPC callback [invalidRets1] - error must the last return value
[WARN] [04-07|01:52:44.363] Cannot register RPC callback [invalidRets3] - maximum 2 return values are allowed, got 3
[WARN] [04-07|01:52:44.363] Cannot register RPC callback [invalidRets2] - error must the last return value
[WARN] [04-07|01:52:44.363] Cannot register RPC callback [invalidRets3] - maximum 2 return values are allowed, got 3
[WARN] [04-07|01:52:44.363] Cannot register RPC callback [invalidRets1] - error must the last return value
[WARN] [04-07|01:52:44.363] Cannot register RPC callback [invalidRets2] - error must the last return value
[WARN] [04-07|01:52:44.363] Cannot register RPC callback [invalidRets3] - maximum 2 return values are allowed, got 3
--- FAIL: TestClientCancelWebsocket (1.00s)
panic: read tcp 127.0.0.1:59236->127.0.0.1:33899: read: connection reset by peer [recovered, repanicked]

goroutine 66 [running]:
testing.tRunner.func1.2({0x117cd40, 0xc000059720})
	/opt/hostedtoolcache/go/1.25.8/x64/src/testing/testing.go:1872 +0x419
testing.tRunner.func1()
	/opt/hostedtoolcache/go/1.25.8/x64/src/testing/testing.go:1875 +0x683
panic({0x117cd40?, 0xc000059720?})
	/opt/hostedtoolcache/go/1.25.8/x64/src/runtime/panic.go:783 +0x132
github.com/erigontech/erigon/rpc.httpTestClient(0xc000324180, {0x11fcef1, 0x2}, 0xc000059120)
	/home/runner/work/erigon/erigon/rpc/client_test.go:619 +0x6b9
github.com/erigontech/erigon/rpc.testClientCancel({0x11fcef1, 0x2}, 0xc000102e00, {0x136dd10, 0xc000300c60})
	/home/runner/work/erigon/erigon/rpc/client_test.go:206 +0x256
github.com/erigontech/erigon/rpc.TestClientCancelWebsocket(0xc000102e00)
	/home/runner/work/erigon/erigon/rpc/client_test.go:166 +0x58
testing.tRunner(0xc000102e00, 0x1241000)
	/opt/hostedtoolcache/go/1.25.8/x64/src/testing/testing.go:1934 +0x21d
created by testing.(*T).Run in goroutine 1
	/opt/hostedtoolcache/go/1.25.8/x64/src/testing/testing.go:1997 +0x9d3

goroutine 1 [chan receive]:
testing.tRunner.func1()
	/opt/hostedtoolcache/go/1.25.8/x64/src/testing/testing.go:1891 +0x9ad
testing.tRunner(0xc00037e1c0, 0xc000053ae0)
	/opt/hostedtoolcache/go/1.25.8/x64/src/testing/testing.go:1940 +0x256
testing.runTests(0xc00035c3c0, {0x198f6a0, 0x28, 0x28}, {0x0?, 0x4bba45?, 0x1999960?})
	/opt/hostedtoolcache/go/1.25.8/x64/src/testing/testing.go:2475 +0x96d
testing.(*M).Run(0xc00037a140)
	/opt/hostedtoolcache/go/1.25.8/x64/src/testing/testing.go:2337 +0xed5
main.main()
	_testmain.go:129 +0x165

goroutine 37 [chan receive]:
```
…mem history reads (#20245) (#20262)

Cherry-pick of #20245 to `release/3.4`.

**Conflict resolution**: `execution/execmodule/executor.go` doesn't
exist in `release/3.4` (the `PipelineExecutor` abstraction was
introduced in `main` after the branch point). The equivalent change —
adding the `ERIGON_IN_MEM_HISTORY` env flag and wiring it to
`SetInMemHistoryReads` — was applied to
`execution/stagedsync/stageloop/stageloop.go` which is the relevant call
site in `release/3.4`.
@AskAlexSharov AskAlexSharov enabled auto-merge April 19, 2026 01:09
@AskAlexSharov AskAlexSharov changed the title cherry-pick 10 commits from release/3.4 to main cherry-pick: from release/3.4 to main Apr 19, 2026

@yperbasis yperbasis left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Regressions — should block merge

  1. agents.md — 43 lines of project guidance deleted

The diff shows only deletions (0 +, 43 -). What's being removed exists on current main (verified git show origin/main:agents.md):

  • Conventions (commit message prefix convention)
  • Pull Requests & Workflows (workflow-dispatch commenting rule)
  • Pre-push / Lint Notes (ruleguard/prealloc/unslice/newDeref/etc. recipes)
  • Workflows (cross-platform shell rule)
  • Go Test Caching (git restore-mtime pattern guidance — tied to .github/actions/setup-erigon/action.yml)

These are not stale — they reference infra that's still live. Since CLAUDE.md is a symlink to agents.md, this silently removes the content AI agents and humans rely on. Origin is the
#20517 cherry-pick commit 6bf62ea: on main (339133d) the same PR kept those sections; on this branch the rebase dropped them.

Fix: restore the deleted sections before merge.

  1. rpc/client_test.go — defer client.Close() silently removed
  • defer server.Stop()
    defer hs.Close()
  • defer client.Close() // ← gone
  • defer server.Stop()
    Effect on current main:
  • Before: client.Close(); hs.Close(); server.Stop() (LIFO)
  • After: server.Stop(); hs.Close() — client is no longer explicitly closed, and server stops before hs.

PR #20368 has already been cherry-picked via batch #20405 (commit 8ceaedc on main adds testClientCancelSynctest with all three defers). Cherry-picking it again from a different
base yields this divergent result.

Fix: drop the rpc/client_test.go hunk from this PR — it's a no-op goal with negative side effects.

🟡 Worth questioning

common/estimate/esitmated_ram.go — IndexSnapshot 4GB → 1GB

//elias-fano index building is single-threaded
// when set it to 3GB - observed OOM-kil at server with 128Gb ram and 32CPU
IndexSnapshot = EstimatedRamPerWorker(1 * datasize.GB)
Workers() = min(CPUs, RAM/2 / estimate) — lower estimate means more concurrent workers. The existing comment says 3 GB already caused OOM; reducing to 1 GB would allow ~3× more
workers on the same box, which naïvely increases OOM risk. Either the comment is stale and should be updated, or the change is wrong. Please clarify in the PR what changed about
elias-fano memory use (e.g. parallel-building now streams, so per-worker RAM dropped).

Cherry-pick hygiene

Many listed PRs (#20262, #20571, #20576, #20518, #20329, #20368) are already on main via earlier batches. The diff matches that — most show as tiny residuals (e.g.
execution/execmodule/exec_module.go is just a blank-line removal). Including them in the stated scope is confusing and, as shown above, can silently overwrite follow-up fixes.
Consider narrowing the description to the PRs that actually produce net changes (primarily #20635, #20565, #20639, #20576) and dropping the duplicates.

Comment tightening in db/integrity/commitment_integrity.go:1011

// Each goroutine needs its own Sampler — the RNG is not goroutine-safe.
sampler := sc.NewSampler()
is replaced with
// Each goroutine needs its own Sampler
sampler := sc.NewWindowSampler(windowStart)
The why ("RNG not goroutine-safe") was the valuable part. Keep it.

Summary

┌───────────────────────┬───────────────────────────────────────────────────────────────────────────────────────────────────┐                                                          
│                       │                                                                                                   │
├───────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Must fix before merge │ agents.md restoration; drop rpc/client_test.go hunk                                               │
├───────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Clarify               │ IndexSnapshot 4→1 GB rationale                                                                    │                                                          
├───────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────┤                                                          
│ Nit                   │ restore "RNG not goroutine-safe" comment; trim scope list in description                          │                                                          
├───────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────┤                                                          
│ Land as-is            │ MaxStep fix, aggregator presets, NewWindowSampler, SaveHeapProfile relocation, unwind-log cleanup │
└───────────────────────┴───────────────────────────────────────────────────────────────────────────────────────────────────┘                                                          

The MaxStep fix alone justifies getting a corrected version of this merged quickly; I'd suggest splitting it out if addressing the regressions will delay this PR.

@AskAlexSharov AskAlexSharov left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed all points in ab0ab68:

  • agents.md: restored all 43 deleted lines (Conventions, Pull Requests & Workflows, Pre-push, Lint Notes, Workflows, Go Test Caching)
  • rpc/client_test.go: restored defer client.Close() and original LIFO order (server.Stophs.Closeclient.Close)
  • common/estimate/esitmated_ram.go: updated comment to say "4GB" (the actual value that caused OOM), keeping IndexSnapshot = 1 * datasize.GB
  • db/integrity/commitment_integrity.go: restored "— the RNG is not goroutine-safe." to the Sampler comment

@AskAlexSharov AskAlexSharov added this pull request to the merge queue Apr 20, 2026
Merged via the queue into main with commit 1146652 Apr 20, 2026
36 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants