Skip to content

db/snapcfg, cmd/utils/app: drop github.com/erigontech/erigon-snapshot dep#21197

Merged
wmitsuda merged 2 commits into
mainfrom
wmitsuda/drop-erigon-snapshot-gomod
May 29, 2026
Merged

db/snapcfg, cmd/utils/app: drop github.com/erigontech/erigon-snapshot dep#21197
wmitsuda merged 2 commits into
mainfrom
wmitsuda/drop-erigon-snapshot-gomod

Conversation

@wmitsuda

@wmitsuda wmitsuda commented May 14, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Fixes #21154. Fixes #19732. Sub-task of #21047.

Summary

Runtime fetch source (raw.githubusercontent.com/erigontech/erigon-snapshot + R2 mirror) and fail-fast behaviour are unchanged. Binary size: −2,973,120 bytes uncompressed (−2.0%) / −1,015,102 bytes gzipped (−1.6%) on darwin/arm64, measured by building before/after a stubbed-empty erigon-snapshot.

Test plan

  • make lint && make erigon integration clean
  • go test ./db/snapcfg/... ./db/downloader/downloadercfg/... pass
  • Manual: --chain=hoodi with both CDN hosts unreachable (HTTPS_PROXY pointed at a dead local port) exits non-zero with the same fail-fast [CRIT] Snapshot hashes for supported networks was not loaded … startup trace
  • Manual: --chain=hoodi with normal network logs Loading remote snapshot hashes chain=hoodi, no Failed to load warning, no [CRIT], and progresses into the downloader (segments begin downloading)
  • Manual: --chain=hoodi against a fresh datadir prepopulated with a real <datadir>/snapshots/preverified.toml (fetched out-of-band) starts cleanly with HTTPS_PROXY pointed at a dead local port — no Loading remote snapshot hashes log line, no [CRIT], downloader brings up — confirming the local-file path bypasses remote fetch
  • Manual: erigon snapshots reset --datadir=<dd> --preverified=embedded --dry-run exits 1 with Error: invalid preverified flag value "embedded"; the --help output shows (remote, local); --preverified=remote and --preverified=local continue to work

@wmitsuda wmitsuda enabled auto-merge May 14, 2026 17:22
@wmitsuda wmitsuda mentioned this pull request May 14, 2026
Open
@yperbasis

yperbasis commented May 27, 2026

Copy link
Copy Markdown
Member

Merged latest main in (c7e153a). The merge itself is clean — only go.mod conflicted: I dropped erigon-snapshot, kept the deps main added since (evmone_precompiles, fastkeccak) and the mdbx-go v0.40.1 bump. go mod tidy is a no-op, make lint and make erigon integration pass.

However, CI is red on race-tests / core-rpc due to pre-existing failures in db/snapshotsync:

  • TestMergeSnapshots — should have 3 item(s), but has 7
  • TestFindMergeRange/big, /small
  • TestBlackListForPruning, _BlocksModeKeepsAllTransactions, _ChainHistoryExpiry

TestMergeSnapshots already failed on the pre-merge branch tip (4e1348722) with the same error, so it's not introduced by the merge — the merge just additionally pulled in main's tightened TestFindMergeRange / TestBlackListForPruning* (from #21342), which fail for the same root cause.

Root cause

This PR makes registry.raw empty at init and loads preverified hashes lazily via LoadRemotePreverified. But the db/snapshotsync merge path needs them synchronously:

NewMerger → snapcfg.KnownCfgOrDevnet(chain) → Cfg.MergeLimit() scans PreverifiedParsed to tell the 500k "old merge" regime apart from the 100k one. Unit tests never call LoadRemotePreverified (and have no network), so PreverifiedParsed is empty, MergeLimit always falls through to Erigon2MergeLimit (100k), and merges up to 500k never happen — producing exactly these failures.

The Get comment assumes callers in the not-yet-loaded window only consult the known boolean, but MergeLimit reads the items themselves, so that assumption doesn't hold for the merger path. Production likely avoids it (the downloader calls LoadRemotePreverified at startup), but the heuristic silently degrades to 100k whenever preverified data isn't loaded (offline / tests).

Options

  • Re-embed the per-chain TOMLs via //go:embed and eager-load at init — drops the external module dep (the PR's goal) while keeping the data synchronously available, fixing MergeLimit everywhere; or
  • Feed the affected db/snapshotsync tests their preverified data via a committed local fixture.

@yperbasis yperbasis added this to the 3.6.0 milestone May 27, 2026
@wmitsuda wmitsuda disabled auto-merge May 29, 2026 01:06
@wmitsuda
db/snapshotsync: vendor full mainnet preverified set as test fixture
24e32c0 
Replaces the minimal 5-entry fixture with the full mainnet.toml copied
from erigon-snapshot v1.3.1-0.20260402120223-7bb412bc89cd — the version
pinned on main when the Go-module dep was dropped. Needed because new
tests added on main (TestBlackListForPruning* from #21342) consult
snapcfg.KnownCfg(mainnet) and assert against state-history step ranges
and 20M+ block tx segments that the minimal fixture doesn't cover.

Updates TestMain doc to explain the prior implicit coupling to the
embedded module and why we froze a copy.
@wmitsuda wmitsuda force-pushed the wmitsuda/drop-erigon-snapshot-gomod branch from c7e153a to 24e32c0 Compare May 29, 2026 01:13
@wmitsuda wmitsuda added this pull request to the merge queue May 29, 2026
@AskAlexSharov

AskAlexSharov commented May 29, 2026

Copy link
Copy Markdown
Collaborator

monorepoooo 🚀

Merged via the queue into main with commit f1ebd67 May 29, 2026
90 checks passed
@wmitsuda wmitsuda deleted the wmitsuda/drop-erigon-snapshot-gomod branch May 29, 2026 02:35
@yperbasis yperbasis modified the milestones: 3.6.0, 3.5.0 May 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@AskAlexSharov AskAlexSharov AskAlexSharov approved these changes

@sudeepdino008 sudeepdino008 Awaiting requested review from sudeepdino008 sudeepdino008 is a code owner

@yperbasis yperbasis Awaiting requested review from yperbasis yperbasis is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

3.5.0

Development

Successfully merging this pull request may close these issues.

db/snapcfg: drop the github.com/erigontech/erigon-snapshot Go-module dependency erigon-snapshot embedded hashes are dead code — remove them

3 participants

@wmitsuda @yperbasis @AskAlexSharov