snapcfg: lazy-parse EmbeddedWebseeds, only parse the chain in use#19722
Merged
Conversation
…ified The webseed.* vars (from erigon-snapshot/webseed) are //go:embed and have never been mutated at runtime — LoadSnapshots() only updates snapshothashes.* vars. Re-parsing the identical embedded bytes after a remote load produced the same map as the package-level init.
Replace eagerly-parsed KnownWebseeds map with KnownWebseedsRaw (raw TOML bytes) and GetKnownWebseeds(chain) that parses on demand. This avoids parsing all 8 chains' webseed TOMLs at init when only 1 is used.
AskAlexSharov
approved these changes
Mar 8, 2026
wmitsuda
added a commit
that referenced
this pull request
Mar 8, 2026
…9722) Follow-up to #19641 — step 2/N towards simplifying TOML reading at startup. ## Summary - **Lazy-parse webseed TOML**: instead of parsing all 8 chains' webseed TOML at init time, store raw bytes in `EmbeddedWebseedsRaw` and parse on demand via `GetEmbeddedWebseeds(chain)` — only the chain actually in use gets parsed. - **Remove no-op re-assignment**: `LoadRemotePreverified` was redundantly re-building the same `KnownWebseeds` map; removed. - **Inline `webseedsParse`**: folded into its sole caller `GetEmbeddedWebseeds`. - **Rename `KnownWebseeds` → `EmbeddedWebseeds`**: clearer naming — `EmbeddedWebseedsRaw` for the raw bytes map, `GetEmbeddedWebseeds()` for parsed access. --- **TODO**: cherry-pick to `release/3.4` after merge.
wmitsuda
added a commit
that referenced
this pull request
Mar 9, 2026
…the chain in use (#19727) Cherry-pick of #19722 (merged to main as a18eb9b) to `release/3.4`. ## Summary - **Lazy-parse webseed TOML**: instead of parsing all 8 chains' webseed TOML at init time, store raw bytes in `EmbeddedWebseedsRaw` and parse on demand via `GetEmbeddedWebseeds(chain)` — only the chain actually in use gets parsed. - **Remove no-op re-assignment**: `LoadRemotePreverified` was redundantly re-building the same `KnownWebseeds` map; removed. - **Inline `webseedsParse`**: folded into its sole caller `GetEmbeddedWebseeds`. - **Rename `KnownWebseeds` → `EmbeddedWebseeds`**: clearer naming — `EmbeddedWebseedsRaw` for the raw bytes map, `GetEmbeddedWebseeds()` for parsed access.
5 tasks
wmitsuda
added a commit
that referenced
this pull request
Mar 11, 2026
## Summary Part 3 of optimizing remote preverified hash loading (after #19641, #19722). - `LoadPreverified` now takes a `chainName` parameter and calls `LoadRemotePreverified` instead of the old bulk variant that fetched all 10 chains - Refactored `webseeds.Verify` to load preverified per-chain inside the iteration loop instead of bulk-loading all chains upfront - Removed unused functions: old bulk `LoadRemotePreverified`, `registry.All`, `registry.ResetRaw`, `GetAllCurrentPreverified` - Renamed `LoadRemotePreverifiedForChain` → `LoadRemotePreverified` since it's now the only variant ## Test plan - [x] Built `erigon` and `downloader` binaries - [x] Tested `erigon seg reset --dry-run` with mainnet and hoodi ephemeral datadirs - [x] Tested `downloader verify_webseeds --chain=chiado --preverified=embedded` to completion - [x] Verified only the requested chain is fetched (confirmed via log output) ## Tasks - [ ] Cherry-pick merge commit to `release/3.4`
wmitsuda
added a commit
that referenced
this pull request
Mar 11, 2026
## Summary Part 3 of optimizing remote preverified hash loading (after #19641, #19722). - `LoadPreverified` now takes a `chainName` parameter and calls `LoadRemotePreverified` instead of the old bulk variant that fetched all 10 chains - Refactored `webseeds.Verify` to load preverified per-chain inside the iteration loop instead of bulk-loading all chains upfront - Removed unused functions: old bulk `LoadRemotePreverified`, `registry.All`, `registry.ResetRaw`, `GetAllCurrentPreverified` - Renamed `LoadRemotePreverifiedForChain` → `LoadRemotePreverified` since it's now the only variant ## Test plan - [x] Built `erigon` and `downloader` binaries - [x] Tested `erigon seg reset --dry-run` with mainnet and hoodi ephemeral datadirs - [x] Tested `downloader verify_webseeds --chain=chiado --preverified=embedded` to completion - [x] Verified only the requested chain is fetched (confirmed via log output) ## Tasks - [ ] Cherry-pick merge commit to `release/3.4`
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #19641 — step 2/N towards simplifying TOML reading at startup.
Summary
EmbeddedWebseedsRawand parse on demand viaGetEmbeddedWebseeds(chain)— only the chain actually in use gets parsed.LoadRemotePreverifiedwas redundantly re-building the sameKnownWebseedsmap; removed.webseedsParse: folded into its sole callerGetEmbeddedWebseeds.KnownWebseeds→EmbeddedWebseeds: clearer naming —EmbeddedWebseedsRawfor the raw bytes map,GetEmbeddedWebseeds()for parsed access.TODO: cherry-pick toDone: #19727release/3.4after merge.