db/snapcfg, cmd/utils, node: add --snap.chaintoml-url override#21584
Merged
Conversation
Add a --snap.chaintoml-url flag to fetch the preverified chain.toml directly from an arbitrary URL instead of the default R2/GitHub CDN. Precedence: ERIGON_REMOTE_PREVERIFIED env (local file) > --snap.chaintoml-url > default CDN fetch. A local preverified.toml in the datadir still takes precedence; delete it to re-fetch from the URL. The URL override has no fallback: a failed fetch returns the error and leaves the registry untouched.
AskAlexSharov
approved these changes
Jun 6, 2026
AskAlexSharov
left a comment
Collaborator
There was a problem hiding this comment.
looks like "Abstraction leak":
- User usually enough to set "Webseed url" (with all what we want inside: chain.toml, manifest.txt, .torrent, files).
- Where do we store our list of canonical hashes - it's technical detail (we can change it from chain.toml to alex.yml, or to v2-chain.toml, or to 10 files).
but oke
taratorio
added a commit
that referenced
this pull request
Jun 8, 2026
…r branch
SonarCloud rejects a branch analysis dated older than the branch's
latest processed one. All merge-queue entries analyze the target branch,
so when entry N's report uploaded after entry N+1's, SonarCloud rejected
it and flipped the commit's SonarCloud check to cancelled ("Date of
analysis cannot be older than the date of the last known analysis").
Seen on 5027d25, where the scans of three stacked queue entries
(#21648, #21584, #21640) raced and #21648's analysis lost.
Restructure so branch analyses can never be submitted out of order:
- merge_group runs keep `make test-sonar-coverage` as a gate check but
no longer scan; they upload coverage-test-all.out as a short-retention
artifact instead.
- A new push-triggered SonarCloud Branch Scan workflow (main and
release/**, matching the branches SonarCloud tracks) downloads that
artifact — the merge queue fast-forwards the target branch to the
already-tested merge commit, so the pushed SHA equals the merge-queue
run's head SHA and the artifact maps to the commit exactly — and runs
only the scanner (~7 min instead of ~25). If no artifact exists
(direct push, expired retention), it falls back to running the tests.
- Scans are serialized per branch FIFO via `concurrency.queue: max`
(GitHub Actions, 2026-05), so every merged commit is analyzed, in
order, with no runs cancelled in merge bursts.
- PR scans (Sonar PR analyses, no main-branch date ordering) and
cache-warming runs are unchanged.
actionlint (≤ v1.7.12) does not know the `queue` key yet, so lint.yml
gains a narrow --ignore alongside the existing workspace one.
A failed analysis upload also no longer involves the gate: scanner
failures now surface on a post-merge run instead, while PR runs keep
exercising the scanner so breakage is still caught before merge.
Sahil-4555
pushed a commit
to Sahil-4555/erigon
that referenced
this pull request
Jun 8, 2026
…r branch (erigontech#21669) ## Problem SonarCloud rejects a branch analysis dated older than the branch's latest processed one. Every merge-queue entry analyzes the target branch, so when queue entry N's report uploads *after* entry N+1's, SonarCloud rejects N's with *"Date of analysis cannot be older than the date of the last known analysis"* and flips that commit's **SonarCloud Code Analysis** check to `cancelled`. Seen on [`5027d259c2`](erigontech@5027d25): three stacked queue entries (erigontech#21648, erigontech#21584, erigontech#21640) scanned concurrently; erigontech#21584 started 6s later but uploaded 21s earlier, so erigontech#21648's analysis was rejected as out of order. No impact on the gate (the scan step is `continue-on-error` and the merge succeeded), but it leaves a spurious failed check on a merged commit. ## Fix Make it structurally impossible to submit branch analyses out of order: - **`merge_group` runs** keep `make test-sonar-coverage` as a gate check but **no longer scan**. They upload `coverage-test-all.out` as a 7-day artifact. - **A new `SonarCloud Branch Scan` workflow** (`push` on `main` + `release/**`, matching the branches SonarCloud tracks) downloads that artifact and runs **only the scanner** (~7 min vs ~25). The merge queue fast-forwards the target branch to the already-tested merge commit, so the pushed SHA equals the merge-queue run's head SHA and the artifact maps to the commit exactly. If no artifact exists (direct push, or expired retention), it falls back to running the tests itself. - Branch scans are serialized per branch **FIFO** via `concurrency.queue: max` ([GitHub Actions, 2026-05](https://github.blog/changelog/2026-05-07-github-actions-concurrency-groups-now-allow-larger-queues/)), so every merged commit is analyzed, in order, with **no runs cancelled** during merge bursts. - **PR scans** (Sonar PR analyses — separate from main-branch date ordering) and **cache-warming** runs are unchanged. A failed analysis upload no longer touches the gate at all: scanner failures now surface on a post-merge run, while PR runs keep exercising the scanner so config/scanner breakage is still caught before merge. ## Notes - `actionlint` (≤ v1.7.12, the latest release) does not know the `queue` concurrency key yet, so `lint.yml` gets a narrow `--ignore` alongside the existing `workspace` one. Removable once actionlint learns the key. - `concurrency.queue: max` is ~1 month old — first real merge-burst after this lands is where FIFO ordering gets exercised live. ## Testing - `make lint`, `actionlint` (CI flags), and `zizmor` (repo config) all clean; the new workflow is finding-free. - The artifact-lookup script was shellchecked and run against live API data: correctly resolves the merge-queue run by head SHA, filters expired artifacts, and degrades to the test-running fallback on a missing artifact or total API failure (never fails the job).
bloxster
added a commit
that referenced
this pull request
Jun 12, 2026
…21687) Addresses code→docs gaps found in weekly maintenance run w24, plus some Fundamentals housekeeping. **Scope: release/3.5 only.** ### Flags & env vars (`configuring-erigon`) - Add `--snap.chaintoml-url` flag (PR #21584), including `ERIGON_REMOTE_PREVERIFIED` override precedence - Add `--snap.p2p-manifest` flag (PR #20526), tagged *(New in v3.5)* - Update `EXEC3_PARALLEL` default to `true` (PR #21591) - Add `--exec.serial` and `--exec.no-prune` flags (new in v3.5); also document `--exec.batched-io` and `--exec.state-cache` - Document `--exec.workers`. Its effective default is the **full CPU-core count** (inherited from the `EXEC3_WORKERS` fallback when the flag is unset); the flag's own `--help` text says "half the CPU cores", which is a known inconsistency in the binary ### Prune modes / snapshots - Update `pruning-modes.md` with the EIP-8252 retention-window breaking change (v3.5): full mode now prunes block bodies/receipts to the last 262,144 blocks (previously kept all post-merge blocks) and will stop serving older block/receipt data; state-history window grows 100k→262k - Add new "Snapshots Management" page under Fundamentals (`seg du`, snapshot categories, node-type estimates, EIP-8252 retention window). The `seg du` example uses an archive datadir, since the estimator only sums on-disk files and the archive row therefore equals the current total - Fix the `erigon snapshots …` ver-format upgrade/downgrade commands in `get-started/installation/upgrading.md` ### Fundamentals section housekeeping - Fix a Mermaid parse error on the **Architecture** page — the Caplin→Execution edge label had unquoted parentheses (`|new blocks<br/>(Engine API)|`), which the flowchart parser rejects; now quoted - Reorder the Fundamentals sidebar into a clean integer reading order. It had grown to ~23 entries with colliding `sidebar_position` values and scattered related pages. New order: concepts first (Architecture, Database, Pruning Modes, Snapshots Management, Caplin) → configuration → operations/tuning → security → integrations - **NAT** moved out of the CLI Reference subfolder to a top-level Fundamentals page (next to Default Ports): `/fundamentals/configuring-erigon/nat` → `/fundamentals/nat`. The old URL is preserved via a client-side redirect (`@docusaurus/plugin-client-redirects`, pinned to 3.10.0); the site is on GitHub Pages, which can't do host-level 301s - The **CLI Reference** page was flattened (`configuring-erigon/index.mdx` → `configuring-erigon.mdx`); its `/fundamentals/configuring-erigon` URL is unchanged ### Mobile UI - Adds a theme-color meta tag to the docs site so mobile browsers tint the address/status bar with the Erigon brand orange — the same behavior the main website, Cocoon and Zilkworm docs already have. Regenerated `llms.txt` / `llms-full.txt` artifacts. --- _Updated after Copilot + @yperbasis review: replaced the `seg du` example with **real mainnet-archive output** (correct ByteCount renderings, `extensions:` line, byte-exact estimates table); documented `other_extensions`; made `--datadir` optional and chain-agnostic; clarified full-mode block-pruning impact; corrected the `--exec.workers` default and tagged all `--exec.*` flags as new in v3.5; added `--exec.batched-io` / `--exec.state-cache`; fixed the RPC cert filename typos in **TLS Authentication** (`RPC key.pem` → `RPC-key.pem`, `RPC.crtv` → `RPC.crt`); gave NAT an integer sidebar position and moved the **Modules** overview first; removed a stray editorial note in the CLI reference; typed the `seg du` code fences as `text`; de-duplicated pruning-mode concepts on the Snapshots page (now deferring to `pruning-modes.md`); and fixed `sidebar_position` collisions._ --------- Co-authored-by: Andy (NanoClaw) <andy@nanoclaw.ai> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: Bloxster <gianni.morselli@erigon.tech>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Lets you point Erigon at a specific (e.g. older)
chain.tomlfrom theerigon-snapshotGitHub repo straight from the CLI. By default Erigon fetchesthe latest published manifest, but for testing you sometimes want a
previous one — for example, to validate against a specific manifest before
changing the step size.
Caveat: this only works as long as the older
.torrents still exist and theunderlying data is still available in the torrent network.
What
Adds a
--snap.chaintoml-urlflag that fetches the preverified snapshotmanifest (
chain.toml) directly from an arbitrary URL, bypassing the defaultR2/GitHub CDN.
Precedence
From highest to lowest:
ERIGON_REMOTE_PREVERIFIEDenv var (local file override)--snap.chaintoml-url(the new override — single plain HTTP GET, no fallback)A local
preverified.tomlin the datadir still wins over all of these — deleteit to re-fetch from the URL.
The URL override has no fallback: a failed fetch returns the error and leaves
the registry untouched.
Changes
db/snapcfg/util.go— newFetchTomlFromURL(plain GET, no CDN headers);LoadRemotePreverifiedgains aforceChainTomlURLparam and its env/remotebranch is rewritten as a 3-way switch.
cmd/utils/flags.go,node/cli/default_flags.go— define and registerSnapChainTomlURLFlag.node/ethconfig/config.go— newSnapshot.ChainTomlURLfield.cmd/downloader/main.go— wire the flag into the standalone downloader.LoadSnapshotsHashes,LoadPreverified, andprovider.go; non-using call sites pass"".Notes
Pure new functionality gated behind an opt-in flag; default behavior is
unchanged when the flag is empty.