Skip to content

db/snapcfg, cmd/utils, node: add --snap.chaintoml-url override#21584

Merged
wmitsuda merged 1 commit into
mainfrom
wmitsuda/snap-chaintoml-url
Jun 6, 2026
Merged

db/snapcfg, cmd/utils, node: add --snap.chaintoml-url override#21584
wmitsuda merged 1 commit into
mainfrom
wmitsuda/snap-chaintoml-url

Conversation

@wmitsuda

@wmitsuda wmitsuda commented Jun 2, 2026

Copy link
Copy Markdown
Member

Why

Lets you point Erigon at a specific (e.g. older) chain.toml from the
erigon-snapshot GitHub repo straight from the CLI. By default Erigon fetches
the latest published manifest, but for testing you sometimes want a
previous one — for example, to validate against a specific manifest before
changing the step size.

Caveat: this only works as long as the older .torrents still exist and the
underlying data is still available in the torrent network.

What

Adds a --snap.chaintoml-url flag that fetches the preverified snapshot
manifest (chain.toml) directly from an arbitrary URL, bypassing the default
R2/GitHub CDN.

Precedence

From highest to lowest:

  1. ERIGON_REMOTE_PREVERIFIED env var (local file override)
  2. --snap.chaintoml-url (the new override — single plain HTTP GET, no fallback)
  3. Default R2/GitHub CDN fetch

A local preverified.toml in the datadir still wins over all of these — delete
it to re-fetch from the URL.

The URL override has no fallback: a failed fetch returns the error and leaves
the registry untouched.

Changes

  • db/snapcfg/util.go — new FetchTomlFromURL (plain GET, no CDN headers);
    LoadRemotePreverified gains a forceChainTomlURL param and its env/remote
    branch is rewritten as a 3-way switch.
  • cmd/utils/flags.go, node/cli/default_flags.go — define and register
    SnapChainTomlURLFlag.
  • node/ethconfig/config.go — new Snapshot.ChainTomlURL field.
  • cmd/downloader/main.go — wire the flag into the standalone downloader.
  • Param threaded through LoadSnapshotsHashes, LoadPreverified, and
    provider.go; non-using call sites pass "".

Notes

Pure new functionality gated behind an opt-in flag; default behavior is
unchanged when the flag is empty.

Add a --snap.chaintoml-url flag to fetch the preverified chain.toml
directly from an arbitrary URL instead of the default R2/GitHub CDN.

Precedence: ERIGON_REMOTE_PREVERIFIED env (local file) > --snap.chaintoml-url
> default CDN fetch. A local preverified.toml in the datadir still takes
precedence; delete it to re-fetch from the URL. The URL override has no
fallback: a failed fetch returns the error and leaves the registry untouched.

@AskAlexSharov AskAlexSharov left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks like "Abstraction leak":

  • User usually enough to set "Webseed url" (with all what we want inside: chain.toml, manifest.txt, .torrent, files).
  • Where do we store our list of canonical hashes - it's technical detail (we can change it from chain.toml to alex.yml, or to v2-chain.toml, or to 10 files).

but oke

@wmitsuda wmitsuda added this pull request to the merge queue Jun 6, 2026
Merged via the queue into main with commit ba724bb Jun 6, 2026
89 checks passed
@wmitsuda wmitsuda deleted the wmitsuda/snap-chaintoml-url branch June 6, 2026 06:23
taratorio added a commit that referenced this pull request Jun 8, 2026
…r branch

SonarCloud rejects a branch analysis dated older than the branch's
latest processed one. All merge-queue entries analyze the target branch,
so when entry N's report uploaded after entry N+1's, SonarCloud rejected
it and flipped the commit's SonarCloud check to cancelled ("Date of
analysis cannot be older than the date of the last known analysis").
Seen on 5027d25, where the scans of three stacked queue entries
(#21648, #21584, #21640) raced and #21648's analysis lost.

Restructure so branch analyses can never be submitted out of order:

- merge_group runs keep `make test-sonar-coverage` as a gate check but
  no longer scan; they upload coverage-test-all.out as a short-retention
  artifact instead.
- A new push-triggered SonarCloud Branch Scan workflow (main and
  release/**, matching the branches SonarCloud tracks) downloads that
  artifact — the merge queue fast-forwards the target branch to the
  already-tested merge commit, so the pushed SHA equals the merge-queue
  run's head SHA and the artifact maps to the commit exactly — and runs
  only the scanner (~7 min instead of ~25). If no artifact exists
  (direct push, expired retention), it falls back to running the tests.
- Scans are serialized per branch FIFO via `concurrency.queue: max`
  (GitHub Actions, 2026-05), so every merged commit is analyzed, in
  order, with no runs cancelled in merge bursts.
- PR scans (Sonar PR analyses, no main-branch date ordering) and
  cache-warming runs are unchanged.

actionlint (≤ v1.7.12) does not know the `queue` key yet, so lint.yml
gains a narrow --ignore alongside the existing workspace one.

A failed analysis upload also no longer involves the gate: scanner
failures now surface on a post-merge run instead, while PR runs keep
exercising the scanner so breakage is still caught before merge.
Sahil-4555 pushed a commit to Sahil-4555/erigon that referenced this pull request Jun 8, 2026
…r branch (erigontech#21669)

## Problem

SonarCloud rejects a branch analysis dated older than the branch's
latest processed one. Every merge-queue entry analyzes the target
branch, so when queue entry N's report uploads *after* entry N+1's,
SonarCloud rejects N's with *"Date of analysis cannot be older than the
date of the last known analysis"* and flips that commit's **SonarCloud
Code Analysis** check to `cancelled`.

Seen on
[`5027d259c2`](erigontech@5027d25):
three stacked queue entries (erigontech#21648, erigontech#21584, erigontech#21640) scanned
concurrently; erigontech#21584 started 6s later but uploaded 21s earlier, so
erigontech#21648's analysis was rejected as out of order. No impact on the gate
(the scan step is `continue-on-error` and the merge succeeded), but it
leaves a spurious failed check on a merged commit.

## Fix

Make it structurally impossible to submit branch analyses out of order:

- **`merge_group` runs** keep `make test-sonar-coverage` as a gate check
but **no longer scan**. They upload `coverage-test-all.out` as a 7-day
artifact.
- **A new `SonarCloud Branch Scan` workflow** (`push` on `main` +
`release/**`, matching the branches SonarCloud tracks) downloads that
artifact and runs **only the scanner** (~7 min vs ~25). The merge queue
fast-forwards the target branch to the already-tested merge commit, so
the pushed SHA equals the merge-queue run's head SHA and the artifact
maps to the commit exactly. If no artifact exists (direct push, or
expired retention), it falls back to running the tests itself.
- Branch scans are serialized per branch **FIFO** via
`concurrency.queue: max` ([GitHub Actions,
2026-05](https://github.blog/changelog/2026-05-07-github-actions-concurrency-groups-now-allow-larger-queues/)),
so every merged commit is analyzed, in order, with **no runs cancelled**
during merge bursts.
- **PR scans** (Sonar PR analyses — separate from main-branch date
ordering) and **cache-warming** runs are unchanged.

A failed analysis upload no longer touches the gate at all: scanner
failures now surface on a post-merge run, while PR runs keep exercising
the scanner so config/scanner breakage is still caught before merge.

## Notes

- `actionlint` (≤ v1.7.12, the latest release) does not know the `queue`
concurrency key yet, so `lint.yml` gets a narrow `--ignore` alongside
the existing `workspace` one. Removable once actionlint learns the key.
- `concurrency.queue: max` is ~1 month old — first real merge-burst
after this lands is where FIFO ordering gets exercised live.

## Testing

- `make lint`, `actionlint` (CI flags), and `zizmor` (repo config) all
clean; the new workflow is finding-free.
- The artifact-lookup script was shellchecked and run against live API
data: correctly resolves the merge-queue run by head SHA, filters
expired artifacts, and degrades to the test-running fallback on a
missing artifact or total API failure (never fails the job).
bloxster added a commit that referenced this pull request Jun 12, 2026
…21687)

Addresses code→docs gaps found in weekly maintenance run w24, plus some
Fundamentals housekeeping. **Scope: release/3.5 only.**

### Flags & env vars (`configuring-erigon`)
- Add `--snap.chaintoml-url` flag (PR #21584), including
`ERIGON_REMOTE_PREVERIFIED` override precedence
- Add `--snap.p2p-manifest` flag (PR #20526), tagged *(New in v3.5)*
- Update `EXEC3_PARALLEL` default to `true` (PR #21591)
- Add `--exec.serial` and `--exec.no-prune` flags (new in v3.5); also
document `--exec.batched-io` and `--exec.state-cache`
- Document `--exec.workers`. Its effective default is the **full
CPU-core count** (inherited from the `EXEC3_WORKERS` fallback when the
flag is unset); the flag's own `--help` text says "half the CPU cores",
which is a known inconsistency in the binary

### Prune modes / snapshots
- Update `pruning-modes.md` with the EIP-8252 retention-window breaking
change (v3.5): full mode now prunes block bodies/receipts to the last
262,144 blocks (previously kept all post-merge blocks) and will stop
serving older block/receipt data; state-history window grows 100k→262k
- Add new "Snapshots Management" page under Fundamentals (`seg du`,
snapshot categories, node-type estimates, EIP-8252 retention window).
The `seg du` example uses an archive datadir, since the estimator only
sums on-disk files and the archive row therefore equals the current
total
- Fix the `erigon snapshots …` ver-format upgrade/downgrade commands in
`get-started/installation/upgrading.md`

### Fundamentals section housekeeping
- Fix a Mermaid parse error on the **Architecture** page — the
Caplin→Execution edge label had unquoted parentheses (`|new
blocks<br/>(Engine API)|`), which the flowchart parser rejects; now
quoted
- Reorder the Fundamentals sidebar into a clean integer reading order.
It had grown to ~23 entries with colliding `sidebar_position` values and
scattered related pages. New order: concepts first (Architecture,
Database, Pruning Modes, Snapshots Management, Caplin) → configuration →
operations/tuning → security → integrations
- **NAT** moved out of the CLI Reference subfolder to a top-level
Fundamentals page (next to Default Ports):
`/fundamentals/configuring-erigon/nat` → `/fundamentals/nat`. The old
URL is preserved via a client-side redirect
(`@docusaurus/plugin-client-redirects`, pinned to 3.10.0); the site is
on GitHub Pages, which can't do host-level 301s
- The **CLI Reference** page was flattened
(`configuring-erigon/index.mdx` → `configuring-erigon.mdx`); its
`/fundamentals/configuring-erigon` URL is unchanged

### Mobile UI
- Adds a theme-color meta tag to the docs site so mobile browsers tint
the address/status bar with the Erigon brand orange — the same behavior
the main website, Cocoon and Zilkworm docs already have.

Regenerated `llms.txt` / `llms-full.txt` artifacts.

---
_Updated after Copilot + @yperbasis review: replaced the `seg du`
example with **real mainnet-archive output** (correct ByteCount
renderings, `extensions:` line, byte-exact estimates table); documented
`other_extensions`; made `--datadir` optional and chain-agnostic;
clarified full-mode block-pruning impact; corrected the `--exec.workers`
default and tagged all `--exec.*` flags as new in v3.5; added
`--exec.batched-io` / `--exec.state-cache`; fixed the RPC cert filename
typos in **TLS Authentication** (`RPC key.pem` → `RPC-key.pem`,
`RPC.crtv` → `RPC.crt`); gave NAT an integer sidebar position and moved
the **Modules** overview first; removed a stray editorial note in the
CLI reference; typed the `seg du` code fences as `text`; de-duplicated
pruning-mode concepts on the Snapshots page (now deferring to
`pruning-modes.md`); and fixed `sidebar_position` collisions._

---------

Co-authored-by: Andy (NanoClaw) <andy@nanoclaw.ai>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Bloxster <gianni.morselli@erigon.tech>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants