Skip to content

[r3.5] docs: add --snap.chaintoml-url, update EXEC3_PARALLEL default#21687

Merged
bloxster merged 46 commits into
release/3.5from
docs/release-3.5-flag-updates-20260609
Jun 12, 2026
Merged

[r3.5] docs: add --snap.chaintoml-url, update EXEC3_PARALLEL default#21687
bloxster merged 46 commits into
release/3.5from
docs/release-3.5-flag-updates-20260609

Conversation

@bloxster

@bloxster bloxster commented Jun 9, 2026

Copy link
Copy Markdown
Collaborator

Addresses code→docs gaps found in weekly maintenance run w24, plus some Fundamentals housekeeping. Scope: release/3.5 only.

Flags & env vars (configuring-erigon)

Prune modes / snapshots

  • Update pruning-modes.md with the EIP-8252 retention-window breaking change (v3.5): full mode now prunes block bodies/receipts to the last 262,144 blocks (previously kept all post-merge blocks) and will stop serving older block/receipt data; state-history window grows 100k→262k
  • Add new "Snapshots Management" page under Fundamentals (seg du, snapshot categories, node-type estimates, EIP-8252 retention window). The seg du example uses an archive datadir, since the estimator only sums on-disk files and the archive row therefore equals the current total
  • Fix the erigon snapshots … ver-format upgrade/downgrade commands in get-started/installation/upgrading.md

Fundamentals section housekeeping

  • Fix a Mermaid parse error on the Architecture page — the Caplin→Execution edge label had unquoted parentheses (|new blocks<br/>(Engine API)|), which the flowchart parser rejects; now quoted
  • Reorder the Fundamentals sidebar into a clean integer reading order. It had grown to ~23 entries with colliding sidebar_position values and scattered related pages. New order: concepts first (Architecture, Database, Pruning Modes, Snapshots Management, Caplin) → configuration → operations/tuning → security → integrations
  • NAT moved out of the CLI Reference subfolder to a top-level Fundamentals page (next to Default Ports): /fundamentals/configuring-erigon/nat/fundamentals/nat. The old URL is preserved via a client-side redirect (@docusaurus/plugin-client-redirects, pinned to 3.10.0); the site is on GitHub Pages, which can't do host-level 301s
  • The CLI Reference page was flattened (configuring-erigon/index.mdxconfiguring-erigon.mdx); its /fundamentals/configuring-erigon URL is unchanged

Mobile UI

  • Adds a theme-color meta tag to the docs site so mobile browsers tint the address/status bar with the Erigon brand orange — the same behavior the main website, Cocoon and Zilkworm docs already have.

Regenerated llms.txt / llms-full.txt artifacts.


Updated after Copilot + @yperbasis review: replaced the seg du example with real mainnet-archive output (correct ByteCount renderings, extensions: line, byte-exact estimates table); documented other_extensions; made --datadir optional and chain-agnostic; clarified full-mode block-pruning impact; corrected the --exec.workers default and tagged all --exec.* flags as new in v3.5; added --exec.batched-io / --exec.state-cache; fixed the RPC cert filename typos in TLS Authentication (RPC key.pemRPC-key.pem, RPC.crtvRPC.crt); gave NAT an integer sidebar position and moved the Modules overview first; removed a stray editorial note in the CLI reference; typed the seg du code fences as text; de-duplicated pruning-mode concepts on the Snapshots page (now deferring to pruning-modes.md); and fixed sidebar_position collisions.

@bloxster bloxster added the docs label Jun 9, 2026
@bloxster bloxster marked this pull request as draft June 9, 2026 07:39
bloxster added 24 commits June 9, 2026 09:40

@yperbasis yperbasis left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified the claims against release/3.5 code; requesting changes for these outstanding items:

snapshots-management.mdx

  • The example output is impossible: duComputeEstimates only sums files already on disk (archiveTotal += f.Size unconditionally), so the archive estimate always equals the total: line — a full-mode node can't show total: 1.2TB with archive 1.4TB. Rows also print in fixed order archive→full→blocks→minimal (the example shows blocks before full), and steps 0–18.500 is ~2.6× too high for mainnet@22M blocks at 390,625 txns/step. Suggest making the example an archive datadir and adding a sentence that estimates for less-pruned modes can't exceed what's on disk.
  • "one step ≈ a fixed number of blocks, depending on chain activity" is self-contradictory — a step is a fixed number of transactions (config3.DefaultStepSize = 390,625 txNums), so the block span is what varies. Please reword.
  • --datadir "defaults to the standard per-network data directory" — the default is paths.DefaultDataDir() (OS-specific, chain-agnostic; seg du has no --chain). Drop "per-network".

NAT page URL

  • configuring-erigon/nat.mdnat.md changes the route /fundamentals/configuring-erigon/nat/fundamentals/nat, and no redirects plugin is configured, so external links/bookmarks 404. Add a @docusaurus/plugin-client-redirects entry (or consciously accept the break) — and correct the PR body's "no files move, all page URLs unchanged" either way.

PR body staleness (it becomes the merge commit message)

  • Body still says --exec.workers defaults to "half the CPU cores"; the diff correctly documents the full CPU count.
  • "Minor formatting fix in the glossary 'Pruning' entry" — no glossary change is in this diff (docs/site/help-center/glossary-of-key-terms.md is untouched; the related fix merged via #21689).

Nits

  • The new Execution section omits --exec.batched-io and --exec.state-cache, both registered in default_flags.go and visible in --help.
  • nat.md at sidebar_position: 10.5 undercuts the "clean 1–23" renumbering — make it an integer.
  • --snap.p2p-manifest is also absent on release/3.4, so it deserves the same (New in v3.5) tag as --snap.chaintoml-url.

Bloxster and others added 3 commits June 10, 2026 14:26
A leftover LLM/editorial line ("Here is the rewritten and merged
Downloader section...") sat above the "Downloader and Synchronization"
heading in the CLI reference and leaked into the generated llms-full.txt
bundles. Remove it and regenerate.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…s, sidebar

snapshots-management.mdx:
- Rewrite the seg du example as an archive datadir: the estimator sums only
  on-disk files, so the archive row always equals the current total and no
  node-type estimate can exceed it. Fix row order (archive→full→blocks→minimal)
  and the step count (~7,000 for mainnet@22M blocks, not 18,500). Add a note
  explaining the estimator can't exceed what's on disk.
- Reword the "step" definition: a step is a fixed number of transactions
  (config3.DefaultStepSize = 390,625 txNums), so blocks-per-step varies.
- Drop "per-network" from the --datadir default (seg du has no --chain;
  default is paths.DefaultDataDir(), chain-agnostic).

configuring-erigon.mdx:
- Document --exec.batched-io and --exec.state-cache (registered, shown in --help).
- Tag --snap.p2p-manifest as (New in v3.5), like --snap.chaintoml-url.

Fundamentals sidebar: give NAT an integer position (11, after Default Ports)
and renumber the following pages so the order stays clean 1–24.

Regenerate llms.txt / llms-full.txt (+ static copies).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
NAT moved from /fundamentals/configuring-erigon/nat to /fundamentals/nat.
The docs site is served as static files on GitHub Pages, which can't do
host-level 301s, so use the official @docusaurus/plugin-client-redirects
(pinned to 3.10.0, matching core) to emit a client-side redirect from the
old URL. Verified the stub builds at the old path.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@bloxster

Copy link
Copy Markdown
Collaborator Author

@yperbasis thanks for the thorough review — all points addressed. Pushed in 54959d9, be7a07b, 0fd365f (branch now up to date with release/3.5, conflicts resolved, docs-site / build green).

snapshots-management.mdx

  • Impossible example → reworked the example as an archive datadir. Since duComputeEstimates sums every on-disk file into archiveTotal unconditionally (snapshots_cmd.go:3784), the archive row now equals the total, and the other modes are smaller. Fixed the row order to the code's fixed archive → full → blocks → minimal (lines 3849–3852) and the step count (~7,000 for mainnet@22M, not 18,500). Added a sentence noting estimates can't exceed what's on disk.
  • "one step ≈ fixed blocks" → reworded: a step is a fixed number of transactions (config3.DefaultStepSize = 390,625 txNums), so blocks-per-step varies.
  • --datadir "per-network" → dropped; now documents paths.DefaultDataDir() (OS-specific, chain-agnostic — seg du has no --chain).

NAT page URL → kept the move but added a client-side redirect via @docusaurus/plugin-client-redirects (pinned 3.10.0) from /fundamentals/configuring-erigon/nat/fundamentals/nat. The site is on GitHub Pages (no host-level 301s), so a client redirect is the available option; verified the stub builds at the old path. PR body corrected — it no longer claims URLs are unchanged.

PR body staleness

Nits

  • Documented --exec.batched-io and --exec.state-cache.
  • NAT given an integer sidebar_position (11, after Default Ports); renumbered the trailing pages to a clean 1–24.
  • Tagged --snap.p2p-manifest as (New in v3.5).

Also removed a stray editorial line ("Here is the rewritten and merged Downloader section…") that Copilot caught leaking into the CLI reference. Ready for another look when you have a moment.

@yperbasis yperbasis left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified the flag/prune/seg du claims against the code on release/3.5 — three remaining issues:

  1. snapshots-management.mdx — example file counts don't add up. The breakdown rows sum to 40,008 files but the total line says 42,810. doDU computes the total as the exact sum of the category rows, so real output can't disagree. (Bytes/percentages are consistent.)

  2. snapshots-management.mdxminimal 0.9TB is not a possible ByteCount rendering. 1254.1 GB − 300.0 GB = 954.1 GB < 1024 GB, so the binary prints 954.1GB (the page itself says sizes use the ByteCount format). full 1.0TB / blocks 1.1TB are fine.

  3. configuring-erigon.mdx — "(New in v3.5)" tagging inconsistent. Only --exec.serial and --exec.no-prune are tagged, but all seven --exec.* flags are new to the erigon binary in v3.5 (none exist in release/3.4's cmd/utils/flags.go; only the integration tool had --exec.workers). Tag the section once, e.g. "all --exec.* flags are new in v3.5; their env-var equivalents existed earlier".

Bloxster and others added 2 commits June 10, 2026 16:06
…* tagging

snapshots-management.mdx:
- Fix the seg du example total file count: rows sum to 40,008, not 42,810
  (doDU prints the total as the exact sum of the category rows).
- Fix the minimal estimate rendering: 1254.1GB − 300.0GB = 954.1GB, which is
  < 1024GB so ByteCount prints "954.1GB", not "0.9TB".

configuring-erigon.mdx:
- All seven --exec.* flags are new to the erigon binary in v3.5 (none exist in
  release/3.4's cmd/utils/flags.go). Tag the section once instead of marking
  only --exec.serial / --exec.no-prune, and note the env-var equivalents
  predate the flags.

Regenerate llms bundles.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@bloxster

Copy link
Copy Markdown
Collaborator Author

@yperbasis all three verified and fixed in 5786533 (branch also updated to current release/3.5):

  1. seg du example file counts — corrected the total to 40,008 (the exact sum of the category rows: 18,200 + 12,400 + 6,300 + 2,100 + 910 + 90 + 8), matching doDU's behaviour. Bytes/percentages were already consistent.
  2. minimal size rendering — fixed to 954.1GB. 1254.1 GB − 300.0 GB = 954.1 GB < 1024 GB, so ByteCount prints 954.1GB, not 0.9TB. (full 1.0TB / blocks 1.1TB unchanged.)
  3. --exec.* "(New in v3.5)" tagging — confirmed against cmd/utils/flags.go: none of the seven exec.* flags exist on release/3.4, all seven are on release/3.5. Removed the inconsistent per-flag tags and added a single section note: "All --exec.* flags are new in v3.5 — they expose, as CLI flags, execution toggles that on earlier releases were settable only via the EXEC3_* / NO_* environment variables."

docs-site / build is green. Thanks for the close reads!

@yperbasis yperbasis left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The three items from my previous review are fixed (verified against release/3.5 code). One remaining blocker, all within the seg du example block in snapshots-management.mdx — it is still not output the binary could produce:

  1. other 0.5GB is not a possible ByteCount rendering. Sub-GiB sizes print in MB (common/bytes.go:26), so a 0.5 GiB category renders as 512.0MB.
  2. Missing extensions: line. With a non-empty other category, duFormatHuman always prints extensions: <sorted list> directly under the other row — the example shows 8 other files but no such line (the page's own table even says extensions are "listed inline").
  3. The sizes can't belong to mainnet | archive at block 22M and contradict the Database page in these same docs (real Nov 2024 mainnet archive @ ~21.2M: snapshots total 2.3TB, ~1.2TB of it block/tx segments, idx 430GB > domain 300GB > history 280GB). The example shows total 1.2TB with domains as the largest category (612GB, 48.8%) and only 410GB of block segments — on mainnet archive, block segments dominate.

Simplest fix that kills the whole class: run erigon seg du against a real mainnet archive datadir and paste the actual output, lightly trimmed. That also fixes the estimates-table column padding, which currently doesn't match the real %-13s %10s %10s %-15s %s format (the breakdown rows, by contrast, are already byte-exact). Remember to regenerate the four llms*.txt artifacts afterwards.

Optional nit: the category table's "domains … usually the largest category" holds for pruned (full/minimal) nodes but not for mainnet archive — consider "typically the largest category on pruned nodes".

…y table)

Address yperbasis's optional nit: "domains … usually the largest category"
holds for pruned (full/minimal) nodes, not mainnet archive where block
segments dominate. The main seg du example block rewrite (real output) will
follow separately.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@bloxster

Copy link
Copy Markdown
Collaborator Author

Thanks @yperbasis — all three confirmed against the code, you're right on each:

  1. other 0.5GB isn't a possible ByteCount rendering (sub-1024-MiB prints in MB, common/bytes.go:28) → should be e.g. 512.0MB.
  2. A non-empty other category always prints the extensions: … line under it (snapshots_cmd.go:4079).
  3. The figures contradict our own Database page — on mainnet archive, block segments dominate (~1.2 TB) with idx 430 GB > domain 300 GB > history 280 GB, total ~2.3 TB; my example had domains largest, which only holds for pruned nodes.

Rather than hand-fabricate the block (and risk the estimate rows being subtly off again), I'm taking your recommended fix: paste real, lightly-trimmed erigon seg du / -v / --json output. I'm syncing a mainnet archive node for it (so the estimates table is meaningful — archive row = on-disk total, others are real subsets); that'll take a couple of days. I'll push the real-output example (which also makes the estimates-table padding byte-exact by construction) and regenerate the four llms*.txt bundles then.

In the meantime I've pushed the optional nit (82e160f): the domains category note now says it's typically largest on pruned (full/minimal) nodes, and that block segments dominate on archive.

Captured from a live mainnet archive node (`--prune.mode archive`,
block 25,296,977) via `erigon seg du` / `-v`. Replaces the fabricated
example, resolving yperbasis's remaining blocker:
- real ByteCount renderings (e.g. other = 9.8MB, not 0.5GB)
- the `extensions:` line under `other` is present
- realistic archive proportions (block segments dominant at 48.3%)
- byte-exact estimates-table column padding (it's actual binary output)

Verbose example now shows real block-segments + domains subcategory trees.
Regenerate llms bundles.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@bloxster

Copy link
Copy Markdown
Collaborator Author

@yperbasis done in 2b7fb01 — replaced the fabricated seg du block with real, verbatim output captured from a live mainnet archive node (--prune.mode archive, ~block 25.3M), via erigon seg du and -v. That resolves all three points at the source:

  • other now renders 9.8MB (real sub-GiB → MB), not 0.5GB.
  • The extensions: .toml, .torrent, .txt line is present under other.
  • Realistic archive proportions — block segments dominate (48.3%), then domains / inverted indices / history / accessors.
  • Estimates-table padding is byte-exact (it's the binary's own output now).

The verbose example shows the real block segments + domains subcategory trees.

Two sanity notes:

  • Total 1.8TB matches our manually-measured archive figure in disk-sizes.json (1.77TB). It's smaller than the database.md "~2.3TB" line because that one is explicitly a Nov 2024 snapshot; this is current (Jun 2026, post-compression-improvements).
  • The estimate rows are the binary's own projections of which on-disk files each mode would keep (archive 1.8TB / blocks 1.2TB / full 361GB / minimal 342GB). minimal lines up with the measured 350GB; full (361GB) is lower than a freshly-synced full node (~920GB) because it's a naive on-disk projection — which the page already caveats. Happy to add an explicit cross-ref to Hardware Requirements if you'd prefer.

docs-site / build green; llms bundles regenerated.

snapshots-management.mdx repeated content that belongs to pruning-modes.md.
Keep pruning-modes.md as the single source of truth and reduce the snapshots
page to its snapshot-specific angle + pointers:
- Drop the Archive/Full/Blocks/Minimal mode table (duplicated the
  pruning-modes table) — keep the seg-du mode-detection + estimates note.
- Collapse the "EIP-8252 retention window change" re-explanation to a brief
  "what changes on disk" framing + a pointer to the pruning-modes
  breaking-change note; keep the "run seg du after the first prune pass" bit.
- "Manual snapshot operations" (snapshot file-format upgrade/downgrade) is
  unchanged — it's snapshot-specific, not duplicated.

Regenerate llms bundles.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 30 out of 33 changed files in this pull request and generated 2 comments.

Files not reviewed (1)
  • docs/site/package-lock.json: Generated file

Comment thread docs/site/docs/fundamentals/snapshots-management.mdx Outdated
Comment thread docs/site/docs/fundamentals/snapshots-management.mdx Outdated
Bloxster and others added 2 commits June 12, 2026 09:35
Copilot review: the two seg du example blocks used bare ``` fences. Type
them as ```text (plain terminal output) to match the repo convention
(logs.md and 14 other files) and the docs-routine §7.3 rule, and to stop
Prism from auto-highlighting the box-drawing/tree characters.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@bloxster bloxster requested a review from yperbasis June 12, 2026 07:48
@bloxster bloxster merged commit fe26547 into release/3.5 Jun 12, 2026
20 checks passed
@bloxster bloxster deleted the docs/release-3.5-flag-updates-20260609 branch June 12, 2026 10:03
bloxster added a commit that referenced this pull request Jun 12, 2026
)

Follow-up to #21687. Adds one cross-link per section of the **CLI
Reference** (`configuring-erigon`) to its dedicated page — placed in the
section explainer where possible, or on the most relevant flag. One link
per target page, no duplicates of the links already present (nat /
supported-networks / pruning-modes / eth_getProof).

**Explainer links:** Database, Snapshots Management, TxPool, Default
Ports, Interacting with Erigon (RPC), MCP, Logs, Sentry, Downloader,
Caplin, Shutter Network, Docker Compose.
**Targeted flag links:** JWT (on `--authrpc.jwtsecret`), Creating a
dashboard (on `--metrics`).

Build passes the `onBrokenLinks: 'throw'` check; `llms.txt` /
`llms-full.txt` regenerated.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Bloxster <gianni.morselli@erigon.tech>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
pull Bot pushed a commit to Dustin4444/erigon that referenced this pull request Jun 12, 2026
…amentals docs) (erigontech#21765)

Forward-ports the docs changes merged to `release/3.5` in erigontech#21687 onto
`main`.

Verified against `origin/main` source per the dual-commit rule — all
documented flags/defaults exist on main: `snap.chaintoml-url`,
`snap.p2p-manifest`, the seven `exec.*` flags, `EXEC3_PARALLEL=true`,
and `exec.workers` Value `NumCPU()/2` (same "half" `--help` text, so the
doc's effective-default note applies identically).

**Includes:**
- New **Snapshots Management** page (`seg du`) with real mainnet-archive
output
- EIP-8252 retention-window breaking-change note in `pruning-modes.md`
- CLI Reference: `--snap.chaintoml-url`, `--snap.p2p-manifest`, and the
`--exec.*` flags (serial / no-prune / batched-io / state-cache), with
the `--exec.workers` effective-default clarification
- CLI Reference **flattened** (`configuring-erigon/index.mdx` →
`configuring-erigon.mdx`; `/fundamentals/configuring-erigon` URL
unchanged)
- **NAT** moved to a top-level page; old URL preserved via
`@docusaurus/plugin-client-redirects`
- TLS Authentication cert-filename typo fix; Fundamentals sidebar
reorder; mobile theme-color
- `llms.txt` / `llms-full.txt` regenerated; build passes `onBrokenLinks:
'throw'`

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Bloxster <gianni.morselli@erigon.tech>
pull Bot pushed a commit to Dustin4444/erigon that referenced this pull request Jun 13, 2026
…gontech#21772)

Main counterpart of erigontech#21763 (which added the same cross-links on
release/3.5). Now that erigontech#21765 (the erigontech#21687 forward-port) has merged, the
flattened CLI Reference page and its link targets exist on `main`, so
this applies cleanly.

Adds one cross-link per section of the **CLI Reference**
(`configuring-erigon`) to its dedicated page — in the section explainer
where possible, or on the most relevant flag. One link per target, no
duplicates of the links already present (nat / supported-networks /
pruning-modes / eth_getProof).

**Explainer links:** Database, Snapshots Management, TxPool, Default
Ports, Interacting with Erigon (RPC), MCP, Logs, Sentry, Downloader,
Caplin, Shutter Network, Docker Compose.
**Targeted flag links:** JWT (on `--authrpc.jwtsecret`), Creating a
dashboard (on `--metrics`).

Build passes the `onBrokenLinks: 'throw'` check; `llms.txt` /
`llms-full.txt` regenerated.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Bloxster <gianni.morselli@erigon.tech>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants