Skip to content

Decentralized snapshot distribution via discv5 + BitTorrent #19660

@mh0lt

Description

@mh0lt

Goal

Trustlessly distribute erigon chain and state snapshots without relying on centralized infrastructure (GitHub repos, R2 buckets).

Problem

Today, snapshot info-hashes are distributed via a centralized path:

  1. The erigon-snapshot GitHub repo maintains canonical hashes
  2. These are embedded into the binary as preverified.toml
  3. At runtime, nodes fall back to fetching from R2/GitHub if no local copy exists

This creates a single point of failure and a central trust dependency for the entire snapshot distribution pipeline.

Solution

Replace the centralized info-hash distribution with a peer-to-peer mechanism:

  1. Publish: Each node creates a BitTorrent torrent of its preverified.toml and advertises the torrent info-hash via a custom discv5 ENR entry
  2. Discover: New nodes find peers via discv5, read their ENR entries to get the latest info-hash, and download preverified.toml via BitTorrent
  3. Bootstrap: Once a node has preverified.toml, the existing downloader pipeline takes over to fetch actual snapshots

Key Design Properties

  • Deterministic: preverified.toml is byte-identical across nodes at the same snapshot step, producing a stable torrent info-hash
  • Versioned: The max frozen transaction number provides monotonic ordering — a new deterministic file is produced for every max step
  • Append-only: Each new version is a strict superset of the previous, enabling cheap integrity checks
  • Backward compatible: Nodes that don't support the new ENR entry fall back to the existing centralized path
  • Incrementally trustable: Trust layers are additive — POC works without trust, statistical consensus adds basic protection, UCAN adds cryptographic guarantees

Implementation Steps

Step 1: POC — Physical Distribution Flow

Prove the mechanics: ENR publishing, torrent creation, peer discovery, download, and integration with the existing downloader pipeline. No trust layer — works on honest-network assumption.

Step 2: Statistical Trust — Threshold Consensus

Add lightweight production trust: collect info-hashes from multiple peers, require majority agreement before accepting. Defeats isolated bad actors. Does not require any cryptographic infrastructure.

Step 3: Identity Trust — UCAN Delegation

Add cryptographic identity trust using UCAN delegation chains. Known trusted publishers sign their updates, nodes verify locally. Eliminates the honest-majority assumption without introducing a central runtime dependency. Defeats Sybil attacks.

Architecture

                    ┌─────────────────────────────────┐
                    │         Node Startup             │
                    └──────────────┬──────────────────┘
                                   │
                    ┌──────────────▼──────────────────┐
                    │  Create torrent of               │
                    │  preverified.toml                 │
                    │  (deterministic info-hash)        │
                    └──────────────┬──────────────────┘
                                   │
                    ┌──────────────▼──────────────────┐
                    │  Set ENR entry:                   │
                    │  {FrozenTx, InfoHash}             │
                    │  (28 bytes, fits in 300B limit)   │
                    └──────────────┬──────────────────┘
                                   │
              ┌────────────────────┼────────────────────┐
              │                    │                     │
    ┌─────────▼─────────┐  ┌──────▼──────┐  ┌──────────▼─────────┐
    │  Seed torrent      │  │  discv5     │  │  Existing          │
    │  via BitTorrent    │  │  discovery  │  │  snapshot          │
    │                    │  │  ← peers    │  │  downloader        │
    └────────────────────┘  └──────┬──────┘  └──────────▲─────────┘
                                   │                     │
                            ┌──────▼──────┐              │
                            │  Read peer  │              │
                            │  ENR entries│              │
                            └──────┬──────┘              │
                                   │                     │
                            ┌──────▼──────┐              │
                            │  Download   │              │
                            │  preverified│──────────────┘
                            │  .toml via  │
                            │  BitTorrent │
                            └─────────────┘

Trust Layers (additive)

Layer Mechanism Threat Model Dependency
None (POC) First peer wins Honest network None
Statistical Threshold consensus (N-of-M) Honest majority None
UCAN Cryptographic delegation chains Byzantine (Sybil-resistant) Root key in binary

Metadata

Metadata

Assignees

No one assigned

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions