Rapidly sync validated blocks and snapshots from KYVE to every Tendermint based Blockchain Application

KSYNC is a fast, efficient synchronization tool that enables blockchain node operators to rapidly bootstrap and sync Tendermint/CometBFT-based nodes using validated data from the KYVE network. Instead of replaying the entire blockchain history, KSYNC retrieves pre-validated blocks and snapshots from KYVE's decentralized storage, dramatically reducing sync time and operational costs.

• Blazing Fast Sync: Reduce node sync time from days/weeks to hours/minutes
• Multiple Sync Strategies: Choose between block-sync, state-sync, or height-sync (combined)
• Multi-Engine Support: Compatible with Tendermint v0.34, CometBFT v0.37/v0.38, and Celestia Core
• Automatic Chain Upgrades: Detects and handles chain upgrades seamlessly (cosmovisor integration)
• Data Integrity: All blocks and snapshots are cryptographically verified against KYVE checksums
• Snapshot Serving: Generate and serve snapshots for KYVE data pools
• Flexible Architecture: Support for both KYVE bundles and direct RPC endpoints
• Built-in Utilities: Backup, prune, and node management tools included
• Cost Effective: Eliminate the need for expensive archival node infrastructure

• Node Bootstrapping: Quickly spin up new validator or full nodes
• Archive Node Alternative: Drop old blocks and re-sync from KYVE when needed
• Chain Migration: Efficiently sync nodes during chain upgrades
• Snapshot Generation: Contribute snapshots to KYVE data pools
• Development & Testing: Rapidly sync test environments to specific block heights

For Users:

• Quick Start
• Installation
• Available Commands
• Usage Examples
• Configuration

For Developers:

• Architecture
• Project Structure
• How KSYNC Works
• Development Setup
• Contributing
• Releases

# Download the latest release for your platform
# Visit: https://github.com/KYVENetwork/ksync/releases

# Example: Sync Osmosis node to latest height using height-sync
ksync height-sync \
  --binary-path=/path/to/osmosisd \
  --home=/path/to/.osmosisd \
  --chain-id=osmosis-1 \
  --source=kyve

# Example: State-sync to a specific height
ksync state-sync \
  --binary-path=/path/to/gaiad \
  --home=/path/to/.gaia \
  --chain-id=cosmoshub-4 \
  --source=kyve \
  --target-height=15000000

Download the latest release for your platform from the releases page:

# Linux AMD64
wget https://github.com/KYVENetwork/ksync/releases/download/vX.X.X/ksync_linux_amd64.tar.gz
tar -xzf ksync_linux_amd64.tar.gz
sudo mv ksync /usr/local/bin/

# macOS
wget https://github.com/KYVENetwork/ksync/releases/download/vX.X.X/ksync_darwin_amd64.tar.gz
tar -xzf ksync_darwin_amd64.tar.gz
sudo mv ksync /usr/local/bin/

Requirements:

• Go 1.22.x
• Make

Build Steps:

git clone https://github.com/KYVENetwork/ksync.git
cd ksync
git checkout tags/vX.X.X -b vX.X.X
make build

This builds the binary in the build/ directory. To install system-wide:

cp build/ksync ~/go/bin/ksync
# or
sudo mv build/ksync /usr/local/bin/ksync

ksync version

Common flags available across all sync commands:

# Initialize the node first
osmosisd init my-node --chain-id osmosis-1

# Sync using height-sync (recommended)
ksync height-sync \
  --binary-path=$(which osmosisd) \
  --home=$HOME/.osmosisd \
  --chain-id=osmosis-1 \
  --source=kyve

# Sync Cosmos Hub to block 15000000
ksync block-sync \
  --binary-path=$(which gaiad) \
  --home=$HOME/.gaia \
  --chain-id=cosmoshub-4 \
  --target-height=15000000

# Quick state-sync for Juno
ksync state-sync \
  --binary-path=/usr/local/bin/junod \
  --home=$HOME/.juno \
  --chain-id=juno-1

# Automatically detect and handle chain upgrades
ksync height-sync \
  --binary-path=$(which osmosisd) \
  --home=$HOME/.osmosisd \
  --chain-id=osmosis-1 \
  --autoselect-binary-version

# Generate and serve snapshots on port 7878
ksync serve-snapshots \
  --binary-path=$(which gaiad) \
  --home=$HOME/.gaia \
  --chain-id=cosmoshub-4 \
  --snapshot-pool-id=1

# Create compressed backup
ksync backup \
  --home=$HOME/.osmosisd \
  --backup-path=$HOME/backups \
  --compress

# Remove all blocks below height 10000000
ksync prune \
  --binary-path=$(which osmosisd) \
  --home=$HOME/.osmosisd \
  --target-height=10000000

# Use RPC endpoint as data source
ksync serve-blocks \
  --binary-path=$(which gaiad) \
  --home=$HOME/.gaia \
  --rpc-server-address=https://rpc.cosmos.network:443

KSYNC respects standard Cosmos SDK environment variables:

export DAEMON_NAME=osmosisd
export DAEMON_HOME=$HOME/.osmosisd

KSYNC works seamlessly with Cosmovisor for automatic binary upgrades:

# Set up Cosmovisor directories
export DAEMON_NAME=osmosisd
export DAEMON_HOME=$HOME/.osmosisd

# Sync with auto-upgrade support
ksync height-sync \
  --binary-path=$DAEMON_HOME/cosmovisor/genesis/bin/osmosisd \
  --home=$DAEMON_HOME \
  --autoselect-binary-version

KSYNC automatically fetches chain configuration from the KYVE Source Registry:

• Chain ID to KYVE pool mapping
• Engine version per block height
• Automatic version detection for upgrades

To use a custom registry:

ksync height-sync \
  --registry-url=https://raw.githubusercontent.com/YOUR_ORG/registry/main \
  ...

KSYNC supports two data sources:

1. KYVE (Recommended): Validated bundles from KYVE network

   • Pro: Cryptographically verified, decentralized, cost-effective
   • Con: Requires KYVE pool support for the chain

2. RPC: Direct sync from RPC endpoints

   • Pro: Works for any chain with RPC access
   • Con: Slower, single point of trust

# Use KYVE (default)
--source=kyve

# Use RPC fallback
--source=rpc --rpc-server-address=https://rpc.example.com:443

KSYNC automatically retrieves data from multiple storage providers:

• Arweave
• Bundlr
• KYVE Storage
• Turbo

No configuration required - KSYNC tries providers in order until successful.

KSYNC follows a layered architecture with clear separation of concerns:

graph TB
    subgraph "External Data Sources"
        KYVE[KYVE Network API<br/>Bundle Metadata & Pool Info]
        STORAGE[Storage Providers<br/>Arweave, Bundlr, Turbo]
        REGISTRY[Source Registry<br/>GitHub - Chain Configs]
    end

    subgraph "CLI Layer"
        CMD[Commands<br/>block-sync, state-sync, height-sync, etc.]
    end

    subgraph "Orchestration Layer"
        ORCH[Sync Orchestrators<br/>BlockSync, StateSync, HeightSync]
        BOOTSTRAP[Bootstrap Manager<br/>Genesis & First Block]
    end

    subgraph "Data Collection Layer"
        COLLECTORS[Collectors<br/>Blocks, Snapshots, Bundles]
        SOURCES[Source Manager<br/>Registry Integration]
    end

    subgraph "Engine Abstraction"
        ENGINE[Engine Interface]
        TM34[Tendermint v0.34]
        CB37[CometBFT v0.37]
        CB38[CometBFT v0.38]
        CEL[Celestia Core]
    end

    subgraph "Binary & Storage"
        BINARY[Blockchain App]
        ABCI[ABCI Interface]
        DBS[(Databases<br/>BlockStore, State)]
    end

    KYVE --> COLLECTORS
    STORAGE --> COLLECTORS
    REGISTRY --> SOURCES
    CMD --> ORCH
    ORCH --> BOOTSTRAP
    ORCH --> COLLECTORS
    COLLECTORS --> SOURCES
    ORCH --> ENGINE
    ENGINE --> TM34 & CB37 & CB38 & CEL
    ENGINE --> ABCI
    ABCI <--> BINARY
    BINARY --> DBS

    classDef external fill:#e1f5ff,stroke:#0288d1
    classDef layer fill:#f3e5f5,stroke:#7b1fa2
    classDef engine fill:#fce4ec,stroke:#c2185b
    classDef storage fill:#fff3e0,stroke:#e65100

    class KYVE,STORAGE,REGISTRY external
    class CMD,ORCH,BOOTSTRAP,COLLECTORS,SOURCES layer
    class ENGINE,TM34,CB37,CB38,CEL engine
    class BINARY,ABCI,DBS storage

For a detailed architecture diagram with all components, see assets/architecture.md.

1. CLI Layer (cmd/ksync/commands/)

   • User-facing commands built with Cobra
   • Input validation and configuration parsing
   • User confirmation prompts

2. Orchestration Layer (blocksync/, statesync/, heightsync/)

   • Business logic for each sync strategy
   • Coordinates between data collection, engine, and binary
   • Handles chain upgrade detection
   • Manages periodic backups

3. Data Collection Layer (collectors/, sources/)

   • Retrieves blocks/snapshots from KYVE bundles
   • Validates checksums against KYVE metadata
   • Integrates with source registry for pool/version mapping
   • Supports pagination and retries

4. Engine Abstraction Layer (engines/)

   • Unified interface across Tendermint/CometBFT versions
   • Database management (BlockStore, StateStore)
   • ABCI communication
   • P2P handling for special cases

5. Binary Process Management (utils/process.go)

   • Starts/stops blockchain application
   • Monitors process health
   • Handles graceful shutdown

6. Storage & Persistence

   • RocksDB for block and state storage
   • Managed by consensus engine
   • Direct database writes during sync

ksync/
├── cmd/ksync/
│   ├── main.go                 # Application entry point
│   └── commands/               # CLI command definitions
│       ├── root.go             # Root command & global flags
│       ├── blocksync.go        # Block sync command
│       ├── statesync.go        # State sync command
│       ├── heightsync.go       # Height sync command
│       ├── servesnapshots.go   # Snapshot serving command
│       ├── backup.go           # Backup utility
│       ├── prune.go            # Pruning utility
│       └── info.go             # Info command
│
├── blocksync/                  # Block synchronization logic
│   ├── blocksync.go            # Main orchestration
│   └── executor.go             # Block execution engine
│
├── statesync/                  # State synchronization logic
│   ├── statesync.go            # Main orchestration
│   └── executor.go             # Snapshot application
│
├── heightsync/                 # Combined sync strategy
│   └── heightsync.go           # State-sync → Block-sync
│
├── bootstrap/                  # Genesis & first block handling
│   ├── bootstrap.go            # Bootstrap orchestration
│   └── p2p.go                  # P2P first block sync
│
├── engines/                    # Consensus engine abstractions
│   ├── engine.go               # Engine interface definition
│   ├── tendermint-v34/         # Tendermint v0.34 implementation
│   ├── cometbft-v37/           # CometBFT v0.37 implementation
│   ├── cometbft-v38/           # CometBFT v0.38 implementation (default)
│   └── celestia-core-v34/      # Celestia support
│
├── collectors/                 # Data retrieval components
│   ├── blocks.go               # Block collector
│   ├── snapshots.go            # Snapshot collector
│   └── bundles.go              # Bundle downloader
│
├── sources/                    # Chain registry integration
│   ├── registry.go             # Registry client
│   └── pool.go                 # Pool ID mapping
│
├── servesnapshots/             # Snapshot generation & serving
│   └── servesnapshots.go       # Snapshot server
│
├── server/                     # HTTP servers
│   ├── snapshot.go             # Snapshot API server (:7878)
│   └── rpc.go                  # RPC server (:7777)
│
├── backup/                     # Backup utilities
│   └── backup.go               # Backup creation & rotation
│
├── types/                      # Type definitions & interfaces
│   ├── engine.go               # Engine interface
│   └── config.go               # Configuration types
│
├── utils/                      # Common utilities
│   ├── logger.go               # Structured logging
│   ├── process.go              # Binary process management
│   ├── rpc.go                  # RPC client utilities
│   └── constants.go            # Global constants
│
├── build/                      # Build artifacts
├── assets/                     # Documentation assets
│   ├── architecture.md         # Detailed architecture diagram
│   └── *.png                   # Existing diagrams
│
├── Makefile                    # Build automation
├── go.mod                      # Go module definition
└── README.md                   # This file

• cmd/: Application entry point and CLI commands
• blocksync/: Block-by-block synchronization logic
• statesync/: State snapshot application logic
• engines/: Abstraction layer for different Tendermint/CometBFT versions
• collectors/: Data retrieval from KYVE bundles
• sources/: Chain registry and pool configuration
• utils/: Shared utilities (logging, RPC, process management)

KSYNC retrieves blockchain data from KYVE's decentralized storage network through the following process:

Step-by-Step Data Retrieval:

1. Source Registry Lookup (sources/registry.go)

   • Query KYVE source registry for chain configuration
   • Map chain-id to KYVE pool IDs (blocks pool, state-sync pool)
   • Retrieve engine version mappings per block height

2. Bundle Metadata Retrieval (collectors/bundles.go)

   • Query KYVE API bundles endpoint (e.g., https://api-eu-1.kyve.network/kyve/v1/bundles/0)
   • Fetch bundle metadata including:
     • storage_id: Location on storage provider
     • data_hash: Checksum for validation
     • from_height / to_height: Block range

3. Bundle Download (collectors/bundles.go)

   • Download bundle data from storage providers:
     • Arweave (https://arweave.net/{storage_id})
     • Bundlr, KYVE Storage, Turbo (fallbacks)
   • Decompress bundle (gzip/snappy)

4. Data Validation (collectors/blocks.go, collectors/snapshots.go)

   • Compute checksum of downloaded data
   • Verify against data_hash from KYVE metadata
   • Parse blocks or snapshot chunks from bundle

5. Data Application (See Data Execution below)

KSYNC replaces the built-in Tendermint/CometBFT consensus process and communicates directly with the blockchain application via ABCI:

Block-Sync Execution (blocksync/executor.go):

1. Initialize Engine (engines/)

   engine.OpenDBs()           // Open BlockStore and StateStore
   engine.DoHandshake()        // ABCI handshake with app

2. Block Application Loop

   for height := startHeight; height <= targetHeight; height++ {
       block := collector.GetBlock(height)
       engine.ApplyBlock(runtime, block)  // ABCI: BeginBlock, DeliverTx, EndBlock, Commit
       engine.SaveBlock(block)             // Write to BlockStore DB
   }

3. Chain Upgrade Detection (blocksync/executor.go)

   • Monitor for upgrade heights from source registry
   • Gracefully stop current binary
   • Switch to new engine version
   • Resume sync with new binary

4. Database Persistence

   • BlockStore: Blocks, commit info, seen commits
   • StateStore: Validators, consensus params, application state root

State-Sync Execution (statesync/executor.go):

1. Snapshot Selection

   snapshots := engine.GetSnapshots()  // Available snapshots from app
   targetSnapshot := findNearestSnapshot(targetHeight, snapshots)

2. Snapshot Download & Application

   for chunkIndex := 0; chunkIndex < totalChunks; chunkIndex++ {
       chunk := collector.GetSnapshotChunk(chunkIndex)
       engine.ApplySnapshotChunk(chunkIndex, chunk)  // ABCI: ApplySnapshotChunk
   }
   engine.OfferSnapshot(snapshot)  // ABCI: OfferSnapshot

3. State Restoration

   • Application restores state from snapshot chunks
   • KSYNC writes minimal block metadata to BlockStore
   • Node ready to continue from snapshot height

When genesis file exceeds 100MB, the TSP (Tendermint Socket Protocol) message size limit is exceeded during InitChain. KSYNC handles this by temporarily acting as a P2P peer:

P2P Bootstrap Process (bootstrap/p2p.go):

1. P2P Server Startup

   • KSYNC starts a minimal P2P server on port 26656
   • Advertises itself as a peer with the first block

2. Application Connection

   • Blockchain app connects to KSYNC as a P2P peer
   • KSYNC sends first block via P2P protocol

3. First Block Execution

   • App executes first block (including large genesis)
   • Genesis state committed to StateStore

4. Switch to ABCI Mode

   • P2P server shuts down
   • KSYNC resumes normal ABCI sync for remaining blocks
   • More efficient for bulk block application

KSYNC supports multiple consensus engine versions through a unified interface (types/engine.go):

type Engine interface {
    // Lifecycle
    OpenDBs() error
    CloseDBs() error
    DoHandshake() error

    // Block Operations
    ApplyBlock(runtime, value []byte) error
    SaveBlock(block, seenCommit, valSet []byte, height int64) error
    GetContinuationHeight() int64

    // State-Sync Operations
    GetSnapshots() []byte
    OfferSnapshot(snapshot []byte) error
    ApplySnapshotChunk(index uint32, chunk []byte) (bool, error)

    // Process Management
    StartProxyApp(homePath string) error
    StopProxyApp() error

    // ... more methods
}

Implementations:

• engines/tendermint-v34/: Tendermint v0.34 (KYVE fork)
• engines/cometbft-v37/: CometBFT v0.37
• engines/cometbft-v38/: CometBFT v0.38 (default)
• engines/celestia-core-v34/: Celestia Data Availability

The engine abstraction allows KSYNC to:

• Support chains across different Tendermint/CometBFT versions
• Handle version upgrades automatically
• Maintain consistent sync logic across implementations

After KSYNC completes synchronization:

1. Stop KSYNC: Close databases and proxy connections
2. Start Normal Node: Launch Tendermint/CometBFT with the app
3. Resume Consensus: Node fetches remaining blocks via P2P
4. Enter Live Mode: Participate in consensus or serve queries

This seamless transition makes KSYNC a drop-in replacement for traditional sync methods, with the same end state and no special configuration required.

• Go: Version 1.22.x (enforced by Makefile)
• Make: For build automation
• Git: For version control
• Docker: Optional, for testing

1. Clone Repository

   git clone https://github.com/KYVENetwork/ksync.git
   cd ksync

2. Install Dependencies

   go mod download
   go mod verify

3. Build

   make build
   # Binary created at: build/ksync

4. Run Locally

   ./build/ksync --help

• Follow standard Go formatting: gofmt and goimports
• Use meaningful variable names
• Add comments for exported functions and types
• Keep functions focused and testable

# Run Docker-based integration tests
make test

# Manual testing against testnet
./build/ksync height-sync \
  --binary-path=/path/to/testnet-binary \
  --home=/tmp/test-node \
  --chain-id=testnet-1 \
  --source=kyve \
  --debug

Enable debug logging for detailed output:

./build/ksync block-sync \
  --binary-path=/path/to/binary \
  --home=/path/to/home \
  --debug

Debug logs include:

• Bundle download progress
• ABCI method calls
• Database operations
• Process lifecycle events

To support a new Tendermint/CometBFT version:

1. Create directory: engines/cometbft-vXX/
2. Implement the Engine interface (types/engine.go)
3. Reference existing implementations for patterns
4. Update engine selection logic in commands
5. Add to source registry version mappings

1. Create file: cmd/ksync/commands/yourcommand.go
2. Define Cobra command with flags
3. Implement command logic or call orchestrators
4. Register in root.go

Example:

var yourCmd = &cobra.Command{
    Use:   "your-command",
    Short: "Short description",
    RunE: func(cmd *cobra.Command, args []string) error {
        // Command logic
        return nil
    },
}

func init() {
    rootCmd.AddCommand(yourCmd)
    // Add flags
}

All contributions should follow these branch naming conventions:

• feat/*: New features
• fix/*: Bug fixes
• refactor/*: Code improvements without logic changes
• docs/*: Documentation updates
• test/*: Test additions or modifications

Use Conventional Commits for all commit messages:

feat: add support for CometBFT v0.39
fix: handle empty bundle response gracefully
docs: update installation instructions
test: add unit tests for snapshot collector

1. Fork & Branch: Create a feature branch from main
2. Develop: Make your changes following code style guidelines
3. Test: Ensure your changes don't break existing functionality
4. Commit: Use conventional commits
5. PR: Open a pull request against main branch
6. Review: Address feedback from maintainers
7. Merge: Once approved, your PR will be merged

• Engine Support: Add new Tendermint/CometBFT versions
• Storage Providers: Add new storage provider integrations
• Performance: Optimize bundle downloads, decompression, or database writes
• Testing: Increase test coverage
• Documentation: Improve guides, add examples, fix typos
• Bug Fixes: Address issues from the issue tracker

KSYNC uses Semantic Versioning for releases:

• MAJOR: Incompatible API changes
• MINOR: New backwards-compatible functionality
• PATCH: Backwards-compatible bug fixes

1. Tag the latest commit with new version: git tag vX.Y.Z
2. Push tag: git push origin vX.Y.Z
3. Create GitHub release with tag as title
4. Release automatically published to pkg.go.dev

See Releases for detailed changelog of each version.

Apache 2.0 License. See LICENSE for details.

• Documentation: docs.kyve.network/ksync
• Discord: discord.com/invite/kyve
• Twitter: @KYVENetwork
• Telegram: t.me/kyvenet
• Issues: GitHub Issues

Built with ❤️ by the KYVE Network team