Skip to content
Xberg.dev

Crawlberg

crawlberg

High-performance web crawling and scraping with a Rust core and generated language packages. Always-on HTML→Markdown conversion, structured metadata, Chromiumoxide and native browser backends, REST/MCP servers, a CLI, and a Docker image.

Quick Start Installation Features Join our Community

Why Crawlberg

  • Flexible Crawling

BFS, DFS, BestFirst, and Adaptive traversal with concurrent fetching, streaming events, and batch crawl/scrape.

  • Always-On Markdown v0.1

Every fetched page is converted to Markdown. MarkdownResult.citations is a boolean; call generate_citations when you need the reference list.

  • Rich Metadata

PageMetadata carries Open Graph, Twitter Card, Dublin Core, article fields, JSON-LD, links, images, feeds, favicons, and hreflang.

  • Browser Fallback v0.3

Use the Chromiumoxide CDP backend or the native browser backend. WAF signals can trigger browser escalation; the vendor is surfaced when the classifier can identify it.

  • MCP & REST Servers

Built-in MCP server for AI agents, REST API for service deployments, both gated behind cargo features.

  • Rust + Generated Packages v0.3

Rust uses the core crate. Generated packages cover Python, TypeScript, WebAssembly, Go, Java, Kotlin Android, C#, Ruby, PHP, Elixir, Dart, Swift, Zig, and C FFI.

→ See all features

Language Support

Language Package Docs
Rust cargo add crawlberg API Reference
Python pip install crawlberg API Reference
TypeScript / Node npm install @xberg-io/crawlberg API Reference
WebAssembly npm install @xberg-io/crawlberg-wasm API Reference
Go go get github.com/xberg-io/crawlberg/packages/go API Reference
Java Maven Central io.xberg.crawlberg:crawlberg API Reference
Kotlin (Android) Maven Central io.xberg.crawlberg:crawlberg-android API Reference
C# dotnet add package Crawlberg API Reference
Ruby gem install crawlberg API Reference
PHP composer require xberg-io/crawlberg API Reference
Elixir {:crawlberg, "~> 0.3.0"} API Reference
Dart / Flutter dart pub add crawlberg API Reference
Swift Swift Package Manager API Reference
Zig zig fetch --save from GitHub API Reference
C (FFI) Shared library + header API Reference
CLI cargo install crawlberg-cli CLI Guide
Docker ghcr.io/xberg-io/crawlberg Docker Guide

Choosing between TypeScript packages

@xberg-io/crawlberg — Native NAPI-RS bindings. Use for Node.js servers and CLI tools. Full feature set including the browser fallback.

@xberg-io/crawlberg-wasm — Pure WebAssembly. Use for browsers, Cloudflare Workers, Deno, Bun, and serverless. No native browser backend, REST server, MCP server, or native streaming crawl wrappers.

Quick Example

src/main.rs
use crawlberg::{CrawlConfig, ContentConfig, create_engine, crawl};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let config = CrawlConfig {
        max_depth: Some(2),
        max_pages: Some(50),
        content: ContentConfig::default(),
        ..Default::default()
    };
    let engine = create_engine(Some(config))?;

    let result = crawl(&engine, "https://example.com").await?;
    for page in &result.pages {
        let title = page.metadata.title.as_deref().unwrap_or("(no title)");
        println!("{} — {}", page.url, title);
    }
    Ok(())
}
main.py
import asyncio
from crawlberg import CrawlConfig, create_engine, crawl

async def main():
    engine = create_engine(CrawlConfig(max_depth=2, max_pages=50))
    result = await crawl(engine, "https://example.com")
    for page in result.pages:
        print(f"{page.url} — {page.metadata.title or '(no title)'}")

asyncio.run(main())
index.ts
import { createEngine, crawl } from "@xberg-io/crawlberg";

const engine = createEngine({ maxDepth: 2, maxPages: 50 });
const result = await crawl(engine, "https://example.com");

for (const page of result.pages) {
  console.log(`${page.url} — ${page.metadata.title ?? "(no title)"}`);
}

Part of Xberg.dev

Document intelligence: text, tables, metadata from 91+ formats with optional OCR.

Managed extraction API with SDKs, dashboards, and observability.

Web crawling and scraping with HTML→Markdown and headless-Chrome fallback.

Fast, lossless HTML→Markdown engine.

Universal LLM API client with native bindings for 14 languages and 143 providers.

Tree-sitter grammars and code-intelligence primitives.

The polyglot binding generator that produces every per-language binding across the 5 polyglot repos.

Explore the Docs

  • Get Started

Install Crawlberg and run your first crawl in under five minutes.

Quick Start

  • Guides

Crawling, scraping, URL discovery, browser automation, WARC output, and deployment.

All Guides

  • Concepts

Public surface, data flow, the binding matrix, feature gates, and the content-extraction pipeline.

Architecture

  • Reference

Per-language API docs, the configuration schema, type catalogue, and error matrix.

References

  • CLI & Servers

The crawlberg CLI, REST API server, and MCP server for AI agents.

CLI Usage

  • Features

Complete feature breakdown: crawl strategies, metadata extraction, browser backends, WARC, MCP, REST.

Features

Getting Help

Edit this page on GitHub

Xberg.dev