Skip to content

python

Directory actions

More options

Directory actions

More options

Latest commit

 

History

History

python

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
crawlberg
crawlberg
 
 
LICENSE
LICENSE
 
 
README.md
README.md
 
 
pyproject.toml
pyproject.toml
 
 

README.md

Xberg

crawlberg

Python bindings for crawlberg — a high-performance Rust web crawling engine for structured data extraction. Powered by PyO3, with async support, full metadata extraction, and Markdown conversion. Wheels published for Linux, macOS, and Windows.

What This Package Provides

  • Same crawler as every binding — one Rust engine behind Python, Node.js, Ruby, Go, Java, .NET, PHP, Elixir, Dart, Kotlin Android, Swift, Zig, WASM, and C FFI.
  • Structured scrape output — HTML, Markdown, metadata, links, assets, response headers, and extraction warnings with consistent field names.
  • Crawl controls — depth, page limits, concurrency, URL filters, robots/sitemap handling, rate limits, and partial failure reporting.
  • Rendering path — optional browser rendering for JavaScript-heavy pages; direct HTTP path for fast static pages.
  • Python package — PyO3 bindings for async crawler workflows and data pipelines.

Installation

pip install crawlberg

Agent plugin

The crawlberg plugin is available via the xberg-io/plugins marketplace.

/plugin marketplace add xberg-io/plugins
/plugin install crawlberg@xberg

Works with Claude Code, Codex, Cursor, Gemini CLI, Factory Droid, GitHub Copilot CLI, and opencode. See the marketplace README for harness-specific install instructions.

Quick Start

import asyncio

from crawlberg import CrawlConfig, create_engine, crawl, scrape

async def main() -> None:
    # Simplest case: scrape a single page with default settings.
    engine = create_engine()
    result = await scrape(engine, "https://example.com/")
    print(f"Title: {result.metadata.title}")
    print(f"Status: {result.status_code}")
    print(f"Links found: {len(result.links)}")

    # Crawl from a seed URL, limited to one hop and a handful of pages.
    crawl_engine = create_engine(CrawlConfig(max_depth=1, max_pages=5))
    crawl_result = await crawl(crawl_engine, "https://en.wikipedia.org/wiki/Web_scraping")
    print(f"Pages crawled: {len(crawl_result.pages)}")

if __name__ == "__main__":
    asyncio.run(main())

API Reference

Full API documentation is available at docs.crawlberg.xberg.io.

Key functions:

  • create_engine(config?) — Create a crawl engine with optional configuration
  • scrape(engine, url) — Scrape a single URL
  • crawl(engine, url) — Crawl a website following links
  • map_urls(engine, url) — Discover all pages on a site
  • batch_scrape(engine, urls) — Scrape multiple URLs concurrently
  • batch_crawl(engine, urls) — Crawl multiple seed URLs concurrently

Contributing

Contributions are welcome! Please see our Contributing Guide for details.

Part of Xberg.dev

  • Xberg — document intelligence: text, tables, metadata from 91+ formats with optional OCR.
  • Xberg Enterprise — managed extraction API with SDKs, dashboards, and observability.
  • crawlberg — web crawling and scraping with HTML→Markdown and headless-Chrome fallback.
  • html-to-markdown — fast, lossless HTML→Markdown engine.
  • liter-llm — universal LLM API client with native bindings for 14 languages and 143 providers.
  • tree-sitter-language-pack — tree-sitter grammars and code-intelligence primitives.
  • alef — the polyglot binding generator that produces every per-language binding across the 5 polyglot repos.
  • Discord — community, roadmap, announcements.

License

This project is licensed under MIT License.

Links