Java bindings for crawlberg — a high-performance Rust web crawling engine. Uses the Panama FFM API (Java 21+) for zero-overhead native calls with full type safety.

• Same crawler as every binding — one Rust engine behind Python, Node.js, Ruby, Go, Java, .NET, PHP, Elixir, Dart, Kotlin Android, Swift, Zig, WASM, and C FFI.
• Structured scrape output — HTML, Markdown, metadata, links, assets, response headers, and extraction warnings with consistent field names.
• Crawl controls — depth, page limits, concurrency, URL filters, robots/sitemap handling, rate limits, and partial failure reporting.
• Rendering path — optional browser rendering for JavaScript-heavy pages; direct HTTP path for fast static pages.
• Java package — Panama FFM binding for direct native calls.

<dependency>
    <groupId>io.xberg.crawlberg</groupId>
    <artifactId>crawlberg</artifactId>
    <version>1.0.3</version>
</dependency>

The crawlberg plugin is available via the xberg-io/plugins marketplace.

/plugin marketplace add xberg-io/plugins
/plugin install crawlberg@xberg

Works with Claude Code, Codex, Cursor, Gemini CLI, Factory Droid, GitHub Copilot CLI, and opencode. See the marketplace README for harness-specific install instructions.

import java.util.Optional;

import io.xberg.crawlberg.CrawlConfig;
import io.xberg.crawlberg.CrawlEngineHandle;
import io.xberg.crawlberg.CrawlResult;
import io.xberg.crawlberg.Crawlberg;
import io.xberg.crawlberg.ScrapeResult;

public final class BasicUsage {
    private BasicUsage() { }

    public static void main(final String[] args) throws Exception {
        // Simplest case: scrape a single page with default settings.
        CrawlEngineHandle engine = Crawlberg.createEngine();
        ScrapeResult result = Crawlberg.scrape(engine, "https://example.com/");
        System.out.println("Title: " + result.metadata().title());
        System.out.println("Status: " + result.statusCode());
        System.out.println("Links found: " + result.links().size());

        // Crawl from a seed URL, limited to one hop and a handful of pages.
        CrawlConfig config = CrawlConfig.builder()
            .withMaxDepth(Optional.of(1L))
            .withMaxPages(Optional.of(5L))
            .build();
        CrawlEngineHandle crawlEngine = Crawlberg.createEngine(config);
        CrawlResult crawlResult = Crawlberg.crawl(
            crawlEngine,
            "https://en.wikipedia.org/wiki/Web_scraping"
        );
        System.out.println("Pages crawled: " + crawlResult.pages().size());
    }
}

Full API documentation is available at docs.crawlberg.xberg.io.

Key functions:

• create_engine(config?) — Create a crawl engine with optional configuration
• scrape(engine, url) — Scrape a single URL
• crawl(engine, url) — Crawl a website following links
• map_urls(engine, url) — Discover all pages on a site
• batch_scrape(engine, urls) — Scrape multiple URLs concurrently
• batch_crawl(engine, urls) — Crawl multiple seed URLs concurrently

Contributions are welcome! Please see our Contributing Guide for details.

• Xberg — document intelligence: text, tables, metadata from 91+ formats with optional OCR.
• Xberg Enterprise — managed extraction API with SDKs, dashboards, and observability.
• crawlberg — web crawling and scraping with HTML→Markdown and headless-Chrome fallback.
• html-to-markdown — fast, lossless HTML→Markdown engine.
• liter-llm — universal LLM API client with native bindings for 14 languages and 143 providers.
• tree-sitter-language-pack — tree-sitter grammars and code-intelligence primitives.
• alef — the polyglot binding generator that produces every per-language binding across the 5 polyglot repos.
• Discord — community, roadmap, announcements.

This project is licensed under MIT License.

• Documentation
• GitHub Repository
• Issue Tracker