Full Changelog: v1.0.0...v1.0.3

Zig

Add to your build.zig.zon:

.dependencies = .{
    .crawlberg-zig = .{\n        .url = \"https://github.com/xberg-io/crawlberg/releases/download/v1.0.3/crawlberg-zig-v1.0.3.tar.gz\",\n        .hash = \"crawlberg-1.0.3-IoY1CkdhfyCxKQPpGFoOCuzdTXjqaSIesDARHZdi0SOI\",\n    },\n},\n```\n

Full Changelog: v1.0.0...v1.0.2

Zig

Add to your build.zig.zon:

.dependencies = .{
    .crawlberg-zig = .{\n        .url = \"https://github.com/xberg-io/crawlberg/releases/download/v1.0.2/crawlberg-zig-v1.0.2.tar.gz\",\n        .hash = \"crawlberg-1.0.2-IoY1Ct1FfyDZy5oEyJFp5M1oOKpNRxAUjADXr0m1AWUg\",\n    },\n},\n```\n

Maintenance release. Version-only bump synced across all manifests; .gitignore ai-rulez block reorganized.

Zig

Add to your build.zig.zon:

.dependencies = .{
    .crawlberg-zig = .{\n        .url = \"https://github.com/xberg-io/crawlberg/releases/download/v1.0.1/crawlberg-zig-v1.0.1.tar.gz\",\n        .hash = \"crawlberg-1.0.1-IoY1Cr6ZeSBqxfzr_BQbn4tvi3vVnP77cuinQNm4A8kT\",\n    },\n},\n```\n

crawlberg 1.0.0 — first stable release. Web crawling/scraping with HTML→Markdown and headless-Chrome fallback. Promotes 1.0.0-rc.2.

Zig

Add to your build.zig.zon:

.dependencies = .{
    .crawlberg-zig = .{\n        .url = \"https://github.com/xberg-io/crawlberg/releases/download/v1.0.0/crawlberg-zig-v1.0.0.tar.gz\",\n        .hash = \"crawlberg-1.0.0-IoY1CtjXeSBM9BTl8x7MWqGD93oEQgl4U8cCLhtaox_J\",\n    },\n},\n```\n

Release candidate 1.0.0-rc.2. Regenerated with alef 0.29.3; NuGet PackageId XbergIo.Crawlberg; manifest version sync.

Zig

Add to your build.zig.zon:

.dependencies = .{
    .crawlberg-zig = .{\n        .url = \"https://github.com/xberg-io/crawlberg/releases/download/v1.0.0-rc.2/crawlberg-zig-v1.0.0-rc.2.tar.gz\",\n        .hash = \"crawlberg-1.0.0-rc.2-IoY1CmHWeSB-IdGdPVcoIZhsoPqLfsXncNiiaYmru5fU\",\n    },\n},\n```\n

Full Changelog: v0.3.0-rc.87...v1.0.0-rc.1

Zig

Add to your build.zig.zon:

.dependencies = .{
    .crawlberg-zig = .{\n        .url = \"https://github.com/xberg-io/crawlberg/releases/download/v1.0.0-rc.1/crawlberg-zig-v1.0.0-rc.1.tar.gz\",\n        .hash = \"crawlberg-1.0.0-rc.1-IoY1CjXBeCC9qwH8LNHnppfvMhq1PZ0qpRJ3ci7sMsVT\",\n    },\n},\n```\n

First stable release. kreuzcrawl ships a Rust core with active bindings for
Python, TypeScript/Node, Ruby, PHP, Go, Java/JNI, C#, Elixir, WebAssembly,
Dart, Kotlin/Android, Swift, Zig, and C FFI, plus a CLI, an HTTP API, and an
MCP server.

Added

• Tiered dispatch engine. The crawl engine chains HTTP → Bypass → Browser
  tiers driven by per-attempt signals rather than a single bypass
  short-circuit. Public kreuzcrawl::types::dispatch surface: Tier,
  EscalationStrategy, EscalationReason, AttemptOutcome, RetryDirective,
  RetryPolicy, WafSignal, WafClassifier, DomainStatePort,
  DomainRecommendation, EscalationBudget, and DispatchProfile (dispatch
  enums are #[non_exhaustive]). CrawlConfig::builder() and
  DispatchProfile::builder() provide fluent construction.
• WAF detection. A TOML fingerprint corpus (rules/waf_fingerprints.toml,
  34 fingerprints) with an Aho-Corasick matcher, TomlClassifier::watch()
  hot-reload (debounced, atomic ArcSwap, Kubernetes ConfigMap-safe), and
  EwmaDomainState for per-domain block-rate tracking that promotes/demotes
  the starting tier.
• SSRF defense. New kreuzcrawl::net::ssrf module — SsrfPolicy,
  HostMatcher (Exact/Suffix/Cidr), SsrfError, and async
  validate_url. CrawlConfig::ssrf plus builder methods
  allow_private_networks(bool) and ssrf_allowlist_host(HostMatcher);
  CrawlError::SsrfPolicyViolation. Exposed as a settable DTO (deny_private,
  max_redirects) across every binding.
• Browser pool injection. BrowserPool/BrowserPoolConfig and
  NativeBrowserExecutor/NativeBrowserExecutorConfig are public;
  CrawlEngineBuilder::with_browser_pool / with_native_executor and
  CrawlEngineHandle::from_engine let consumers construct and warm() a pool
  once and reuse it across all crawl jobs.
• Public substrate parsers. kreuzcrawl::robots and kreuzcrawl::sitemap
  are public (parse_robots_txt, is_path_allowed, RobotsRules,
  parse_sitemap_xml, parse_sitemap_index, is_sitemap_index) — usable
  without spinning up the engine.
• Pluggable proxy rotation. ProxyProvider trait + StaticProxyProvider
  baseline, wired into the reqwest fetch path via
  CrawlEngineBuilder::with_proxy_provider; called per request and taking
  precedence over the static CrawlConfig::proxy value.
• CLI. batch-scrape, batch-crawl, download, citations, and
  version subcommands, bringing the CLI to 1:1 with the core and MCP
  surfaces.
• MCP server. Tools are 1:1 with the CLI (batch_crawl,
  generate_citations, …), each declaring read_only/destructive/
  open_world safety annotations, and are served over both stdio and rmcp
  Streamable HTTP at /mcp when the binary is built with the api + mcp
  features.
• Observability. OpenTelemetry counters
  kreuzcrawl_waf_fingerprint_matches_total and
  kreuzcrawl_escalations_total, plus property tests, cargo-fuzz targets, and
  Criterion benchmarks covering the WAF subsystem.

Changed

• Memory-bounded streaming crawl. crawl_stream / batch_crawl_stream
  move each page into its CrawlEvent::Page and drop it instead of
  accumulating every page, bounding peak memory on large crawls (≈2.5 GB →
  ≈20 MB working set). crawl()'s batch result is unchanged.
• Dispatch model. CrawlError::WafBlocked is now a struct variant
  ({ vendor, message }); DomainStatePort moved to an observation model
  (recommend/observe); SimpleRetryPolicy's off-by-one is fixed
  (max_retries=3 yields 3 retries); #[non_exhaustive] added to
  CrawlError, NetworkErrorKind, and the dispatch enums so future variants
  are non-breaking.
• Asset downloads route through http_fetch, so every file fetch is
  subject to the SSRF policy.

Fixed

• Crawl loop materializes downloaded documents. The download_documents
  flag was previously honored only by single-page scrape(); the crawl loop
  now builds CrawlPageResult.downloaded_document for linked PDFs/DOCX via a
  shared helper instead of fetching, flagging, and discarding the bytes.
• SSRF rollout hardening. Follow-up fixes to the SSRF refactor: redirect
  final_url is tracked again (per-hop re-validation moved into
  follow_redirects), within-batch URL dedup no longer races, crawl
  child-depth is incremented (restoring max_depth and include_paths
  semantics), and CrawlConfig JSON deserialization honors
  KREUZCRAWL_ALLOW_PRIVATE_NETWORK through a SsrfPolicy::from_env serde
  default. Each is covered by a regression test.
• MCP server exposed zero tools. The handler was missing rmcp's
  #[tool_handler], so tools/list/tools/call returned an empty list over
  both stdio and HTTP; it now delegates to the generated tool router.

Security

• SSRF defense, enabled by default. scrape(), crawl(),
  batch_crawl(), sitemap fetch, robots.txt fetch, and asset download refuse
  URLs resolving to loopback (127.0.0.0/8), RFC1918 private networks,
  link-local (169.254.0.0/16), cloud metadata (0.0.0.0/8), multicast
  (224.0.0.0/4), IPv6 ULA (fc00::/7), IPv6 link-local (fe80::/10), IPv6
  multicast (ff00::/8), or any non-http(s) scheme. Includes DNS-rebinding
  mitigation (every resolved IP must pass the policy), redirect-chain
  re-validation (bounded by ssrf.max_redirects, default 5), and
  link-enqueue validation with bounded concurrency. Opt out via
  KREUZCRAWL_ALLOW_PRIVATE_NETWORK=1 or
  CrawlConfig::allow_private_networks(true).

Build

• Bindings, facades, READMEs, docs, stubs, and e2e suites are generated by
  alef (pinned at 0.26.6) across all 14 language targets.
• Publish-pipeline hardening: a native per-arch Docker matrix that drops QEMU
  emulation, Flutter-free Dart native builds for pub.dev, Swift artifactbundle
  checksum injection and Apple system-framework linking, and
  lockfile-preserving source publishes for the Elixir NIF, PHP extension, and
  Ruby gem.

Zig

Add to your build.zig.zon:

.dependencies = .{
    .kreuzcrawl-zig = .{\n        .url = \"https://github.com/kreuzberg-dev/kreuzcrawl/releases/download/v0.3.0/kreuzcrawl-zig-v0.3.0.tar.gz\",\n        .hash = \"kreuzcrawl-0.3.0-l-oqNoO5eCDhkMWBHtd-su6btmha_K-hk0D2x5Ooq7B9\",\n    },\n},\n```\n

Fixed

• Docker: multi-arch publish no longer times out on the arm64 leg. Publish Docker Images built linux/amd64,linux/arm64 in a single job with the arm64 image compiled under QEMU emulation, which routinely ran the full Rust build right up against the 120-minute job timeout (rc.86 squeaked through at 103 min; rc.87 tipped over and was cancelled, leaving no 0.3.0-rc.87 image on GHCR). The job now builds each architecture natively in a matrix (amd64 on ubuntu-latest, arm64 on ubuntu-24.04-arm), pushes each by digest, and merges them into a single manifest list with docker buildx imagetools create — matching the canonical infra pattern and removing QEMU entirely. Per-arch GHA cache scopes also fix a cache-from/cache-to scope mismatch that previously defeated cache reuse. (.github/workflows/publish-docker.yaml)

Build

• Regenerated all bindings against alef 0.26.6 (pin bumped from 0.26.3). Folds in the accumulated 0.26.4→0.26.6 codegen changes (including the pyo3 Python trait-callback reliability fix and the Dart mirror/opaque/from_json declaration fix).

Zig

Add to your build.zig.zon:

.dependencies = .{
    .kreuzcrawl-zig = .{\n        .url = \"https://github.com/kreuzberg-dev/kreuzcrawl/releases/download/v0.3.0-rc.88/kreuzcrawl-zig-v0.3.0-rc.88.tar.gz\",\n        .hash = \"kreuzcrawl-0.3.0-rc.88-l-oqNv3EeCARU_oSw7q0Ey1p26mbnSkJ7NBo160AuQcR\",\n    },\n},\n```\n

Added

• MCP: Streamable HTTP transport at /mcp. When the binary is built with both the api and mcp features (the CLI is), the REST API server now also exposes the MCP server over rmcp's Streamable HTTP transport, mounted outside the REST middleware stack so request-timeout/compression layers don't break MCP SSE. Each tool also declares its safety annotations (read_only/destructive/open_world hints). (crates/kreuzcrawl/src/mcp, crates/kreuzcrawl/src/api/router.rs)

Fixed

• MCP: the server exposed zero tools. The impl ServerHandler was missing rmcp's #[tool_handler], so tools/list/tools/call silently returned an empty list over both stdio and HTTP — every MCP client through rc.86 saw a tool-less server. The handler now delegates to the generated tool router. (crates/kreuzcrawl/src/mcp/server.rs)
• Swift: RustBridgeC.h no longer regresses to the placeholder. rc.86 shipped the typedef-only placeholder header (reverting the populated header from rc.85), so every SwiftPM consumer of the source package failed to compile with thousands of cannot find '__swift_bridge__$…' in scope errors. The alef 0.26.3 bump below preserves an already-populated header across alef all --clean, and the umbrella header (994 swift-bridge C declarations) is repopulated here.

Build

• Regenerated all bindings against alef 0.26.3 (pin bumped from 0.25.60). Beyond the Swift header preservation fix above, this folds in the accumulated 0.25.60→0.26.3 codegen changes.

Zig

Add to your build.zig.zon:

.dependencies = .{
    .kreuzcrawl-zig = .{\n        .url = \"https://github.com/kreuzberg-dev/kreuzcrawl/releases/download/v0.3.0-rc.87/kreuzcrawl-zig-v0.3.0-rc.87.tar.gz\",\n        .hash = \"kreuzcrawl-0.3.0-rc.87-l-oqNlnHeSAlSjd6Z9WxFtrTMIzeT11cfH85hL9n9W5q\",\n    },\n},\n```\n

Build

• Regenerated all bindings against alef 0.25.60 (pin bumped from 0.25.59). Notable codegen fixes: the generated Rust e2e/test-app common.rs now resolves the mock-server binary via env!("CARGO_BIN_EXE_mock-server") instead of a hardcoded target/release/mock-server path (so cargo test debug builds spawn it correctly), and the Kotlin Android MockServerListener now parses the MOCK_SERVERS env map on the preset path so per-fixture mockServer.<id> lookups resolve under the registry-mode test runner.

Zig

Add to your build.zig.zon:

.dependencies = .{
    .kreuzcrawl-zig = .{\n        .url = \"https://github.com/kreuzberg-dev/kreuzcrawl/releases/download/v0.3.0-rc.86/kreuzcrawl-zig-v0.3.0-rc.86.tar.gz\",\n        .hash = \"kreuzcrawl-0.3.0-rc.86-l-oqNnHJeSBVbHIEt2GQxgeXjxKLiAHHXSOKdNd5taQj\",\n    },\n},\n```\n