Unobtanium Search 🔍

A stand alone and independent search engine with its own crawler.

Try it | Documentation | Libraries

What is in here ?

This repository contains:

• The unobtanium crawler and summarizer in crawler/
• The unobtanium frontend in viewer/
• The application library in lib-unobtanium

How does it work? The quick overview:

Unobtanium works in multiple stages:

• Crawling from the web to the crawler database (unobtanium-crawler crawl)
• Summarizing from the crawler database to the summary database (unobtanium-crawler summarize)
• Serving the frontend from the summary database (unobtanium-viewer)

Note: If you want to try the SQL-based token index (suing token:"<query>" in the search field) you have to generate it using unobtanium-crawler regenerate-token-index.

Also see the documentation on components.

If you want to try hosting unobtanium yourself please get in touch, there are some underdocumented gotchas in there.

Dependencies

To be able to compile unobtanium you need a working rust compiler along with development files for openssl and sqlite.

Under Alpine:

apk add rust cargo pkgconf openssl-dev sqlite-dev

Under Debian trixie:

apt install rustc cargo pkg-config libssl-dev libsqlite3-dev

For older debian versions the rust compiler is too old, please consider upgrading to trixie or newer.

Help! It doesn't compile

Unobtanium is developed together with its components and while usually these are released to crates.io, during development unobtanium will point to the repositories as colocated folders (see the Cargo.toml file) as often the APIs will be expanded on demand to fit the needed usecases.

There are two solutions:

• Check out and use a tagged version, these have their dependencies guaranteed on crates.io
• Clone those repositories so that they're next to this repository will make the code compile (Usually).

License

The core unobtanium code in this repository is licensed under AGPL-3.0-only.

Copyright:

• 2023 - 2025 Slatian