Skip to content

magnus919/SlopSearX

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

72 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

SlopSearX

Cloud-native, stateless, AI-agent-first meta search engine. Drop-in SearXNG replacement for the GroktoCrawl stack.

MIT License Docker


SlopSearX is a horizontally scalable, stateless meta search engine designed for AI agent consumption. It replaces SearXNG in the GroktoCrawl stack with:

  • JSON output by default — structured responses designed for programmatic consumption
  • YAML+Markdown native output — structured + readable for AI agent contexts via format=yaml
  • SearXNG-compatible API — drop-in replacement for existing consumers
  • Plugin engine adapters — one file per engine, @register_engine, zero orchestrator changes
  • Category routing — SearXNG-compatible taxonomy with sub-categories and env-var overrides
  • Distributed rate limiting — Valkey-backed sliding windows, correct at 50+ replicas
  • Response caching — Valkey-backed, category-aware TTL, 150x speedup on cache hits
  • OpenMetrics observability/metrics endpoint, per-engine counters + latency + status
  • Stateless, cloud-native — no local volumes, no persistent DB, all replicas interchangeable
  • Hybrid config — env vars for secrets, optional mounted file for engine tuning

API

Endpoint Description
GET /search?q=...&format=json SearXNG-compatible JSON (default)
GET /search?q=...&format=yaml YAML+Markdown agent-native output
GET /search?q=...&categories=science,news Filter by category (OR semantics)
GET /search?q=...&engines=brave,wikipedia Explicit engine selection
GET /health Per-engine health check with status
GET /metrics OpenMetrics for Prometheus scraping
GET /config Categories→engines mapping for runtime discovery

Engines (48)

General / Web

Engine Type Auth Categories
Brave Search API ENGINE_BRAVE_API_KEY general, news, science, images
DuckDuckGo Scrape None general, news
Google Scrape None general, news
Hacker News API None general, news
Reddit API None general, social, reddit:subreddit
Wikipedia API None general, science, reference

Developer / Package Registries

Engine Type Auth Categories
Crates.io API None general, it, reference, packages
Docker Hub API None general, it, reference, packages
GitHub API GITHUB_TOKEN general, reference, github:code, github:issues, github:prs
npm API None general, it, reference, packages
PyPI API None general, it, reference, packages
Repology API None general, it, reference, packages
RubyGems API None general, it, reference, packages
Stack Exchange API Optional general, reference, science, stackexchange:code, stackexchange:serverfault

Science & Research

Engine Type Auth Categories
arXiv API None general, science, reference
HuggingFace API HF_TOKEN (optional) general, science, huggingface:datasets, huggingface:papers
OpenAlex API None general, science, reference
Open Library API None general, books, reference
Semantic Scholar API Optional general, science, reference
UniProt API None general, science, reference, biology, medical
Internet Archive API None reference, web:archive, historical

Medical / Health

Engine Type Auth Categories
ClinicalTrials.gov API None general, medical, health, science
openFDA API None general, medical, health, science, government
PubChem API None general, science, reference, chemistry, medical
PubMed API None general, science, reference, medical, health

Security / Threat Intelligence

Engine Type Auth Categories
AbuseIPDB API ENGINE_ABUSEIPDB_API_KEY security, threat-intel
AlienVault OTX API ENGINE_OTX_API_KEY security, threat-intel
Censys API ENGINE_CENSYS_API_KEY + _API_SECRET it, security
CRT.sh API None it, security
CVE Program (MITRE) API None it, security
DeHashed API ENGINE_DEHASHED_API_KEY security, threat-intel
Exploit-DB Scrape None security, exploit
FIRST EPSS API None security, threat-intel
GreyNoise API ENGINE_GREYNOISE_API_KEY (optional) security, threat-intel
Have I Been Pwned API ENGINE_HIBP_API_KEY security, reference
IntelX API ENGINE_INTELX_API_KEY security, threat-intel
MITRE ATT&CK API None security, reference
NVD (NIST) API ENGINE_NVD_API_KEY (optional) it, security
Shodan API ENGINE_SHODAN_API_KEY it, security
URLhaus API None security, threat-intel
VirusTotal API ENGINE_VIRUSTOTAL_API_KEY security, malware
VulnCheck API ENGINE_VULNCHECK_API_KEY security, threat-intel

Finance / Economics

Engine Type Auth Categories
FRED API ENGINE_FRED_API_KEY general, finance, reference, economics
SEC EDGAR API None general, finance, reference

Media & Entertainment

Engine Type Auth Categories
MusicBrainz API None general, music, reference
TMDB API ENGINE_TMDB_API_KEY general, movies, entertainment

Geography / GIS

Engine Type Auth Categories
Nominatim (OSM) API None general, geography, reference

Legal

Engine Type Auth Categories
Oyez (SCOTUS) API None general, reference, legal

Adding a new engine: See docs/ENGINE_ADAPTERS.md for the full adapter reference — contract rules, data types, lifecycle hooks, and the category system.

Quick Start

Pre-built Docker images are available from GitHub Container Registry. Builds run automatically on every push to main (latest, unstable) and on version tags (stable, X, X.Y, X.Y.Z).

# Pull and run with Valkey for caching and rate limiting
docker run -d --name valkey valkey/valkey:8-alpine
docker run -d --name slopsearx -p 8080:8080 \
  -e VALKEY_URL=redis://valkey:6379/0 \
  --link valkey \
  ghcr.io/magnus919/slopsearx:latest

# Try it
curl 'http://localhost:8080/search?q=hello+world&format=json'

License

MIT — see LICENSE.

About

Cloud-native, stateless, AI-agent-first meta search engine. Drop-in SearXNG replacement built for the GroktoCrawl stack.

Resources

License

Contributing

Stars

Watchers

Forks

Packages

 
 
 

Contributors