feat(tools): add SearXNG as web search backend#6071
Conversation
Add SearXNG as a self-hosted, open-source federated search option for web_search_tool. Users who run their own SearXNG instance can now set SEARXNG_URL and select the "searxng" backend via `hermes tools`. - Search via SearXNG JSON API with normalized result format - Backend selection, availability check, and config registration - Graceful fallback for extract/crawl (search-only backend) - 14 unit tests covering URL handling, search, dispatch, and config Closes NousResearch#5941
|
Only issue is that hermes agent architecture right now requires a web search tool to use the same backend for web scrape tool. Does SearXNG have the ability to scrape the pages too? Otherwise anyone who selects it will be left without a way to scrape the page |
…selected SearXNG is search-only, so web_extract_tool and web_crawl_tool now gracefully fall through to Firecrawl instead of returning an error. This ensures users who select SearXNG still have full extract/crawl capabilities via the default Firecrawl backend. Addresses review feedback from teknium1.
|
Good catch — thanks for the feedback! SearXNG is search-only and doesn't support extraction or crawling. I've updated the implementation so that when SearXNG is selected as the backend, The fallback also handles the case where Firecrawl isn't configured — it catches the error gracefully and reports it per-URL. Added tests for both the happy path and the error case. See commit b69a1a5. |
Sorry, I don't quite understand why this would be necessary. I think there's no reason for the Hermes agent to directly support Searchng, since it needs to perform search and extract web page content. You might not know that Firecrawl can easily use Searchng as a search backend; just read the documentation, Good luck, buddy. |
Adds SearXNG (https://docs.searxng.org) as a self-hosted, privacy-first web search backend alongside Firecrawl, Tavily, Exa, and Parallel. SearXNG is a meta-search engine that aggregates results from 70+ search engines. No API key needed -- just set SEARXNG_URL to your instance. Changes: - tools/web_tools.py: _get_searxng_url(), _searxng_search(), search dispatch, extract falls back to Firecrawl (SearXNG is search-only) - hermes_cli/tools_config.py: SearXNG provider in web tool picker - hermes_cli/config.py: SEARXNG_URL env var, diagnostics, set command - tests/tools/test_web_tools_searxng.py: 15 tests - optional-skills/research/searxng-search/: agent-guided skill - Docs: configuration.md, environment-variables.md, skills catalogs Based on #6071 by @gnanam1990, #8106 by @cro, #2572 by @bhovig, #2710 and #9961 by @StreamOfRon, #7258 by @coldxiangyu163
|
Merged via PR #11562 which consolidates SearXNG integration from multiple community PRs. Your core search logic, the |
Adds SearXNG (https://docs.searxng.org) as a self-hosted, privacy-first web search backend alongside Firecrawl, Tavily, Exa, and Parallel. SearXNG is a meta-search engine that aggregates results from 70+ search engines. No API key needed -- just set SEARXNG_URL to your instance. Changes: - tools/web_tools.py: _get_searxng_url(), _searxng_search(), search dispatch, extract falls back to Firecrawl (SearXNG is search-only) - hermes_cli/tools_config.py: SearXNG provider in web tool picker - hermes_cli/config.py: SEARXNG_URL env var, diagnostics, set command - tests/tools/test_web_tools_searxng.py: 15 tests - optional-skills/research/searxng-search/: agent-guided skill - Docs: configuration.md, environment-variables.md, skills catalogs Based on NousResearch#6071 by @gnanam1990, NousResearch#8106 by @cro, NousResearch#2572 by @bhovig,
Adds SearXNG (https://docs.searxng.org) as a self-hosted, privacy-first web search backend alongside Firecrawl, Tavily, Exa, and Parallel. SearXNG is a meta-search engine that aggregates results from 70+ search engines. No API key needed -- just set SEARXNG_URL to your instance. Changes: - tools/web_tools.py: _get_searxng_url(), _searxng_search(), search dispatch, extract falls back to Firecrawl (SearXNG is search-only) - hermes_cli/tools_config.py: SearXNG provider in web tool picker - hermes_cli/config.py: SEARXNG_URL env var, diagnostics, set command - tests/tools/test_web_tools_searxng.py: 15 tests - optional-skills/research/searxng-search/: agent-guided skill - Docs: configuration.md, environment-variables.md, skills catalogs Based on NousResearch#6071 by @gnanam1990, NousResearch#8106 by @cro, NousResearch#2572 by @bhovig,
Summary
SEARXNG_URLpointing to their instance and selectsearxngviahermes toolsChanges
tools/web_tools.py_searxng_search()— queries SearXNG JSON API, normalizes results to the standard{success, data: {web: [...]}}format_get_searxng_url()— reads and validatesSEARXNG_URLenv var_get_backend(),_is_backend_available(),_web_requires_env()to includesearxngweb_search_tooland graceful unsupported message inweb_extract_tool(SearXNG is search-only)hermes_cli/tools_config.pyhermes_cli/config.pySEARXNG_URLto env var registry, diagnostics display, andsetcommandtests/tools/test_web_tools_searxng.pyMotivation
Closes #5941. SearXNG is a popular self-hosted metasearch engine that aggregates results from 70+ sources. This gives users a free, privacy-first alternative that requires no API key — just a running instance.
Limitations
SearXNG is a search-only backend. It does not support
web_extract_toolorweb_crawl_tool. When selected, extract returns a clear error message guiding users to use a different backend orweb_search_tool.Test plan
test_auxiliary_named_custom_providers)