feat(web): Add support for SearXNG search and native backend to reduce dependency on third party APIs#2710
Closed
StreamOfRon wants to merge 4 commits into
Closed
Conversation
6aad651 to
a08fa56
Compare
3d00a58 to
4c03d43
Compare
…iguration - Split web.backend into web.search.backend (with URL) and web.extract.backend - Add SearXNG as search backend with URL template support - Add 'native' extract backend for direct HTTP requests with html-to-markdown - Implement _get_search_backend() and _get_extract_backend() precedence functions - Add v10→v11 config migration with interactive SearXNG URL prompt - Update tools_config.py with SearXNG and Native HTTP provider options - Add html-to-markdown as core dependency - Update check_web_api_key() to handle native backend (no API key required) - All tests passing (2 pre-existing failures unrelated to this change) Plan: .opencode/plans/1774319582434-stellar-lagoon.md
Test coverage includes: - _get_search_backend() — 29 tests covering tool-specific config, generic fallback, env detection - _get_extract_backend() — 21 tests with same precedence logic minus searxng - _searxng_search() — 8 tests for URL template substitution, error handling, limits, API keys - _native_extract() — 4 structural tests (async function verification) - check_web_api_key() — 5 tests for native backend (no key) and SearXNG support - Backend compatibility — 2 tests for _get_backend() alias - Config migration v10→v11 — 7 tests for automatic splitting, SearXNG prompting - Config schema — 6 tests for DEFAULT_CONFIG web keys - Environment variables — 5 tests for SEARXNG_URL and SEARXNG_API_KEY Total: 72 new tests, all passing. Full suite: 6152 passed, 1 pre-existing failure. Plan: .opencode/plans/1774319582434-stellar-lagoon.md
Remove: - .opencode/plans/ (plan documents) - .opencode/web-config-v2-todos.md (todo tracking) - docs/superpowers/plans/ (documentation) - test_results.txt (test output)
93b37fd to
7addb38
Compare
kshitijk4poor
pushed a commit
that referenced
this pull request
Apr 17, 2026
Adds SearXNG (https://docs.searxng.org) as a self-hosted, privacy-first web search backend alongside Firecrawl, Tavily, Exa, and Parallel. SearXNG is a meta-search engine that aggregates results from 70+ search engines. No API key needed -- just set SEARXNG_URL to your instance. Changes: - tools/web_tools.py: _get_searxng_url(), _searxng_search(), search dispatch, extract falls back to Firecrawl (SearXNG is search-only) - hermes_cli/tools_config.py: SearXNG provider in web tool picker - hermes_cli/config.py: SEARXNG_URL env var, diagnostics, set command - tests/tools/test_web_tools_searxng.py: 15 tests - optional-skills/research/searxng-search/: agent-guided skill - Docs: configuration.md, environment-variables.md, skills catalogs Based on #6071 by @gnanam1990, #8106 by @cro, #2572 by @bhovig, #2710 and #9961 by @StreamOfRon, #7258 by @coldxiangyu163
5 tasks
Collaborator
|
Merged via PR #11562 which consolidates SearXNG integration from multiple community PRs. Your skill and documentation structure ideas informed the optional skill shipped with the final implementation. Thank you for the contribution! |
This was referenced May 3, 2026
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Splits the monolithic
web.backendconfig key into separateweb.search.backendandweb.extract.backendkeys, allowing independent control of which provider handles search vs. content extraction. Also adds two new backends: SearXNG (self-hosted search, no API key) and Native HTTP (direct extraction with html-to-markdown, no API key).Includes several bug fixes to the LLM post-processing pipeline discovered during local testing.
Changes
New backends
web.search.urlin config.yaml orSEARXNG_URLenv var. Requires no API key.Configuration schema (v10 → v11)
Selection priority for each tool call:
web.search.backend/web.extract.backend)web.backend)Existing configs are automatically migrated from v10 to v11. The
web.backendkey continues to work as a fallback, so no breaking changes.Bug fixes (discovered during local testing)
_call_summarizer_llmpermanent error short-circuit — errors likeno endpoints available,404, and guardrail rejections (e.g. from OpenRouter) now bail out immediately instead of retrying for ~62 seconds (2+4+8+16+32s backoff).Primary model fallback — on a permanent aux LLM error, retries once with the user's configured primary model (
_read_main_model()) before giving up. Ensures summarization works even when the default auxiliary model is unavailable or restricted.process_single_resultsafety net — any unhandled summarization exception now falls back to returning raw extracted content rather than failing the entire tool call. The user gets unsummarized-but-valid content instead of an error.web_extract_toolouter exception logging — promoted fromDEBUGtoWARNINGso tool-level failures are visible in~/.hermes/logs/errors.log._native_extractSSL error surfacing — SSL failures now log atWARNINGand return a clear diagnostic message per-URL instead of silently returning empty content.Files changed
tools/web_tools.py_get_search_backend(),_get_extract_backend(),_searxng_search(),_native_extract(), updatedcheck_web_api_key(); bug fixes to_call_summarizer_llm,process_single_result,web_extract_toolhermes_cli/config.pyDEFAULT_CONFIG, v10→v11 migration, SearXNG env var metadatahermes_cli/tools_config.pypyproject.tomlhtml-to-markdowndependencytests/tools/test_web_config_v2.pytests/hermes_cli/test_config_v2_migration.pywebsite/docs/user-guide/configuration.mdwebsite/docs/user-guide/features/tools.mdTesting
72 new unit tests covering: backend selection precedence, SearXNG URL encoding, native extraction,
check_web_api_key()with all backend combinations, config migration, and schema validation. Full suite passing with no regressions.