Skip to content

feat: add configurable custom web search backend#10414

Closed
Kolektori wants to merge 1 commit into
NousResearch:mainfrom
Kolektori:feat/custom-web-search-backend
Closed

feat: add configurable custom web search backend#10414
Kolektori wants to merge 1 commit into
NousResearch:mainfrom
Kolektori:feat/custom-web-search-backend

Conversation

@Kolektori

@Kolektori Kolektori commented Apr 15, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

Adds a custom backend for web_search so Hermes can query any JSON search endpoint defined in ~/.hermes/config.yaml

The scope is intentionally narrow:

  • web_search only
  • GET only
  • fixed config-driven mapping into Hermes' existing search result shape

The web_search tool schema stays unchanged. This makes local or self-hosted search APIs usable through the existing tool without adding another provider-specific backend. Examples include 4get and SearXNG, but the implementation stays generic and does not hardcode either provider

Related Issue

Fixes #10284

Type of Change

  • 🐛 Bug fix (non-breaking change that fixes an issue)
  • ✨ New feature (non-breaking change that adds functionality)
  • 🔒 Security fix
  • 📝 Documentation update
  • ✅ Tests (adding or improving test coverage)
  • ♻️ Refactor (no behavior change)
  • 🎯 New skill (bundled or hub)

Changes Made

  • Updated tools/web_tools.py to recognize web.backend: custom
  • Added config validation for web.custom.url_template, results_path, title_field, url_field, description_field, optional headers, and optional timeout
  • Added config-driven search execution for custom JSON backends using URL-encoded %s substitution
  • Normalized custom backend responses into Hermes' existing web_search result format
  • Extended backend availability checks so check_web_api_key() can treat a valid custom config as available without an API key
  • Expanded tests/tools/test_web_tools_config.py to cover request formatting, headers, truncation, malformed item skipping, config validation, and runtime error paths

How to Test

  1. Add a custom backend config to ~/.hermes/config.yaml, for example:
    web:
      backend: custom
      custom:
        url_template: http://127.0.0.1:8080/api/v1/web?s=%s
        results_path: web
        title_field: title
        url_field: url
        description_field: description
        headers:
          Accept: application/json
        timeout: 20
  1. Run the focused test suite:
python -m pytest tests/tools/test_web_tools_config.py -q
  1. Start Hermes with the web tool enabled and run a query that triggers web_search

  2. Confirm the custom endpoint receives a GET request with the encoded query and Hermes returns normalized results with title, url, description, and position

Checklist

Code

  • I've read the Contributing Guide
  • My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
  • I searched for existing PRs to make sure this isn't a duplicate
  • My PR contains only changes related to this fix/feature (no unrelated commits)
  • I've run pytest tests/ -q and all tests pass
  • I've added tests for my changes (required for bug fixes, strongly encouraged for features)
  • I've tested on my platform: macOS 26.3.1

Documentation & Housekeeping

  • I've updated relevant documentation (README, docs/, docstrings) — or N/A. N/A for this change
  • I've updated cli-config.yaml.example if I added/changed config keys — or N/A. N/A because this is a read-path config addition with no default config change
  • I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — or N/A. N/A
  • I've considered cross-platform impact (Windows, macOS) per the compatibility guide — or N/A. This only adds httpx calls and config parsing
  • I've updated tool descriptions/schemas if I changed tool behavior — or N/A. The web_search schema stays unchanged by design

Screenshots / Logs

Focused validation:

$ python -m pytest tests/tools/test_web_tools_config.py -q
63 passed, 11 warnings in 2.12s

Note: I did not run pytest tests/ -q for this PR draft, so that checklist item is intentionally left unchecked

Add a config-driven custom backend for web_search.

The backend uses a GET URL template, expects JSON, and maps results into Hermes' existing search response format. It validates the custom config up front so availability checks and runtime failures stay predictable.

The tests now cover headers, template params, malformed results, limit truncation, and the main config and error paths.
@kshitijk4poor

Copy link
Copy Markdown
Collaborator

Merged via PR #11562 which consolidates SearXNG integration from multiple community PRs. Your self-hosted-first philosophy and documentation approach informed the final implementation. Thank you for the contribution!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature]: Support a configurable custom JSON search backend for web_search

2 participants