test(plugins): cover xAI web search plugin in bundled-plugin contract by briandevans · Pull Request #29056 · NousResearch/hermes-agent

briandevans · 2026-05-20T03:15:32Z

What does this PR do?

The xAI web search plugin landed in commit a0c0312 ("feat(web): add xAI Web Search provider plugin") as the 8th bundled web provider, but tests/plugins/web/test_web_search_provider_plugins.py still expected exactly the original seven (brave-free, ddgs, exa, firecrawl, parallel, searxng, tavily). The test_all_seven_plugins_present_in_registry assertion fails on every post-merge run of main, e.g. https://github.com/NousResearch/hermes-agent/actions/runs/26138063691/job/76877433972 and https://github.com/NousResearch/hermes-agent/actions/runs/26138014706/job/76877288367.

This PR extends the bundled-plugin contract to cover xai:

Rename test_all_seven_plugins_present_in_registry → test_all_bundled_plugins_present_in_registry (number-neutral so the next plugin addition doesn't bit-rot the name) and add "xai" to the expected sorted list.
Add ("xai", True, False, False) to the capability-flags parametrize (search-only — no extract or crawl, matching plugins/web/xai/provider.py's supports_* impl).
Add "xai" to the name / display-name and setup-schema parametrize lists so every bundled plugin has the same contract coverage.

Related Issue

None — this aligns the test contract to the same-day landing of a0c031299.

Type of Change

🐛 Bug fix (non-breaking change that fixes an issue)
✨ New feature (non-breaking change that adds functionality)
🔒 Security fix
📝 Documentation update
✅ Tests (adding or improving test coverage)
♻️ Refactor (no behavior change)
🎯 New skill (bundled or hub)

Changes Made

tests/plugins/web/test_web_search_provider_plugins.py — extend the four TestBundledPluginsRegister tests (presence, capability flags, name/display, setup schema) to cover the xAI plugin.

How to Test

Reproduce on main:

uv run --with pytest --with pytest-xdist --with pytest-asyncio --with pytest-timeout python3 -m pytest tests/plugins/web/test_web_search_provider_plugins.py::TestBundledPluginsRegister::test_all_seven_plugins_present_in_registry -v

→ fails with assert ['brave-free', ..., 'xai'] == ['brave-free', ..., 'tavily'].

With this PR applied:

uv run --with pytest --with pytest-xdist --with pytest-asyncio --with pytest-timeout python3 -m pytest tests/plugins/web/test_web_search_provider_plugins.py -v

→ all 48 tests pass.

Checklist

Code

I've read the Contributing Guide
My commit messages follow Conventional Commits (test(plugins):)
I searched for existing PRs to make sure this isn't a duplicate
My PR contains only changes related to this fix (no unrelated commits)
I've run focused tests for the touched code and all pass (48/48)
I've added tests for my changes (the change is the test alignment)
I've tested on my platform: macOS 15.x

Documentation & Housekeeping

I've updated relevant documentation — N/A
I've updated cli-config.yaml.example if I added/changed config keys — N/A
I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — N/A
I've considered cross-platform impact — N/A (pure-Python test parametrize change)
I've updated tool descriptions/schemas if I changed tool behavior — N/A

Related / Positioning

Open PR feat(web): add Gemini Google Search Grounding provider #26021 (feat(web): add Gemini Google Search Grounding provider) also touches this file to add a 9th provider (gemini). Cleanly stackable: if feat(web): add Gemini Google Search Grounding provider #26021 lands first, the rebase here just adds "xai" next to "gemini" in the same lists; if this PR lands first, feat(web): add Gemini Google Search Grounding provider #26021 adds "gemini" next to the now-present "xai". Either order produces the same final state.

Audited siblings: the only related entry-point for "bundled web plugin contract" is this one file. tools/web_tools.py already references "xai" in _is_backend_available() (added in a0c0312); agent/web_search_registry.py discovers plugins dynamically. No widening needed beyond this test file.

The xAI web search plugin landed in commit a0c0312 ("feat(web): add xAI Web Search provider plugin") as the 8th bundled web provider, but test_web_search_provider_plugins.py still expected exactly the original seven (brave-free, ddgs, exa, firecrawl, parallel, searxng, tavily), causing test_all_seven_plugins_present_in_registry to fail on every post-merge CI run. Extend the contract to cover xai: - Rename test_all_seven_plugins_present_in_registry → test_all_bundled_plugins_present_in_registry (number-neutral so the next plugin addition doesn't bit-rot the test name) and add "xai" to the expected sorted list. - Add ("xai", True, False, False) to the capability-flags parametrize (xAI is search-only — no extract or crawl, matching its provider impl). - Add "xai" to the name / display-name and setup-schema parametrize lists so every bundled plugin has the same contract coverage. Verified by running tests/plugins/web/test_web_search_provider_plugins.py: all 48 tests pass (was 47 passing + 1 failing pre-fix).

Copilot

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Updates the web search provider plugin test suite to account for the newly bundled xai provider and to remove outdated “seven plugins” wording.

Changes:

Renamed test/docstrings to refer to “bundled” plugins instead of “seven”.
Added xai to the expected registry provider list and related parametrized tests.
Added capability-flag expectations for xai (search-only).

Comments suppressed due to low confidence (1)

tests/plugins/web/test_web_search_provider_plugins.py:1

The bundled plugin name list is now duplicated in multiple places (expected registry list + two parametrizations). This increases the chance of future drift when adding/removing providers. Consider extracting a single module-level constant (e.g., BUNDLED_PLUGIN_NAMES) and reusing it across these tests (and deriving the expected list from it if appropriate) so updates are one-touch.

"""Plugin-side tests for the web search provider migration (PR #25182).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

    @pytest.mark.parametrize(
        "plugin_name",
-        ["brave-free", "ddgs", "searxng", "exa", "parallel", "tavily", "firecrawl"],
+        ["brave-free", "ddgs", "searxng", "exa", "parallel", "tavily", "firecrawl", "xai"],


    @pytest.mark.parametrize(
        "plugin_name",
-        ["brave-free", "ddgs", "searxng", "exa", "parallel", "tavily", "firecrawl"],
+        ["brave-free", "ddgs", "searxng", "exa", "parallel", "tavily", "firecrawl", "xai"],


briandevans · 2026-05-20T04:07:45Z

CI audit — the one failure on this PR's run is tests/hermes_cli/test_update_hangup_protection.py::TestInstallHangupProtection::test_wraps_stdout_and_stderr_with_mirror, which is unrelated to the xAI plugin contract scope of this PR. It's a baseline flake from a module-reload ordering issue in xdist (a captured _UpdateOutputStream reference at the test file's top-level becomes stale after tests/hermes_cli/test_skills_subparser.py does del sys.modules['hermes_cli.main']).

Reproduces on every recent open PR's CI on clean origin/main (e.g. #29048, #29049, #28991). The xAI plugin contract assertion this PR fixes (test_all_seven_plugins_present_in_registry) is no longer in the failure set, which is the intended effect of this change. The hangup_protection flake will need its own follow-up.

briandevans · 2026-05-20T17:11:56Z

Closing — superseded by @camaragon's #29224, which is the better fix here:

Covers both CI baseline failures in one PR (xAI plugin contract and the test_wraps_stdout_and_stderr_with_mirror xdist double-import race). This PR only handles the first.
Adds XAI_API_KEY to _clear_web_env so the new plugin's is_available() is properly isolated across tests — a coverage gap I missed.
CI is green on theirs.

Thanks @camaragon!

Copilot AI review requested due to automatic review settings May 20, 2026 03:15

briandevans mentioned this pull request May 20, 2026

fix(gateway): recognize Windows drive-letter paths in extract_local_files() bare-path uploads #28991

Closed

19 tasks

Copilot AI reviewed May 20, 2026

View reviewed changes

alt-glitch added type/test Test coverage or test infrastructure P3 Low — cosmetic, nice to have comp/plugins Plugin system and bundled plugins tool/web Web search and extraction provider/xai xAI (Grok) labels May 20, 2026

alt-glitch mentioned this pull request May 20, 2026

fix(tests): update plugin registry count for xAI + fix hangup test xdist flake #29382

Closed

briandevans closed this May 20, 2026

briandevans mentioned this pull request May 21, 2026

test(plugins): add xai to bundled web search plugin test suite #29587

Closed

19 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test(plugins): cover xAI web search plugin in bundled-plugin contract#29056

test(plugins): cover xAI web search plugin in bundled-plugin contract#29056
briandevans wants to merge 1 commit into
NousResearch:mainfrom
briandevans:fix/test-web-plugins-include-xai

briandevans commented May 20, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

briandevans commented May 20, 2026

Uh oh!

briandevans commented May 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

briandevans commented May 20, 2026

What does this PR do?

Related Issue

Type of Change

Changes Made

How to Test

Checklist

Code

Documentation & Housekeeping

Related / Positioning

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

briandevans commented May 20, 2026

Uh oh!

briandevans commented May 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants