fix(tools): strip slash-containing enum values for xAI Responses#27907
Closed
gbarany wants to merge 5 commits into
Closed
fix(tools): strip slash-containing enum values for xAI Responses#27907gbarany wants to merge 5 commits into
gbarany wants to merge 5 commits into
Conversation
added 2 commits
May 18, 2026 10:40
xAI's /v1/responses schema validator rejects any tool-parameter string enum value containing a / character (e.g. "application/json", "*/*", "text/plain"). It returns HTTP 200 with a single SSE "event: error" frame whose message is the opaque "Invalid arguments passed to the model.", then closes the stream, aborting every agent turn. The Brave Search MCP triggers this out of the box: its brave_llm_context and brave_place_search tools advertise an HTTP-header "accept" parameter with enum ["application/json", "*/*"]. Other Responses-compatible backends (OpenAI Codex) accept these enums, so the strip is gated by a new is_xai_responses parameter plumbed from the existing detection in conversation_loop and codex_runtime through the transport into _preflight_codex_api_kwargs. The new strip_xai_incompatible_enum_values helper lives next to strip_pattern_and_format in tools/schema_sanitizer.py and emits a parallel INFO log line when it fires.
… gating - 6 unit tests for the new sanitizer covering OpenAI and Responses tool formats, mixed safe+unsafe enums, nested schemas, and idempotence on clean input. - 2 integration tests on _preflight_codex_api_kwargs verifying the strip runs only when is_xai_responses=True (and is a no-op for openai-codex and other Responses-compatible backends).
added 3 commits
May 18, 2026 11:13
Round of fixes from the multi-agent PR review on NousResearch#27907: * Deep-copy normalized_tools before stripping in the xAI branch of _preflight_codex_api_kwargs so the in-place mutation can't leak back through the aliased `parameters` reference into the caller's upstream tool registry (would have affected non-xAI turns served by the same Python process). * Wrap the strip call in try/except + logger.warning so a malformed schema can't crash the request mid-preflight — the request falls back to sending the original tools and xAI's real error surfaces instead of a Hermes-side traceback. * Extract `agent_uses_xai_responses(agent)` in agent.codex_responses_adapter so the predicate stays in sync across conversation_loop.py and codex_runtime.py. Future provider keys land in one place. * Drop the `getattr(agent, '_base_url_hostname', None)` softening — a missing attribute is a real initialization bug and silencing it would hide the very mis-route this PR is meant to fix. * Log non-list (malformed) `enum` values at DEBUG instead of silently ignoring them, and demote the stripped-count log from INFO to DEBUG (fires every turn on affected setups; not a notable event). * Tighten docstring: pin the mutation/return contract and replace the brittle Brave tool-name references with the underlying header-enum shape that triggers the rejection. Tests: * 7 new unit tests covering caller-deepcopy contract, list-reference return, array `items` with enum, anyOf union, mixed-type enum values, all-stripped enum with sibling `default` survival, and the malformed (non-list) enum path. * 4 new helper tests covering `agent_uses_xai_responses` (provider match, hostname-only match, negative providers, missing-attrs tolerance). * 3 transport-wrapper tests pinning that ResponsesApiTransport.preflight_kwargs forwards `is_xai_responses` and that the caller's tool registry is preserved across the call. All previously-passing tests on the touched modules still pass (563 of 577; the 14 unrelated failures in tests/agent/test_anthropic_adapter.py also fail on upstream main from this checkout — they read ambient Claude Code OAuth tokens from the host keychain).
Per the silent-failure-hunter review: getattr fallbacks would silently route a mis-initialized agent (one that skipped agent_init and so never got provider / _base_url_hostname set) as non-xAI, re-triggering the exact opaque-SSE-error bug this PR exists to prevent. Bare attribute access surfaces AttributeError loudly so the real init bug bubbles up. Matches the inline-expression pattern already used in agent/chat_completion_helpers.py:287. Test now asserts pytest.raises(AttributeError) on a Bare() agent stub instead of the previous silent-False behavior.
…boundary Move strip_xai_incompatible_enum_values out of the preflight path and into agent/chat_completion_helpers.build_api_kwargs alongside the existing strip_pattern_and_format call. Both xAI-specific tool-schema transforms now live in the same is_xai_responses block that already existed there, on a deep-copied tool list so the shared registry stays valid for non-xAI turns. Removes the plumbing introduced for the earlier review round: * agent_uses_xai_responses helper in agent/codex_responses_adapter.py * is_xai_responses kwarg on ResponsesApiTransport.preflight_kwargs * is_xai_responses kwarg on _preflight_codex_api_kwargs * derivation + pass-through in conversation_loop.py and codex_runtime.py Preflight is once again provider-neutral validation. xAI compatibility is now owned end-to-end by the request builder, which is the only place that also knows whether the call is bound for xAI. Strip semantics: if any string value in an enum contains "/", drop the entire enum constraint rather than narrowing the list. Preserving the non-slash subset would silently change the model-visible value space for that property; dropping the constraint keeps the type-string typing so free-form values still validate. Tests now exercise the behavior end-to-end through agent._build_api_kwargs instead of probing preflight. The transport-level test class is renamed to TestCodexPreflightProviderNeutral and pins that preflight does not apply provider policy.
3 tasks
teknium1
pushed a commit
that referenced
this pull request
May 29, 2026
…27907) The xAI tool-schema sanitizers (strip_slash_enum, strip_pattern_and_format) mutate their input in place — that's their documented contract. The two call sites (chat_completion_helpers.build_api_kwargs and the auxiliary client) were passing agent.tools straight through, so the first xAI request would permanently strip slash-containing enum constraints and pattern/format keywords from the per-agent tool registry. Effect: any subsequent non-xAI call from the same agent (auxiliary task routed to Anthropic, OpenRouter fallback, mid-session model switch) saw the already-stripped schema with no way for the user to notice from their config. Fix: deepcopy tools_for_api before sanitizing at both call sites. The slash-enum bug itself (xAI 400ing on enums with '/') was fixed earlier by #32443 (Nami4D) — that PR landed the strip but used the sanitizers directly without copying. This salvages #27907's correctness contribution (the deepcopy) while skipping its redundant parallel sanitizer (strip_xai_incompatible_enum_values is functionally equivalent to the existing strip_slash_enum) and its preflight- neutrality argument (we chose model-gated preflight in #32443). 3 new tests in tests/run_agent/test_run_agent_codex_responses.py: - strips_slash_enum_from_outgoing_request — outgoing kwargs has no slash-containing enum values (functional contract preserved). - does_not_mutate_agent_tools — headline #27907 regression. Snapshot agent.tools before build_api_kwargs, assert it survives intact after. Pre-fix this assertion would have caught the mutation. - is_idempotent_across_repeated_calls — three xAI requests in a row each strip cleanly AND don't progressively erode the source schema. 344/344 across tests/agent/test_auxiliary_client.py, tests/agent/transports/test_codex_transport.py, tests/run_agent/test_run_agent_codex_responses.py, and tests/tools/test_schema_sanitizer.py. Co-authored-by: Gabor Barany <barany.gabor@gmail.com>
Contributor
|
Closing — the deepcopy correctness fix from your PR was salvaged into #34416 (#34416) with your authorship preserved ( What landed:
What didn't land, and why:
Thank you for the careful empirical work mapping which xAI enum values get rejected ( |
KKT-OPT
pushed a commit
to KKT-OPT/hermes-agent
that referenced
this pull request
May 31, 2026
…ousResearch#27907) The xAI tool-schema sanitizers (strip_slash_enum, strip_pattern_and_format) mutate their input in place — that's their documented contract. The two call sites (chat_completion_helpers.build_api_kwargs and the auxiliary client) were passing agent.tools straight through, so the first xAI request would permanently strip slash-containing enum constraints and pattern/format keywords from the per-agent tool registry. Effect: any subsequent non-xAI call from the same agent (auxiliary task routed to Anthropic, OpenRouter fallback, mid-session model switch) saw the already-stripped schema with no way for the user to notice from their config. Fix: deepcopy tools_for_api before sanitizing at both call sites. The slash-enum bug itself (xAI 400ing on enums with '/') was fixed earlier by NousResearch#32443 (Nami4D) — that PR landed the strip but used the sanitizers directly without copying. This salvages NousResearch#27907's correctness contribution (the deepcopy) while skipping its redundant parallel sanitizer (strip_xai_incompatible_enum_values is functionally equivalent to the existing strip_slash_enum) and its preflight- neutrality argument (we chose model-gated preflight in NousResearch#32443). 3 new tests in tests/run_agent/test_run_agent_codex_responses.py: - strips_slash_enum_from_outgoing_request — outgoing kwargs has no slash-containing enum values (functional contract preserved). - does_not_mutate_agent_tools — headline NousResearch#27907 regression. Snapshot agent.tools before build_api_kwargs, assert it survives intact after. Pre-fix this assertion would have caught the mutation. - is_idempotent_across_repeated_calls — three xAI requests in a row each strip cleanly AND don't progressively erode the source schema. 344/344 across tests/agent/test_auxiliary_client.py, tests/agent/transports/test_codex_transport.py, tests/run_agent/test_run_agent_codex_responses.py, and tests/tools/test_schema_sanitizer.py. Co-authored-by: Gabor Barany <barany.gabor@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Fixes a 100%-broken state for any user on the xAI provider (
xai/xai-oauth) who also has the Brave Search MCP enabled. Brave'sbrave_llm_contextandbrave_place_searchtools advertise an HTTP-headeracceptparameter as{"type": "string", "enum": ["application/json", "*/*"]}. xAI's/v1/responsesschema validator rejects any stringenumvalue containing/, replies withHTTP 200plus a single SSEevent: errorframe whosemessageis the opaque"Invalid arguments passed to the model.", then closes the stream. Hermes retries three times and aborts the turn.The fix is a new
strip_xai_incompatible_enum_values()sanitizer intools/schema_sanitizer.pythat drops anyenumconstraint containing a slash-bearing string value. It's wired intoagent/chat_completion_helpers.build_api_kwargsalongside the existingstrip_pattern_and_formatcall (same xAI gate, same deep-copy-then-strip pattern). Other Responses-compatible backends are not touched.Type of Change
Changes Made
tools/schema_sanitizer.py— newstrip_xai_incompatible_enum_values(tools)helper. Walks both OpenAI-format and Responses-format tool entries. If any stringenumvalue contains/, drops the entireenumconstraint (rather than narrowing the list, which would silently change the model-visible value space). Logs malformed (non-list) enums at DEBUG. Emits a sibling DEBUG log when the strip fires.agent/chat_completion_helpers.py— adds the new sanitizer to the existing xAI tool-schema transform pass, on a deep-copied tool list so the shared registry stays valid for non-xAI turns served by the same process.tests/tools/test_schema_sanitizer.py— 13 new unit tests (OpenAI format, Responses format, drops whole enum on mixed values, nested schemas, no-op when clean, empty list, arrayitemsenum,anyOfunion, mixed non-string members, all-stripped with siblingdefault, malformed enum, caller-deepcopy contract, returned list-reference identity).tests/run_agent/test_run_agent_codex_responses.py— 3 integration tests verifying_build_api_kwargsstrips for xAI agents, leaves slash enums untouched for non-xAI, and that preflight remains provider-neutral.tests/agent/transports/test_codex_transport.py— renamed test class toTestCodexPreflightProviderNeutral, pinning that preflight does not apply provider policy and does not mutate caller-owned schemas.How to Test
Reproduce against xAI directly without Hermes:
Drop
*/*(or any value containing/) from the enum and the same call streamsresponse.completed.In Hermes itself: enable the Brave Search MCP and pick
xai-oauth+grok-4.3— every turn fails onmainbefore this PR and succeeds after.Confirmed empirically (any string enum value containing
/triggers the rejection):"application/json""*/*""text/plain""text/*""a/b""foo*bar""foo","bar"Test runs
Checklist
Code
fix(tools):,refactor(tools):,test(tools):)nousresearch/hermes-agent:mainDocumentation & Housekeeping
cli-config.yaml.example— N/A (no new config keys)CONTRIBUTING.mdorAGENTS.md— N/A (no workflow/architecture change)Notes for reviewers
chat_completion_helpers.build_api_kwargs) alongside the existingstrip_pattern_and_formatcall — same gate, same deep-copy, same try/except. Preflight remains provider-neutral validation.["fast", "fast/precise", "precise"]), the entireenumconstraint is dropped rather than narrowed to["fast", "precise"]. Narrowing would silently change the accepted value space for that property in ways a user couldn't see from their schema — dropping the constraint keepstype: stringtyping, so free-form values still validate, but doesn't pretend the model never knew about the missing values.event: errorframe'smessagefield in the user-visible error along with the offending tool name when available. Today the openai SDK collapses the frame into the generic_StreamErrorEventsummary so neither Hermes nor the user can tell which tool is bad.Screenshots / Logs
Before (on
nousresearch/hermes-agent:main, freshhello):Captured wire response (first SSE frame from xAI):
After the patch — same request body succeeds, with the sanitizer logging: