Skip to content

fix(transports/codex): gate xAI reasoning.effort per model#23106

Closed
zakame wants to merge 1 commit into
NousResearch:mainfrom
zakame:fix/23088-xai-reasoning-effort-gating
Closed

fix(transports/codex): gate xAI reasoning.effort per model#23106
zakame wants to merge 1 commit into
NousResearch:mainfrom
zakame:fix/23088-xai-reasoning-effort-gating

Conversation

@zakame

@zakame zakame commented May 10, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

  • xAI's Responses API rejects `reasoning.effort` on most Grok models with HTTP 400; only `grok-4.3` and `grok-4.20-multi-agent` accept it per xAI's reasoning capability docs. The codex transport was sending it unconditionally for every xAI model, which was breaking primary-model calls on `grok-4-1-fast` and forcing fallback on every request.
  • Add a per-model allowlist helper `grok_supports_reasoning_effort()` in `agent/model_metadata.py` using exact-or-hyphen-delimited matching (with optional OpenRouter `x-ai/` prefix strip) and gate `kwargs["reasoning"]` behind it. For all xAI models with reasoning enabled, `kwargs["include"]` is always set to `[]` (upstream already removed `reasoning.encrypted_content` requests — the OAuth/SuperGrok surface rejects replayed encrypted items on turn 2+). Unsupported models silently skip the `reasoning` key.
  • Tighten the allowlist: use `name == prefix or name.startswith(prefix + "-")` so `grok-4.30-pro` cannot leak into the `grok-4.3` entry; remove `grok-3-mini` (no longer in official xAI reasoning docs).
  • Existing happy-path behavior on `grok-4.3` and `grok-4.20-multi-agent` is preserved; no other provider paths (OpenAI, Anthropic, OpenRouter, Kimi, GitHub Models, LM Studio) are touched.

Fixes #23088.

Test plan

  • `uv run pytest tests/agent/transports/test_codex_transport.py tests/agent/test_model_metadata.py -q` — 162 passed across `TestCodexBuildKwargs` and the new `TestCodexXaiReasoningEffortGating` class.
  • `uv run pytest tests/agent/ tests/providers/ -q` — 1041 passed; 2 pre-existing failures in unrelated Bedrock/Anthropic adapter tests.
  • Adversarial check: stash the fix, regression tests fail (including the literal `grok-4-1-fast` repro); restore, all green.
  • Boundary tests pin: `grok-4.30` does NOT match `grok-4.3`; `x-ai/grok-4.3` (OpenRouter) does match; mixed-case and empty/None model handled correctly.
  • Live smoke test against `grok-4-1-fast` with a real `XAI_API_KEY` — to be run by the issue reporter after this lands.

@alt-glitch alt-glitch added type/bug Something isn't working P1 High — major feature broken, no workaround provider/xai xAI (Grok) comp/agent Core agent loop, run_agent.py, prompt builder labels May 10, 2026
@zakame zakame mentioned this pull request May 10, 2026
Closed
1 task
@zakame

zakame commented May 10, 2026

Copy link
Copy Markdown
Contributor Author

Live test looks ok! 🚀

hermes@hermes-agent:~/.hermes/hermes-agent$ hermes -m grok-4-1-fast --provider xai -z "is this thing on? show the Grok model that made this."
Yep, mic check 1-2-1-2. This Hermes Agent response is powered by grok-4-1-fast from xAI. What's up?

This was referenced May 10, 2026
Closed
Open
@liuhao1024 liuhao1024 mentioned this pull request May 10, 2026
Open
12 tasks
@zakame zakame force-pushed the fix/23088-xai-reasoning-effort-gating branch from 6be6600 to 88274a9 Compare May 11, 2026 04:10
@zakame

zakame commented May 11, 2026

Copy link
Copy Markdown
Contributor Author

Looks superseded by #23435 but I've rebased with it:

  • retain rejection of grok-3-mini and grok-3-mini-fast to pass reasoning.effort against, as its undocumented behavior and likely an oversight on xAI's part
  • retain gate of both reasoning effort and encrypted content only to the models documented to support it
  • keep the comprehensive testsuite and doc updates

@zakame zakame force-pushed the fix/23088-xai-reasoning-effort-gating branch from 88274a9 to e9becd1 Compare May 12, 2026 06:35
@zakame

zakame commented May 12, 2026

Copy link
Copy Markdown
Contributor Author

Now removed debug logging and consolidated tests

@zakame zakame force-pushed the fix/23088-xai-reasoning-effort-gating branch from e9becd1 to 1a13fb4 Compare May 12, 2026 11:53
@zakame
fix(transports/codex): gate xAI reasoning.effort per model
f0ddde3 
Fixes NousResearch#23088.

xAI's Responses API rejects `reasoning.effort` on most Grok models with
HTTP 400; only `grok-4.3` and `grok-4.20-multi-agent` accept it per the
current official capability table. The codex transport was sending it
(plus `reasoning.encrypted_content` include) unconditionally for every
xAI model with reasoning enabled, breaking primary-model calls on
`grok-4-1-fast` and similar.

Gate both `kwargs["reasoning"]` and `kwargs["include"]` behind
`grok_supports_reasoning_effort()` from `agent/model_metadata` — the
canonical per-model allowlist using exact-or-hyphen-delimited prefix
matching (with optional OpenRouter `x-ai/` strip). Unsupported models
silently skip both keys with no logging.

Also tighten `grok_supports_reasoning_effort` itself:
- Switch from bare `startswith(prefix)` to
  `name == prefix or name.startswith(prefix + "-")` so that
  `grok-4.30-pro` cannot leak into the `grok-4.3` allowlist entry.
- Remove `grok-3-mini` from `_GROK_EFFORT_CAPABLE_PREFIXES` — no longer
  listed in the official xAI reasoning docs; conservative omission avoids
  future 400s.
- Fix stale docstring example (`grok-3-mini` → `grok-4.3`).

Existing happy-path behavior on `grok-4.3` and `grok-4.20-multi-agent`
is preserved; no other provider paths are touched.

Tests: 56 tests across `TestCodexBuildKwargs` and the new
`TestCodexXaiReasoningEffortGating`; 8 duplicate/overlapping tests
removed so each scenario is covered exactly once. Aggregator prefix test
updated to use `grok-4.3`; `grok-3-mini` test updated to assert effort
is now dropped.
@zakame zakame force-pushed the fix/23088-xai-reasoning-effort-gating branch from 1a13fb4 to f0ddde3 Compare May 19, 2026 05:14
@zakame

zakame commented May 19, 2026

Copy link
Copy Markdown
Contributor Author

Rebased - reasoning.encrypted_content no longer included at all.

@teknium1

teknium1 commented May 24, 2026

Copy link
Copy Markdown
Contributor

Closing — your underlying bug (#23088) is already fixed on main by PR #23435, which landed the same approach you'd outlined: gate reasoning.effort behind a per-model allowlist (grok_supports_reasoning_effort() in agent/model_metadata.py) and skip the kwarg in agent/transports/codex.py when it returns False.

Thanks for filing the bug AND submitting a fix on the same day — the issue you wrote was load-bearing for the eventual resolution. Sorry the parallel PR landed first; your patch shape was equivalent.

@teknium1 teknium1 closed this May 24, 2026
@zakame

zakame commented May 25, 2026

Copy link
Copy Markdown
Contributor Author

@teknium1 no problem, I was going to close this one out anyway since the last rebase already shipped only the extra tests for #23435 now as well as the model helper.

Glad the issue guidance helped! 🎉 🚀

@zakame zakame deleted the fix/23088-xai-reasoning-effort-gating branch May 25, 2026 06:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

comp/agent Core agent loop, run_agent.py, prompt builder P1 High — major feature broken, no workaround provider/xai xAI (Grok) type/bug Something isn't working

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Bug]: xAI grok-4-1-fast returns HTTP 400 — "does not support parameter reasoningEffort"

3 participants

@zakame @teknium1 @alt-glitch