Skip to content

fix(models): add xai to _MODELS_DEV_PREFERRED; refresh stale curated IDs (#16699)#16734

Closed
briandevans wants to merge 1 commit into
NousResearch:mainfrom
briandevans:fix/xai-stale-models-dev-preferred-16699
Closed

fix(models): add xai to _MODELS_DEV_PREFERRED; refresh stale curated IDs (#16699)#16734
briandevans wants to merge 1 commit into
NousResearch:mainfrom
briandevans:fix/xai-stale-models-dev-preferred-16699

Conversation

@briandevans

@briandevans briandevans commented Apr 27, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Add "xai" to _MODELS_DEV_PREFERRED so the /model picker merges fresh models.dev entries on top of curated, matching the existing pattern for opencode-go, deepseek, zai, gemini, etc.
  • Replace stale curated entries (grok-4.20-reasoning, grok-4-1-fast-reasoning — both 400 on selection) with current valid IDs taken from the live models.dev xai catalog (filtered to tool_call=True).
  • Add regression tests against provider_model_ids("xai") for both the live-merge and offline-fallback paths.

The bug

_PROVIDER_MODELS["xai"] in hermes_cli/models.py contains two slugs that no longer match xAI's catalog:

"xai": [
    "grok-4.20-reasoning",       # ← xAI no longer accepts this name
    "grok-4-1-fast-reasoning",   # ← never existed in xAI's catalog
],

Reproduction (from the issue, verified against live xAI catalog):

  1. Configure xAI provider via XAI_API_KEY and base https://api.x.ai/v1.

  2. Run hermes, open /model, select xAI, pick either entry.

  3. Result:

    HTTP 400: Unknown Model, please check the model code.

Verified against a fresh https://models.dev/api.json fetch — the xai provider's tool_call=True IDs are date-suffixed: grok-4.20-0309-reasoning, grok-4.20-0309-non-reasoning, plus grok-4-fast, grok-4-fast-non-reasoning, grok-code-fast-1, etc. None of those match the curated list.

The fix

Two changes — each is incomplete on its own:

  1. Add "xai" to _MODELS_DEV_PREFERRED. xAI's catalog drifts (the grok-4.20 family was renamed to date-suffixed IDs upstream). Adding xai to the preferred set causes provider_model_ids and the gateway /model picker to merge fresh models.dev entries on top of the curated list — the exact pattern already in place for opencode-go, opencode-zen, deepseek, zai, gemini, etc. Without this, refreshing the curated list just locks us into another stale snapshot the next time xAI renames a model.

  2. Refresh the curated fallback to current valid IDs. This list is the offline / restricted-network fallback (when models.dev is unreachable, this is the only catalog users see), so it has to contain real current IDs that don't 400 on selection. New entries are pulled from the live models.dev xai catalog filtered to tool_call=True:

    "xai": [
    "grok-4.20-0309-reasoning",
    "grok-4.20-0309-non-reasoning",
    "grok-4-fast",
    "grok-4-fast-non-reasoning",
    "grok-code-fast-1",
],

xAI doesn't fit either documented exclusion (openrouter — too many models to dump; nous — Portal /models endpoint is the source of truth), so adding it is consistent with the existing block comment.

Test plan

  • tests/hermes_cli/test_models_dev_preferred_merge.py — 14 passed (3 new):

  • Adjacent suite: test_model_catalog, test_user_providers_model_switch, test_api_key_providers, test_model_validation, test_model_metadata — 354 passed.

  • Regression guard: ran the 3 new tests against unmodified hermes_cli/models.py (only test file diffed) — all 3 fail with the exact reporter symptom:

    AssertionError: assert 'grok-4.20-0309-reasoning' in ['grok-4.20-reasoning', 'grok-4-1-fast-reasoning']

    After applying the source fix, all 3 pass.

Notes

  • Doc reference to /model grok-4-1-fast-reasoning in website/docs/integrations/providers.md and the test fixture in tests/agent/test_model_metadata.py (substring-matching context-length lookup) are intentionally not touched here — both are independent of the picker behaviour and would broaden scope. Worth a small follow-up.
  • The VERCEL_AI_GATEWAY_MODELS and OpenRouter slug entries (xai/grok-4.20-reasoning, x-ai/grok-4.20) are gateway-specific aliases, not direct xAI API IDs, so they are left alone here.

Related

@briandevans @claude
fix(models): add xai to _MODELS_DEV_PREFERRED; refresh stale curated …
e087ad3 
…IDs (NousResearch#16699)

The curated `_PROVIDER_MODELS["xai"]` list contained two slugs that no
longer match xAI's catalog (`grok-4.20-reasoning`, `grok-4-1-fast-reasoning`).
Selecting either from the `/model` picker yields HTTP 400 "Unknown Model".

Two changes, applied together because each is incomplete on its own:

1. Add `"xai"` to `_MODELS_DEV_PREFERRED`. xAI's catalog drifts (e.g. the
   `grok-4.20` family was renamed to date-suffixed IDs like
   `grok-4.20-0309-reasoning`), so the picker should merge fresh
   models.dev entries on top of the curated list — the same pattern
   already in place for opencode-go, opencode-zen, deepseek, zai, gemini,
   etc. Without this, even refreshing the curated list locks us into
   another stale snapshot the moment xAI renames again.

2. Replace the curated entries with current valid IDs (taken from the
   live models.dev `xai` provider catalog, filtered to `tool_call=True`).
   This is the offline / restricted-network fallback path — when
   models.dev is unreachable, users still see the curated list, so it
   must contain real current IDs that don't 400 on selection.

Regression test asserts:
- xAI is in `_MODELS_DEV_PREFERRED`.
- Fresh models.dev entries surface via `provider_model_ids("xai")` (the
  reported picker bug).
- Offline fallback contains current IDs and excludes the two stale
  entries (`grok-4.20-reasoning`, `grok-4-1-fast-reasoning`).

The new tests fail on clean origin/main with
`assert 'grok-4.20-0309-reasoning' in ['grok-4.20-reasoning', 'grok-4-1-fast-reasoning']`,
which is the exact symptom reporters hit.

Fixes NousResearch#16699

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings April 27, 2026 23:14
Copilot AI reviewed Apr 27, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Updates Hermes’ model catalog behavior for the xai provider to prevent stale curated IDs from breaking /model selection, while keeping an offline fallback.

Changes:

  • Add xai to _MODELS_DEV_PREFERRED so provider_model_ids("xai") and the /model picker merge fresh models.dev entries on top of curated IDs.
  • Refresh _PROVIDER_MODELS["xai"] curated fallback to currently valid xAI model IDs.
  • Add regression tests covering preferred-merge and offline-fallback behavior for xai.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File Description
hermes_cli/models.py Updates xAI curated fallback IDs and enables models.dev-preferred merging for xai.
tests/hermes_cli/test_models_dev_preferred_merge.py Adds tests to prevent regressions in xAI model merging and offline curated fallback behavior.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/cli CLI entry point, hermes_cli/, setup wizard provider/xai xAI (Grok) labels Apr 27, 2026
@briandevans

briandevans commented Apr 28, 2026

Copy link
Copy Markdown
Contributor Author

CI audit — all 16 test job failures are pre-existing baselines on clean origin/main. Zero failures intersect with touched code (hermes_cli/models.py, tests/hermes_cli/test_models_dev_preferred_merge.py).

Touched-code tests pass locally: tests/hermes_cli/test_models_dev_preferred_merge.py — 14/14 passed.

The 16 failures (env-specific WSL detection, tput: No value for $TERM in CI runner, schema regression in prompt_caching category, two flaky 30s timeouts in test_agent_cache / test_web_server::PtyWebSocket, config_env_expansion TypeError: string indices must be integers, tool_arg_coercion assert 'inf' == inf, make_agent_passes_resolved_provider / session_resume kwarg drift, etc.) are the same set documented in the #16666 baseline table — they reproduce identically on every recent unrelated PR (16701, 16666, 16555, …).

@alt-glitch alt-glitch mentioned this pull request Apr 28, 2026
Closed
@briandevans briandevans mentioned this pull request Apr 28, 2026
Closed
3 tasks
@briandevans

briandevans commented Apr 28, 2026

Copy link
Copy Markdown
Contributor Author

Closing — superseded by @vominh1919's #16934 (static refresh, merged 6c78305) and @teknium1's follow-up a83f669 (auto-derive xAI list from $HERMES_HOME/models_dev_cache.json), both for #16699.

The auto-derive approach is strictly better:

Aspect This PR (#16734) Merged a83f669 + 6c78305
Fixes the stale grok-4.20 IDs ✓ ✓
Adds xai to _MODELS_DEV_PREFERRED merge ✓ ✗ (not needed — _xai_curated_models() reads cache directly)
Refreshes hardcoded list ✓ (manual) ✗ (eliminated; cache becomes the source)
Self-heals on next models.dev refresh partial (only via merge) ✓ (no PR needed)
Mirrors existing _codex_curated_models() pattern ✗ ✓

The Teknium follow-up is the cleaner shape — replacing the static list with a cache-backed helper means the next xAI rename won't require any code change at all. Thanks @vominh1919 + @teknium1.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments

Assignees

No one assigned

Labels

comp/cli CLI entry point, hermes_cli/, setup wizard P2 Medium — degraded but workaround exists provider/xai xAI (Grok) type/bug Something isn't working

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Bug] _PROVIDER_MODELS["xai"] is stale — grok-4.20-reasoning renamed; broader fragility from hardcoded provider lists

3 participants

@briandevans @alt-glitch