Skip to content

/model picker uses static Copilot list, ignores live catalog — newer/tier-gated models invisible #22990

@idanshimon

Description

@idanshimon

Summary

The /model picker (and hermes model) shows a hardcoded list of 17 Copilot models from _PROVIDER_MODELS["copilot"] in hermes_cli/models.py. It does not call the live GitHub Copilot catalog (/models on the configured base_url), so any model the user's account is entitled to but that isn't in the static list is invisible from the picker.

This matters because the Copilot model catalog is tier-dependent: different subscription/SKU tiers (Pro, Business, Enterprise) and different base URLs (api.githubcopilot.com, api.business.githubcopilot.com, api.enterprise.githubcopilot.com) return different model sets, and the catalog evolves continuously as GitHub ships new models without a Hermes release.

Concrete examples missing from the picker

Confirmed returned by _fetch_github_models() against a configured Copilot base URL but absent from _PROVIDER_MODELS["copilot"]:

  • claude-opus-4.7, claude-opus-4.7-high, claude-opus-4.7-xhigh
  • claude-opus-4.5, claude-opus-4.6
  • gpt-5.5, gpt-5.2

These are valid, working Copilot models — the API serves them at runtime when set as model.default, but they cannot be selected via the picker.

Why the static list can't fix this long-term

Catalog membership is a function of the user's Copilot tier × base URL × GitHub's current rollout. Examples of tier/endpoint differences a user can hit today:

  • A Copilot Pro user on api.githubcopilot.com sees one catalog.
  • A Copilot Business org on api.business.githubcopilot.com may see additional models (e.g. higher-effort Claude variants) that aren't on Pro.
  • A Copilot Enterprise org on api.enterprise.githubcopilot.com may see yet another superset (long-context variants, additional reasoning tiers).

Hermes already supports configuring base_url per provider, so users on all three tiers actively run the agent — but the picker shows the same 17 models to all of them, regardless of what their account actually has.

Root cause

hermes_cli/model_switch.py::list_authenticated_providers() has two sections:

  • Section 1 (lines ~1200-1264): handles auth_type == "api_key" providers via PROVIDER_TO_MODELS_DEV. Uses the static curated list.
  • Section 2 (lines ~1266-1377): handles HERMES_OVERLAYS. Has a Copilot special-case at line 1346 that calls provider_model_ids("copilot")_fetch_github_models() → live catalog.

Copilot has auth_type == "api_key" (env vars COPILOT_GITHUB_TOKEN, GH_TOKEN, GITHUB_TOKEN), so Section 1 always wins and adds copilot to seen_slugs. Section 2's dedup then skips it — the live-catalog code path is dead for Copilot.

Reproduction

>>> from hermes_cli.models import provider_model_ids
>>> ids = provider_model_ids("copilot")        # live catalog
>>> any("opus-4.7" in m for m in ids)
True

>>> from hermes_cli.model_switch import list_authenticated_providers
>>> picker = next(p for p in list_authenticated_providers(current_provider="copilot")
...               if p["slug"] == "copilot")
>>> picker["total_models"]
17
>>> any("opus-4.7" in m for m in picker["models"])
False                                          # bug

Suggested fix

Special-case copilot / copilot-acp in Section 1 to use the live catalog the same way Section 2 already does:

# inside the section-1 loop, after has_creds check:
if hermes_id in {"copilot", "copilot-acp"}:
    model_ids = provider_model_ids(hermes_id)   # falls back to static on network error
else:
    model_ids = curated.get(hermes_id, [])
    if hermes_id in _MODELS_DEV_PREFERRED:
        model_ids = _merge_with_models_dev(hermes_id, model_ids)

provider_model_ids("copilot") already gracefully falls back to the static curated list on network failure, so offline behavior is preserved.

Environment

  • Hermes version: current main (behind by ~145 commits at time of writing — happy to verify on tip)
  • Provider: copilot
  • Python: 3.9
  • macOS 26.4

Happy to send a PR if a maintainer confirms the proposed direction.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Medium — degraded but workaround existscomp/cliCLI entry point, hermes_cli/, setup wizardprovider/copilotGitHub Copilot (ACP + Chat)sweeper:implemented-on-mainSweeper: behavior already present on current maintype/bugSomething isn't working

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions