Skip to content

feat: model capability pre-flight validation#5443

Open
kshitijk4poor wants to merge 1 commit into
NousResearch:mainfrom
kshitijk4poor:feat/model-capability-preflight
Open

feat: model capability pre-flight validation#5443
kshitijk4poor wants to merge 1 commit into
NousResearch:mainfrom
kshitijk4poor:feat/model-capability-preflight

Conversation

@kshitijk4poor

@kshitijk4poor kshitijk4poor commented Apr 6, 2026

Copy link
Copy Markdown
Collaborator

Fixes #5437. Also addresses #5452 (model deprecation warnings) and #5455 (API key format validation).

Summary

Adds advisory pre-flight validation covering model capabilities, deprecation status, and API key format — all before the first API call.

1. Model capability pre-flight

Problem: Hermes sends tool schemas, vision content, and reasoning params to all models regardless of capability. Failures discovered only at API call time.

Introduces agent/model_capabilities.py with:

  • ModelCapabilities dataclass and KNOWN_CAPABILITIES (30+ regex patterns)
  • get_model_capabilities(): Pure pattern-matching lookup, no API calls
  • validate_preflight(): Warns about tools/vision/reasoning mismatches
  • Advisory only — never blocks requests, never modifies schemas or messages

2. Model deprecation warnings

Problem: No warning when a model is deprecated. Users discover it via API errors.

Adds check_model_deprecation():

  • Queries models.dev registry (already cached in-memory) for status
  • Emits a warning if model is marked "deprecated"
  • Returns None when data unavailable — best-effort, never blocks

3. API key format validation

Problem: has_usable_secret() only filters obvious placeholders. A garbled or wrong-provider key passes and fails with a cryptic 401 on first API call.

Adds validate_api_key_format():

  • Checks key prefixes against known patterns (OpenAI sk-, Anthropic sk-ant-, OpenRouter sk-or-, DeepSeek sk-, Groq gsk_)
  • Emits a warning if key doesn't match expected format
  • Advisory only — never blocks, unknown providers pass through

Design Decisions

  • Advisory only throughout: Warnings, not errors. Never blocks requests.
  • No network: Capability patterns are pure regex. Deprecation uses already-cached models.dev data.
  • Exception-safe: All preflight blocks wrapped in try/except.
  • Conservative for unknowns: Unknown models get no warnings.

Independent of #5441 (P0) and #5442 (P1) — no cross-dependencies.

Tests

84 tests covering pattern matching, preflight warnings, deprecation checks (with mocked models.dev), API key format validation for all known providers, and edge cases.

@kshitijk4poor kshitijk4poor force-pushed the feat/model-capability-preflight branch from 64cb628 to 9ceb66a Compare April 6, 2026 07:49
@drewbitt

drewbitt commented Apr 6, 2026

Copy link
Copy Markdown

Not sure this is worth the complexity.

@alt-glitch alt-glitch added type/feature New feature or request P3 Low — cosmetic, nice to have comp/agent Core agent loop, run_agent.py, prompt builder labels May 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent loop, run_agent.py, prompt builder P3 Low — cosmetic, nice to have type/feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Model capability pre-flight validation

3 participants