Summary

Salvages #40403 (sanidhyasin) onto current main and completes the fix across both client paths, fixing #40033.

A custom OpenAI-compatible provider can fail with an opaque HTTP 502: Upstream access forbidden (or a 4xx) even though the identical API key, base URL, model, and JSON body succeed with curl. The cause is an upstream gateway/WAF rejecting the OpenAI Python SDK's identifying headers (User-Agent: OpenAI/Python ..., X-Stainless-*). There was no supported way to override them.

This adds a model.default_headers config key whose entries are merged onto OpenAI-wire clients with user values taking precedence over provider/SDK defaults, so a custom endpoint can swap in a plain User-Agent.

model:
  provider: custom
  default: <model>
  base_url: http://<host>/v1
  default_headers:
    User-Agent: "curl/8.7.1"

Why two commits

1. fix(agent): honor model.default_headers … (sanidhyasin, cherry-picked, authorship preserved) — applies the override to the main agent OpenAI client at construction (agent/agent_init.py) and across credential swaps / client rebuilds (run_agent.py::_apply_client_headers_for_base_url). No-op for native Anthropic/Bedrock.

2. fix(aux): honor model.default_headers on auxiliary client too (follow-up) — the original fix only covered the main client. The auxiliary client (title generation, context compression, vision routing) builds its own OpenAI clients, so for a custom WAF'd endpoint the main turn would succeed while auxiliary calls to the same endpoint still failed. This adds a shared agent.auxiliary_client._apply_user_default_headers() and applies it at every OpenAI-wire construction site — config-level model.provider: custom (_try_custom_endpoint), named custom providers (custom_providers/providers entries, incl. the anthropic-SDK-missing OpenAI-wire fallback), the api-key-provider, async-conversion, and main resolve fallback branches. To prevent the two clients ever drifting on precedence/value handling, the main-path method now delegates the config read + merge to this single shared helper.

E2E verified

With a real isolated HERMES_HOME + config.yaml, the override reaches the OpenAI client on all three paths:

• main agent client ✓
• config-level provider: custom auxiliary client ✓
• named custom_providers auxiliary client ✓

Tests

• tests/run_agent/test_provider_attribution_headers.py — 4 tests from fix(agent): honor model.default_headers for custom OpenAI-compatible providers (#40033) #40403 (main path): custom override, user-wins-over-provider-defaults, unconfigured no-op, Anthropic skip.
• tests/agent/test_auxiliary_user_default_headers.py — 8 new tests: helper merge semantics (user-wins, no-op, None-handling), config-level custom aux, named-custom aux.
• Full aux + attribution header suites green (295 passed). ruff clean.

Design notes

• User-always-wins on the OpenAI wire (over kimi/copilot/nvidia provider headers too) is intentional and consistent between the main and auxiliary paths — model.default_headers is a documented global escape hatch. A reviewer flagged this; it's a low-risk, by-design tradeoff, not a footgun in practice (one wouldn't set a global curl UA while routing aux to kimi.com).
• ${ENV_VAR} interpolation in header values is intentionally out of scope (separate concern, one-PR-one-concern) — can be a follow-up.

Credit

• @sanidhyasin — fix(agent): honor model.default_headers for custom OpenAI-compatible providers (#40033) #40403, the model.default_headers config key + correct precedence + main-path implementation + docs + main-path tests.

Original issue #40033 reported by @danfye.

Closes #40403
Closes #40033