Summary

Fixes #40033 — a custom OpenAI-compatible provider can fail with an opaque HTTP 502: Upstream access forbidden (or a 4xx) even though the identical API key, base URL, model, and JSON body succeed with curl. The reporter traced it to the OpenAI Python SDK's default identifying headers — User-Agent: OpenAI/Python ... and the X-Stainless-* family — being rejected by an upstream gateway/WAF in front of the OpenAI-compatible endpoint.

There was no supported way to override those headers, so the only fix was to patch the local install.

Root cause

For OpenAI-wire providers, default_headers is set from a fixed set of base-URL/provider rules (agent/agent_init.py at construction; AIAgent._apply_client_headers_for_base_url on credential swaps / client rebuilds). A generic custom endpoint matches none of those rules, so it inherits the SDK's built-in identifying headers with no user-facing override.

Fix

Add a model.default_headers config key. Its entries are merged onto the OpenAI client's default_headers with user values taking precedence over provider- and SDK-supplied defaults, so a custom endpoint can swap in a plain User-Agent (or override X-Stainless-*) and reach the upstream.

• New AIAgent._apply_user_default_headers() reads model.default_headers via the existing cfg_get / load_config helpers and merges it on top of whatever headers were already resolved.
• Called both at initial client construction (agent/agent_init.py) and at the end of _apply_client_headers_for_base_url, so the override survives credential rotation and client rebuilds — not just the first request.
• No-op for native Anthropic / Bedrock modes (they don't use the OpenAI client) and when the key is unset, so existing behaviour and the provider/attribution defaults are unchanged.

Config (now works)

model:
  provider: custom
  default: <model>
  base_url: http://<host>/v1
  default_headers:
    User-Agent: "curl/8.7.1"

This matches suggested fix option 1 in the issue. Documented in cli-config.yaml.example under the model: section.

Tests

tests/run_agent/test_provider_attribution_headers.py — 4 new tests:

• custom endpoint: model.default_headers overrides the SDK User-Agent and adds extra headers (the Custom OpenAI-compatible provider can fail when upstream blocks OpenAI Python SDK default headers #40033 reproduction)
• user headers win over provider defaults while leaving the untouched ones (e.g. OpenRouter HTTP-Referer) intact
• unconfigured → provider defaults untouched, nothing injected
• Anthropic mode → skipped entirely

Full file passes (13 tests). Ran with the project venv on macOS (Python 3.11). ruff check clean on the changed files.

Platforms

Logic is platform-independent (header dict merge + config read). Tested on macOS; no OS-specific code paths touched.