Skip to content

Cross-provider fallback should respect provider trust boundaries for images/attachments #43481

@mitchmcalister

Description

@mitchmcalister

Problem

When a model fallback retry crosses providers (e.g. Anthropic → OpenAI), user-supplied images and attachments are forwarded to the fallback provider. Users who configure cross-provider fallbacks for text resilience may not expect their image data to be sent to a different provider.

The current behavior is binary: configured providers receive everything. There's no mechanism to distinguish "I trust this provider for text fallback" from "I trust this provider with my image data."

Proposed solution

Add provider-level trust configuration so users can control what data crosses provider boundaries on fallback:

  • A config field like agents.defaults.model.fallbackTrustPolicy or per-provider trust levels
  • resolveRetryImages checks whether the fallback is cross-provider and whether the target provider is trusted for image data
  • Default behavior: preserve current behavior (forward everything) to avoid breaking existing setups, but allow users to restrict it

Context

Flagged by security review on #43331. The behavior is pre-existing (prior to #43331, images were stripped unconditionally on fallback — a compatibility hack, not a privacy boundary), but worth addressing properly.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions