Skip to content

Implement audio and video context support #671

@nabinchha

Description

@nabinchha

Follow-up implementation issue for the design plan tracked in #668 and added in PR #669.

Scope:

  • Add first-class audio and video context config models following the existing image-context pattern.
  • Support context values as URL sources or base64 source data.
  • In explicit URL mode (data_type=url), allow local path and file:// URI values to pass through for colocated model endpoints that can read the same filesystem path.
  • In auto-detect mode, handle audio/video local-looking paths like image context handles unresolved local-looking paths: do not pass them through implicitly. Require data_type=url for local path pass-through.
  • Preserve existing image-context behavior and backward compatibility.
  • Add provider adapter translation for supported media blocks.
  • Normalize provider errors at adapter boundaries. Use canonical unsupported-capability errors when the adapter can determine the incompatibility before transport, and otherwise preserve provider 400 responses so provider/model-specific modality limitations are visible to users.
  • Add focused config, engine, adapter, and regression tests described in the plan.

Out of scope:

  • Audio or video generation columns.
  • Data Designer-managed local file resolution for audio/video contexts, including base_path lookup or loading local audio/video files into base64.
  • Provider file upload lifecycle, file IDs, or config-layer filenames.
  • Automatic video frame extraction.

References:

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions