Follow-up implementation issue for the design plan tracked in #668 and added in PR #669.
Scope:
- Add first-class audio and video context config models following the existing image-context pattern.
- Support context values as URL sources or base64 source data.
- In explicit URL mode (
data_type=url), allow local path and file:// URI values to pass through for colocated model endpoints that can read the same filesystem path.
- In auto-detect mode, handle audio/video local-looking paths like image context handles unresolved local-looking paths: do not pass them through implicitly. Require
data_type=url for local path pass-through.
- Preserve existing image-context behavior and backward compatibility.
- Add provider adapter translation for supported media blocks.
- Normalize provider errors at adapter boundaries. Use canonical unsupported-capability errors when the adapter can determine the incompatibility before transport, and otherwise preserve provider 400 responses so provider/model-specific modality limitations are visible to users.
- Add focused config, engine, adapter, and regression tests described in the plan.
Out of scope:
- Audio or video generation columns.
- Data Designer-managed local file resolution for audio/video contexts, including
base_path lookup or loading local audio/video files into base64.
- Provider file upload lifecycle, file IDs, or config-layer filenames.
- Automatic video frame extraction.
References:
Follow-up implementation issue for the design plan tracked in #668 and added in PR #669.
Scope:
data_type=url), allow local path andfile://URI values to pass through for colocated model endpoints that can read the same filesystem path.data_type=urlfor local path pass-through.Out of scope:
base_pathlookup or loading local audio/video files into base64.References: