Description

Follow up to #5977

Strategy

1. Set a break point in /packages/provider-utils/src/response-handler.ts
   in order to inspect the API response body across all providers
2. Run the examples at examples/ai-core/generate-image/*.ts
3. For providers that include any meta data in the response body, update the response schema accordingly in separate pull requests

Progress

• @ai-sdk/amazon-bedrock

  Update packages/amazon-bedrock/src/bedrock-image-model.ts

  • no meta data. The response body only has an images key which is an array of base64 strings

• @ai-sdk/deepinfra

  Update packages/deepinfra/src/deepinfra-image-model.ts

  • no meta data. The response body only has an images key which is an array of data:image/png,... strings

• feat (fal): Set .providerMetaData for image model responses #6406

• @ai-sdk/fireworks

  Update packages/fireworks/src/fireworks-image-model.ts

  • no meta data. Fireworks responds with a binary response body which is the image itself. However there is some information in response headers that might be of interest

    {
        "access-control-allow-origin": "*",
        "alt-svc": "h3=\":443\"; ma=86400",
        "cf-cache-status": "DYNAMIC",
        "cf-ray": "942e60a20fb0979e-LAX",
        connection: "keep-alive",
        "content-length": "1348913",
        "content-type": "image/png",
        date: "Tue, 20 May 2025 19:52:59 GMT",
        "finish-reason": "SUCCESS",
        "fireworks-middleware-version": "v2",
        "fireworks-request-generation-ms": "750",
        "fireworks-request-postprocessing-ms": "398",
        "fireworks-request-preprocessing-ms": "1",
        "fireworks-request-queued-ms": "0",
        "fireworks-request-runtime-ms": "1150",
        id: "",
        seed: "1674419546",
        server: "cloudflare",
        "server-timing": "total;dur=1270.0;desc=\"Total Response Time\"",
        vary: "Accept-Encoding",
        "x-billing-information": "{\"model_name\": \"stable-diffusion-xl-1024-v1-0\", \"num_steps\": 30, \"num_samples\": 1, \"height\": 1024, \"width\": 1024, \"control_net_name\": null}",
        "x-envoy-upstream-service-time": "1152",
        "x-fireworks-billing-idempotency-id": "123e4567-e89b-12d3-a456-426655440000",
        "x-latent-code-uri": "",
    }

    See x-billing-information which has the following data

    {
        "model_name": "stable-diffusion-xl-1024-v1-0",
        "num_steps": 30,
        "num_samples": 1,
        "height": 1024,
        "width": 1024,
        "control_net_name": null
    }

    However, the x-billing-information header does not seem to be documented at https://docs.fireworks.ai

• feat (google-vertex): Set .providerMetaData for image model responses #6409

• @ai-sdk/luma

  Update packages/luma/src/luma-image-model.ts

  • example response

    {
        id: "123e4567-e89b-12d3-a456-426655440000",
        generation_type: "image",
        state: "queued",
        failure_reason: null,
        created_at: "2025-05-20T20:28:39.912392Z",
        assets: null,
        model: "photon-flash-1",
        request: {
            generation_type: "image",
            model: "photon-flash-1",
            prompt: "A salamander at dusk in a forest pond, in the style of ukiyo-e",
            aspect_ratio: "1:1",
            format: "jpg",
            callback_url: null,
            image_ref: null,
            style_ref: null,
            character_ref: null,
            modify_image_ref: null,
            sync: false,
            sync_timeout: 60,
        },
    }

    request.prompt is the original prompt. The actual image is downloaded in a 2nd step

• @ai-sdk/openai-compatible

  Update packages/openai-compatible/src/openai-compatible-image-model.ts

  • question: should we add revisedPrompt like we did for the openai provider?

• @ai-sdk/replicate

  Update packages/replicate/src/replicate-image-model.ts

  • example response from example 1

    {
    id: "skpp6pb15hrme0cpxxmvzz85a0",
    model: "black-forest-labs/flux-schnell",
    version: "hidden",
    input: {
        num_outputs: 1,
        // this is the original prompt
        prompt: "The Loch Ness Monster getting a manicure",
    },
    logs: "",
    output: [
        "https://replicate.delivery/xezq/jYf6JrdHNsUXe01C4GmQdJGpK2CTGnIwNBdDguQMva4HnpuUA/out-0.webp",
    ],
    data_removed: false,
    error: null,
    status: "processing",
    created_at: "2025-05-20T20:36:22.956Z",
    urls: {
        cancel:
        "https://api.replicate.com/v1/predictions/skpp6pb15hrme0cpxxmvzz85a0/cancel",
        get: "https://api.replicate.com/v1/predictions/skpp6pb15hrme0cpxxmvzz85a0",
        stream:
        "https://stream.replicate.com/v1/files/bcwr-zpp6rdytkpibyeinhly2ilrcjfh2s7kgrbugysbrorjhgxcbo3yq",
    },
    }
    

  • example response from example 2

    {
    id: "1ccwc9rewdrme0cpxxp99bb8e8",
    model: "black-forest-labs/flux-schnell",
    version: "hidden",
    input: {
        aspect_ratio: "16:9",
        num_outputs: 1,
        prompt: "The Loch Ness Monster getting a manicure",
    },
    logs: "",
    output: [
        "https://replicate.delivery/xezq/fXyzUk0Y522QBaTNkBhB5zp2p4zChX0aqfaiSkvKCJJ3ppuUA/out-0.webp",
    ],
    data_removed: false,
    error: null,
    status: "processing",
    created_at: "2025-05-20T20:39:18.499Z",
    urls: {
        cancel:
        "https://api.replicate.com/v1/predictions/1ccwc9rewdrme0cpxxp99bb8e8/cancel",
        get: "https://api.replicate.com/v1/predictions/1ccwc9rewdrme0cpxxp99bb8e8",
        stream:
        "https://stream.replicate.com/v1/files/bcwr-jkwad4i5wchwc2ckuzoxlpqn7rgsc7j4pyaeyql5d2pz3mzujwaq",
    },
    }
    

  • example response from example 3

    {
    id: "4pkwd87fmxrme0cpxxn95be9y8",
    model: "recraft-ai/recraft-v3",
    version: "hidden",
    input: {
        num_outputs: 1,
        prompt: "The Loch Ness Monster getting a manicure",
        size: "1365x1024",
        style: "realistic_image",
    },
    logs: "",
    output:
        "https://replicate.delivery/xezq/znZ6Gf1fBfLLlIFBbbaczaVq17iChcqVuDVriedDzfvUGN1lC/tmp8i1905ua.webp",
    data_removed: false,
    error: null,
    status: "processing",
    created_at: "2025-05-20T20:38:04.967Z",
    urls: {
        cancel:
        "https://api.replicate.com/v1/predictions/4pkwd87fmxrme0cpxxn95be9y8/cancel",
        get: "https://api.replicate.com/v1/predictions/4pkwd87fmxrme0cpxxn95be9y8",
        stream:
        "https://stream.replicate.com/v1/files/bcwr-lzkgaqd6trjw6avkyzjkn35hvmwt26sp66caxottrezongnbetma",
    },
    }
    

• @ai-sdk/togetherai

  Update packages/togetherai/src/togetherai-image-model.ts

  • example response

    {
        id: "nuqneku-e89b12-d3a4564266554400-PDX",
        model: "black-forest-labs/FLUX.1-dev",
        object: "list",
        data: [
            {
            timings: {
                inference: 3.9200671669095755,
            },
            index: 0,
                b64_json: "/9j/4AAQSk..."
            },
        ],
    }

AI SDK Version

ai: 5.0.0-alpha.3