Noting the questions I asked in #1556 (comment), especially:

• Do we distinguish between inline data and URLs? Note that e.g. OpenAI's image_url field doesn't, it just uses a data: URL for inline data. This seems pretty simple and convenient. Maybe an additional boolean to indicate which it is? Or people could just check if the URL starts with data:. Alternatively, we have two different types, e.g. media_inline and media_url, or 2 per media type e.g. image_inline etc.

You've chosen two types:

class BlobPart(BaseModel):
    type: Literal["blob"] = Field(description="The type of the content captured in this part.")
    mime_type: str = Field(description="The IANA MIME type of the attached data.")
    data: bytes = Field(description="base64 encoded bytes of the attached data.")

    class Config:
        extra = "allow"

class FileDataPart(BaseModel):
    type: Literal["file_data"] = Field(description="The type of the content captured in this part.")
    mime_type: str = Field(description="The IANA MIME type of the attached data.")
    file_uri: str = Field(description="A URI referencing to reference attached data. Should be recorded without modification, as it was sent to the model.")

    class Config:
        extra = "allow"

The only difference between these part types is the data vs file_uri. What do you think is the advantage of having an extra type for this instead of using data URIs containing base64?

What do you think is the advantage of having an extra type for this instead of using data URIs containing base64?

Hadn't considered this before, mainly because I'm more familiar with the Gemini format that inspired this. But a few thoughts

• Having the type as data: bytes would encode to OTLP protobuf as AnyValue.bytes_value, so you don't have to pay base64 size overhead. I think this would apply to many storage formats like parquet vs requiring base64 URI in a string.

  The way I wrote the description requires base64, so I would need to tweak it.

• Using data URLs for raw bytes does feel a little arbitrary because you could make the same argument for text: data:text/plain,This%20is%20a%20plain%20text%20string. Ofc, the escaping is ugly and takes extra space.

I see bytes in the protobuf, but not the spec, although byte arrays are allowed: https://opentelemetry.io/docs/specs/otel/logs/data-model/#type-any

Is bytes not a byte array? This applies to attributes too https://opentelemetry.io/docs/specs/otel/common/attribute-type-mapping/#byte-sequences.

I guess that must be right and I misinterpreted.

Do we have any precedent/guidance for bytes that can end up inside a complex attribute on a span? Right now it has to convert to JSON, which means base64 encoding, and then to the backend the field will just look like a string. The backend then has to know that it should decode that based on semconv if it wants to end up with the same thing as if it had received the attribute in a log body using protobuf instead of JSON.

cc @lmolkova for the above question

I guess that must be right and I misinterpreted.

Do we have any precedent/guidance for bytes that can end up inside a complex attribute on a span? Right now it has to convert to JSON, which means base64 encoding, and then to the backend the field will just look like a string. The backend then has to know that it should decode that based on semconv if it wants to end up with the same thing as if it had received the attribute in a log body using protobuf instead of JSON.

It's temporary that we serialize to json string, byte array is a good long-term solution and would work nicely with spec changes that are in-flight - open-telemetry/opentelemetry-specification#4651, specifically

https://github.com/open-telemetry/opentelemetry-specification/blob/4f92d3dce6224797045da43e680a1a3dcebeac89/specification/common/README.md?plain=1#L38

One other thing to note, we have this code for non-complex attributes which attempts to interpret python bytes as unicode and drops the attribute if not. I think this predates bytes attributes in OTLP. I think we can remove it independently of open-telemetry/opentelemetry-specification#4651

How should this be instrumented? https://platform.openai.com/docs/guides/images-vision?api-mode=responses&format=base64-encoded#analyze-images

response = client.responses.create(
    model="gpt-4.1",
    input=[
        {
            "role": "user",
            "content": [
                { "type": "input_text", "text": "what's in this image?" },
                {
                    "type": "input_image",
                    "image_url": f"data:image/jpeg;base64,{base64_image}",
                },
            ],
        }
    ],
)

A FileDataPart with image_url kept as is, or a BlobPart where the data URL is decoded?

My preference would be to generate a Blob for the reason above about bytes being serialized more efficiently. I appreciate the ambiguity though.

@alexmojaki do you have a strong preference for using data URLs?

Not strongly, but it feels like something should be stated in the PR.