Use MEAI for AI telemetry by JamesNK · Pull Request #11237 · dotnet/aspire

JamesNK · 2025-09-03T10:27:17Z

Description

Aspire chat integrations enable telemetry. Each integration enables telemetry in the respective implementation. This PR changes that to always use MEAI for telemetry.

The reason is implementations aren't keeping up with the GenAI semantic convention spec:

Azure.AI.Interfence is two major changes behind the current spec: https://github.com/Azure/azure-sdk-for-net/blob/1af7c19fe8dcb7f7f843730117accbd6927ad2a4/sdk/ai/Azure.AI.Inference/src/Telemetry/OpenTelemetryScope.cs
OpenAI is in an even worse state. It doesn't record input/output messages at all: https://github.com/openai/openai-dotnet/blob/dd6aa1303b0c7e9aaaedd3ce4208078ae8fe6cca/src/Utility/Telemetry/OpenTelemetryScope.cs

The reality is that only MEAI is keeping up with GenAI semantic convention spec.

I don't want to enable both MEAI and OpenAI/Azure because then you'll see duplicate spans in tracing and duplicate counters in metrics.

The downside is there could be some implementation specific telemetry that is missed out on by only recording MEAI. That could be solved longer term by MEAI folks could reach out to implementations and agree on a way for them to add tags to the GenAI span, even if they didn't create it.

For example, Azure has azure.resource_provider.namespace that could be added to the active GenAI span. Azure could check to see if there is a current activity, and check if it is for GenAI (e.g. has gen_ai.system tag), and then add implementation specific metadata.

Checklist

Is this feature complete?
- Yes. Ready to ship.
- No. Follow-up changes expected.
Are you including unit tests for the changes and scenario tests if relevant?
- Yes
- No
Did you add public API?
- Yes
  - If yes, did you have an API Review for it?
    - Yes
    - No
  - Did you add <remarks /> and <code /> elements on your triple slash comments?
    - Yes
    - No
- No
Does the change make any security assumptions or guarantees?
- Yes
  - If yes, have you done a threat model and had a security review?
    - Yes
    - No
- No
Does the change require an update in our Aspire docs?
- Yes
  - Link to aspire-docs issue (consider using one of the following templates):
- No

github-actions · 2025-09-03T10:27:24Z

🚀 Dogfood this PR with:

curl -fsSL https://raw.githubusercontent.com/dotnet/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 11237

Or

Run remotely in PowerShell:

iex "& { $(irm https://raw.githubusercontent.com/dotnet/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 11237"

Copilot

Pull Request Overview

This PR updates the Aspire OpenAI, Azure OpenAI, and Azure AI Inference components to use Microsoft.Extensions.AI (MEAI) for AI telemetry instead of the previous experimental OpenAI telemetry implementation. The change introduces a new EnableSensitiveTelemetryData configuration option and updates telemetry source names to work with both experimental and stable MEAI versions.

Key changes:

Adds EnableSensitiveTelemetryData configuration option across all AI components
Updates telemetry source names from "OpenAI.*" to "Experimental.Microsoft.Extensions.AI" and "Microsoft.Extensions.AI"
Removes experimental telemetry configuration logic and simplifies settings properties

Reviewed Changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
`src/Components/Aspire.OpenAI/OpenAISettings.cs`	Removes experimental telemetry logic, adds EnableSensitiveTelemetryData property
`src/Components/Aspire.OpenAI/AspireOpenAIExtensions.cs`	Updates telemetry source names to MEAI sources
`src/Components/Aspire.OpenAI/AspireOpenAIClientBuilder.cs`	Adds EnableSensitiveTelemetryData parameter to constructor
`src/Components/Aspire.OpenAI/AspireOpenAIClientBuilderChatClientExtensions.cs`	Configures EnableSensitiveData on OpenTelemetryChatClient
`src/Components/Aspire.OpenAI/AspireOpenAIClientBuilderEmbeddingGeneratorExtensions.cs`	Configures EnableSensitiveData on OpenTelemetryEmbeddingGenerator
`src/Components/Aspire.Azure.AI.OpenAI/AzureOpenAISettings.cs`	Similar changes to OpenAI settings with MEAI telemetry
`src/Components/Aspire.Azure.AI.OpenAI/AspireAzureOpenAIExtensions.cs`	Updates to use MEAI telemetry sources
`src/Components/Aspire.Azure.AI.Inference/ChatCompletionsClientSettings.cs`	Adds EnableSensitiveTelemetryData and removes experimental logic
`src/Components/Aspire.Azure.AI.Inference/AspireAzureAIInferenceExtensions.cs`	Updates telemetry sources and configures sensitive data handling
`tests/Aspire.OpenAI.Tests/OpenAIPublicApiTests.cs`	Updates test constructor calls to include new parameter
Configuration schema files	Adds EnableSensitiveTelemetryData property definitions

mitchdenny · 2025-09-03T10:38:12Z

src/Components/Aspire.Azure.AI.Inference/AspireChatCompletionsClientBuilder.cs

    string? deploymentName,
-    bool disableTracing)
+    bool disableTracing,
+    bool enableSensitiveTelemetryData)


Minor breaking change but I doubt folks are using this class directly. Probably not worth providing a constructor overload?

We should add an overload and avoid breaking this.

Oh wait, this is prerelease . Carry on

Why do the builders use constructors? There will always be changes to types like these. For example, this PR. And literally every change to this type is going to be breaking.

Wouldn't init properties be better?

eerhardt · 2025-09-03T14:13:44Z

src/Components/Aspire.OpenAI/AspireOpenAIExtensions.cs

        {
            builder.Services.AddOpenTelemetry()
-                .WithTracing(traceBuilder => traceBuilder.AddSource("OpenAI.*"));
+                .WithTracing(traceBuilder => traceBuilder.AddSource(telemetrySources));


I'm confused at what layers of telemetry are the "right ones" to listen to.

Developers can use Aspire.OpenAI and not talk through Microsoft.Extensions.AI. They can grab the OpenAIClient from DI and use it directly. If they did that, they wouldn't get any telemetry right? Since we would only enable MEAI telemetry and MEAI isn't being invoked.

cc @stephentoub @lmolkova

Since we would only enable MEAI telemetry and MEAI isn't being invoked.

Correct. I think the solution is to clarify in the docs that telemetry is only available when OpenAI client is wrapped in Microsoft.Extensions.AI client.

I think the best approach is to standardize on MEAI as the provider of AI telemetry. The reality is that only MEAI is keeping up with GenAI semantic convention spec.

Azure.AI.Interfence is two major changes behind the current spec: https://github.com/Azure/azure-sdk-for-net/blob/1af7c19fe8dcb7f7f843730117accbd6927ad2a4/sdk/ai/Azure.AI.Inference/src/Telemetry/OpenTelemetryScope.cs

OpenAI is in an even worse state. It doesn't record input/output messages at all: https://github.com/openai/openai-dotnet/blob/dd6aa1303b0c7e9aaaedd3ce4208078ae8fe6cca/src/Utility/Telemetry/OpenTelemetryScope.cs

I don't want to enable both MEAI and OpenAI/Azure because then you'll see duplicate spans in tracing and duplicate counters in metrics.

The downside is there could be some implementation specific telemetry that is missed out on by only recording MEAI. That could be solved longer term by MEAI folks could reach out to implementations and agree on a way for them to add tags to the GenAI span, even if they didn't create it.

For example, Azure has azure.resource_provider.namespace that could be added to the active GenAI span. Azure could check to see if there is a current activity, and check if it is for GenAI (e.g. has gen_ai.system tag), and then add implementation specific metadata.

Would it make sense if those libraries didn't have their own telemetry at all? The current state of the world is rather confusing.

It's worth considering. Another issue is what happens if implementors have APIs that aren't exposed by MEAI. They might need their own telemetry then. Out of scope of this PR.

I think at least for Aspire integration we can take an opiniated stance that our GenAI telemetry settings are related to MEAI.

Lets start there an open discussions with the right teams and people.

JamesNK · 2025-09-04T23:11:05Z

This is ready to review. It is a blocker for #11227 so should be looked at promptly.

tests/Aspire.Azure.AI.OpenAI.Tests/ConformanceTests.cs

tests/Aspire.OpenAI.Tests/ConformanceTests.cs

…n AspireOpenAIClientBuilder.

JamesNK added 3 commits September 3, 2025 16:23

Use MEAI for AI telemetry

4333b20

Update

c58bc88

Update

708a26f

Copilot AI review requested due to automatic review settings September 3, 2025 10:27

JamesNK requested review from eerhardt, radical and sebastienros as code owners September 3, 2025 10:27

github-actions bot added the area-integrations Issues pertaining to Aspire Integrations packages label Sep 3, 2025

Copilot AI reviewed Sep 3, 2025

View reviewed changes

Update

83e5d57

mitchdenny reviewed Sep 3, 2025

View reviewed changes

eerhardt reviewed Sep 3, 2025

View reviewed changes

eerhardt mentioned this pull request Sep 3, 2025

Improve dashboard GenAI telemetry UI #11227

Merged

18 tasks

JamesNK added 3 commits September 4, 2025 10:50

WIP

1422cae

Fix

5b0912d

Simplify

5ce4525

JamesNK requested a review from stephentoub September 4, 2025 03:29

Clean up

fb689dc

davidfowl approved these changes Sep 5, 2025

View reviewed changes

eerhardt approved these changes Sep 5, 2025

View reviewed changes

tests/Aspire.Azure.AI.OpenAI.Tests/ConformanceTests.cs Outdated Show resolved Hide resolved

tests/Aspire.OpenAI.Tests/ConformanceTests.cs Outdated Show resolved Hide resolved

Apply suggestions from code review

154e680

eerhardt enabled auto-merge (squash) September 5, 2025 17:07

Use the correct config section name when looking for deploymentName i…

dc1f1f3

…n AspireOpenAIClientBuilder.

eerhardt merged commit 2365bd5 into main Sep 5, 2025
305 checks passed

eerhardt deleted the jamesnk/ai-telemetry branch September 5, 2025 17:55

dotnet-policy-service bot added this to the 9.5 milestone Sep 5, 2025

This was referenced Sep 8, 2025

Update Aspire AI integrations with telemetry changes dotnet/docs-aspire#4518

Closed

Update AspireOllamaChatClientExtensions.AddChatClient to allow configuring sensitive data CommunityToolkit/Aspire#842

Closed

MihuBot mentioned this pull request Sep 28, 2025

[✨ Triage] dotnet/aspire#11697 by thangchung - Generative AI visualizer feature in .NET Aspire 9.5 was not working MihuBot/runtime-utils#1546

Closed

github-actions bot locked and limited conversation to collaborators Oct 6, 2025

Conversation

JamesNK commented Sep 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklist

Uh oh!

github-actions bot commented Sep 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JamesNK Sep 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JamesNK Sep 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JamesNK commented Sep 4, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

JamesNK commented Sep 3, 2025 •

edited

Loading

github-actions bot commented Sep 3, 2025 •

edited

Loading

JamesNK Sep 3, 2025 •

edited

Loading

JamesNK Sep 3, 2025 •

edited

Loading