aigw: split access logs and map request headers#1797
Merged
mathetake merged 6 commits intoenvoyproxy:mainfrom Jan 22, 2026
Merged
aigw: split access logs and map request headers#1797mathetake merged 6 commits intoenvoyproxy:mainfrom
mathetake merged 6 commits intoenvoyproxy:mainfrom
Conversation
Split Envoy access logs by request type (LLM vs MCP) using CEL matchers, add request-header→attribute mapping env vars for logs/spans/metrics, and update examples/tests to validate session.id propagation. Signed-off-by: Adrian Cole <adrian@tetrate.io>
Contributor
Author
|
@mathetake @nacx might be glitches here I need to look at again tomorrow, but hopefully the overall direction is sensible. if not, lemme know. |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #1797 +/- ##
==========================================
+ Coverage 84.04% 84.06% +0.02%
==========================================
Files 117 118 +1
Lines 12990 13213 +223
==========================================
+ Hits 10917 11108 +191
- Misses 1418 1433 +15
- Partials 655 672 +17 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Contributor
Author
|
@nacx @mathetake once this is in, I can add OTLP access log from gateway in aigw/standalone mode without a massive PR |
Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com>
mathetake
approved these changes
Jan 22, 2026
027b8ec to
17b07cd
Compare
Member
|
/retest |
mathetake
pushed a commit
that referenced
this pull request
Jan 26, 2026
**Description** Default span/log request‑header mappings to `agent-session-id:session.id` so agent frameworks like Goose get session correlation with zero config, while still allowing explicit overrides (different mapping or empty to disable). Metrics never default to session IDs because they are high cardinality. The default mapping is in the new ENV variable `OTEL_AIGW_REQUEST_HEADER_ATTRIBUTES`, so those who want no `agent-session-id:session.id` should set `OTEL_AIGW_REQUEST_HEADER_ATTRIBUTES=` (empty string) to clear it. Refactor request‑header mapping handling so defaults/merging live only in extproc; `aigw` and controller/helm just pass flags through. Ordering is normalized everywhere (request → span → metrics → log) and docs/examples describe defaults without explicitly setting `agent-session-id:session.id`. **Related Issues/PRs (if applicable)** Related: #1797 **Special notes for reviewers (if applicable)** Ran the examples/goose with OTEL console env and --debug. Since goose now propagates agent-session-id by default, we can see in the telemetry `agent-session-id=20260123_19`: MCP span (`tools/list`) showing `session.id` set from the header: ``` {"Name":"ListTools","SpanContext":{"TraceID":"6412e470a771ede413f3318b984b65f5","SpanID":"bf7a7b2b20ce85a9","TraceFlags":"01","TraceState":"","Remote":false},"Parent":{"TraceID":"00000000000000000000000000000000","SpanID":"0000000000000000","TraceFlags":"00","TraceState":"","Remote":false},"SpanKind":3,"StartTime":"2026-01-23T16:01:20.669792+09:00","EndTime":"2026-01-23T16:01:21.313477459+09:00","Attributes":[{"Key":"mcp.protocol.version","Value":{"Type":"STRING","Value":"2025-06-18"}},{"Key":"mcp.transport","Value":{"Type":"STRING","Value":"http"}},{"Key":"mcp.request.id","Value":{"Type":"STRING","Value":"{1}"}},{"Key":"mcp.method.name","Value":{"Type":"STRING","Value":"tools/list"}},{"Key":"session.id","Value":{"Type":"STRING","Value":"20260123_19"}}],"Events":[{"Name":"route to backend","Attributes":[{"Key":"mcp.backend.name","Value":{"Type":"STRING","Value":"kiwi"}},{"Key":"mcp.session.id","Value":{"Type":"STRING","Value":"f9e80f73-bc48-4797-afae-045ef0e57e7d"}},{"Key":"mcp.session.new","Value":{"Type":"BOOL","Value":false}}],"DroppedAttributeCount":0,"Time":"2026-01-23T16:01:21.303264+09:00"}],"Links":null,"Status":{"Code":"Ok","Description":""},"DroppedAttributes":0,"DroppedEvents":0,"DroppedLinks":0,"ChildSpanCount":0,"Resource":[{"Key":"service.name","Value":{"Type":"STRING","Value":"ai-gateway"}},{"Key":"telemetry.sdk.language","Value":{"Type":"STRING","Value":"go"}},{"Key":"telemetry.sdk.name","Value":{"Type":"STRING","Value":"opentelemetry"}},{"Key":"telemetry.sdk.version","Value":{"Type":"STRING","Value":"1.39.0"}}],"InstrumentationScope":{"Name":"envoyproxy/ai-gateway","Version":"","SchemaURL":"","Attributes":null},"InstrumentationLibrary":{"Name":"envoyproxy/ai-gateway","Version":"","SchemaURL":"","Attributes":null}} ``` MCP access log showing `session.id` on a tool call: ``` {"bytes_received":341,"bytes_sent":8720,"connection_termination_details":null,"downstream_local_address":"127.0.0.1:10088","downstream_remote_address":"127.0.0.1:50643","duration":1247,"jsonrpc.request.id":"4","mcp.method.name":"tools/call","mcp.provider.name":"kiwi","mcp.session.id":"f9e80f73-bc48-4797-afae-045ef0e57e7d","method":"POST","request.path":"/","response_code":200,"session.id":"20260123_19","start_time":"2026-01-23T07:01:33.553Z","upstream_cluster":"httproute/default/ai-eg-mcp-br-mcp-route-kiwi/rule/0","upstream_host":"146.75.115.52:443","upstream_local_address":"192.168.23.60:50644","upstream_transport_failure_reason":null,"user-agent":"Go-http-client/1.1","x-envoy-origin-path":"/mcp","x-envoy-upstream-service-time":"613","x-forwarded-for":null,"x-request-id":"bd29074f-3ab0-41b3-a184-e0ec87a3809b"} ``` LLM access log showing `session.id` on a chat completion: ``` {"bytes_received":14807,"bytes_sent":47214,"connection_termination_details":null,"downstream_local_address":"127.0.0.1:1975","downstream_remote_address":"127.0.0.1:50651","duration":3560,"gen_ai.provider.name":"default/openai/route/aigw-run/rule/0/ref/0","gen_ai.request.model":"qwen3:1.7b","gen_ai.response.model":"qwen3:1.7b","gen_ai.usage.input_tokens":3227,"gen_ai.usage.output_tokens":253,"method":"POST","request.path":"/v1/chat/completions","response_code":200,"session.id":"20260123_19","start_time":"2026-01-23T07:01:29.980Z","upstream_cluster":"httproute/default/aigw-run/rule/0","upstream_host":"127.0.0.1:11434","upstream_local_address":"127.0.0.1:50653","upstream_transport_failure_reason":null,"user-agent":null,"x-envoy-origin-path":"/v1/chat/completions","x-envoy-upstream-service-time":null,"x-forwarded-for":"192.168.23.60","x-request-id":"2b430167-040d-43ef-a48e-de0ebaa0fcdc"} ``` Minor improvements: - normalized all example header/attributes and order of trace, metrics and logs - Add `AIGW_DEBUG` so docker compose examples can actually show debug output - align data‑plane tests to use the aigw func‑e download location instead of re-downloading --------- Signed-off-by: Adrian Cole <adrian@tetrate.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Split stdout/file access logs by request type using CEL over request headers (x-ai-eg-model for LLM, x-ai-eg-mcp-backend for MCP) so MCP-only fields never appear on LLM logs and vice‑versa. This avoids relying on
/mcppaths, which are not present on backend‑listener requests.Add OTEL request-header mapping env vars for users:
OTEL_AIGW_REQUEST_HEADER_ATTRIBUTES(base mapping) andOTEL_AIGW_LOG_REQUEST_HEADER_ATTRIBUTES(log override). These are merged and wired through ext_proc and the Envoy Gateway extension server so access logs can includesession.idwithout per‑request app changes. MCP uses JSON‑RPCparams._metafor POST requests, but GET streams have no JSON‑RPC payload, so compose examples also send HTTP headers for access-log mapping.Rename access‑log keys to the OTEL‑style
gen_ai.*andmcp.provider.namewhile preserving gateway‑specific metadata fields, addrequest.pathto the common access‑log fields, and update examples, fixtures, and docs accordingly. Compose examples pass user/session IDs via compose args so log output can be verified end‑to‑end (append--debugto theaigwcommand for verbose logs).Ensure original downstream paths are preserved in access logs by setting
x-envoy-original-pathandx-ai-eg-original-pathfrom the incoming request path across LLM and MCP flows.Related Issues/PRs (if applicable)
Related: #1303
Special notes for reviewers (if applicable)
Access log samples from docker compose runs (Envoy stdout):