Makes sense to consolidate. The trace context work here goes past what #16865 covered.

The log transport sharing (logger.ts) was verified e2e in #16865 with a live OTLP collector + Loki, not just unit tests. The unit tests alone didn't catch the original module-isolation bug, so worth a runtime check before merging.

I can run the e2e validation on this branch if you want.

Makes sense to consolidate. The trace context work here goes past what #16865 covered.

The log transport sharing (logger.ts) was verified e2e in #16865 with a live OTLP collector + Loki, not just unit tests. The unit tests alone didn't catch the original module-isolation bug, so worth a runtime check before merging.

I can run the e2e validation on this branch if you want.

@leonnardo thanks, given how core this is i need to triple check these changes but your welcomed to test ans report back (ideally with screenshots)

Spun up an isolated dev gateway with OTEL fully enabled.

Setup

OPENCLAW_CONFIG_PATH (isolated config)
OTLP HTTP collector → Victoria Metrics + Loki
service.name: openclaw-gateway-dev-otel-test
Node v24.13.0 / Linux 6.8.0-100-generic (x64)

Reproduce

git checkout vincentkoc-code/dedupe-otel-trace-context-cluster
pnpm install && pnpm build

OPENCLAW_CONFIG_PATH=/tmp/otel-test/openclaw.json \
OPENCLAW_SKIP_CHANNELS=1 \
node scripts/run-node.mjs --dev gateway

Config:

{
  "diagnostics": {
    "enabled": true,
    "otel": {
      "enabled": true,
      "endpoint": "http://<collector>:4318",
      "serviceName": "openclaw-gateway-dev-otel-test",
      "traces": true, "metrics": true, "logs": true
    }
  }
}

LGTM on logs + metrics:

Greptile Summary

This PR consolidates overlapping OTEL diagnostics work from three related PRs, adding trace context infrastructure and fixing log transport global state persistence.

Key Changes

Trace Context Infrastructure

• Added AsyncLocalStorage-based trace context management (src/infra/diagnostic-trace-context.ts) for propagating trace IDs and span IDs across async boundaries
• Entry points (dispatchReplyFromConfig, runEmbeddedAttempt) now establish trace contexts using wrapper functions
• Added optional traceId and parentSpanId fields to diagnostic events for span correlation

OTEL Service Enhancements

• OTEL spans now accept parent trace context to establish hierarchical relationships (extensions/diagnostics-otel/src/service.ts:369-394)
• Added GenAI semantic convention attributes (gen_ai.operation.name, gen_ai.system, etc.) for model usage spans
• Span names now include model information (e.g., "chat claude-opus-4-6")

Log Transport Fix

• Fixed log transport registry to persist across module bundles using globalThis (src/logging/logger.ts:42-61)
• Registry now tracks all active logger instances and ensures transports attach to all of them
• Uses WeakMap to prevent duplicate transport attachments

Session State Tracking

• Added traceId and currentSpanId fields to session state for tracking trace context across message lifecycle
• logWebhookReceived now generates a trace ID at the start of each message flow
• logMessageQueued stores the trace ID in session state
• logMessageProcessed generates a span ID and stores it in session state

Test Coverage

Comprehensive tests added for:

• AsyncLocalStorage trace context creation and propagation
• Trace context fields passing through diagnostic events
• Log transport registration with multiple logger instances
• OTEL API mocks updated to support context.active(), trace.setSpanContext(), and TraceFlags.SAMPLED

Notes

The trace context infrastructure is foundational work that establishes the plumbing for hierarchical span relationships. The AsyncLocalStorage context is set up at entry points but appears to be intended for integration with code paths not modified in this PR (likely model usage emission). Session state provides a separate mechanism for tracking trace IDs across the webhook→message lifecycle.

The implementation is backward compatible and all tests pass.

Confidence Score: 4/5

• Safe to merge - well-tested foundational work with backward compatible changes
• Score reflects solid implementation of trace context infrastructure and log transport fixes, with comprehensive test coverage. The code is backward compatible and introduces no breaking changes. Deducting one point because the trace context integration appears partial - AsyncLocalStorage contexts are established but the integration with all diagnostic logging paths isn't fully visible in this PR, suggesting this may be foundational work with full integration coming from other consolidated PRs or follow-up work.
• No files require special attention - changes are well-structured and tested

_{Last reviewed commit: 0d53d29}

This modification is very helpful. By the way, will the calls from the main agent to the sub-agents be correctly chained here?

Hello! Any plans to move this work forward?

Any updates on the otel fix?

Thanks for the work here. We split the OTEL/diagnostics foundation into smaller mainline commits instead of landing this branch as-is, so this is now resolved on main.

The relevant landed pieces are:

• GenAI token usage metrics: 7bbd473
• GenAI operation duration metrics: dc19069
• GenAI model-usage span attributes + semconv opt-in behavior: cd7a8f8 and 5815ca9
• Tool-loop, context-assembly, and memory diagnostic OTEL exports: d6ef1fc, ff172f4, and b8a4173
• Trusted trace parenting / propagation guardrails: 5fe06f3, 84f183b, and 89c5298
• Public docs update: 64582bb

Closing this as superseded/resolved by the mainline commits. Appreciate the push here — it helped shape the final, safer split.