MongoDB 3.7.0 OTel Instrumentation Analysis

Context

MongoDB.Driver 3.7.0 adds built-in OpenTelemetry instrumentation. When used with our existing dd-trace-dotnet custom instrumentation, this creates duplicate/overlapping spans. We need to determine whether the OTel spans can replace our custom instrumentation, or whether we should disable them.

This analysis is based on the diff at mongodb.diff, which compares the SchemaV0 snapshot for the 3_0 package version (before) against the new output with MongoDB 3.7.0 (after).

Span Count Change

The span count more than doubles: +30 new OTel spans (15 L1 + 15 L2).

Note: the "before" count has 16 mongodb.query spans vs 15 in "after". The initial find that was directly under Main() (Id_2) now has an OTel L1 parent (Id_6), and there's one fewer mongodb.query span for the countDocuments operations - actually looking more carefully, the count is the same (15 DD spans each for the 3 groups of sync/async x 5 operations + 1 initial find = 16... let me recount). Actually both before and after have 15 mongodb.query spans - same count. The diff just renumbers IDs.

New Span Hierarchy

Before (2-tier):

App span (e.g., "sync-calls")
  └── mongodb.query (our custom instrumentation)

After (4-tier):

App span (e.g., "sync-calls")
  └── OTel L1 - "client.request" (logical operation)
       └── mongodb.query (our custom instrumentation, RE-PARENTED)
            └── OTel L2 - "client.request" (wire protocol command)

Our mongodb.query spans are now sandwiched between two OTel spans. The OTel L1 span becomes the parent of our span, and our span becomes the parent of the OTel L2 span.

Two Levels of OTel Spans

OTel Level 1 — Logical Operation Spans

Tags:

• db.collection.name: e.g., employees
• db.namespace: e.g., test-db
• db.operation.name: e.g., find, delete, insert, countDocuments
• db.operation.summary: e.g., find test-db.employees
• db.system.name: mongodb
• otel.library.name: MongoDB.Driver
• otel.library.version: 3.7.0
• otel.status_code: STATUS_CODE_OK

No metrics. No host/port info at this level.

OTel Level 2 — Wire Protocol Command Spans

Tags:

• db.collection.name: e.g., employees
• db.command.name: e.g., find, delete, insert, aggregate
• db.mongodb.lsid: Session ID (BSON)
• db.namespace: e.g., test-db
• db.query.summary: e.g., find test-db.employees
• db.system.name: mongodb
• network.transport: tcp
• server.address: e.g., mongo
• otel.library.name: MongoDB.Driver
• otel.library.version: 3.7.0
• otel.status_code: STATUS_CODE_OK

Metrics:

• db.mongodb.driver_connection_id: e.g., 3.0
• db.mongodb.server_connection_id: e.g., 7.0
• server.port: e.g., 27017.0

Tag-by-Tag Comparison: OTel Spans vs DD Custom Spans

DD Custom mongodb.query span tags:

DD Custom span properties:

Tags present in OTel but NOT in DD custom spans:

• db.operation.name / db.command.name
• db.operation.summary / db.query.summary
• db.system.name
• db.mongodb.lsid (session ID - L2 only)
• network.transport (L2 only)
• server.address (L2 only — similar to out.host)
• otel.library.name, otel.library.version
• otel.status_code

Metrics present in OTel but NOT in DD custom spans:

• db.mongodb.driver_connection_id (L2 only)
• db.mongodb.server_connection_id (L2 only)
• server.port (L2 only — similar to out.port)

Operation Name Mapping: countDocuments vs aggregate

Notable: The OTel L1 span uses the logical operation name countDocuments, while our DD custom span (and OTel L2) use the wire protocol command aggregate. This is because MongoDB implements countDocuments() as an aggregate pipeline internally. The OTel L1 span provides the more user-friendly logical name.

Critical Gaps if Replacing DD Custom with OTel

1. mongodb.query tag — MISSING from OTel

The full BSON query body is the most significant tag in our custom instrumentation. Neither OTel level provides this. The OTel spans only include db.query.summary (e.g., find test-db.employees) which is just the operation + namespace, not the actual query filter/pipeline.

2. Span Type — http instead of mongodb

Both OTel span levels use Type http, while our custom spans correctly use Type mongodb. This affects categorization in the Datadog UI (DB queries vs HTTP calls).

3. Span Name — client.request instead of mongodb.query

Generic OTel name vs our specific operation name.

4. Service Name — No separate service (SchemaV0)

In SchemaV0, our custom spans use Samples.MongoDB-mongodb (separate service), while OTel spans use the app service name Samples.MongoDB. This means no dedicated MongoDB service in the service map.

5. Resource Name — Different format

• DD: find test-db (operation + database)
• OTel L1: find test-db.employees (operation + db.collection — more specific, arguably better)
• OTel L2: find (just the command — less useful)

6. component tag — MISSING from OTel

Our DD spans set component: MongoDb. OTel spans don't have this.

What OTel Does Better

1. Collection name in resource: find test-db.employees is more informative than find test-db
2. Logical operation names: countDocuments instead of aggregate (L1 only)
3. Connection metadata: driver_connection_id, server_connection_id, network.transport
4. Session tracking: db.mongodb.lsid
5. Semantic conventions: Uses standard OTel DB semantic conventions (db.namespace, db.system.name, etc.)

Summary / Recommendation

Cannot replace custom instrumentation with OTel spans. Key blockers:

1. No query body (mongodb.query) — This is the most valuable tag for debugging and is completely absent from OTel spans
2. Wrong span type (http instead of mongodb) — Would break DB categorization in Datadog UI
3. Wrong span name (client.request instead of mongodb.query) — Loses MongoDB-specific identification
4. No separate service name in SchemaV0 — Breaks service map topology
5. Doubled span count — 30 extra spans per trace adds overhead

Recommended approach: Disable the OTel instrumentation for MongoDB 3.7.0+ to maintain the existing behavior. This likely means either:

• Suppressing the MongoDB.Driver OTel ActivitySource at startup
• Or adjusting the integration to detect and skip OTel span creation when our instrumentation is active

If keeping both is desired for any reason, the OTel spans should at minimum be filtered out from the test snapshots, and ideally suppressed at runtime to avoid the 2.5x span multiplication.