Skip to content

Add otlp sink #5663

@huypham612

Description

@huypham612

Is your feature request related to a problem? Please describe.
Data Prepper currently lacks a native OTLP-compatible sink plugin for sending traces to AWS X-Ray or other OTLP-compliant backends.

Describe the solution you'd like
Add a new otlp sink plugin that accepts OpenTelemetry trace payloads (ExportTraceServiceRequest) and forwards them via HTTP to OTLP endpoints, with support for:

  • AWS SigV4 signing (for sending to X-Ray)
  • gzip compression
  • Retry logic with backoff
  • Configurable thresholds for batch size, event count, and flush timeout

The initial release supports exporting spans to AWS X-Ray. Future releases will support sending spans, metrics, and logs to any OTLP Protobuf-compatible endpoint.

Describe alternatives you've considered (Optional)

  • Using the http sink to forward traces: This approach requires additional enhancements that are not natively supported, such as OTLP Protobuf encoding, AWS SigV4 signing, and retry handling. As a result, it increases configuration complexity and risks incorrect behavior under production loads.

Additional context
The implementation follows the Data Prepper plugin model and supports configuration via YAML. It is tested against high-throughput trace ingestion scenarios and shows strong performance characteristics under load.

Metadata

Metadata

Assignees

Labels

enhancementNew feature or requestplugin - sinkA plugin to write data to a destination.

Type

No type

Projects

Status

Done

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions