-
Notifications
You must be signed in to change notification settings - Fork 313
Add otlp sink #5663
Description
Is your feature request related to a problem? Please describe.
Data Prepper currently lacks a native OTLP-compatible sink plugin for sending traces to AWS X-Ray or other OTLP-compliant backends.
Describe the solution you'd like
Add a new otlp sink plugin that accepts OpenTelemetry trace payloads (ExportTraceServiceRequest) and forwards them via HTTP to OTLP endpoints, with support for:
- AWS SigV4 signing (for sending to X-Ray)
- gzip compression
- Retry logic with backoff
- Configurable thresholds for batch size, event count, and flush timeout
The initial release supports exporting spans to AWS X-Ray. Future releases will support sending spans, metrics, and logs to any OTLP Protobuf-compatible endpoint.
Describe alternatives you've considered (Optional)
- Using the http sink to forward traces: This approach requires additional enhancements that are not natively supported, such as OTLP Protobuf encoding, AWS SigV4 signing, and retry handling. As a result, it increases configuration complexity and risks incorrect behavior under production loads.
Additional context
The implementation follows the Data Prepper plugin model and supports configuration via YAML. It is tested against high-throughput trace ingestion scenarios and shows strong performance characteristics under load.
Metadata
Metadata
Assignees
Labels
Type
Projects
Status