-
Notifications
You must be signed in to change notification settings - Fork 313
Support OpenTelemetry OTLP/HTTP as addition to OTLP/gRPC #4983
Description
Is your feature request related to a problem? Please describe.
The current OpenTelemetry specification describes two transport channels:
- OTLP/gRPC, which is currently supported by DataPrepper for logs, metrics, and traces;
- OTLP/HTTP, using a binary(protobuf) or JSON format.
All formats use the same protobuf scheme. OTLP/HTTP is currently not supported by DataPrepper. The OpenTelemetry specification has changed since the original implementation of the DataPrepper OTel sources. It now recommends OTLP/HTTP + protobuf to be the default protocol per opentelemetry-specification#1885. Many OpenTelemetry SDKs nowadays use OTLP/HTTP that way and not all OpenTelemetry instrumentations even support OTLP/gRPC. Connecting such solutions to DataPrepper is not possible without a protocol translation.
Describe the solution you'd like
DataPrepper should add support for OTLP/HTTP to the Otel*Sources. The configuration should enable a selection of the protocol to be supported. Ideally, it is possible to use both protocols simultaneously. This enables connecting different services to the same DataPrepper instance.
Describe alternatives you've considered (Optional)
-
The OpenTelemetry Collector can be used to translate between OTLP/gRPC and OTLP/HTTP and vice versa. However, this requires an additional component in the signal stream. Direct support by DataPrepper would be a better approach.
-
The HTTP source could be used for the OTLP/HTTP with JSON format. It would require a complex JSON parsing configuration due to the nested arrays in the OTel data structures. In this configuration, the data would also not pass through the OTel processors easily.
Additional context
There has been a previous issue about OTLP/HTTP support in the OpenDistro project: opendistro-for-elasticsearch/data-prepper#283.
Metadata
Metadata
Assignees
Labels
Type
Projects
Status