Skip to content

Support OpenTelemetry OTLP/HTTP as addition to OTLP/gRPC #4983

@KarstenSchnitter

Description

@KarstenSchnitter

Is your feature request related to a problem? Please describe.
The current OpenTelemetry specification describes two transport channels:

  • OTLP/gRPC, which is currently supported by DataPrepper for logs, metrics, and traces;
  • OTLP/HTTP, using a binary(protobuf) or JSON format.

All formats use the same protobuf scheme. OTLP/HTTP is currently not supported by DataPrepper. The OpenTelemetry specification has changed since the original implementation of the DataPrepper OTel sources. It now recommends OTLP/HTTP + protobuf to be the default protocol per opentelemetry-specification#1885. Many OpenTelemetry SDKs nowadays use OTLP/HTTP that way and not all OpenTelemetry instrumentations even support OTLP/gRPC. Connecting such solutions to DataPrepper is not possible without a protocol translation.

Describe the solution you'd like
DataPrepper should add support for OTLP/HTTP to the Otel*Sources. The configuration should enable a selection of the protocol to be supported. Ideally, it is possible to use both protocols simultaneously. This enables connecting different services to the same DataPrepper instance.

Describe alternatives you've considered (Optional)

  1. The OpenTelemetry Collector can be used to translate between OTLP/gRPC and OTLP/HTTP and vice versa. However, this requires an additional component in the signal stream. Direct support by DataPrepper would be a better approach.

  2. The HTTP source could be used for the OTLP/HTTP with JSON format. It would require a complex JSON parsing configuration due to the nested arrays in the OTel data structures. In this configuration, the data would also not pass through the OTel processors easily.

Additional context
There has been a previous issue about OTLP/HTTP support in the OpenDistro project: opendistro-for-elasticsearch/data-prepper#283.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    Status

    Done

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions