Skip to content

OTLP Source unified endpoint for logs, traces and metrics#5677

Merged
kkondaka merged 27 commits intoopensearch-project:mainfrom
ps48:otel-telemetry-source
Jun 26, 2025
Merged

OTLP Source unified endpoint for logs, traces and metrics#5677
kkondaka merged 27 commits intoopensearch-project:mainfrom
ps48:otel-telemetry-source

Conversation

@ps48
Copy link
Copy Markdown
Member

@ps48 ps48 commented May 4, 2025 •
edited
Loading

Description

OTLP Source unifies the three existing source OTel-logs,traces and metrics sources. This provides ease of use for a single endpoint for the three sources separated by path. It re-uses most of the existing APIs and services including the latest http-common library for GRPCserver creation.

Screenshot 2025-05-06 at 11 03 06 AM

Major decision points made in the PR:

  1. Port is kept as 21893, logically made sense and 21890/1/2 are taken up by logs, traces and metrics.
  2. The paths are separated out in the config for each type of telemetry and is made optional.
    3. The codec not configurable to the older format. My argument here is we need to move the new uses to the new OTel format. Updated to the below based on comments
  3. The codec is now configurable defaulting to otel, can be configured to opensearch format if required by the users.

Note: All the above decisions can be changed if, needed.

Adding some sample configuration on usage from README.md here:

otel-telemetry-pipeline:
  source:
    otlp:
      ssl: false
  route:
    - logs: 'getEventType() == "LOG"'
    - traces: 'getEventType() == "TRACE"'
    - metrics: 'getEventType() == "METRIC"'
  sink:
    - pipeline:
        name: "logs-pipeline"
        routes:
          - "logs"
    - pipeline:
        name: "traces-pipeline"
        routes:
          - "traces"
    - pipeline:
        name: "metrics-pipeline"
        routes:
          - "metrics"

Issues Resolved

#5596

Check List

  • New functionality includes testing.
  • New functionality has a documentation issue. Please link to it in this PR.
    • New functionality has javadoc added
  • Commits are signed with a real name per the DCO

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@KarstenSchnitter KarstenSchnitter KarstenSchnitter left review comments

@dlvenable dlvenable dlvenable approved these changes

@kkondaka kkondaka kkondaka approved these changes

@sb2k16 sb2k16 Awaiting requested review from sb2k16 sb2k16 is a code owner

@chenqi0805 chenqi0805 Awaiting requested review from chenqi0805

@engechas engechas Awaiting requested review from engechas

@san81 san81 Awaiting requested review from san81 san81 is a code owner

@srikanthjg srikanthjg Awaiting requested review from srikanthjg srikanthjg is a code owner

@graytaylor0 graytaylor0 Awaiting requested review from graytaylor0 graytaylor0 is a code owner

@dinujoh dinujoh Awaiting requested review from dinujoh dinujoh is a code owner

@oeyh oeyh Awaiting requested review from oeyh oeyh is a code owner

Assignees

No one assigned

Labels

backport 2.12

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@ps48 @dlvenable @KarstenSchnitter @kkondaka