feat(otlp-sink): Add OTLP sink plugin for exporting spans to AWS X-Ray by huypham612 · Pull Request #5664 · opensearch-project/data-prepper

huypham612 · 2025-04-30T17:51:16Z

Description

This PR introduces a new otlp sink plugin for Data Prepper that supports exporting OpenTelemetry spans to AWS X-Ray over HTTP using OTLP.

Key features:

Sends ExportTraceServiceRequest messages over HTTP to X-Ray-compatible endpoints
Supports gzip compression and AWS SigV4 signing
Retries with exponential backoff and jitter on retryable HTTP errors
Metrics integration (delivery latency, payload size, retries, errors, etc.)
Batching support with max_events, max_batch_size, and flush_timeout thresholds
Fully unit-tested at 100% coverage
End-to-end and Performance tested with a local pipeline sending spans to a SigV4-signed OTLP endpoint (AWS X-Ray)

Issues Resolved

Resolves #5663

Check List

New functionality includes testing.
New functionality has a documentation issue. Please link to it in this PR.
- New functionality has javadoc added
Commits are signed with a real name per the DCO

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

* feat(xray-otlp-sink): Add X-Ray OTLP Sink plugin skeleton - Added test resources and support for Span records - Added sample pipeline config and OTLP test span JSON under `src/test/resources` - Verified local pipeline ingest and logging using `grpcurl` - Added README with developer instructions for running and testing locally These changes establish the foundation for local testing and future X-Ray e2e testing. Signed-off-by: huy pham <huyp@amazon.com>

Signed-off-by: huy pham <huyp@amazon.com>

KarstenSchnitter

Let's have a discussion in the minimal feature set. I am missing mTLS and gRPC support. I will do a more thorough review of this PR next week.

dlvenable

Thank you @huypham612 for this contribution!

...p-sink/src/main/java/org/opensearch/dataprepper/plugins/sink/otlp/buffer/OtlpSinkBuffer.java

data-prepper-plugins/otlp-sink/build.gradle

...tlp-sink/src/main/java/org/opensearch/dataprepper/plugins/sink/otlp/http/OtlpHttpSender.java

KarstenSchnitter

I support @dlvenable suggestion to use another http client. I also suggest to be more distinctive on the error metrics, so we can distinguish message drops because of processing errors from those rejected by the target. Please also have a look at the retry-after header as described here: https://opentelemetry.io/docs/specs/otlp/#otlphttp-throttling

data-prepper-plugins/otlp-sink/README.md

...p-sink/src/main/java/org/opensearch/dataprepper/plugins/sink/otlp/buffer/OtlpSinkBuffer.java

...tlp-sink/src/main/java/org/opensearch/dataprepper/plugins/sink/otlp/http/OtlpHttpSender.java

...otlp-sink/src/main/java/org/opensearch/dataprepper/plugins/sink/otlp/http/ThreadSleeper.java

...sink/src/main/java/org/opensearch/dataprepper/plugins/sink/otlp/metrics/OtlpSinkMetrics.java

huypham612 · 2025-05-01T18:51:24Z

Thank you for your feedback @dlvenable and @KarstenSchnitter, I will go through them and revise the code accordingly.

...p-sink/src/main/java/org/opensearch/dataprepper/plugins/sink/otlp/buffer/OtlpSinkBuffer.java

...java/org/opensearch/dataprepper/plugins/sink/otlp/configuration/AwsAuthenticationConfig.java

...tlp-sink/src/main/java/org/opensearch/dataprepper/plugins/sink/otlp/http/OtlpHttpSender.java

huypham612 · 2025-05-08T20:44:24Z

I support @dlvenable suggestion to use another http client. I also suggest to be more distinctive on the error metrics, so we can distinguish message drops because of processing errors from those rejected by the target. Please also have a look at the retry-after header as described here: https://opentelemetry.io/docs/specs/otlp/#otlphttp-throttling

Thank you, I will address this comment and update the code to handle this header. Good catch.

huypham612 · 2025-05-16T00:03:42Z

Hi @dlvenable, @KarstenSchnitter, and @kkondaka,

I've pushed the updated changes based on the feedback. Here’s a summary of the key updates:

Migrated to Armeria HTTP client.
Added failedSpansCount metric for improved observability.
Introduced ExecutorService to manage the buffer queue and ensure resilience.
Allowed unbounded max_events by supporting 0 as a valid value.
Reused the shared AwsConfig class for consistency with other sinks.
Added unit tests to cover 100%.
Re-ran performance tests and confirmed the implementation works as expected.

Note: I evaluated using the Retry-After header for dynamic backoff, but decided against it due to the following trade-offs:

Armeria’s retry rule API only supports boolean predicates or fixed Backoff.
Supporting Retry-After would require a custom Backoff implementation, which adds complexity with minimal benefit for most OTLP endpoints.
The existing exponential backoff with jitter strategy already handles typical retry scenarios effectively.

Please take another look when you have time — thank you!

...p-sink/src/main/java/org/opensearch/dataprepper/plugins/sink/otlp/buffer/OtlpSinkBuffer.java

data-prepper-plugins/otlp-sink/README.md

data-prepper-plugins/otlp-sink/build.gradle

...src/main/java/org/opensearch/dataprepper/plugins/sink/otlp/configuration/OtlpSinkConfig.java

...s/otlp-sink/src/main/java/org/opensearch/dataprepper/plugins/sink/otlp/http/SigV4Signer.java

...tlp-sink/src/main/java/org/opensearch/dataprepper/plugins/sink/otlp/http/OtlpHttpSender.java

...r-plugins/otlp-sink/src/main/java/org/opensearch/dataprepper/plugins/sink/otlp/OtlpSink.java

...p-sink/src/main/java/org/opensearch/dataprepper/plugins/sink/otlp/buffer/OtlpSinkBuffer.java

...r-plugins/otlp-sink/src/main/java/org/opensearch/dataprepper/plugins/sink/otlp/OtlpSink.java

...src/main/java/org/opensearch/dataprepper/plugins/sink/otlp/configuration/OtlpSinkConfig.java

Signed-off-by: huy pham <huyp@amazon.com>

- Introduce `ThresholdConfig` with `max_events`, `max_batch_size`, and `flush_timeout` - Implement `OtlpSinkBuffer` using a bounded `LinkedBlockingQueue` driven by threshold settings - Enhance `OtlpSinkMetrics` to register queue gauges and cover new payload/latency metrics - Update README with configuration options and note on AWS region derivation - Add comprehensive unit tests for config, buffer, metrics, and HTTP sender - Removed unused integ test package and resources Signed-off-by: huy pham <huyp@amazon.com>

- Changed logic to get the aws region value from the provided endpoint. Signed-off-by: huy pham <huyp@amazon.com>

- Retries on 429, 502, 503, and 504 per OTEL spec. Logs other errors without retry. - See: https://opentelemetry.io/docs/specs/otlp/exporter/#retrying-on-failure Signed-off-by: huy pham <huyp@amazon.com>

- removed log.info statements - Add description to exception Signed-off-by: huy pham <huyp@amazon.com>

- Updated worker loop to retry on interrupt until queue is empty - Avoid logging error on expected shutdown interrupt - test: Added unit tests to verify final flush Signed-off-by: huy pham <huyp@amazon.com>

- Removed unused metrics Signed-off-by: huy pham <huyp@amazon.com>

Signed-off-by: huy pham <huyp@amazon.com>

…ments Signed-off-by: huy pham <huyp@amazon.com>

…ialsSupplier. Signed-off-by: huy pham <huyp@amazon.com>

- Updated README - Fixed init bug - Performed e2e and perf tests Signed-off-by: huy pham <huyp@amazon.com>

Signed-off-by: huy pham <huyp@amazon.com>

kkondaka · 2025-05-27T05:14:01Z

...tlp-sink/src/main/java/org/opensearch/dataprepper/plugins/sink/otlp/http/OtlpHttpSender.java

+        }
+
+        // Defensive copy to avoid ConcurrentModificationException
+        final List<Pair<ResourceSpans, EventHandle>> immutableBatch = List.copyOf(batch);


Usually, we use synchronization (using locks) to avoid concurrent modification. Not sure if making a copy is the best way to handle this. What do you think @dlvenable ?

I'm thinking because Armeria is async, the List.copyOf ensures immutability without locking overhead. Given the typical batch size is small (1MB, even up to 5MB), this is a practical tradeoff for correctness and performance in a trace sink.

This is a shallow copy. So we are just copying the list. The underlying ResourceSpans and EventHandle objects are not copied.

Signed-off-by: huy pham <huyp@amazon.com>

…entials Signed-off-by: huy pham <huyp@amazon.com>

dlvenable

Thank you @huypham612 for this great new feature!

…ject#5664) Add OTLP sink plugin for exporting spans to AWS X-Ray (opensearch-project#5664) Feature/xray init feat(xray-otlp-sink): Add X-Ray OTLP Sink plugin skeleton - Added test resources and support for Span records - Added sample pipeline config and OTLP test span JSON under `src/test/resources` - Verified local pipeline ingest and logging using `grpcurl` - Added README with developer instructions for running and testing locally These changes establish the foundation for local testing and future X-Ray e2e testing. Signed-off-by: huy pham <huyp@amazon.com> Signed-off-by: Heli <desaiheli17@gmail.com> Co-authored-by: Heli <desaiheli17@gmail.com> Signed-off-by: Jeffrey Aaron Jeyasingh <jeffreyaaron06@gmail.com>

huypham612 added 2 commits April 30, 2025 09:11

Add a simple integration-test (#2)

5978b82

Signed-off-by: huy pham <huyp@amazon.com>

huypham612 requested review from KarstenSchnitter, chenqi0805, dinujoh, dlvenable, engechas, graytaylor0, kkondaka, oeyh, san81, sb2k16 and srikanthjg as code owners April 30, 2025 17:51

KarstenSchnitter reviewed May 1, 2025

View reviewed changes

dlvenable requested changes May 1, 2025

View reviewed changes

KarstenSchnitter reviewed May 1, 2025

View reviewed changes

kkondaka reviewed May 2, 2025

View reviewed changes

huypham612 requested review from KarstenSchnitter, dlvenable and kkondaka May 16, 2025 00:05

kkondaka reviewed May 20, 2025

View reviewed changes

dlvenable requested changes May 21, 2025

View reviewed changes

huypham612 requested review from dlvenable and kkondaka May 22, 2025 04:42

dlvenable requested changes May 23, 2025

View reviewed changes

...r-plugins/otlp-sink/src/main/java/org/opensearch/dataprepper/plugins/sink/otlp/OtlpSink.java Outdated Show resolved Hide resolved

...src/main/java/org/opensearch/dataprepper/plugins/sink/otlp/configuration/OtlpSinkConfig.java Outdated Show resolved Hide resolved

huypham612 requested a review from dlvenable May 23, 2025 23:26

dlvenable previously approved these changes May 24, 2025

View reviewed changes

huypham612 and others added 12 commits May 23, 2025 18:49

feature: add gzip and percentile metrics (#13)

4608a77

Signed-off-by: huy pham <huyp@amazon.com>

refactor(aws-region): get aws region from endpoint value

28bd811

- Changed logic to get the aws region value from the provided endpoint. Signed-off-by: huy pham <huyp@amazon.com>

refactor(http-retry-code): follow retryable code

e85cd1c

- Retries on 429, 502, 503, and 504 per OTEL spec. Logs other errors without retry. - See: https://opentelemetry.io/docs/specs/otlp/exporter/#retrying-on-failure Signed-off-by: huy pham <huyp@amazon.com>

refactor(log/exception): only use log.error

3477932

- removed log.info statements - Add description to exception Signed-off-by: huy pham <huyp@amazon.com>

fix(otlp-sink-buffer): Ensure final flush drains queue on shutdown

6506627

- Updated worker loop to retry on interrupt until queue is empty - Avoid logging error on expected shutdown interrupt - test: Added unit tests to verify final flush Signed-off-by: huy pham <huyp@amazon.com>

Update README

242ca15

- Removed unused metrics Signed-off-by: huy pham <huyp@amazon.com>

Removed unused Output codec

390f951

Signed-off-by: huy pham <huyp@amazon.com>

Update OTLP sink with Armeria, metrics, buffering, and config improve…

01671b1

…ments Signed-off-by: huy pham <huyp@amazon.com>

Release EventHandle on both success and failure; switch to AwsCredent…

2728591

…ialsSupplier. Signed-off-by: huy pham <huyp@amazon.com>

Avoid breaking change by making the aws config block required.

977b524

- Updated README - Fixed init bug - Performed e2e and perf tests Signed-off-by: huy pham <huyp@amazon.com>

Validate the aws config via the @AssertTrue pattern

bb4ffb0

Signed-off-by: huy pham <huyp@amazon.com>

huypham612 force-pushed the otlp-sink-plugin branch from 7a9bebc to bb4ffb0 Compare May 24, 2025 01:51

kkondaka reviewed May 27, 2025

View reviewed changes

Add missing unit test to reach 100%

01851ea

Signed-off-by: huy pham <huyp@amazon.com>

huypham612 dismissed dlvenable’s stale review via 01851ea May 30, 2025 14:51

huypham612 requested review from dlvenable and kkondaka May 30, 2025 14:59

dlvenable previously approved these changes May 30, 2025

View reviewed changes

Resolve merge conflicts with upstream main

af11f60

Signed-off-by: huy pham <huyp@amazon.com>

huypham612 dismissed dlvenable’s stale review via af11f60 May 30, 2025 15:58

huypham612 requested a review from dlvenable May 30, 2025 16:00

dlvenable previously approved these changes May 30, 2025

View reviewed changes

Mock AWS credentials provider to allow tests to run without real cred…

8527a38

…entials Signed-off-by: huy pham <huyp@amazon.com>

huypham612 dismissed dlvenable’s stale review via 8527a38 May 30, 2025 18:05

huypham612 requested a review from dlvenable May 30, 2025 18:10

kkondaka approved these changes May 30, 2025

View reviewed changes

dlvenable approved these changes Jun 3, 2025

View reviewed changes

dlvenable merged commit 35795aa into opensearch-project:main Jun 3, 2025
67 of 74 checks passed

Conversation

huypham612 commented Apr 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Issues Resolved

Check List

Uh oh!

KarstenSchnitter left a comment

Choose a reason for hiding this comment

Uh oh!

dlvenable left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

KarstenSchnitter left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

huypham612 commented May 1, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

huypham612 commented May 8, 2025

Uh oh!

huypham612 commented May 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kkondaka May 27, 2025

Choose a reason for hiding this comment

Uh oh!

huypham612 May 27, 2025

Choose a reason for hiding this comment

Uh oh!

dlvenable May 27, 2025

Choose a reason for hiding this comment

Uh oh!

dlvenable left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

huypham612 commented Apr 30, 2025 •

edited

Loading

KarstenSchnitter left a comment •

edited

Loading

huypham612 commented May 16, 2025 •

edited

Loading