Add Add X-Request-Id header to uniquely identify a search request by sgup432 · Pull Request #19798 · opensearch-project/OpenSearch

sgup432 · 2025-10-28T17:48:44Z

Description

This change adds support for a unique header which can be sent by the user to track a request.
This change is specifically tested/written for a search request, and also adds X-Request-Id support in search slow log as well.

Sample search slow log while the change was manually tested

[2025-10-28T03:57:33,764][TRACE][i.s.s.query              ] [runTask-0] [products][0] took[21.7ms], took_millis[21], total_hits[2 hits], stats[], search_type[QUERY_THEN_FETCH], total_shards[1], source[{"query":{"match":{"category":{"query":"electronics","operator":"OR","prefix_length":0,"max_expansions":50,"fuzzy_transpositions":true,"lenient":false,"zero_terms_query":"NONE","auto_generate_synonyms_phrase_query":true,"boost":1.0}}}}], id[], request-id[4bf92f3577b34da6a3ce929d0e0e4736]

Related Issues

#18512

Check List

Functionality includes testing.
~~[] API changes companion pull request created, if applicable.~~
Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Summary by CodeRabbit

New Features
- Added W3C Trace Context (traceparent) support and trace ID extraction/propagation across search, REST responses, transport requests, thread context, and task headers.
- Trace IDs are now included in slow logs and surfaced in responses/task metadata for improved observability.
Tests
- Added validators and extensive unit/integration tests for traceparent format and trace ID propagation.
- Tests updated to exercise header propagation; test suites can disable telemetry via feature flags.

_{✏️ Tip: You can customize this high-level summary in your review settings.}

Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

github-actions · 2025-10-28T18:19:31Z

❌ Gradle check result for 7dc64ff: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

github-actions · 2025-10-29T19:36:03Z

❌ Gradle check result for c994206: null

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

github-actions · 2025-10-30T08:58:04Z

❌ Gradle check result for 5f7eeb1: null

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

reta

@sgup432 My apologies, but I don't understand what the problem this pull request solves

the OpenSearch with OpenTelemetry plugin installed will automatically pick the trace from HTTP headers, we support all formats that OpenTelemetry provides [1]
the tracing context is being passed from threat to thread using ThreadContext and ThreadContextBasedTracerContextStorage [2], that would use one of the default context propagator OpenTelementry provides [2]
the tracing context is being injected into downstream HTTP calls using ThreadContext
the tracing context is being injected into downstream transport calls using ThreadContext as well

There is no need to introduce any custom headers here - we purely rely on the telemetry provider (in this case, OpenTelementry) for end to end context propagation, that already works (at least it did), if not - we have to fix that. Please let me know if you have any further questions.

Thank you.

[1] https://opentelemetry.io/docs/specs/otel/context/api-propagators/#propagators-distribution
[2] https://github.com/opensearch-project/OpenSearch/blob/main/server/src/main/java/org/opensearch/telemetry/tracing/ThreadContextBasedTracerContextStorage.java

sgup432 · 2025-11-03T06:49:11Z

@reta
While OpenTelemetry plugin does provide support to pick http headers and capture traces, but its optional and not part of OpenSearch core. So it does not come by default.

By having traceId accessible at the Task level as part of core, it can be logged in slow logs(which is the intention here). This enables users to correlate slow or failing requests directly to the same trace ID they passed via traceparent. And this can be done without explicitly installing Opentelemetry plugin.

In addition, once traceId is available at the Task level, it can be reused by other components (e.g., metrics, profiling, custom plugins).

There is no need to introduce any custom headers here

These are not custom headers, traceparent header is a standardized HTTP header defined by the W3C Trace Context specification. And OpenTelemetry has the support for same if passed by the user.

…ensearch-project#19798) * Add traceparent/traceId support Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Fixing SlowLogTests failure Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Fixing UT Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Adding granular checks on traceparent Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Adding changelog Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Addressing some comments Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Fixing UT Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Removing uppercase HEX digits support for traceparent Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Minor refactor Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Update javadoc Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Fixing unit test due to tracing Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Fix netty unit test Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Minor refactor Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Muting test until fix Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Removing awaitsFix annotation Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Fixing http smoke test Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Retrigger gradle build Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Typo correction Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Minor correction Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Retrigger build Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Retrigger build again Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Using X-Request-Id to uniquely identify a request Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Update changelog Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Remvoing unnecessary changes Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * removing unncessary changes part 2 Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Update server/src/main/java/org/opensearch/index/SearchSlowLog.java Co-authored-by: Andriy Redko <drreta@gmail.com> Signed-off-by: Sagar <99425694+sgup432@users.noreply.github.com> * Addressing minor comments Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Fix unit test Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Minor correction in naming Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Minor correction in javadoc Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Retrigger build Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Retrigger build Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> --------- Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> Signed-off-by: Sagar <99425694+sgup432@users.noreply.github.com> Co-authored-by: Andriy Redko <drreta@gmail.com>

Add traceparent/traceId support

7dc64ff

Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

sgup432 requested review from a team, Bukhtawar, CEHENKLE, Rishikesh1159, anasalkouz, andrross, ashking94, cwperks, dbwiddis, gbbafna, jed326, kotwanikunal, mch2, msfroh, owaiskazi19, peternied, reta, sachinpkale, saratvemulapalli, shwetathareja and sohami as code owners October 28, 2025 17:48

sgup432 changed the title ~~Add traceparent/traceId support~~ Add traceparent/traceId support for search Oct 28, 2025

Fixing SlowLogTests failure

c994206

Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

Fixing UT

5f7eeb1

Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

reta requested changes Oct 30, 2025

View reviewed changes

tmanninger mentioned this pull request Feb 20, 2026

[Feature Request] Make X-Request-Id validation configurable #20688

Closed

opensearch-ci-bot mentioned this pull request Feb 23, 2026

[AUTOCUT] Gradle Check Flaky Test Report for Gradle Test Run :test:framework:test #20708

Open

sgup432 mentioned this pull request Feb 23, 2026

[RFC] Comprehensive Search Latency Breakdown #20693

Open

cwperks mentioned this pull request Mar 9, 2026

[PROPOSAL] 3.5.1 Patch Release - X-Request-ID errors opensearch-project/technical-steering#90

Closed

This was referenced Mar 9, 2026

[AUTOCUT] Gradle Check Flaky Test Report for Gradle Test Run :server:test #19729

Closed

[AUTOCUT] Gradle Check Flaky Test Report for QueryPhaseTests #14551

Open

opensearch-ci-bot mentioned this pull request Mar 19, 2026

[AUTOCUT] Gradle Check Flaky Test Report for DetailedErrorsDisabledIT #20034

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Add X-Request-Id header to uniquely identify a search request#19798

Add Add X-Request-Id header to uniquely identify a search request#19798
reta merged 39 commits intoopensearch-project:mainfrom
sgup432:traceparent_support

sgup432 commented Oct 28, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Oct 28, 2025

Uh oh!

github-actions bot commented Oct 29, 2025

Uh oh!

github-actions bot commented Oct 30, 2025

Uh oh!

reta left a comment

Uh oh!

sgup432 commented Nov 3, 2025 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

sgup432 commented Oct 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issues

Check List

Summary by CodeRabbit

Uh oh!

github-actions bot commented Oct 28, 2025

Uh oh!

github-actions bot commented Oct 29, 2025

Uh oh!

github-actions bot commented Oct 30, 2025

Uh oh!

reta left a comment

Choose a reason for hiding this comment

Uh oh!

sgup432 commented Nov 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

sgup432 commented Oct 28, 2025 •

edited

Loading

sgup432 commented Nov 3, 2025 •

edited

Loading