[RW]: Adopt client_golang/exp/api/remote types for receiving RW1 and RW2 by pipiland2612 · Pull Request #17197 · prometheus/prometheus

pipiland2612 · 2025-09-16T08:02:50Z

Which issue(s) does the PR fix:

Solves part of [RW2]: Prometheus client_golang remote API adoption #17192
Continues work on write_handler: Use client_golang/exp/api/remote for handling remote writes #16086

Scope

This PR migrates Prometheus to use the official github.com/prometheus/client_golang/exp/api/remote package types for remote write protocol handling. This change covers:

What's included:

Receiving side: Write handler now uses exp/api/remote types for processing both RW1 (prometheus.WriteRequest) and RW2 (io.prometheus.write.v2.Request) incoming requests
Sending side: Queue manager and write client updated to use exp/api/remote types when sending remote write requests (both RW1 and RW2)
Replaces config.RemoteWriteProtoMsg with remoteapi.WriteMessageType throughout the codebase
Removes duplicate protocol type definitions from Prometheus config package

What's NOT included:

No changes to protocol behavior or specifications
No functional changes to how remote write works
The Remote Write protocols (1.0 and 2.0) remain stable and unchanged

Does this PR introduce a user-facing change?

NONE

bboreham · 2025-09-16T11:07:03Z

Thanks for this. I guess @bwplotka should take a look.

The exp area is declared Experimental, so can we rely on it in Prometheus?

pipiland2612 · 2025-09-16T11:32:25Z

Hi @bboreham, thanks for check out my PR. I think @bwplotka has already explained this quite carefully here. You can check it out: #16086 (comment)

storage/remote/client.go

storage/remote/write_handler.go

saswatamcode

LGTM! Great work 💪🏻

Some small nits!

Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> # Conflicts: # storage/remote/write_handler.go

Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com>

pipiland2612 · 2025-09-24T11:39:22Z

cc @bwplotka

Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com>

bwplotka

Initial look - looks great! Some small comments, thanks!

cmd/prometheus/main.go

storage/remote/client.go

storage/remote/write_handler.go

bwplotka

Checked the rest, generally LGTM, modulo above comments, great work! 💪🏽

Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com>

pipiland2612 · 2025-10-06T11:48:11Z

Hi @bwplotka, thanks for pointing those out. Please take a look again when you have time. I have fixed at 0639af2

bwplotka

LGTM, thanks!

Before merging, let's ensure (wait for response) @bboreham is still objecting this change after our explanation, also pinged on prometheus-dev https://cloud-native.slack.com/archives/C01AUBA4PFE/p1759757051305119

bboreham · 2025-10-07T10:25:35Z

In regard to "experimental", I would prefer if the comment was modified to say something different to "The module may be contain breaking changes or be removed in the future."
What about "This code is used by released versions of Prometheus, but should not be relied upon outside of the Prometheus project" ?

bboreham · 2025-10-07T10:27:16Z

Can you clarify the title "Use client_golang_exp/api/remote for handling remote writes" please?
Looks like this is focused on configuration and RW2. I do not see RW1 structs in client_golang.

EDIT: according to a Slack comment, this PR is for receiving, and sending will be changed later.
Please make the scope clear.

bboreham · 2025-10-07T10:28:33Z

Is RW2 covered by PromBench?

bwplotka · 2025-10-07T11:06:24Z

Great points, do you mind @pipiland2612 helping with client_golang/exp module documentation and this PR title description? (@bboreham is right that "handling" remote write can mean both sending and receiving, in this case it's receiving).

Is RW2 covered by PromBench?

Yes (prometheus/test-infra#845) -- although for sending, which we don't change here.

We don't have prombench for receiving for anything AFAIK

storage/remote/write_handler.go

Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com>

pipiland2612 · 2025-10-17T07:24:15Z

Can you clarify the title "Use client_golang_exp/api/remote for handling remote writes" please? Looks like this is focused on configuration and RW2. I do not see RW1 structs in client_golang.

EDIT: according to a Slack comment, this PR is for receiving, and sending will be changed later. Please make the scope clear.

Hi @bboreham, Indeed, thanks for suggesting. I have updated the PR title and descriptions

pipiland2612 · 2025-10-17T07:39:12Z

In regard to "experimental", I would prefer if the comment was modified to say something different to "The module may be contain breaking changes or be removed in the future." What about "This code is used by released versions of Prometheus, but should not be relied upon outside of the Prometheus project" ?

Thanks! Making changes at https://github.com/prometheus/client_golang/pull/1894/files. PTAL

bboreham · 2025-10-17T10:14:28Z

Thanks, the scope and impact is much clearer now.

bwplotka

Thanks!

bwplotka · 2025-10-24T09:32:32Z

I changed title and squashed commit msg to include "receiving" word, otherwise it looks like we adopt this lib for all RW interactions in this PR. Next time, please ensure clear PR titles and descriptions, it's quite important. @pipiland2612 (:

## Summary This PR upgrades the `github.com/prometheus/prometheus` dependency from v0.307.3 to v0.308.0. ## Changes Made #### Fixed `RemoteWriteProtoMsg` unrecognized In prometheus/prometheus v0.308.0, the remote write protocol message types were moved from `github.com/prometheus/prometheus/config` to `github.com/prometheus/client_golang/exp/api/remote` as part of [prometheus/prometheus#17197](prometheus/prometheus#17197). #### Fixed `CreatedTimestamp` → `StartTimestamp` In [prometheus/prometheus#17411](prometheus/prometheus#17411), the Remote Write 2.0 spec was updated to 2.0-rc.4. The `CreatedTimestamp` field was renamed to `StartTimestamp` and moved from `TimeSeries` to individual `Sample` and `Histogram` messages. #### Fixed not enough arguments in`api_v1.NewAPI` The `api_v1.NewAPI` function signature changed due to [prometheus/prometheus#17470](prometheus/prometheus#17470) - Added `appendMetadata bool` parameter for remote write metadata handling. Added the new parameter `appendMetadata = false` - prometheusreceiver scrapes targets, doesn't receive remote write requests ## Remaining Issues ### `EnableNativeHistogramsIngestion` removed from `scrape.Options` There's one change in this upgrade that I'd love to get the community's input on before finalizing. In [prometheus/prometheus#17315](prometheus/prometheus#17315), Prometheus removed the `EnableNativeHistogramsIngestion` field from `scrape.Options`. Native histogram ingestion is now controlled through the Prometheus scrape config option `scrape_native_histograms` rather than programmatically. This creates an interesting situation for us. Currently, we have the feature gate `receiver.prometheusreceiver.EnableNativeHistograms` that controls whether native histograms are converted to OTel exponential histograms. But with this Prometheus change, the *ingestion* of native histograms during scraping is now a separate concern controlled by Prometheus config. I see two paths forward here. We could automatically inject `scrape_native_histograms: true` into the scrape config whenever our feature gate is enabled, preserving the current behavior where enabling the feature gate is all users need to do. Alternatively, we could simply remove the line and require users to configure `scrape_native_histograms: true` in their Prometheus scrape config themselves, which is more explicit but adds an extra configuration step. What do you think? I'm also wondering if this constitutes a breaking change that should be called out in the changelog. Would appreciate any thoughts! --------- Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com>

pipiland2612 requested review from bwplotka, cstyan and tomwilkie as code owners September 16, 2025 08:02

pipiland2612 changed the title ~~revert~~ Use client_golang_exp/api/remote for handling remote writes Sep 16, 2025

pipiland2612 closed this Sep 16, 2025

pipiland2612 reopened this Sep 16, 2025

roidelapluie mentioned this pull request Sep 16, 2025

remote_write: return 422 when native histograms are sent but disabled; add tests #17196

Open

saswatamcode mentioned this pull request Sep 22, 2025

write_handler: Use client_golang/exp/api/remote for handling remote writes #16086

Closed

saswatamcode reviewed Sep 22, 2025

View reviewed changes

storage/remote/client.go Show resolved Hide resolved

saswatamcode reviewed Sep 22, 2025

View reviewed changes

storage/remote/write_handler.go Outdated Show resolved Hide resolved

saswatamcode reviewed Sep 22, 2025

View reviewed changes

storage/remote/write_handler.go Outdated Show resolved Hide resolved

saswatamcode approved these changes Sep 22, 2025

View reviewed changes

pipiland2612 added 2 commits September 23, 2025 14:55

resolves conflicts

104a9d3

Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> # Conflicts: # storage/remote/write_handler.go

add comment

64a7843

Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com>

pipiland2612 force-pushed the Official_client branch from dbf9b5e to 64a7843 Compare September 23, 2025 11:58

pipiland2612 added 2 commits September 23, 2025 15:04

fix

d24d90e

Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com>

solved conflicts

85a0734

Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com>

pipiland2612 force-pushed the Official_client branch from 5adabdf to 85a0734 Compare September 24, 2025 11:27

fix failling test

bfe489a

Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com>

pipiland2612 closed this Sep 24, 2025

pipiland2612 reopened this Sep 24, 2025

fix conflicts

11d610a

Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com>

pipiland2612 mentioned this pull request Oct 4, 2025

feat(promtool): add RW2 support to promtool push metrics using client_golang library #17280

Merged

bwplotka mentioned this pull request Oct 6, 2025

[RW2]: Prometheus client_golang remote API adoption #17192

Open

bwplotka reviewed Oct 6, 2025

View reviewed changes

nit_fixing

0639af2

Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com>

fix comment

7f2ed8a

Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com>

bwplotka approved these changes Oct 6, 2025

View reviewed changes

bwplotka reviewed Oct 7, 2025

View reviewed changes

storage/remote/write_handler.go Show resolved Hide resolved

bwplotka reviewed Oct 7, 2025

View reviewed changes

storage/remote/write_handler.go Show resolved Hide resolved

fix conflict

59ea74d

Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com>

pipiland2612 changed the title ~~Use client_golang_exp/api/remote for handling remote writes~~ [RW]: Adopt client_golang/exp/api/remote types for RW1 and RW2 Oct 17, 2025

pipiland2612 mentioned this pull request Oct 17, 2025

Clarify exp library stability prometheus/client_golang#1894

Merged

bwplotka approved these changes Oct 24, 2025

View reviewed changes

bwplotka changed the title ~~[RW]: Adopt client_golang/exp/api/remote types for RW1 and RW2~~ [RW]: Adopt client_golang/exp/api/remote types for receiving RW1 and RW2 Oct 24, 2025

bwplotka merged commit f070e35 into prometheus:main Oct 24, 2025
31 checks passed

pipiland2612 deleted the Official_client branch October 24, 2025 09:55

ArthurSens mentioned this pull request Dec 9, 2025

Bump prometheus/prometheus to v0.308.0 open-telemetry/opentelemetry-collector-contrib#44861

Merged

kgeckhart mentioned this pull request Dec 10, 2025

v3.8.0: "we got 2xx status code from the Receiver yet statistics indicate some data was not written; investigation needed" #17659

Closed

Conversation

pipiland2612 commented Sep 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue(s) does the PR fix:

Scope

Does this PR introduce a user-facing change?

Uh oh!

bboreham commented Sep 16, 2025

Uh oh!

pipiland2612 commented Sep 16, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

saswatamcode left a comment

Choose a reason for hiding this comment

Uh oh!

pipiland2612 commented Sep 24, 2025

Uh oh!

bwplotka left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bwplotka left a comment

Choose a reason for hiding this comment

Uh oh!

pipiland2612 commented Oct 6, 2025

Uh oh!

bwplotka left a comment

Choose a reason for hiding this comment

Uh oh!

bboreham commented Oct 7, 2025

Uh oh!

bboreham commented Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bboreham commented Oct 7, 2025

Uh oh!

bwplotka commented Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pipiland2612 commented Oct 17, 2025

Uh oh!

pipiland2612 commented Oct 17, 2025

Uh oh!

bboreham commented Oct 17, 2025

Uh oh!

bwplotka left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

bwplotka commented Oct 24, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

pipiland2612 commented Sep 16, 2025 •

edited

Loading

bboreham commented Oct 7, 2025 •

edited

Loading

bwplotka commented Oct 7, 2025 •

edited

Loading