Remote Write Receive Fix: Remove duplicate labels when type-and-unit-label feature is on by pipiland2612 · Pull Request #17546 · prometheus/prometheus

pipiland2612 · 2025-11-17T18:30:40Z

Which issue(s) does the PR fix:

Fixes [RW2] type-and-unit-labels not visible when using Prometheus Agent with Prometheus Remote Write v2 #17480

Manual testing

I use the setup at [RW2] type-and-unit-labels not visible when using Prometheus Agent with Prometheus Remote Write v2 #17480 (comment) and the result I get:

{
  "__name__": "node_cpu_seconds_total",
  "__type__": "counter",
  "cpu": "0",
  "instance": "localhost:9100",
  "job": "node-exporter",
  "mode": "idle"
}

and the sender log doesn't show any error anymore:

time=2025-11-17T20:23:41.327+02:00 level=INFO source=watcher.go:292 msg="Replaying WAL" component=remote remote_name="Remote write to prometheus" url=http://127.0.0.1:9090/api/v1/write queue="Remote write to prometheus"
time=2025-11-17T20:23:41.327+02:00 level=DEBUG source=watcher.go:312 msg="Tailing WAL" component=remote remote_name="Remote write to prometheus" url=http://127.0.0.1:9090/api/v1/write lastCheckpoint="" checkpointIndex=0 currentSegment=0 lastSegment=5
time=2025-11-17T20:23:41.327+02:00 level=DEBUG source=watcher.go:318 msg="Processing segment" component=remote remote_name="Remote write to prometheus" url=http://127.0.0.1:9090/api/v1/write currentSegment=0
time=2025-11-17T20:23:41.334+02:00 level=DEBUG source=manager.go:305 msg="Starting provider" component="discovery manager scrape" provider=static/0 subs=map[node-exporter:{}]
time=2025-11-17T20:23:41.334+02:00 level=INFO source=main.go:1550 msg="Completed loading of configuration file" db_storage=375ns remote_storage=436.25µs web_handler=542ns query_engine=166ns scrape=7.400666ms scrape_sd=42.667µs notify=708ns notify_sd=500ns rules=250ns tracing=3.042µs filename=prometheus-agent.yml totalDuration=8.082166ms
time=2025-11-17T20:23:41.334+02:00 level=INFO source=main.go:1286 msg="Server is ready to receive web requests."
time=2025-11-17T20:23:41.334+02:00 level=DEBUG source=manager.go:345 msg="Discoverer channel closed" component="discovery manager scrape" provider=static/0
time=2025-11-17T20:23:48.290+02:00 level=INFO source=watcher.go:538 msg="Done replaying WAL" component=remote remote_name="Remote write to prometheus" url=http://127.0.0.1:9090/api/v1/write duration=6.963371792s
time=2025-11-17T20:24:01.328+02:00 level=DEBUG source=queue_manager.go:1184 msg=QueueManager.calculateDesiredShards component=remote remote_name="Remote write to prometheus" url=http://127.0.0.1:9090/api/v1/write dataInRate=103.44000000000001 dataOutRate=129.3 dataKeptRatio=1 dataPendingRate=-25.86 dataPending=0 dataOutDuration=0.0010581416999999999 timePerSample=8.183617169373549e-06 desiredShards=0.00084651336 highestSent=1.763403828e+09 highestRecv=1.763403828e+09
time=2025-11-17T20:24:01.328+02:00 level=DEBUG source=queue_manager.go:1202 msg=QueueManager.updateShardsLoop component=remote remote_name="Remote write to prometheus" url=http://127.0.0.1:9090/api/v1/write lowerBound=0.7 desiredShards=0.00084651336 upperBound=1.3
time=2025-11-17T20:24:33.268+02:00 level=DEBUG source=watcher.go:442 msg="Watcher is reading the WAL due to timeout, haven't received any write notifications recently" component=remote remote_name="Remote write to prometheus" url=http://127.0.0.1:9090/api/v1/write timeout=15s

Does this PR introduce a user-facing change?

[BUGFIX] Remote Write Receive: Remove duplicate labels when type-and-unit-label feature is on

bwplotka

Thanks! This is close but the bug is on receiving side I think

storage/remote/queue_manager.go

Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com>

bwplotka

Amazing thanks!

Do you mind updating PR title and Release notes to mention RW2 receiving?

bwplotka · 2025-11-17T21:32:22Z

Also wonder if OTLP receiving has similar bug (:

pipiland2612 · 2025-11-18T07:11:20Z

Amazing thanks!

Do you mind updating PR title and Release notes to mention RW2 receiving?

Thanks for suggesting, PTAL thanks!

Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com>

pipiland2612 · 2025-11-18T07:41:46Z

Also wonder if OTLP receiving has similar bug (:

Seems like it's not, otlp receiver use builder.Set:

prometheus/storage/remote/otlptranslator/prometheusremotewrite/helper.go

Line 152 in cefefc6

c.builder.Set(model.MetricTypeLabel, strings.ToLower(string(meta.Type)))

and builder.Set overwrites the value

prometheus/model/labels/labels_common.go

Line 192 in cefefc6

if a.Name == n {

so I think the duplicate is not happening at OTLP

bwplotka

Thanks!

bboreham · 2025-12-18T17:45:51Z

Hi, checking through the changelog for next release, I don't see this as "remove duplicate labels". We are choosing to overwrite any __type__ or __unit__ that comes in with the equivalent information generated from metadata.

"avoid duplicate labels" would be more correct, but what does it mean if they are supplied but inconsistent with metadata?
Looking at the issue, should Prometheus Agent avoid sending them?

##### [\`v3.9.0\`](https://github.com/prometheus/prometheus/releases/tag/v3.9.0) #### Note for users of Native Histograms In version 3.9, Native Histograms is no longer experimental, and the feature flag `native-histogram` has no effect. You must now turn on the config setting `scrape_native_histograms` to collect Native Histogram samples from exporters. #### Changelog - \[CHANGE] Native Histograms are no longer experimental! Make the `native-histogram` feature flag a no-op. Use `scrape_native_histograms` config option instead. [#17528](prometheus/prometheus#17528) - \[CHANGE] API: Add maximum limit of 10,000 sets of statistics to TSDB status endpoint. [#17647](prometheus/prometheus#17647) - \[FEATURE] API: Add /api/v1/features for clients to understand which features are supported. [#17427](prometheus/prometheus#17427) - \[FEATURE] Promtool: Add `start_timestamp` field for unit tests. [#17636](prometheus/prometheus#17636) - \[FEATURE] Promtool: Add `--format seriesjson` option to `tsdb dump` to output just series labels in JSON format. [#13409](prometheus/prometheus#13409) - \[FEATURE] Add `--storage.tsdb.delay-compact-file.path` flag for better interoperability with Thanos. [#17435](prometheus/prometheus#17435) - \[FEATURE] UI: Add an option on the query drop-down menu to duplicate that query panel. [#17714](prometheus/prometheus#17714) - \[ENHANCEMENT]: TSDB: add flag `--storage.tsdb.block-reload-interval` to configure TSDB Block Reload Interval. [#16728](prometheus/prometheus#16728) - \[ENHANCEMENT] UI: Add graph option to start the chart's Y axis at zero. [#17565](prometheus/prometheus#17565) - \[ENHANCEMENT] Scraping: Classic protobuf format no longer requires the unit in the metric name. [#16834](prometheus/prometheus#16834) - \[ENHANCEMENT] PromQL, Rules, SD, Scraping: Add native histograms to complement existing summaries. [#17374](prometheus/prometheus#17374) - \[ENHANCEMENT] Notifications: Add a histogram `prometheus_notifications_latency_histogram_seconds` to complement the existing summary. [#16637](prometheus/prometheus#16637) - \[ENHANCEMENT] Remote-write: Add custom scope support for AzureAD authentication. [#17483](prometheus/prometheus#17483) - \[ENHANCEMENT] SD: add a `config` label with job name for most `prometheus_sd_refresh` metrics. [#17138](prometheus/prometheus#17138) - \[ENHANCEMENT] TSDB: New histogram `prometheus_tsdb_sample_ooo_delta`, the distribution of out-of-order samples in seconds. Collected for all samples, accepted or not. [#17477](prometheus/prometheus#17477) - \[ENHANCEMENT] Remote-read: Validate histograms received via remote-read. [#17561](prometheus/prometheus#17561) - \[PERF] TSDB: Small optimizations to postings index. [#17439](prometheus/prometheus#17439) - \[PERF] Scraping: Speed up relabelling of series. [#17530](prometheus/prometheus#17530) - \[PERF] PromQL: Small optimisations in binary operators. [#17524](prometheus/prometheus#17524), [#17519](prometheus/prometheus#17519). - \[BUGFIX] UI: PromQL autocomplete now shows the correct type and HELP text for OpenMetrics counters whose samples end in `_total`. [#17682](prometheus/prometheus#17682) - \[BUGFIX] UI: Fixed codemirror-promql incorrectly showing label completion suggestions after the closing curly brace of a vector selector. [#17602](prometheus/prometheus#17602) - \[BUGFIX] UI: Query editor no longer suggests a duration unit if one is already present after a number. [#17605](prometheus/prometheus#17605) - \[BUGFIX] PromQL: Fix some "vector cannot contain metrics with the same labelset" errors when experimental delayed name removal is enabled. [#17678](prometheus/prometheus#17678) - \[BUGFIX] PromQL: Fix possible corruption of PromQL text if the query had an empty `ignoring()` and non-empty grouping. [#17643](prometheus/prometheus#17643) - \[BUGFIX] PromQL: Fix resets/changes to return empty results for anchored selectors when all samples are outside the range. [#17479](prometheus/prometheus#17479) - \[BUGFIX] PromQL: Check more consistently for many-to-one matching in filter binary operators. [#17668](prometheus/prometheus#17668) - \[BUGFIX] PromQL: Fix collision in unary negation with non-overlapping series. [#17708](prometheus/prometheus#17708) - \[BUGFIX] PromQL: Fix collision in label\_join and label\_replace with non-overlapping series. [#17703](prometheus/prometheus#17703) - \[BUGFIX] PromQL: Fix bug with inconsistent results for queries with OR expression when experimental delayed name removal is enabled. [#17161](prometheus/prometheus#17161) - \[BUGFIX] PromQL: Ensure that `rate`/`increase`/`delta` of histograms results in a gauge histogram. [#17608](prometheus/prometheus#17608) - \[BUGFIX] PromQL: Do not panic while iterating over invalid histograms. [#17559](prometheus/prometheus#17559) - \[BUGFIX] TSDB: Reject chunk files whose encoded chunk length overflows int. [#17533](prometheus/prometheus#17533) - \[BUGFIX] TSDB: Do not panic during resolution reduction of invalid histograms. [#17561](prometheus/prometheus#17561) - \[BUGFIX] Remote-write Receive: Avoid duplicate labels when experimental type-and-unit-label feature is enabled. [#17546](prometheus/prometheus#17546) - \[BUGFIX] OTLP Receiver: Only write metadata to disk when experimental metadata-wal-records feature is enabled. [#17472](prometheus/prometheus#17472)

##### [\`v3.9.1\`](https://github.com/prometheus/prometheus/releases/tag/v3.9.1) - \[BUGFIX] Agent: fix crash shortly after startup from invalid type of object. [#17802](prometheus/prometheus#17802) - \[BUGFIX] Scraping: fix relabel keep/drop not working. [#17807](prometheus/prometheus#17807) --- ##### [\`v3.9.0\`](https://github.com/prometheus/prometheus/releases/tag/v3.9.0) #### Note for users of Native Histograms In version 3.9, Native Histograms is no longer experimental, and the feature flag `native-histogram` has no effect. You must now turn on the config setting `scrape_native_histograms` to collect Native Histogram samples from exporters. #### Changelog - \[CHANGE] Native Histograms are no longer experimental! Make the `native-histogram` feature flag a no-op. Use `scrape_native_histograms` config option instead. [#17528](prometheus/prometheus#17528) - \[CHANGE] API: Add maximum limit of 10,000 sets of statistics to TSDB status endpoint. [#17647](prometheus/prometheus#17647) - \[FEATURE] API: Add /api/v1/features for clients to understand which features are supported. [#17427](prometheus/prometheus#17427) - \[FEATURE] Promtool: Add `start_timestamp` field for unit tests. [#17636](prometheus/prometheus#17636) - \[FEATURE] Promtool: Add `--format seriesjson` option to `tsdb dump` to output just series labels in JSON format. [#13409](prometheus/prometheus#13409) - \[FEATURE] Add `--storage.tsdb.delay-compact-file.path` flag for better interoperability with Thanos. [#17435](prometheus/prometheus#17435) - \[FEATURE] UI: Add an option on the query drop-down menu to duplicate that query panel. [#17714](prometheus/prometheus#17714) - \[ENHANCEMENT]: TSDB: add flag `--storage.tsdb.block-reload-interval` to configure TSDB Block Reload Interval. [#16728](prometheus/prometheus#16728) - \[ENHANCEMENT] UI: Add graph option to start the chart's Y axis at zero. [#17565](prometheus/prometheus#17565) - \[ENHANCEMENT] Scraping: Classic protobuf format no longer requires the unit in the metric name. [#16834](prometheus/prometheus#16834) - \[ENHANCEMENT] PromQL, Rules, SD, Scraping: Add native histograms to complement existing summaries. [#17374](prometheus/prometheus#17374) - \[ENHANCEMENT] Notifications: Add a histogram `prometheus_notifications_latency_histogram_seconds` to complement the existing summary. [#16637](prometheus/prometheus#16637) - \[ENHANCEMENT] Remote-write: Add custom scope support for AzureAD authentication. [#17483](prometheus/prometheus#17483) - \[ENHANCEMENT] SD: add a `config` label with job name for most `prometheus_sd_refresh` metrics. [#17138](prometheus/prometheus#17138) - \[ENHANCEMENT] TSDB: New histogram `prometheus_tsdb_sample_ooo_delta`, the distribution of out-of-order samples in seconds. Collected for all samples, accepted or not. [#17477](prometheus/prometheus#17477) - \[ENHANCEMENT] Remote-read: Validate histograms received via remote-read. [#17561](prometheus/prometheus#17561) - \[PERF] TSDB: Small optimizations to postings index. [#17439](prometheus/prometheus#17439) - \[PERF] Scraping: Speed up relabelling of series. [#17530](prometheus/prometheus#17530) - \[PERF] PromQL: Small optimisations in binary operators. [#17524](prometheus/prometheus#17524), [#17519](prometheus/prometheus#17519). - \[BUGFIX] UI: PromQL autocomplete now shows the correct type and HELP text for OpenMetrics counters whose samples end in `_total`. [#17682](prometheus/prometheus#17682) - \[BUGFIX] UI: Fixed codemirror-promql incorrectly showing label completion suggestions after the closing curly brace of a vector selector. [#17602](prometheus/prometheus#17602) - \[BUGFIX] UI: Query editor no longer suggests a duration unit if one is already present after a number. [#17605](prometheus/prometheus#17605) - \[BUGFIX] PromQL: Fix some "vector cannot contain metrics with the same labelset" errors when experimental delayed name removal is enabled. [#17678](prometheus/prometheus#17678) - \[BUGFIX] PromQL: Fix possible corruption of PromQL text if the query had an empty `ignoring()` and non-empty grouping. [#17643](prometheus/prometheus#17643) - \[BUGFIX] PromQL: Fix resets/changes to return empty results for anchored selectors when all samples are outside the range. [#17479](prometheus/prometheus#17479) - \[BUGFIX] PromQL: Check more consistently for many-to-one matching in filter binary operators. [#17668](prometheus/prometheus#17668) - \[BUGFIX] PromQL: Fix collision in unary negation with non-overlapping series. [#17708](prometheus/prometheus#17708) - \[BUGFIX] PromQL: Fix collision in label\_join and label\_replace with non-overlapping series. [#17703](prometheus/prometheus#17703) - \[BUGFIX] PromQL: Fix bug with inconsistent results for queries with OR expression when experimental delayed name removal is enabled. [#17161](prometheus/prometheus#17161) - \[BUGFIX] PromQL: Ensure that `rate`/`increase`/`delta` of histograms results in a gauge histogram. [#17608](prometheus/prometheus#17608) - \[BUGFIX] PromQL: Do not panic while iterating over invalid histograms. [#17559](prometheus/prometheus#17559) - \[BUGFIX] TSDB: Reject chunk files whose encoded chunk length overflows int. [#17533](prometheus/prometheus#17533) - \[BUGFIX] TSDB: Do not panic during resolution reduction of invalid histograms. [#17561](prometheus/prometheus#17561) - \[BUGFIX] Remote-write Receive: Avoid duplicate labels when experimental type-and-unit-label feature is enabled. [#17546](prometheus/prometheus#17546) - \[BUGFIX] OTLP Receiver: Only write metadata to disk when experimental metadata-wal-records feature is enabled. [#17472](prometheus/prometheus#17472)

pipiland2612 requested review from bwplotka, cstyan and tomwilkie as code owners November 17, 2025 18:30

bwplotka requested changes Nov 17, 2025

View reviewed changes

storage/remote/queue_manager.go Outdated Show resolved Hide resolved

drop extra label from receiver

3350af5

Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com>

pipiland2612 force-pushed the fix_type_and_unit_agent branch from 5206502 to 3350af5 Compare November 17, 2025 20:05

pipiland2612 requested a review from bwplotka November 17, 2025 20:07

bwplotka approved these changes Nov 17, 2025

View reviewed changes

pipiland2612 changed the title ~~[RW2] Fix: Remove duplicate labels when type-and-unit-label feature is on~~ Remote Write Receive Fix: Remove duplicate labels when type-and-unit-label feature is on Nov 18, 2025

used constant

0f4e101

Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com>

bwplotka approved these changes Nov 18, 2025

View reviewed changes

bwplotka merged commit 5087a25 into prometheus:main Nov 18, 2025
28 checks passed

pipiland2612 deleted the fix_type_and_unit_agent branch November 18, 2025 11:02

SungJin1212 mentioned this pull request Dec 8, 2025

Skip attaching __unit__ and __type__ label if exist cortexproject/cortex#7145

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remote Write Receive Fix: Remove duplicate labels when type-and-unit-label feature is on#17546

Remote Write Receive Fix: Remove duplicate labels when type-and-unit-label feature is on#17546
bwplotka merged 2 commits intoprometheus:mainfrom
pipiland2612:fix_type_and_unit_agent

pipiland2612 commented Nov 17, 2025 •

edited

Loading

Uh oh!

bwplotka left a comment

Uh oh!

Uh oh!

bwplotka left a comment

Uh oh!

bwplotka commented Nov 17, 2025

Uh oh!

pipiland2612 commented Nov 18, 2025

Uh oh!

pipiland2612 commented Nov 18, 2025

Uh oh!

bwplotka left a comment

Uh oh!

Uh oh!

bboreham commented Dec 18, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

pipiland2612 commented Nov 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue(s) does the PR fix:

Manual testing

Does this PR introduce a user-facing change?

Uh oh!

bwplotka left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

bwplotka left a comment

Choose a reason for hiding this comment

Uh oh!

bwplotka commented Nov 17, 2025

Uh oh!

pipiland2612 commented Nov 18, 2025

Uh oh!

pipiland2612 commented Nov 18, 2025

Uh oh!

bwplotka left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

bboreham commented Dec 18, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

pipiland2612 commented Nov 17, 2025 •

edited

Loading