Skip to content

fix(prometheus.remote_write): Fix sent_batch_duration_seconds measuring before the request was sent [backport]#5698

Merged
kgeckhart merged 1 commit intorelease/v1.14from
backport/pr-5697-to-v1.14
Mar 2, 2026
Merged

fix(prometheus.remote_write): Fix sent_batch_duration_seconds measuring before the request was sent [backport]#5698
kgeckhart merged 1 commit intorelease/v1.14from
backport/pr-5697-to-v1.14

Conversation

@grafana-alloybot
Copy link
Contributor

Backport of #5697

This PR backports #5697 to release/v1.14.

Original PR Author

@kgeckhart

Description

prometheus_remote_storage_sent_batch_duration_seconds was measuring before the HTTP request was sent rather than after, causing the metric to reflect encoding/serialization time rather than the actual send duration.

Applies the fix from prometheus/prometheus#18214 via a fork replace directive pointing to https://github.com/grafana/prometheus/tree/fix-sent-batch-duration-v0.309.1.

Remove the replace directive when upstream PR #18214 is merged and Prometheus is upgraded.


This backport was created automatically.

…ng before the request was sent (#5697)

`prometheus_remote_storage_sent_batch_duration_seconds` was measuring
before the HTTP request was sent rather than after, causing the metric
to reflect encoding/serialization time rather than the actual send
duration.

Applies the fix from prometheus/prometheus#18214
via a fork replace directive pointing to
https://github.com/grafana/prometheus/tree/fix-sent-batch-duration-v0.309.1.

Remove the replace directive when upstream PR #18214 is merged and
Prometheus is upgraded.

(cherry picked from commit 10cfb6c)
@grafana-alloybot grafana-alloybot bot requested a review from a team as a code owner March 2, 2026 17:50
@github-actions
Copy link
Contributor

github-actions bot commented Mar 2, 2026

🔍 Dependency Review

github.com/prometheus/prometheus v0.309.1 -> github.com/grafana/prometheus v1.8.2-0.20260302171028-8cf60eef5463 — ✅ Safe
  • What changed

  • Scope and impact

    • The change is localized to Prometheus remote-write internals (storage/remote). It does not modify exported APIs that consumers (such as the OpenTelemetry Collector’s prometheusreceiver) import.
    • No breaking changes to package structure, public types, or function signatures were introduced between upstream v0.309.1 and this forked commit.
    • Behavioral change is limited to the semantics of the sent_batch_duration_seconds metric:
      • Previously included time before the request was sent.
      • Now only measures the duration after the request is sent.
      • If you consume/alert on this metric, expect values to decrease; otherwise, no code changes are required.
  • Evidence

  • Code changes required in this repository

    • None. This is a compatible, forked patch-level change with no public API changes.
    • If you have dashboards/alerts tied to sent_batch_duration_seconds, you may want to recalibrate thresholds due to the corrected measurement scope.
  • Illustrative patch from the fork (conceptual)

    • The fix delays starting the histogram timer until right before the HTTP request is sent:
      - start := time.Now()
      - defer func() { sentBatchDuration.Observe(time.Since(start).Seconds()) }()
      + var start time.Time
      + defer func() {
      +   if !start.IsZero() {
      +     sentBatchDuration.Observe(time.Since(start).Seconds())
      +   }
      + }()
        ...
      + // right before sending the HTTP request
      + start = time.Now()
        resp, err := httpClient.Do(req)
    • No function signatures or imports that downstream users rely on are altered.
  • Recommended validation

    • Run existing tests for components depending on github.com/prometheus/prometheus (e.g., prometheusreceiver).
    • If you monitor sent_batch_duration_seconds, verify expected drop in observed duration due to corrected timing.

Notes

  • This is a temporary forked replacement to pick up a specific bug fix. The comments in go.mod/builder-config.yaml indicate it should be removed once the upstream PR is merged and the project upgrades Prometheus accordingly.

@kgeckhart kgeckhart enabled auto-merge (squash) March 2, 2026 18:34
@kgeckhart kgeckhart merged commit 150aecb into release/v1.14 Mar 2, 2026
42 checks passed
@kgeckhart kgeckhart deleted the backport/pr-5697-to-v1.14 branch March 2, 2026 18:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant