promql(histograms): Fix counter reset hint handling when aggregating#17312
Merged
promql(histograms): Fix counter reset hint handling when aggregating#17312
Conversation
Member
Author
krajorama
reviewed
Oct 9, 2025
krajorama
requested changes
Oct 9, 2025
Member
krajorama
left a comment
There was a problem hiding this comment.
Getting there, I think I understand the changes now. I have some nit comments and request some clarifying docstrings.
Fixes #17308. As explained adding the warn-annotation about conflicting counter reset hints doesn't happen consistently. Furthermore, because of incremental mean calculation being used so far (which includes subtraction), avg calculation always created gauge histograms. The fix is to make Sub behave like Add WRT counter reset handling, and then set the result of a subtraction to gauge explicitly in actual PromQL subtraction (rather than using Sub for something else, like incremental mean calculation). Also, track the presence of a CounterReset hint and a NotCounterReset hint separately for the entirety of aggregated histograms and create the warn-annotation based on that. As a minor fix, this commit also consistently creates the warn annotation in aggregation to be about "aggregation" rather than "subtraction" or "addition", because the latter are just internal operations within the aggregation, which is not of interest for the user. Signed-off-by: beorn7 <beorn@grafana.com>
317ac5b to
51e0982
Compare
charleskorn
reviewed
Oct 13, 2025
Comment on lines
+1103
to
+1108
| switch h.H.CounterResetHint { | ||
| case histogram.CounterReset: | ||
| counterResetSeen = true | ||
| case histogram.NotCounterReset: | ||
| notCounterResetSeen = true | ||
| } |
Contributor
There was a problem hiding this comment.
Does this need to be done for s.Histograms[0].H as well? Otherwise if the first histogram is CounterReset and the rest are NotCounterReset (or vice versa), then the annotation won't be emitted.
Member
Author
There was a problem hiding this comment.
You are right. I actually did this right for avg_over_time (and I vaguely remember that I even wrote a test for it, but now I'm confused because I run the same tests for avg_over_time and sum_over_time). I will investigate and create a fix with a test.
charleskorn
added a commit
to grafana/mimir
that referenced
this pull request
Oct 13, 2025
charleskorn
added a commit
to grafana/mimir
that referenced
this pull request
Oct 13, 2025
krajorama
reviewed
Oct 14, 2025
|
|
||
| counterResetCollision = hasCounterResetCollision(h, other) | ||
|
|
||
| h.CounterResetHint = GaugeType |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #17308.
As explained adding the warn-annotation about conflicting counter reset hints doesn't happen consistently. Furthermore, because of incremental mean calculation being used so far (which includes subtraction), avg calculation always created gauge histograms.
The fix is to make Sub behave like Add WRT counter reset handling, and then set the result of a subtraction to gauge explicitly in actual PromQL subtraction (rather than using Sub for something else, like incremental mean calculation). Also, track the presence of a CounterReset hint and a NotCounterReset hint separately for the entirety of aggregated histograms and create the warn-annotation based on that.
As a minor fix, this commit also consistently creates the warn annotation in aggregation to be about "aggregation" rather than "subtraction" or "addition", because the latter are just internal operations within the aggregation, which is not of interest for the user.
Does this PR introduce a user-facing change?