Skip to content

promql: Make it possible to add custom details in annotations and summarise multiple of the same annotation#15577

Merged
beorn7 merged 11 commits intoprometheus:mainfrom
zenador:enrich-anno-new
Feb 3, 2026
Merged

promql: Make it possible to add custom details in annotations and summarise multiple of the same annotation#15577
beorn7 merged 11 commits intoprometheus:mainfrom
zenador:enrich-anno-new

Conversation

@zenador
Copy link
Contributor

@zenador zenador commented Dec 11, 2024

Previously, annotations contained all the info within the error string, and could not be combined, e.g. if you have these annotations, they could only show up as separate annotations:

  • PromQL warning: diff of 3.5 from source X
  • PromQL warning: diff of 4.5 from source Y
  • PromQL warning: diff of 4.0 from source Z
  • PromQL warning: diff of 3.0 from source Z

So if you wanted to give more details, you would end up spamming the user with many of the same annotation type which is difficult to understand, and to avoid that you would have to simplify the warning to something like PromQL warning: diff found with no detail.

This PR makes it possible to combine them so you can be more detailed without spamming the user, e.g. PromQL warning: 4 instances of diff of 3.0 to 4.5 from sources X, Y, Z, and this can be customised for each specific annotation as desired. Existing annotations which don't need custom treatment can be left as default. See TestAnnotations_AsStrings for an example in practice.

This can be used to improve annotations like HistogramQuantileForcedMonotonicityInfo: #15578

Supersedes #14339

This has no user-facing effects yet:

NONE

@zenador
Copy link
Contributor Author

zenador commented Dec 11, 2024

Tagging @beorn7 and @krajorama because you reviewed the original PR

@zenador
Copy link
Contributor Author

zenador commented Dec 11, 2024

The only test failure seems flaky

@beorn7 beorn7 requested review from beorn7 and krajorama December 11, 2024 12:24
@beorn7
Copy link
Member

beorn7 commented Dec 11, 2024

Huh, very weird test failure. (Not the race that we got hit by recently, see #15538. )

Rerunning the tests to see if it is flaky or consistent.

And yeah, will review ASAP.

Copy link
Member

@beorn7 beorn7 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I understand now what you are up to, but it took me a while to understand.

First of all, let's try to implement this without involving the user of the annotations package. Ideally, we stick with the idea that annotations are just errors and handle all the other stuff behind the scenes (using type switches, type casts, or maybe just errors.As and errors.Is). This would also remove the AddRaw method, which is just cognitive ballast from the point of view of the user.

I have tried to sketch out the idea in comments. It might or might not work that way. Maybe you can try it out. If it doesn't work, I'll work on this more deeply and will come up with a more detailed (or different) suggestion. However, I'm about to leave for my Xmas vacations, so my next response will have to wait until the new (Gregorian) year. (Back in action 2025-01-07.) Sorry for that.

@github-actions github-actions bot added the stale label Mar 2, 2025
@bboreham
Copy link
Member

Hello from the bug-scrub! Do you think you will come back to this?

My reaction is it seems like a lot of work for something of very low benefit to the user.

@github-actions github-actions bot removed the stale label Jun 29, 2025
@zenador
Copy link
Contributor Author

zenador commented Jul 23, 2025

Hi! Would like to come back to this eventually as I think it could be useful (open up the possibility of adding more detail in annotations in future), but it's a low priority as there's no immediate need, so I'll look at it again when there's more time. I agree it's not worth spending too much time on this for now (unless a good use case appears), so I'll timebox it, and will probably abandon it if it turns out to be too complicated / involve too many changes.

Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
This reverts commit 824f219.

Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
This reverts commit ac9d848.

Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
@zenador
Copy link
Contributor Author

zenador commented Oct 2, 2025

@beorn7 after quite some time, finally followed your suggestions and the code changes are much simpler now!

Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
@beorn7
Copy link
Member

beorn7 commented Oct 2, 2025

Thanks. On my list.

@beorn7
Copy link
Member

beorn7 commented Oct 15, 2025

This is still on my review list. However, I might only get to it after PromCon.

@beorn7
Copy link
Member

beorn7 commented Nov 13, 2025

Sorry for further delay. I hope to get to this before this week ends.

@beorn7
Copy link
Member

beorn7 commented Nov 25, 2025

Apologies for further delay. This is still very close to the top of my list.

Copy link
Member

@beorn7 beorn7 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice. I just have a few documentation and style nits.

@beorn7
Copy link
Member

beorn7 commented Jan 22, 2026

FYI: After holidays and sickness, I'm back in action. @zenador it seems there is very little left to get this past the finish line. Will you be able to work on this any time soon?

Co-authored-by: Björn Rabenstein <github@rabenste.in>
Signed-off-by: zenador <zenador@users.noreply.github.com>
@zenador zenador requested a review from a team as a code owner February 1, 2026 20:06
Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
@zenador
Copy link
Contributor Author

zenador commented Feb 2, 2026

@beorn7 Finally updated :) Didn't realise there was so little, should have looked at it earlier

Copy link
Member

@beorn7 beorn7 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks.

@beorn7
Copy link
Member

beorn7 commented Feb 3, 2026

Not sure why the DCO test has left us. I'll squash this anyway.

@beorn7 beorn7 merged commit c0ad853 into prometheus:main Feb 3, 2026
31 checks passed
wbollock pushed a commit to wbollock/prometheus that referenced this pull request Feb 6, 2026
…marise multiple of the same annotation (prometheus#15577)

Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
Signed-off-by: zenador <zenador@users.noreply.github.com>
Co-authored-by: Björn Rabenstein <github@rabenste.in>
Signed-off-by: Will Bollock <wbollock@linode.com>
wbollock pushed a commit to wbollock/prometheus that referenced this pull request Feb 6, 2026
…marise multiple of the same annotation (prometheus#15577)

Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
Signed-off-by: zenador <zenador@users.noreply.github.com>
Co-authored-by: Björn Rabenstein <github@rabenste.in>
wbollock pushed a commit to wbollock/prometheus that referenced this pull request Feb 6, 2026
…marise multiple of the same annotation (prometheus#15577)

Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
Signed-off-by: zenador <zenador@users.noreply.github.com>
Co-authored-by: Björn Rabenstein <github@rabenste.in>
wbollock pushed a commit to wbollock/prometheus that referenced this pull request Feb 6, 2026
…marise multiple of the same annotation (prometheus#15577)

Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
Signed-off-by: zenador <zenador@users.noreply.github.com>
Co-authored-by: Björn Rabenstein <github@rabenste.in>
wbollock pushed a commit to wbollock/prometheus that referenced this pull request Feb 6, 2026
…marise multiple of the same annotation (prometheus#15577)

Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
Signed-off-by: zenador <zenador@users.noreply.github.com>
Co-authored-by: Björn Rabenstein <github@rabenste.in>
wbollock pushed a commit to wbollock/prometheus that referenced this pull request Feb 6, 2026
…marise multiple of the same annotation (prometheus#15577)

Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
Signed-off-by: zenador <zenador@users.noreply.github.com>
Co-authored-by: Björn Rabenstein <github@rabenste.in>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants