prometheus: alert dashboard links with fixed timestamps#17034
Conversation
|
Notifying subscribers in CODENOTIFY files for diff 1851376...ffa11b5. No notifications. |
Codecov Report
@@ Coverage Diff @@
## main #17034 +/- ##
==========================================
- Coverage 51.98% 51.98% -0.01%
==========================================
Files 1703 1703
Lines 84786 84788 +2
Branches 7524 7666 +142
==========================================
- Hits 44079 44077 -2
- Misses 36806 36808 +2
- Partials 3901 3903 +2
|
|
@bobheadxi You can use something like |
|
I considered that (see PR description):
It's not a great experience if the alert spans, say, 24 hours of problems (the link would only show a few minutes or whatever value we set there) |
|
I see, I did not notice that in the body my bad. I think defaulting to 1h or inferring Eg if CPU utilization alerts when its |
Unfortunately you cannot do arithmetic in alertmanager templates :/ However, I think I found a nice middle-ground strategy with e3f166f:
|
But couldnt we set that in the generated alert file when its generated by the generator? |
We can't infer |
I meant the |
|
Merging for now, can see how it works in practice :) this will be nice for looking at past alerts |
While working on https://github.com/sourcegraph/sourcegraph/pull/17014 I added a relative timestamp to the dashboard link in alerts, did a bit of fenangling to make the link completely fixed to timestamps associated with the delivered alert.
We can depend on
(index .Alerts 0)because our grouping strategy ensures each group delivered only has one alert.I'm still a bit unsure about this now that I've got it working, the experience is a bit less than ideal for alerts that e.g. are spikes lasting a second, since then we get a link to a panel that has a tiny window. An alternative is
timeandtime.window, but that might not be great for alerts lasting longer. It is not possible to do arithmetic on these timestamps in alertmanager templates: prometheus/alertmanager#1188=> update: see https://github.com/sourcegraph/sourcegraph/pull/17034#issuecomment-756598154