Codecov Report

Merging #12087 into master will decrease coverage by 0.00%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master   #12087      +/-   ##
==========================================
- Coverage   50.67%   50.66%   -0.01%     
==========================================
  Files        1420     1420              
  Lines       80371    80360      -11     
  Branches     6831     6668     -163     
==========================================
- Hits        40727    40718       -9     
+ Misses      36064    36061       -3     
- Partials     3580     3581       +1     

Specificity: I'm opting to disallow broad silences (ie all of name, level, service are required and a user cannot mute all "service": "gitserver" alerts). Regex is possible, but not allowed at the moment. Open to thoughts on this though

This sounds good to me, bulk silencing seems like a really bad practice in general so I'd rather not have this and find out we need it later - than enable someone to shoot themselves in the foot so-to-speak :)

Implementation detail: Alertmanager requires a start and end date for each alert. Right now, I've set this to 10 years... should be fine right?

Seems reasonable to me, I was just thinking that it is quite cool that with this approach we can un-silence alerts or notify admins in the future to un-silence them if we e.g. see they are flaky on many instances and fix them in an upgrade. That's a cool property of having this in site config :)

there does not seem to be a clear way to get the name of an alert at the moment - not sure if this is within the scope of this PR to address, and if there's a good way to go about it (maybe replace the human-readable description in the main table with alert name?)

This is indeed unfortunate.. I really like having the human-readable names, though. What about having admins first go to the solutions doc (as they would to consider if it might be a real issue for them), and having one of the pre-generated solutions there being "Silence the alert in your site config: <name>"?

This would also mean you could have the site config be this format instead:

{
  "observability.silenceAlerts": [
    "warning_gitserver_disk_space_remaining",
    "critical_gitserver_command_duration_test"
  ]
}

WDYT?

This is indeed unfortunate.. I really like having the human-readable names, though. What about having admins first go to the solutions doc (as they would to consider if it might be a real issue for them), and having one of the pre-generated solutions there being "Silence the alert in your site config: "?

This would also mean you could have the site config be this format instead

Sounds good!

Looks good!
Just wanted to confirm that

{
  "observability.silenceAlerts": [
    "warning_gitserver_disk_space_remaining",
    "critical_gitserver_command_duration_test"
  ]
}

in the site config will be the typical way of enabling this.

@daxmc99 yep, should be unless anyone has any other thoughts on that! Working on an update to this PR to enable that, but it shouldn't be a significant change implementation-wise

2c3f7b9 , e5fa2a0 makes the change to specify silences by alertname and generates alertname in the alert solutions documentation for easier silencing :)

Ran it through creating/deleting silences locally with the new format

@slimsag let me know if there's any follow-ups you want me to make!