What did you do?

We use the Slack integration to send notifications to a RocketChat server, which is reasonably compatible with Slack for this purpose. This has worked well for 4+ years, however after our last update we keep getting an ever increasing list of "Resolved" alerts every 5 minutes.

What did you expect to see?

We expect to see an alert posted, then a resolved posted, then silence.

What did you see instead? Under which circumstances?

We see the alert posted, then the resolved posted over and over again (every 5 minutes) until the AlertManager instance is restarted.

Environment

We believe that change #3121 triggered this condition. That change checks a 200 status code to ensure if contains "ok": true (previously 200 meant OK without further checking).

Our RocketChat instances returns a different message on success:

$ curl ...
< HTTP/1.1 200 OK
< Content-Type: application/json
...
{"success":true}

We believe that this new code identifies this (incorrectly) as failed POST and thus doesn't remove the now resolved alert from its list, and dutifully tries again every 5 minutes for eternity.

(Credit to @bg-govau for pin-pointing the issue. PR forthcoming.)