Description

Recently we noticed a dead lock issue happened in our running alert managers.
Reference:

• dispather: https://github.com/prometheus/alertmanager/blob/main/dispatch/dispatch.go#L143
• mem.Alerts.Subscribe:

  alertmanager/provider/mem/mem.go

  Line 151 in 1da134a

  func (a *Alerts) Subscribe() provider.AlertIterator {

• claim store.Alerts.mtx:

  alertmanager/provider/mem/mem.go

  Line 157 in 1da134a

  alerts = a.alerts.List()

  
  

  alertmanager/store/store.go

  Line 119 in 1da134a

  a.Lock()

• gc:

  alertmanager/store/store.go

  Line 72 in 1da134a

  func (a *Alerts) gc() {

• mem.gc.callback:

  alertmanager/provider/mem/mem.go

  Line 112 in 1da134a

  a.mtx.Lock()

Proposal fix

• Have subscribe function here

  alertmanager/provider/mem/mem.go

  Line 151 in 1da134a

  func (a *Alerts) Subscribe() provider.AlertIterator {

  to claim store.Alerts.mux first before claim mem.Alerts.mux