Resolved -
Summary: Between 14 January at 21:30 CET and 15 January at 09:15 CET, a small subset of CEP trigger automations was delayed for up to 11 hours. This subset accounted for less than 2% of all CEP trigger automations.
Impact for the affected trigger automations:
* Event processing was delayed by up to 11 hours.
* As a consequence, the delivery of emails, SMS, and push notifications was delayed by up to 11 hours.
Root Cause: one instance of our event processing pipeline was unable to process certain events due to an edge case in their data, which slowed down the processing speed for that specific instance.
Resolution: a mitigation was applied on 15 January at 08:36 CET, which restored normal processing performance. By 09:10 CET, the backlog was cleared and processing returned to real-time.
Jan 14, 20:30 UTC