Purpose

This issue talks about chaining the Monitors in the Alerting plugin - how to re-use the findings and alerts generated by one monitor to another monitor. It highlights the need for a new kind of monitor to handle more elaborate use cases and complex queries which are not catered to by the current implementation of Alerting and proposes a solution to achieve that.

Background

Alerting plugin is used to monitor user data and create alerts that trigger when conditions occur in one or more indexes. Monitors are used to create triggers. Bucket level monitors create triggers which have conditions that alert on data aggregated into buckets. Query level monitors support triggers which have conditions that generate alerts on query criteria. Document level monitor can create triggers that alert on individual document matching query conditions. Each monitor performs a very specific and limited function.

• Users of alerting plugin have complex queries which are not being catered to by a single monitor and need to use a combination of triggers provided by different types of monitors.
• Users need a single view of data source during the monitor execution run to be fanned out across multiple monitors, to attain finer consistency of results. (In the current implementation of alerting plugin we cannot achieve this as all monitors run as individual jobs and we can't guarantee they are looking at the same data set)
• If monitors are chained the findings generated by one monitor can be used as the source data for a second monitor that can do further granular analysis and generate more meaningful of alerts.
• Users need the ability to define pipelines of rules and queries that will run as single execution, internally running a sequence of monitors and generating a chain of alerts.

To achieve the above vision we would need to have a more abstract way to monitor data - composite monitors. Composite monitors allow users to execute delegate monitors in sequential fashion hence providing more control over the data seen by composite monitors which will enable creation of more granular findings and chained alerts.

Challenges Solved By Composite Monitors

Chained Findings

Let’s consider the following example where a composite monitor allows user to chain findings to extract more value from monitor analyses and create more meaningful alerts.:

• User creates Monitor 1, a Document level monitor on a set of logs indices with the following queries - 1. Request payload size >100 kb. 2. Response Status != 200 3. Response contains a specific header. On every execution, monitor would query and generate findings mentioning which documents matched how many queries.
• Although individually each of the above 3 queries is a data metric for the user, user deems that any documents matching all 3 queries would require further analysis - say, he wants to aggregate these logs by client IP and check how many such IPs are there and how many such requests are being fired from these IPs. These cannot be catered by Monitor 1 itself and would require further processing.
• User creates Monitor 2, a bucket level monitor which analyses the findings of above mentioned document level monitor and create triggers and alert notifications for said conditions.
• User creates a composite monitor which accepts a sequence of monitors to execute. User defines the sequence to execute monitor1 followed by monitor 2, which accepts a config to use Monitor 1 findings as its data set.
• This would let user define condition to execute a monitor based on another monitor’s findings.

Chained Alerts

Let’s consider the following use case:

• User creates Monitor 1, to monitor CPU Utilization of their service’s worker nodes. Monitor has trigger conditions to create alerts on high CPU threshold. But high CPU maybe due to background processes, high traffic, cluster instabilities etc. Simply notifying on high CPU is not of much value to user.
• User creates Monitor 2, to monitor incoming request count in the same time window. Monitor has trigger conditions to create alerts on high request count threshold. But if the cluster is able to handle the heavy traffic smoothly, there is no value in creating notifications.
• User creates a composite monitor with a sequence to run monitor 1 and monitor 2 and a monitor 3. Monitor 3 accepts a config to read alerts created by both monitor 1 and monitor 2 in the current execution and if yes, create an alert and send out a notification that their service is degrading due to high traffic. This alert would help customer scale their service and serves as a more meaningful and critical alert.

Running multiple monitors in one execution

User simply wants to run multiple monitors as a part of a composite monitor at once to ensure all the monitors are deterministically querying data in the same time range and control the sequence of execution of monitors. This also ensures that an entire workflow or pipeline created by users is solved by a single composite monitor.