[ResponseOps][Alerting V2] ESQL views#255968
Conversation
|
Pinging @elastic/response-ops (Team:ResponseOps) |
💔 Build Failed
Failed CI StepsTest Failures
Metrics [docs]
History
cc @adcoelho |
kdelemme
left a comment
There was a problem hiding this comment.
Approving because the flagged issue might not happen before we start using the views, and only the first time kibana starts (or in integration tests).
| initializeESQLViews({ | ||
| logger, | ||
| esClient, | ||
| }); |
There was a problem hiding this comment.
What happen if we try to use the views before they are created? We had a problem with the resources (datastreams & index setup) and the dispatcher that was trying to use an index before it was mapped properly...
With the other resources, at least we can do await resourceManager.waitUntilReady(); but that's not possible with the esql view initialization service. Since no other service is using the view it is fine for now, but worth flagging.
And as a more generic question (but maybe not for you), can't we just wait for both the initializeResources et initializeESQLViews to be done?
There was a problem hiding this comment.
Thank you for the review!
What happens if we try to use the views before they are created? We had a problem with the resources (datastreams & index setup) and the dispatcher that was trying to use an index before it was mapped properly...
This definitely makes sense, but I deliberately did not include waitUntilReady() in this PR because, for now, there is no intent to use the views in the backend.
They are created during startup, but only the UI will use them(alert episodes table). Error handling can be done there, and the idea is to try to use what is provided by ES APIs as much as possible.
## Summary ### Key capabilities - **ES|QL-native rule evaluation** — Rules are defined as ES|QL queries with optional WHERE clause conditions, evaluated on a configurable schedule - **Alert lifecycle management** — Full episode tracking with pending → active → recovering → inactive state transitions, including configurable alert delay (consecutive breaches / duration) - **Event-driven architecture** — Alert events and actions are stored in dedicated data streams (`.alerting-events`, `.alerting-actions`) with ES|QL views for querying - **Notification dispatch pipeline** — A multi-step dispatcher that matches alert episodes to notification policies, handles throttling/suppression, and triggers Kibana Workflows using encrypted API keys - **Notification policies** — CRUD APIs and UI for creating notification policies with KQL-based rule matching, workflow integration, and API key management - **Rule authoring UI** — A shared rule form package (`@kbn/alerting-v2-rule-form`) usable standalone or embedded in Discover, with ES|QL editor, WHERE clause condition editing, recovery configuration, and live query preview - **Rule management UI** — Full rule list with pagination, enable/disable, clone, edit, and delete operations - **APM instrumentation** — Middleware and decorators for tracing rule execution and client operations ### Architecture highlights - **InversifyJS DI** — All services use constructor injection with typed tokens, scoped per-request or singleton as appropriate - **Pipeline pattern** — Rule executor and dispatcher use composable step-based pipelines - **Saved Objects** — Rules stored as hidden saved objects; notification policies stored as encrypted saved objects (for API key protection) - **Feature privileges** — Dedicated Kibana feature with read/all privileges for RBAC --- ## Contained PRs <details> <summary><strong>Core Engine & Plugin Init</strong> (12 PRs)</summary> - #247283 — Init alerting v2 plugin (@cnasikas) - #247452 — Add the alerting v2 feature privileges (@cnasikas) - #247673 — Director (@cnasikas) - #248306 — Create basic services (@cnasikas) - #248696 — Initialize all resources (@cnasikas) - #250023 — Schema package (@cnasikas) - #250010 — YML Editor (@cnasikas) - #251064 — Remove index.mode: lookup for RnA alert indices (@cnasikas) - #251707 — Simplify task registration pattern (@kdelemme) - #251876 — Dedicated user service (@cnasikas) - #252073 — Use `kbn/data-streams` in alerting_v2 (@cnasikas) - #255120 — Update alerting-v2 owner to new rna project team (@cnasikas) </details> <details> <summary><strong>Rule Execution Pipeline</strong> (12 PRs)</summary> - #247472 — Add alerting v2 Rule Executor (@darnautov) - #248285 — Alerting v2 rule HTTP APIs (@darnautov) - #248728 — Add basic alert actions route (@darnautov) - #250161 — Refactor rule executor to use a pipeline pattern (@darnautov) - #252292 — Implement the CountTimeframeStrategy for the director (@cnasikas) - #252544 — Add support of streaming in the rule executor (@darnautov) - #252754 — Update rule attributes (@kdelemme) - #253355 — Add getRules client method (@kdelemme) - #253668 — Make evaluation.query.condition optional (@kdelemme) - #254031 — Add recovery event generation to rule execution pipeline (@kdelemme) - #255968 — ES|QL views (@adcoelho) - #256697 — Create episodes ES|QL view (@adcoelho) </details> <details> <summary><strong>Alert Suppression & Episodes</strong> (3 PRs)</summary> - #252174 — Alert suppression (@kdelemme) - #256486 — Fix suppression query (@kdelemme) - #256527 — Store 'unmatched' action for unmatched alert episodes (@kdelemme) </details> <details> <summary><strong>Dispatcher & Notification Engine</strong> (6 PRs)</summary> - #250822 — Alerting v2 dispatcher (@kdelemme) - #251529 — Use query service in dispatcher (@kdelemme) - #251679 — Dispatcher task (@kdelemme) - #252758 — Dispatcher notification policy (@kdelemme) - #255332 — Wait for resources before scheduling dispatcher task (@kdelemme) - #256536 — Use stored encrypted API keys from Notification Policy in dispatcher step (@kdelemme) </details> <details> <summary><strong>Notification Policies (Server)</strong> (4 PRs)</summary> - #251336 — Introduce notification policy CRUD APIs and client (@cnasikas) - #253134 — Update notification policy (@cnasikas) - #254808 — Store API key owner on Notification Policy (@kdelemme) - #256940 — Make notification policies global with optional rule-label scoping (@kdelemme) </details> <details> <summary><strong>Notification Policies UI</strong> (1 PR)</summary> - #255599 — Add notification policies UI and Storybook form story (@adcoelho) </details> <details> <summary><strong>Rule Authoring UI</strong> (13 PRs)</summary> - #250961 — Add create rule flyout in Discover (@adcoelho) - #255111 — Add activation configuration fields to alerting V2 rule form (@yiannisnikolopoulos) - #255427 — Rule form: provide services via context (@dominiqueclarke) - #255876 — MVP rule form, Split evaluation condition, and Recovery configuration (@dominiqueclarke) - #256260 — Foundational rule list (@dominiqueclarke) - #256756 — Wire up edit flow (@dominiqueclarke) - #256801 — Move consecutive breaches max to shared constants (@yiannisnikolopoulos) - #256818 — Preview query and design parity (@dominiqueclarke) - #256938 — Allow clearing number inputs in state transition fields (@yiannisnikolopoulos) - #257017 — Add enable/disable and clone rule to rule list (@dominiqueclarke) - #257246 — Remove all React.FC (@dominiqueclarke) - #257415 — Rule form - fix test (@dominiqueclarke) - #257454 — Block comma key in number input component (@yiannisnikolopoulos) </details> <details> <summary><strong>API Documentation & Schema</strong> (2 PRs)</summary> - #254901 — Rename indexes for alert events and actions (@adcoelho) - #255810 — OAS for alert action routes (@adcoelho) </details> <details> <summary><strong>Observability & Monitoring</strong> (3 PRs)</summary> - #254925 — Add ApmMiddleware to the rule executor (@adcoelho) - #255115 — Add the withAPM decorator and apply it to the rules_client (@adcoelho) - #255999 — Fix linting problem in apm middleware (@adcoelho) </details> <details> <summary><strong>CI & Maintenance</strong> (2 PRs)</summary> - #257409 — Refactor SO services to use inversify DI for client initialization (@darnautov) - Fix alerting-v2-schema jest config (@darnautov) </details> --- --------- Co-authored-by: Dima Arnautov <dmitrii.arnautov@elastic.co> Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Kevin Delemme <kevin.delemme@elastic.co> Co-authored-by: Mike Côté <mikecote@users.noreply.github.com> Co-authored-by: Antonio <antonio.coelho@elastic.co> Co-authored-by: Kevin Delemme <kdelemme@gmail.com> Co-authored-by: Dominique Clarke <dominique.clarke@elastic.co> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Jason Rhodes <jason.rhodes@elastic.co> Co-authored-by: Yiannis Nikolopoulos <yiannis.nikolopoulos@elastic.co> Co-authored-by: Alexi Doak <109488926+doakalexi@users.noreply.github.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Bailey Cash <bailey.cash@elastic.co> Co-authored-by: Anna Davydova <ana.davydova@elastic.co> Co-authored-by: Umberto Pepato <umbopepato@users.noreply.github.com> Co-authored-by: Jason Rhodes <jason.matthew.rhodes@gmail.com> Co-authored-by: Joana Cardoso <169058851+joana-cps@users.noreply.github.com>
## Summary ### Key capabilities - **ES|QL-native rule evaluation** — Rules are defined as ES|QL queries with optional WHERE clause conditions, evaluated on a configurable schedule - **Alert lifecycle management** — Full episode tracking with pending → active → recovering → inactive state transitions, including configurable alert delay (consecutive breaches / duration) - **Event-driven architecture** — Alert events and actions are stored in dedicated data streams (`.alerting-events`, `.alerting-actions`) with ES|QL views for querying - **Notification dispatch pipeline** — A multi-step dispatcher that matches alert episodes to notification policies, handles throttling/suppression, and triggers Kibana Workflows using encrypted API keys - **Notification policies** — CRUD APIs and UI for creating notification policies with KQL-based rule matching, workflow integration, and API key management - **Rule authoring UI** — A shared rule form package (`@kbn/alerting-v2-rule-form`) usable standalone or embedded in Discover, with ES|QL editor, WHERE clause condition editing, recovery configuration, and live query preview - **Rule management UI** — Full rule list with pagination, enable/disable, clone, edit, and delete operations - **APM instrumentation** — Middleware and decorators for tracing rule execution and client operations ### Architecture highlights - **InversifyJS DI** — All services use constructor injection with typed tokens, scoped per-request or singleton as appropriate - **Pipeline pattern** — Rule executor and dispatcher use composable step-based pipelines - **Saved Objects** — Rules stored as hidden saved objects; notification policies stored as encrypted saved objects (for API key protection) - **Feature privileges** — Dedicated Kibana feature with read/all privileges for RBAC --- ## Contained PRs <details> <summary><strong>Core Engine & Plugin Init</strong> (12 PRs)</summary> - elastic#247283 — Init alerting v2 plugin (@cnasikas) - elastic#247452 — Add the alerting v2 feature privileges (@cnasikas) - elastic#247673 — Director (@cnasikas) - elastic#248306 — Create basic services (@cnasikas) - elastic#248696 — Initialize all resources (@cnasikas) - elastic#250023 — Schema package (@cnasikas) - elastic#250010 — YML Editor (@cnasikas) - elastic#251064 — Remove index.mode: lookup for RnA alert indices (@cnasikas) - elastic#251707 — Simplify task registration pattern (@kdelemme) - elastic#251876 — Dedicated user service (@cnasikas) - elastic#252073 — Use `kbn/data-streams` in alerting_v2 (@cnasikas) - elastic#255120 — Update alerting-v2 owner to new rna project team (@cnasikas) </details> <details> <summary><strong>Rule Execution Pipeline</strong> (12 PRs)</summary> - elastic#247472 — Add alerting v2 Rule Executor (@darnautov) - elastic#248285 — Alerting v2 rule HTTP APIs (@darnautov) - elastic#248728 — Add basic alert actions route (@darnautov) - elastic#250161 — Refactor rule executor to use a pipeline pattern (@darnautov) - elastic#252292 — Implement the CountTimeframeStrategy for the director (@cnasikas) - elastic#252544 — Add support of streaming in the rule executor (@darnautov) - elastic#252754 — Update rule attributes (@kdelemme) - elastic#253355 — Add getRules client method (@kdelemme) - elastic#253668 — Make evaluation.query.condition optional (@kdelemme) - elastic#254031 — Add recovery event generation to rule execution pipeline (@kdelemme) - elastic#255968 — ES&elastic#124;QL views (@adcoelho) - elastic#256697 — Create episodes ES&elastic#124;QL view (@adcoelho) </details> <details> <summary><strong>Alert Suppression & Episodes</strong> (3 PRs)</summary> - elastic#252174 — Alert suppression (@kdelemme) - elastic#256486 — Fix suppression query (@kdelemme) - elastic#256527 — Store 'unmatched' action for unmatched alert episodes (@kdelemme) </details> <details> <summary><strong>Dispatcher & Notification Engine</strong> (6 PRs)</summary> - elastic#250822 — Alerting v2 dispatcher (@kdelemme) - elastic#251529 — Use query service in dispatcher (@kdelemme) - elastic#251679 — Dispatcher task (@kdelemme) - elastic#252758 — Dispatcher notification policy (@kdelemme) - elastic#255332 — Wait for resources before scheduling dispatcher task (@kdelemme) - elastic#256536 — Use stored encrypted API keys from Notification Policy in dispatcher step (@kdelemme) </details> <details> <summary><strong>Notification Policies (Server)</strong> (4 PRs)</summary> - elastic#251336 — Introduce notification policy CRUD APIs and client (@cnasikas) - elastic#253134 — Update notification policy (@cnasikas) - elastic#254808 — Store API key owner on Notification Policy (@kdelemme) - elastic#256940 — Make notification policies global with optional rule-label scoping (@kdelemme) </details> <details> <summary><strong>Notification Policies UI</strong> (1 PR)</summary> - elastic#255599 — Add notification policies UI and Storybook form story (@adcoelho) </details> <details> <summary><strong>Rule Authoring UI</strong> (13 PRs)</summary> - elastic#250961 — Add create rule flyout in Discover (@adcoelho) - elastic#255111 — Add activation configuration fields to alerting V2 rule form (@yiannisnikolopoulos) - elastic#255427 — Rule form: provide services via context (@dominiqueclarke) - elastic#255876 — MVP rule form, Split evaluation condition, and Recovery configuration (@dominiqueclarke) - elastic#256260 — Foundational rule list (@dominiqueclarke) - elastic#256756 — Wire up edit flow (@dominiqueclarke) - elastic#256801 — Move consecutive breaches max to shared constants (@yiannisnikolopoulos) - elastic#256818 — Preview query and design parity (@dominiqueclarke) - elastic#256938 — Allow clearing number inputs in state transition fields (@yiannisnikolopoulos) - elastic#257017 — Add enable/disable and clone rule to rule list (@dominiqueclarke) - elastic#257246 — Remove all React.FC (@dominiqueclarke) - elastic#257415 — Rule form - fix test (@dominiqueclarke) - elastic#257454 — Block comma key in number input component (@yiannisnikolopoulos) </details> <details> <summary><strong>API Documentation & Schema</strong> (2 PRs)</summary> - elastic#254901 — Rename indexes for alert events and actions (@adcoelho) - elastic#255810 — OAS for alert action routes (@adcoelho) </details> <details> <summary><strong>Observability & Monitoring</strong> (3 PRs)</summary> - elastic#254925 — Add ApmMiddleware to the rule executor (@adcoelho) - elastic#255115 — Add the withAPM decorator and apply it to the rules_client (@adcoelho) - elastic#255999 — Fix linting problem in apm middleware (@adcoelho) </details> <details> <summary><strong>CI & Maintenance</strong> (2 PRs)</summary> - elastic#257409 — Refactor SO services to use inversify DI for client initialization (@darnautov) - Fix alerting-v2-schema jest config (@darnautov) </details> --- --------- Co-authored-by: Dima Arnautov <dmitrii.arnautov@elastic.co> Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Kevin Delemme <kevin.delemme@elastic.co> Co-authored-by: Mike Côté <mikecote@users.noreply.github.com> Co-authored-by: Antonio <antonio.coelho@elastic.co> Co-authored-by: Kevin Delemme <kdelemme@gmail.com> Co-authored-by: Dominique Clarke <dominique.clarke@elastic.co> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Jason Rhodes <jason.rhodes@elastic.co> Co-authored-by: Yiannis Nikolopoulos <yiannis.nikolopoulos@elastic.co> Co-authored-by: Alexi Doak <109488926+doakalexi@users.noreply.github.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Bailey Cash <bailey.cash@elastic.co> Co-authored-by: Anna Davydova <ana.davydova@elastic.co> Co-authored-by: Umberto Pepato <umbopepato@users.noreply.github.com> Co-authored-by: Jason Rhodes <jason.matthew.rhodes@gmail.com> Co-authored-by: Joana Cardoso <169058851+joana-cps@users.noreply.github.com>
- **ES|QL-native rule evaluation** — Rules are defined as ES|QL queries with optional WHERE clause conditions, evaluated on a configurable schedule - **Alert lifecycle management** — Full episode tracking with pending → active → recovering → inactive state transitions, including configurable alert delay (consecutive breaches / duration) - **Event-driven architecture** — Alert events and actions are stored in dedicated data streams (`.alerting-events`, `.alerting-actions`) with ES|QL views for querying - **Notification dispatch pipeline** — A multi-step dispatcher that matches alert episodes to notification policies, handles throttling/suppression, and triggers Kibana Workflows using encrypted API keys - **Notification policies** — CRUD APIs and UI for creating notification policies with KQL-based rule matching, workflow integration, and API key management - **Rule authoring UI** — A shared rule form package (`@kbn/alerting-v2-rule-form`) usable standalone or embedded in Discover, with ES|QL editor, WHERE clause condition editing, recovery configuration, and live query preview - **Rule management UI** — Full rule list with pagination, enable/disable, clone, edit, and delete operations - **APM instrumentation** — Middleware and decorators for tracing rule execution and client operations - **InversifyJS DI** — All services use constructor injection with typed tokens, scoped per-request or singleton as appropriate - **Pipeline pattern** — Rule executor and dispatcher use composable step-based pipelines - **Saved Objects** — Rules stored as hidden saved objects; notification policies stored as encrypted saved objects (for API key protection) - **Feature privileges** — Dedicated Kibana feature with read/all privileges for RBAC --- <details> <summary><strong>Core Engine & Plugin Init</strong> (12 PRs)</summary> - elastic#247283 — Init alerting v2 plugin (@cnasikas) - elastic#247452 — Add the alerting v2 feature privileges (@cnasikas) - elastic#247673 — Director (@cnasikas) - elastic#248306 — Create basic services (@cnasikas) - elastic#248696 — Initialize all resources (@cnasikas) - elastic#250023 — Schema package (@cnasikas) - elastic#250010 — YML Editor (@cnasikas) - elastic#251064 — Remove index.mode: lookup for RnA alert indices (@cnasikas) - elastic#251707 — Simplify task registration pattern (@kdelemme) - elastic#251876 — Dedicated user service (@cnasikas) - elastic#252073 — Use `kbn/data-streams` in alerting_v2 (@cnasikas) - elastic#255120 — Update alerting-v2 owner to new rna project team (@cnasikas) </details> <details> <summary><strong>Rule Execution Pipeline</strong> (12 PRs)</summary> - elastic#247472 — Add alerting v2 Rule Executor (@darnautov) - elastic#248285 — Alerting v2 rule HTTP APIs (@darnautov) - elastic#248728 — Add basic alert actions route (@darnautov) - elastic#250161 — Refactor rule executor to use a pipeline pattern (@darnautov) - elastic#252292 — Implement the CountTimeframeStrategy for the director (@cnasikas) - elastic#252544 — Add support of streaming in the rule executor (@darnautov) - elastic#252754 — Update rule attributes (@kdelemme) - elastic#253355 — Add getRules client method (@kdelemme) - elastic#253668 — Make evaluation.query.condition optional (@kdelemme) - elastic#254031 — Add recovery event generation to rule execution pipeline (@kdelemme) - elastic#255968 — ES&elastic#124;QL views (@adcoelho) - elastic#256697 — Create episodes ES&elastic#124;QL view (@adcoelho) </details> <details> <summary><strong>Alert Suppression & Episodes</strong> (3 PRs)</summary> - elastic#252174 — Alert suppression (@kdelemme) - elastic#256486 — Fix suppression query (@kdelemme) - elastic#256527 — Store 'unmatched' action for unmatched alert episodes (@kdelemme) </details> <details> <summary><strong>Dispatcher & Notification Engine</strong> (6 PRs)</summary> - elastic#250822 — Alerting v2 dispatcher (@kdelemme) - elastic#251529 — Use query service in dispatcher (@kdelemme) - elastic#251679 — Dispatcher task (@kdelemme) - elastic#252758 — Dispatcher notification policy (@kdelemme) - elastic#255332 — Wait for resources before scheduling dispatcher task (@kdelemme) - elastic#256536 — Use stored encrypted API keys from Notification Policy in dispatcher step (@kdelemme) </details> <details> <summary><strong>Notification Policies (Server)</strong> (4 PRs)</summary> - elastic#251336 — Introduce notification policy CRUD APIs and client (@cnasikas) - elastic#253134 — Update notification policy (@cnasikas) - elastic#254808 — Store API key owner on Notification Policy (@kdelemme) - elastic#256940 — Make notification policies global with optional rule-label scoping (@kdelemme) </details> <details> <summary><strong>Notification Policies UI</strong> (1 PR)</summary> - elastic#255599 — Add notification policies UI and Storybook form story (@adcoelho) </details> <details> <summary><strong>Rule Authoring UI</strong> (13 PRs)</summary> - elastic#250961 — Add create rule flyout in Discover (@adcoelho) - elastic#255111 — Add activation configuration fields to alerting V2 rule form (@yiannisnikolopoulos) - elastic#255427 — Rule form: provide services via context (@dominiqueclarke) - elastic#255876 — MVP rule form, Split evaluation condition, and Recovery configuration (@dominiqueclarke) - elastic#256260 — Foundational rule list (@dominiqueclarke) - elastic#256756 — Wire up edit flow (@dominiqueclarke) - elastic#256801 — Move consecutive breaches max to shared constants (@yiannisnikolopoulos) - elastic#256818 — Preview query and design parity (@dominiqueclarke) - elastic#256938 — Allow clearing number inputs in state transition fields (@yiannisnikolopoulos) - elastic#257017 — Add enable/disable and clone rule to rule list (@dominiqueclarke) - elastic#257246 — Remove all React.FC (@dominiqueclarke) - elastic#257415 — Rule form - fix test (@dominiqueclarke) - elastic#257454 — Block comma key in number input component (@yiannisnikolopoulos) </details> <details> <summary><strong>API Documentation & Schema</strong> (2 PRs)</summary> - elastic#254901 — Rename indexes for alert events and actions (@adcoelho) - elastic#255810 — OAS for alert action routes (@adcoelho) </details> <details> <summary><strong>Observability & Monitoring</strong> (3 PRs)</summary> - elastic#254925 — Add ApmMiddleware to the rule executor (@adcoelho) - elastic#255115 — Add the withAPM decorator and apply it to the rules_client (@adcoelho) - elastic#255999 — Fix linting problem in apm middleware (@adcoelho) </details> <details> <summary><strong>CI & Maintenance</strong> (2 PRs)</summary> - elastic#257409 — Refactor SO services to use inversify DI for client initialization (@darnautov) - Fix alerting-v2-schema jest config (@darnautov) </details> --- --------- Co-authored-by: Dima Arnautov <dmitrii.arnautov@elastic.co> Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Kevin Delemme <kevin.delemme@elastic.co> Co-authored-by: Mike Côté <mikecote@users.noreply.github.com> Co-authored-by: Antonio <antonio.coelho@elastic.co> Co-authored-by: Kevin Delemme <kdelemme@gmail.com> Co-authored-by: Dominique Clarke <dominique.clarke@elastic.co> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Jason Rhodes <jason.rhodes@elastic.co> Co-authored-by: Yiannis Nikolopoulos <yiannis.nikolopoulos@elastic.co> Co-authored-by: Alexi Doak <109488926+doakalexi@users.noreply.github.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Bailey Cash <bailey.cash@elastic.co> Co-authored-by: Anna Davydova <ana.davydova@elastic.co> Co-authored-by: Umberto Pepato <umbopepato@users.noreply.github.com> Co-authored-by: Jason Rhodes <jason.matthew.rhodes@gmail.com> Co-authored-by: Joana Cardoso <169058851+joana-cps@users.noreply.github.com>
- **ES|QL-native rule evaluation** — Rules are defined as ES|QL queries with optional WHERE clause conditions, evaluated on a configurable schedule - **Alert lifecycle management** — Full episode tracking with pending → active → recovering → inactive state transitions, including configurable alert delay (consecutive breaches / duration) - **Event-driven architecture** — Alert events and actions are stored in dedicated data streams (`.alerting-events`, `.alerting-actions`) with ES|QL views for querying - **Notification dispatch pipeline** — A multi-step dispatcher that matches alert episodes to notification policies, handles throttling/suppression, and triggers Kibana Workflows using encrypted API keys - **Notification policies** — CRUD APIs and UI for creating notification policies with KQL-based rule matching, workflow integration, and API key management - **Rule authoring UI** — A shared rule form package (`@kbn/alerting-v2-rule-form`) usable standalone or embedded in Discover, with ES|QL editor, WHERE clause condition editing, recovery configuration, and live query preview - **Rule management UI** — Full rule list with pagination, enable/disable, clone, edit, and delete operations - **APM instrumentation** — Middleware and decorators for tracing rule execution and client operations - **InversifyJS DI** — All services use constructor injection with typed tokens, scoped per-request or singleton as appropriate - **Pipeline pattern** — Rule executor and dispatcher use composable step-based pipelines - **Saved Objects** — Rules stored as hidden saved objects; notification policies stored as encrypted saved objects (for API key protection) - **Feature privileges** — Dedicated Kibana feature with read/all privileges for RBAC --- <details> <summary><strong>Core Engine & Plugin Init</strong> (12 PRs)</summary> - elastic#247283 — Init alerting v2 plugin (@cnasikas) - elastic#247452 — Add the alerting v2 feature privileges (@cnasikas) - elastic#247673 — Director (@cnasikas) - elastic#248306 — Create basic services (@cnasikas) - elastic#248696 — Initialize all resources (@cnasikas) - elastic#250023 — Schema package (@cnasikas) - elastic#250010 — YML Editor (@cnasikas) - elastic#251064 — Remove index.mode: lookup for RnA alert indices (@cnasikas) - elastic#251707 — Simplify task registration pattern (@kdelemme) - elastic#251876 — Dedicated user service (@cnasikas) - elastic#252073 — Use `kbn/data-streams` in alerting_v2 (@cnasikas) - elastic#255120 — Update alerting-v2 owner to new rna project team (@cnasikas) </details> <details> <summary><strong>Rule Execution Pipeline</strong> (12 PRs)</summary> - elastic#247472 — Add alerting v2 Rule Executor (@darnautov) - elastic#248285 — Alerting v2 rule HTTP APIs (@darnautov) - elastic#248728 — Add basic alert actions route (@darnautov) - elastic#250161 — Refactor rule executor to use a pipeline pattern (@darnautov) - elastic#252292 — Implement the CountTimeframeStrategy for the director (@cnasikas) - elastic#252544 — Add support of streaming in the rule executor (@darnautov) - elastic#252754 — Update rule attributes (@kdelemme) - elastic#253355 — Add getRules client method (@kdelemme) - elastic#253668 — Make evaluation.query.condition optional (@kdelemme) - elastic#254031 — Add recovery event generation to rule execution pipeline (@kdelemme) - elastic#255968 — ES&elastic#124;QL views (@adcoelho) - elastic#256697 — Create episodes ES&elastic#124;QL view (@adcoelho) </details> <details> <summary><strong>Alert Suppression & Episodes</strong> (3 PRs)</summary> - elastic#252174 — Alert suppression (@kdelemme) - elastic#256486 — Fix suppression query (@kdelemme) - elastic#256527 — Store 'unmatched' action for unmatched alert episodes (@kdelemme) </details> <details> <summary><strong>Dispatcher & Notification Engine</strong> (6 PRs)</summary> - elastic#250822 — Alerting v2 dispatcher (@kdelemme) - elastic#251529 — Use query service in dispatcher (@kdelemme) - elastic#251679 — Dispatcher task (@kdelemme) - elastic#252758 — Dispatcher notification policy (@kdelemme) - elastic#255332 — Wait for resources before scheduling dispatcher task (@kdelemme) - elastic#256536 — Use stored encrypted API keys from Notification Policy in dispatcher step (@kdelemme) </details> <details> <summary><strong>Notification Policies (Server)</strong> (4 PRs)</summary> - elastic#251336 — Introduce notification policy CRUD APIs and client (@cnasikas) - elastic#253134 — Update notification policy (@cnasikas) - elastic#254808 — Store API key owner on Notification Policy (@kdelemme) - elastic#256940 — Make notification policies global with optional rule-label scoping (@kdelemme) </details> <details> <summary><strong>Notification Policies UI</strong> (1 PR)</summary> - elastic#255599 — Add notification policies UI and Storybook form story (@adcoelho) </details> <details> <summary><strong>Rule Authoring UI</strong> (13 PRs)</summary> - elastic#250961 — Add create rule flyout in Discover (@adcoelho) - elastic#255111 — Add activation configuration fields to alerting V2 rule form (@yiannisnikolopoulos) - elastic#255427 — Rule form: provide services via context (@dominiqueclarke) - elastic#255876 — MVP rule form, Split evaluation condition, and Recovery configuration (@dominiqueclarke) - elastic#256260 — Foundational rule list (@dominiqueclarke) - elastic#256756 — Wire up edit flow (@dominiqueclarke) - elastic#256801 — Move consecutive breaches max to shared constants (@yiannisnikolopoulos) - elastic#256818 — Preview query and design parity (@dominiqueclarke) - elastic#256938 — Allow clearing number inputs in state transition fields (@yiannisnikolopoulos) - elastic#257017 — Add enable/disable and clone rule to rule list (@dominiqueclarke) - elastic#257246 — Remove all React.FC (@dominiqueclarke) - elastic#257415 — Rule form - fix test (@dominiqueclarke) - elastic#257454 — Block comma key in number input component (@yiannisnikolopoulos) </details> <details> <summary><strong>API Documentation & Schema</strong> (2 PRs)</summary> - elastic#254901 — Rename indexes for alert events and actions (@adcoelho) - elastic#255810 — OAS for alert action routes (@adcoelho) </details> <details> <summary><strong>Observability & Monitoring</strong> (3 PRs)</summary> - elastic#254925 — Add ApmMiddleware to the rule executor (@adcoelho) - elastic#255115 — Add the withAPM decorator and apply it to the rules_client (@adcoelho) - elastic#255999 — Fix linting problem in apm middleware (@adcoelho) </details> <details> <summary><strong>CI & Maintenance</strong> (2 PRs)</summary> - elastic#257409 — Refactor SO services to use inversify DI for client initialization (@darnautov) - Fix alerting-v2-schema jest config (@darnautov) </details> --- --------- Co-authored-by: Dima Arnautov <dmitrii.arnautov@elastic.co> Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Kevin Delemme <kevin.delemme@elastic.co> Co-authored-by: Mike Côté <mikecote@users.noreply.github.com> Co-authored-by: Antonio <antonio.coelho@elastic.co> Co-authored-by: Kevin Delemme <kdelemme@gmail.com> Co-authored-by: Dominique Clarke <dominique.clarke@elastic.co> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Jason Rhodes <jason.rhodes@elastic.co> Co-authored-by: Yiannis Nikolopoulos <yiannis.nikolopoulos@elastic.co> Co-authored-by: Alexi Doak <109488926+doakalexi@users.noreply.github.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Bailey Cash <bailey.cash@elastic.co> Co-authored-by: Anna Davydova <ana.davydova@elastic.co> Co-authored-by: Umberto Pepato <umbopepato@users.noreply.github.com> Co-authored-by: Jason Rhodes <jason.matthew.rhodes@gmail.com> Co-authored-by: Joana Cardoso <169058851+joana-cps@users.noreply.github.com>
## Summary ### Key capabilities - **ES|QL-native rule evaluation** — Rules are defined as ES|QL queries with optional WHERE clause conditions, evaluated on a configurable schedule - **Alert lifecycle management** — Full episode tracking with pending → active → recovering → inactive state transitions, including configurable alert delay (consecutive breaches / duration) - **Event-driven architecture** — Alert events and actions are stored in dedicated data streams (`.alerting-events`, `.alerting-actions`) with ES|QL views for querying - **Notification dispatch pipeline** — A multi-step dispatcher that matches alert episodes to notification policies, handles throttling/suppression, and triggers Kibana Workflows using encrypted API keys - **Notification policies** — CRUD APIs and UI for creating notification policies with KQL-based rule matching, workflow integration, and API key management - **Rule authoring UI** — A shared rule form package (`@kbn/alerting-v2-rule-form`) usable standalone or embedded in Discover, with ES|QL editor, WHERE clause condition editing, recovery configuration, and live query preview - **Rule management UI** — Full rule list with pagination, enable/disable, clone, edit, and delete operations - **APM instrumentation** — Middleware and decorators for tracing rule execution and client operations ### Architecture highlights - **InversifyJS DI** — All services use constructor injection with typed tokens, scoped per-request or singleton as appropriate - **Pipeline pattern** — Rule executor and dispatcher use composable step-based pipelines - **Saved Objects** — Rules stored as hidden saved objects; notification policies stored as encrypted saved objects (for API key protection) - **Feature privileges** — Dedicated Kibana feature with read/all privileges for RBAC --- ## Contained PRs <details> <summary><strong>Core Engine & Plugin Init</strong> (12 PRs)</summary> - elastic#247283 — Init alerting v2 plugin (@cnasikas) - elastic#247452 — Add the alerting v2 feature privileges (@cnasikas) - elastic#247673 — Director (@cnasikas) - elastic#248306 — Create basic services (@cnasikas) - elastic#248696 — Initialize all resources (@cnasikas) - elastic#250023 — Schema package (@cnasikas) - elastic#250010 — YML Editor (@cnasikas) - elastic#251064 — Remove index.mode: lookup for RnA alert indices (@cnasikas) - elastic#251707 — Simplify task registration pattern (@kdelemme) - elastic#251876 — Dedicated user service (@cnasikas) - elastic#252073 — Use `kbn/data-streams` in alerting_v2 (@cnasikas) - elastic#255120 — Update alerting-v2 owner to new rna project team (@cnasikas) </details> <details> <summary><strong>Rule Execution Pipeline</strong> (12 PRs)</summary> - elastic#247472 — Add alerting v2 Rule Executor (@darnautov) - elastic#248285 — Alerting v2 rule HTTP APIs (@darnautov) - elastic#248728 — Add basic alert actions route (@darnautov) - elastic#250161 — Refactor rule executor to use a pipeline pattern (@darnautov) - elastic#252292 — Implement the CountTimeframeStrategy for the director (@cnasikas) - elastic#252544 — Add support of streaming in the rule executor (@darnautov) - elastic#252754 — Update rule attributes (@kdelemme) - elastic#253355 — Add getRules client method (@kdelemme) - elastic#253668 — Make evaluation.query.condition optional (@kdelemme) - elastic#254031 — Add recovery event generation to rule execution pipeline (@kdelemme) - elastic#255968 — ES&elastic#124;QL views (@adcoelho) - elastic#256697 — Create episodes ES&elastic#124;QL view (@adcoelho) </details> <details> <summary><strong>Alert Suppression & Episodes</strong> (3 PRs)</summary> - elastic#252174 — Alert suppression (@kdelemme) - elastic#256486 — Fix suppression query (@kdelemme) - elastic#256527 — Store 'unmatched' action for unmatched alert episodes (@kdelemme) </details> <details> <summary><strong>Dispatcher & Notification Engine</strong> (6 PRs)</summary> - elastic#250822 — Alerting v2 dispatcher (@kdelemme) - elastic#251529 — Use query service in dispatcher (@kdelemme) - elastic#251679 — Dispatcher task (@kdelemme) - elastic#252758 — Dispatcher notification policy (@kdelemme) - elastic#255332 — Wait for resources before scheduling dispatcher task (@kdelemme) - elastic#256536 — Use stored encrypted API keys from Notification Policy in dispatcher step (@kdelemme) </details> <details> <summary><strong>Notification Policies (Server)</strong> (4 PRs)</summary> - elastic#251336 — Introduce notification policy CRUD APIs and client (@cnasikas) - elastic#253134 — Update notification policy (@cnasikas) - elastic#254808 — Store API key owner on Notification Policy (@kdelemme) - elastic#256940 — Make notification policies global with optional rule-label scoping (@kdelemme) </details> <details> <summary><strong>Notification Policies UI</strong> (1 PR)</summary> - elastic#255599 — Add notification policies UI and Storybook form story (@adcoelho) </details> <details> <summary><strong>Rule Authoring UI</strong> (13 PRs)</summary> - elastic#250961 — Add create rule flyout in Discover (@adcoelho) - elastic#255111 — Add activation configuration fields to alerting V2 rule form (@yiannisnikolopoulos) - elastic#255427 — Rule form: provide services via context (@dominiqueclarke) - elastic#255876 — MVP rule form, Split evaluation condition, and Recovery configuration (@dominiqueclarke) - elastic#256260 — Foundational rule list (@dominiqueclarke) - elastic#256756 — Wire up edit flow (@dominiqueclarke) - elastic#256801 — Move consecutive breaches max to shared constants (@yiannisnikolopoulos) - elastic#256818 — Preview query and design parity (@dominiqueclarke) - elastic#256938 — Allow clearing number inputs in state transition fields (@yiannisnikolopoulos) - elastic#257017 — Add enable/disable and clone rule to rule list (@dominiqueclarke) - elastic#257246 — Remove all React.FC (@dominiqueclarke) - elastic#257415 — Rule form - fix test (@dominiqueclarke) - elastic#257454 — Block comma key in number input component (@yiannisnikolopoulos) </details> <details> <summary><strong>API Documentation & Schema</strong> (2 PRs)</summary> - elastic#254901 — Rename indexes for alert events and actions (@adcoelho) - elastic#255810 — OAS for alert action routes (@adcoelho) </details> <details> <summary><strong>Observability & Monitoring</strong> (3 PRs)</summary> - elastic#254925 — Add ApmMiddleware to the rule executor (@adcoelho) - elastic#255115 — Add the withAPM decorator and apply it to the rules_client (@adcoelho) - elastic#255999 — Fix linting problem in apm middleware (@adcoelho) </details> <details> <summary><strong>CI & Maintenance</strong> (2 PRs)</summary> - elastic#257409 — Refactor SO services to use inversify DI for client initialization (@darnautov) - Fix alerting-v2-schema jest config (@darnautov) </details> --- --------- Co-authored-by: Dima Arnautov <dmitrii.arnautov@elastic.co> Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Kevin Delemme <kevin.delemme@elastic.co> Co-authored-by: Mike Côté <mikecote@users.noreply.github.com> Co-authored-by: Antonio <antonio.coelho@elastic.co> Co-authored-by: Kevin Delemme <kdelemme@gmail.com> Co-authored-by: Dominique Clarke <dominique.clarke@elastic.co> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Jason Rhodes <jason.rhodes@elastic.co> Co-authored-by: Yiannis Nikolopoulos <yiannis.nikolopoulos@elastic.co> Co-authored-by: Alexi Doak <109488926+doakalexi@users.noreply.github.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Bailey Cash <bailey.cash@elastic.co> Co-authored-by: Anna Davydova <ana.davydova@elastic.co> Co-authored-by: Umberto Pepato <umbopepato@users.noreply.github.com> Co-authored-by: Jason Rhodes <jason.matthew.rhodes@gmail.com> Co-authored-by: Joana Cardoso <169058851+joana-cps@users.noreply.github.com>
Summary
The focus of this PR is the view creation logic. I will create a follow-up PR with an
$.alerting-episodesview when we decide which fields we want on the alerts table.This PR adds startup initialization of ES|QL views in the Alerting v2 plugin(in addition to the existing data stream resource setup). Views are created in Elasticsearch via the
/_query/viewAPI, so alerting can use them.There is currently a separate PR in the Elasticsearch repo that updates the Kibana system user permissions to allow for the ESQL view management.
The views created by this PR are: