Related to:

#93729
#95903
#98353
elastic/elasticsearch#72181

Summary

There's more and more questions and concerns being raised regarding rule monitoring implementation for RAC. I'm working on #98353 which implements "event log" abstraction within rule registry that will be used for writing and reading both alerts and rule execution logs.

This RFC proposes some naming and structure for RAC indices, hierarchy for rule registries, and lists a few open questions and concerns.

Proposal

Index aliases naming convention would be similar to Elastic data stream naming scheme:

{prefix}-{consumer}.{additional.log.name}-{kibana space}

where:

• {prefix} will be .alerts by default. Users will be able to override it in Kibana config and set it to any other value, e.g. .alerts-xyz or .whatever (for compatibility with legacy Kibana multitenancy).
• {consumer} will indicate the rule/alert consumer in terms of Alerting framework (more info). We will probably have security, observability and stack as consumers.
• {additional.log.name} will be used to specify concrete indices for alerts, execution events, metrics and whatnot. We will probably have only alerts and events as concrete logs. It might be good to have short and sweet names without - or . in them (perhaps _ is ok).
• {kibana space} will indicate space id, for example default. In case of space agnostic alerts, the no.space placeholder can be used.

Examples of concrete index names

For clarity, this section contains the concrete index names that are created the Security and Observability solutions.

Security:

.alerts-security.alerts-{kibana space}-000001        // (1) 
.alerts-security.events-{kibana space}-000001        // (2) 

(1) The Alert documents that support human workflow and are updatable.
(2) Rule-specific execution events and metrics created by Security rules to enhance our observability of alerting.

Technically it will be possible to derive child logs from these alerts and events, e.g. .alerts-security.alerts.ml-{kibana space}. Although we don't think that we need this in Security at this point.

Observability:

.alerts-observability.{apm,uptime,metrics,logs}.alerts-{kibana space}-000001   // (1)
.alerts-observability.{apm,uptime,metrics,logs}.events-{kibana space}-000001   // (2)

.alerts-observability.metrics.alerts-no.space-000001   // (3)

(1) The Alert documents that are updatable. Same exact semantics as for Security (1)
(2) Supporting documents (evaluation) for the Alerts + execution logs to be used for the Observability of Alerting
(3) Example of space agnostic alert index. This can be used by the space-agnostic Rule types like the Stack Monitoring might need. The no.space "space" is not a valid Kibana space name, so this pattern can be used as a placeholder.

Diagram

Here's a structural diagram showing some rule execution dependencies in the context of RAC and how the proposed indices fit the whole picture:

diagram source