Related to: #91265

Summary

Build Security Rule Execution Log on top of the event_log plugin. Implement logging and querying the log under the hood of the app.

Important!

• event_log plugin does not have all the needed APIs and doesn't provide support for RBAC that we need. We should collaborate on extending it with the Alerting team. More on that in the sub-task [Security Solution] Extend event_log plugin with functionality required for Rule Execution Log #106347.

DOs:

• Use this proposal as a foundation: [Security Solution][Detections] Proposal: building Rule Execution Log on top of Event Log and ECS #94143.
• Define a schema for rule execution events and metrics.
• Implement a rule execution log abstraction that would provide a simple api for writing to the log and executing queries, hiding non-important details from the rest of security_solution.
• Inject an instance of rule execution log into executor functions of Security rule types and into Security route handlers.
• Implement actual execution logging in rule executors (writing events and metrics to the log) and route handlers.
• Update route handlers to start fetching execution events and metrics from the log instead of custom saved objects.
• Hide the new execution log implementation under its own feature switch. If the switch is disabled, we use the old custom saved objects. If it's enabled, we use the new execution log.
• Finally, remove the old implementation and custom SO definition from the code.

DONTs:

• No changes in the UI is expected.
• The plan was to implement this new log within the new incarnation of Detection Engine built on top of rule_registry, so no need to change the current implementation of Detection Engine.
• Backwards compatibility with our custom Saved Objects (of type siem-detection-engine-rule-status) is not required. On the Security side, when we release the new Rule Execution Log, it’d be acceptable to drop the historical data stored in custom SOs (1 current status + 5 failures per rule). This would likely allow us to not invest time into developing migrations and fallbacks for custom SOs, especially considering breaking changes in SO ids in the upcoming 8.0 ([Meta] Breaking saved object type conversions in 8.0 #100489).

Sub-tasks

Further clarification of requirements:

• Figure out requirements for RBAC for Rule Execution Log from the product standpoint. Mind not only current requirements in Security, but also future requirements in Observability/Stack/other solutions in the context of Unified Rule Monitoring.
• Related: communicate to the Observability team that probably requirements for RBAC for "evaluations" in Observability should be figured out as well.

Implementation:

• [RAC][Security Solution][Detections] Implement a Rule Execution Log abstraction for use in Security Solution #106461
• [RAC][Security Solution][Detections] Integrate Rule Execution Log into Detection Engine #106465
• [RAC][Security Solution][Detections] Integrate Rule Execution Log into HTTP route handlers #106466
• Make sure existing UI (Rule Management page, Rule Details page) is working well with the new Rule Execution Log. We expect that we update existing API routes without any breaking changes, so no further integration work will be needed. Thus, there's no ticket for that.
• [Security Solution] Extend event_log plugin with functionality required for Rule Execution Log #106347 [partially implemented]
• [RAC][Security Solution][Detections] Integrate the extended event_log plugin into the Rule Execution Log #106469 [partially implemented]
• [Alerting] Storing custom searchable rule execution data inside the rule #112193

Handling breaking changes in Saved Object ids by 7.16 FF (#105819):

• In Rule Status SOs. If we ship Rule Execution Log in 7.16 with the feature flag being disabled, we will need to make sure to at least specify SO references when saving Rule Status SOs. See [Security Solution][Detections] Migrate to using Saved Object References array to reference alert saved objects from security saved objects #107068
• In rule execution events in event_log. We don't have our own ticket for that; Alerting team has Research approach for event log when making rules and connectors share-capable #100666