Summary
Consolidate #98353 and #98935 into a single implementation.
We ended up having 4 implementations of index management/writing/reading related/similar to the problem we're trying to solve in RAC: two in rule_registry (RuleDataService and EventLogService), one in security_solution (for .siem-signals index), one in event_log plugin. We should compare them, mind their strong and weak parts and build a consolidated implementation in rule_registry.
High-level plan:
Regarding robust index bootstrapping, consider this:
- Race conditions during index bootstrapping should be handled one way or another. Possible options: a) robust idempotent logic with error handling; b) leveraging
task_manager to make sure bootstrapping procedure runs only once at a time; c) using some sort of distributed lock, and d) maybe something else I'm missing. Maybe we could check how Saved Objects service bootstraps .kibana index.
- Errors should be handled correctly. Pay special attention to errors from Elasticsearch APIs.
When the consolidated implementation is ready, make sure to update all references to it from plugins which already use it: security_solution, observability, apm, uptime, etc.
Tasks for 7.15
Tasks for 7.16
Backlog
Indexing and index bootstrapping logic:
API enhancements for RuleDataService and RuleDataClient:
User-defined resources::
Misc:
Consider these as well:
Summary
Consolidate #98353 and #98935 into a single implementation.
We ended up having 4 implementations of index management/writing/reading related/similar to the problem we're trying to solve in RAC: two in
rule_registry(RuleDataServiceandEventLogService), one insecurity_solution(for.siem-signalsindex), one inevent_logplugin. We should compare them, mind their strong and weak parts and build a consolidated implementation inrule_registry.High-level plan:
RuleDataServiceandRuleDataClient. Enhance it, improve its API for better and safer DX, improve its implementation, especially the part responsible for index bootstrapping.EventLogService) from rule_registry..siem-signalsimplementation in Security once we migrate the Detection Engine torule_registry.event_logplugin for their monitoring needs. Also, we're planning to implement Rule Execution Log in Security based onevent_log([Security Solution] Extend event_log plugin with functionality required for Rule Execution Log #106347, [RAC][Security Solution][Detections] Rule Execution Log - technical implementation #101013).Regarding robust index bootstrapping, consider this:
task_managerto make sure bootstrapping procedure runs only once at a time; c) using some sort of distributed lock, and d) maybe something else I'm missing. Maybe we could check how Saved Objects service bootstraps.kibanaindex.When the consolidated implementation is ready, make sure to update all references to it from plugins which already use it:
security_solution,observability,apm,uptime, etc.Tasks for 7.15
ruleDataClient.bulk()- ✔️ @banderrorTasks for 7.16
getAuthorizedAlertsIndicesincludes Kibana space id into the index name #111154Backlog
Indexing and index bootstrapping logic:
API enhancements for RuleDataService and RuleDataClient:
RuleDataClientUser-defined resources::
Misc:
Consider these as well: