Epic: #174168
Caused by: https://github.com/elastic/security-team/issues/11822, elastic/detection-rules#4150 (comment), https://github.com/elastic/ia-trade-team/issues/552
Related to: #201500

Summary

Since last summer of 2024, we've had multiple incidents in Cloud where Kibana would crash with an OOM because of the installation of the package with prebuilt rules. We've done a lot of work to optimize package installation logic in Fleet and recently re-added all historical rule versions to the package. However, the most recent incident has shown that this wasn't enough. As a quick mitigation, we again reduced the package size to having at most 4 assets per rule in package v8.17.5. This change is final and we're not going to ever increase this limit.

The fact that we're not gonna have many historical rule versions in the package means that the situation of not having the base version for a rule will be pretty normal. It won't be an edge case, as we originally planned. This forces us to revisit our initial reasoning about handling missing base versions during customization, upgrading, and importing prebuilt rules.

Current behavior

When the base version of a currently installed or imported prebuilt rule is missing among the security-rule asset saved objects:

• On editing of any prebuilt rule, we mark the rule as customized unconditionally in all cases. For example, adding a new tag will mark the rule as customized. Then, if the user removes this tag, the rule will remain to be marked as customized. A related discussion we've had in the past: [Security Solution] Prebuilt Rule Marked as Customized By Just Clicking "Edit rule settings" And Clicking Save Without Any Changes #203315 (comment).

• On import of any prebuilt rule, we mark the rule as customized unconditionally in all cases. For example, exporting a non-customized prebuilt rule and re-importing it right away with "overwrite" will mark it as customized.

• On upgrade of any prebuilt rule, every rule field that has an update in the target version (or, strictly speaking, is different in the target version), will be marked as a solvable conflict.
  • User will have to upgrade the rule with preview via the flyout.
  • User won't be able to quickly upgrade the rule without preview directly from the table - neither one by one, nor in bulk.

• On upgrade of a customized prebuilt rule, every rule field that has an update in the target version (or, strictly speaking, is different in the target version), will be marked as a solvable conflict. Also, we will suggest the target field version as the final update proposal and will not incorporate the user customization into it. We have an existing bug for that: [Security Solution] Prebuilt rule customization is lost on upgrade when Base version is missing #201500.

Proposed behavior

Editing of prebuilt rules with missing base versions

When the base version of a currently installed prebuilt rule is missing among the security-rule asset saved objects, and the user edits this rule:

• We should mark the rule as customized, only if the new rule settings are different from the current rule settings.
  • For example, adding a new tag should mark the rule as customized. Then, if the user removes this tag, the rule should remain to be marked as customized. This matches the current behavior.
  • However, if the user saves the rule without making any changes to it, it should keep its is_customized field as is. This is different from the current behavior.

Importing of prebuilt rules with missing base versions

When the base version of a prebuilt rule that is being imported is missing among the security-rule asset saved objects, and the user imports this rule:

• If this rule is not installed, it should be created with is_customized field set to false.
• If this rule is already installed, it should be updated.
  • Its is_customized field should be set to true if the rule from the import payload is not equal to the installed rule.
  • Its is_customized field should be be kept unchanged (false or true) if the rule from the import payload is equal to the installed rule.

Upgrading of prebuilt rules with missing base versions

When the base version of a currently installed prebuilt rule is missing among the security-rule asset saved objects, and there is a new version available for this rule, and the user tries to upgrade this rule to the latest version, there will be two different logic flows for upgrading depending on whether the prebuilt rule is customized or not.

• When the rule has a missing base version and is NOT marked as customized:
  • We should return all the target fields from the diff algorithm as NO_CONFLICT (a change from the current behavior)
  • This will automatically allow users to bulk upgrade such rules without previewing them in the Rule Upgrade flyout, as if they were non-customized rules with a base version.
• When the rule has a missing base version and is marked as customized:
  • We should attempt to merge all non-functional mergeable fields (any field that doesn't have consequences with how the rule runs e.g. tags) and return them as SOLVABLE_CONFLICT.
  • We should pick the target version for all functional mergeable fields (e.g. index) and non-mergeable fields and return them as SOLVABLE_CONFLICT.

These changes will necessitate some modification of the current diff algorithm logic as well as requiring us to pass down the rule object in order to calculate the is_customized branching.

There will be an impact on the current rule upgrade UI. Now, when a rule has a missing base version:

• Upgrading individual rules one by one:
  • We will change the "Update rule" button to a "Review rule" button for rules that require review before upgrading (i.e. rules with unsolvable conflicts and rules with solvable conflicts, including customized rules without a base version). This button will just open the Rule Upgrade flyout instead of actually updating the rule as it does currently. Rules without conflicts will keep the current "Update rule" button.
  • We will show a callout in the Rule Upgrade flyout that will explain that the rule has gone a while without being updated (maybe calling it a stale rule?) and because of that, the upgrade experience will be a bit different.
  • We will follow the above logic and mark all fields as SOLVABLE_CONFLICT if the rule is marked as customized and NO_CONFLICT if it isn't customized.
• Upgrading multiple rules in bulk:
  • We will allow users to bulk upgrade prebuilt rules with solvable conflicts (if they choose to do so), but warn them about this operation being potentially dangerous. Currently, users can only bulk upgrade non-conflicting rules from the UI.
  • We won't allow users to bulk upgrade prebuilt rules with non-solvable conflicts.

Also, we will add a new sorting column and a new filter to the Rule Upgrade table for filtering and sorting:

• Rules with no conflicts
• Rules with solvable conflicts
• Rules with non-solvable conflicts

This allows the user to have more granularity over the ways they upgrade their rules and allow them ways to not let rules that require updating block them from keeping other rules up to date.

Incentivizing the users to upgrade their rules when possible

Since we now only have a max of 4 rule versions in a given package, we need to be more proactive with getting users to upgrade their rules, especially if they want to customize them. This will minimize the chances the base version will be missing when users are customizing or upgrading their rules. The ways to do this will be:

• Showing a callout on the Rule Management page that will motivate users to keep their rules up to date, and upgrade all the rules that can be upgraded, sooner rather than later.
• Showing a callout on the Rule Details page if this is a prebuilt rule that can be upgraded to a new version. This callout will motivate the user to upgrade the rule.
  • This will require refactoring of the upgrade/_review endpoint to take an id or array of ids so that we're not calling the entire endpoint whenever we need to see if a rule has an upgrade. (cc @xcrzx)
• Letting users open the Rule Upgrade flyout from the Rule Details page:
  • This will also require refactoring of the upgrade/_review endpoint
  • It would be complex to redirect the user to the rule's update table, filter the table, and open the rule upgrade flyout for a specific rule, much easier to let the user upgrade their rule directly from the details page, especially if we already have the review data
• Showing a callout on the Rule Editing page if this is a prebuilt rule that can be upgraded to a new version. This callout will motivate the user to upgrade the rule before editing it.