Epics: https://github.com/elastic/security-team/issues/1974 (internal), #174168
Forked from: #144060

Summary

We need to figure out what rule fields we’re going to show in the rule upgrade flyout.

• Which fields will be "customizable" (the user will be able to see a diff and resolve conflicts in the flyout)?
• Which fields will be "technical" (we will handle their update under the hood)?

You can see a detailed breakdown of all the existing rule fields below. They are categorized by multiple dimensions which will hopefully help us with reasoning about each field and making decisions.

Please scroll down to find fields to pay attention to and discuss.

Rule fields

Kind:

• Technical: is needed for the app to work
• Domain: is needed for the users' use cases and/or detection rule business logic to work

Change:

• Static: changes at least once (e.g. on rule creation) or at most every time a user updates this field in a rule
• Dynamic: may change often and/or in the background without any user actions

Level:

• Framework: defined at the Alerting Framework level, is known to the Framework, is common to many rule types including Observability and Stack rules
• Solution: defined on the Security Solution side, in most cases stored in the rule params object
• Both: defined at both levels simultaneously (sometimes it's one part at the Framework and another one at the Solution level)

Prebuilt?

• ✅: supported by the prebuilt rule schema and can be specified in https://github.com/elastic/detection-rules
• ✅ (unused): supported by the schema but is not used in https://github.com/elastic/detection-rules
• ✅⁉️: supported by the schema but probably shouldn't be
• ✅⁉️ (unused): supported by the schema but probably shouldn't be + not used in https://github.com/elastic/detection-rules
• ❌: is specific to the app and not supported by the prebuilt rule schema

User editable?

• Editable: currently, users can create and update the field via the UI or the API
• Editable (1): currently, users can specify this field only once during rule creation, they cannot update it after a rule is created
• Editable (no UI): currently, users can create and update the field only via the API
• Readonly: currently, users can only read the field

Customizable?

• ✅: users will be able to edit this field in prebuilt rules and then be able to resolve potential conflicts during the rule upgrade process (the field will be visible in the rule upgrade UI)
• ⚙️ (auto): users will be able to edit this field in prebuilt rules, but conflicts during the rule upgrade process will be resolved automatically under the hood (the field won't be visible in the rule upgrade UI)
• ❌: users won't be able to edit this field in prebuilt rules (the field won't be visible in the rule upgrade UI)

Fields common to all rule types

Fields specific to Custom Query and Saved Query rules

Fields specific to EQL rules

Fields specific to Indicator Match rules

Fields specific to Threshold rules

Fields specific to Machine Learning rules

Fields specific to New Terms rules

Fields specific to ES|QL Rules

Rule fields to discuss

Fields common to all rule types:

• type ― Confirm that we're not going to support changing the rule type on upgrade.
• related_integrations, required_fields, setup ― These can be specified in https://github.com/elastic/detection-rules but we don't support editing them in the app. Since we don't have UI for editing them, I'm wondering if it's time to finally build it. When we build the UI we will be able to support these fields in the rule upgrade flyout.
• author, license ― Are we going to allow the user to customize these? Could there be any legal restrictions on that?
• actions, throttle ― We allow the user to specify actions for installed prebuilt rules, but no prebuilt rules actually have actions in https://github.com/elastic/detection-rules. Can we remove support for actions from the prebuilt rule schema and handle the upgrade for these 2 fields under the hood (always take the current user's version)?
• exceptions_list ― We allow the user to specify exceptions for installed prebuilt rules, but only one prebuilt rule actually have exceptions in https://github.com/elastic/detection-rules. It's the "Endpoint Security" rule. How should we handle potential conflicts between Elastic's and user's versions of this field? Can we use some simple heuristic and handle this under the hood instead of building some kind of UI?
• max_signals ― This is a circuit breaker value. A few prebuilt rules have it in https://github.com/elastic/detection-rules. Users can modify it via the API. We don't have a UI for it. How do we resolve conflicts in this field?
• output_index ― This is a name of a concrete target index where the rule is supposed to write alerts. I believe this field is deprecated and unused in the app. No prebuilt rules have it in https://github.com/elastic/detection-rules. Users can still modify it via the API because it's in the schema. I don't think we have a UI for it. What should we do with this field?
• namespace ― This is a namespace (a suffix) of the alerts-as-data index where the rule is supposed to write alerts. AFAIR this is a replacement for output_index but it looks to be unused in the app. No prebuilt rules have it in https://github.com/elastic/detection-rules. Users can still modify it via the API because it's in the schema. I don't think we have a UI for it. What should we do with this field?

Fields specific to Custom Query and Saved Query rules:

• response_actions ― This is fully user-editable and supported by the prebuilt rule schema, but no prebuilt rules have it in https://github.com/elastic/detection-rules. Our suggestion would be to remove it from the schema and handle it under the hood of the upgrade workflow.

Fields specific to EQL rules:

• language ― Specifically for EQL rules, do we allow the user to change this? From the rule schema, it looks like this can accept only one value: "eql".
  • EDIT: No, we don't want users to be able to change language for EQL (and neither for ES|QL) rules. Language should be eql and esql respectively. The field might change if a rule changes its type from an upstream update, but a user should not be able to manually force this.

Fields specific to Indicator Match rules:

• threat_indicator_path ― What is this field used for? Do we have a UI for it? A few prebuilt rules have it in https://github.com/elastic/detection-rules.
  • EDIT: corresponds to Indicator prefix override field, is currently editable in the UI and should be customizable.
• concurrent_searches ― This one is user-editable via API but we have no UI for it. No prebuilt rules have it in https://github.com/elastic/detection-rules. How should we handle it in the upgrade workflow?
• items_per_search ― This one is user-editable via API but we have no UI for it. No prebuilt rules have it in https://github.com/elastic/detection-rules. How should we handle it in the upgrade workflow?