Summary

The purpose of this PR is to update the detection engine to work with the new exception list schema. Previously, exceptions lived directly in the rule, now they are stored separately, and the rule only stores a reference to said exception lists.

Changes are concentrated in two areas. First x-pack/plugins/lists:

• Updates the EntryList schema so it now takes a list id and type (as needed for the detections engine logic) as opposed to before where it was just a string for the id
• Updates the lists plugin unit tests to account for the updated EntryList schema
• Exposes the exceptions list client (in the lists plugin) to be used by the detection engine

Second in x-pack/plugins/security_solution:

• Updates the detection engine exceptions_list schema. Previously, exceptions sat directly on the rule itself, now that exceptions are stored separately, the exceptions_list property needs to only keep reference to the exceptions list
• Updates schema unit tests
• Updates routes to use new exceptions list schema
• Updates route unit tests
• Removes old exception list scripts that are no longer necessary

To Do

• Update big loop logic (issue described below)
• Update UI to now dynamically read from rule exception_list [SIEM][Detection Engine] - Update UI to read rule exceptions_list #69939

Right now we have logic in build_exceptions_query.ts that takes exceptions that don't include large value lists and appends them to the main KQL query which is then translated into the elastic query dsl. We also have logic in filter_events_wtih_list.ts that uses inverse logic to filter events search results against any exceptions with large value lists. On their own, each of these does what is expected. However, after all expansion of the exceptions list functionality, it is clear that these two separate pieces of logic would not suffice.

This PR does not address that issue (it's in a follow up PR), but thought it worth noting here. It gets confusing with the boolean logic, but to try to simplify the issue we came upon with our existing solution is the following:

Imagine we had a rule, with exceptions_list: [{ id: 'exception_list', namespace_type: 'single'}]. The exception_list has 2 exception list items:

// exception list item 1
a && b && c(large value list)

// exception list item 2
d && e(large value list)

The initial search would query (query && a && b) || (query && d). Then the results of that search would get filtered through the large value lists logic, so it would filter for (c || e). What this would result in, is signals that were filtered for:

(query && a && b) || (query && d) || c || e

when what we want is:

(query && a && b && c) || (query && d && e)

These are not logically equivalent. Our current logic would work if there was a single exception item, but as soon as there is more than one as shown in the simplified example above, the logic breaks down. This issue is being addressed in another PR.

Testing

To turn on lists plugin - in kibana.dev.yml

# Enable lists feature
xpack.lists.enabled: true
xpack.lists.listIndex: '.lists-yara'
xpack.lists.listItemIndex: '.items-yara'

Add export ELASTIC_XPACK_SIEM_LISTS_FEATURE=true to your bash file.

Use the scripts in x-pack/plugins/lists/server/scripts to create some sample exception lists and items. You can use the following:

If you've previously played around with lists, run ./hard_reset.sh (this will delete any lists you've created).

Create large value list:

• Create large value list ./post_list.sh
• Create large value list item ./post_list_item.sh (I modified the value to be "value": "10.4.2.140")

Create exception list:

• Create exception list ./post_exception_list.sh
• Create exception list item ./post_exception_list_item.sh ./exception_lists/new/exception_list_item_with_list.json

Use the scripts in x-pack/plugins/security_solution/server/lib/detection_engine/scripts to create rule:

• Run ./post_rule.sh ./rules/queries/query_with_list.json (Makes reference to the exception list created in step above)

In the Alerts table, you should see something like the following where you only see events where the event.module is zeek and source.ip is 10.4.2.140 (or whatever ip you specified).

Checklist

• Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
• Unit or functional tests were updated or added to match the most common scenarios

For maintainers

• This was checked for breaking API changes and was labeled appropriately