Skip to content

[RAC][Security Solution] Register Security Detection Rules with Rule Registry#96015

Merged
spong merged 63 commits intoelastic:masterfrom
spong:reference-rule-playground
May 28, 2021
Merged

[RAC][Security Solution] Register Security Detection Rules with Rule Registry#96015
spong merged 63 commits intoelastic:masterfrom
spong:reference-rule-playground

Conversation

@spong
Copy link
Copy Markdown
Member

@spong spong commented Apr 1, 2021 •
edited
Loading

Summary

This PR starts the migration of the Security Solution rules to use the rule-registry introduced in #95903. This is a pathfinding effort in porting over the existing Security Solution rules, and may include some temporary reference rules for testing out different paradigms as we move the rules over. See #95735 for details

Enable via the following feature flags in your kibana.dev.yml:

# Security Solution Rules on Rule Registry
xpack.ruleRegistry.index: '.kibana-[USERNAME]-alerts' # Only necessary to scope from other devs testing, if not specified defaults to `.alerts-security-solution`
xpack.securitySolution.enableExperimental: ['ruleRegistryEnabled']

Note: if setting a custom xpack.ruleRegistry.index, for the time being you must also update the DEFAULT_ALERTS_INDEX in order for the UI to display alerts within the alerts table.

Three reference rule types have been added (query, eql, threshold), along with scripts for creating them located in:

x-pack/plugins/security_solution/server/lib/detection_engine/reference_rules/scripts/

Main Detection page TGrid queries have been short-circuited to query .alerts-security-solution* for displaying alerts from the new alerts as data indices.

To test, checkout, enable the above feature flag(s), and run one of the scripts from the above directory, e.g. ./create_reference_rule_query.sh (ensure your ENV vars as set! :)

Alerts as data within the main Detection Page 🎉

cc @madirey @dgieselaar @pmuellr @yctercero @dhurley14 @marshallmain

@spong spong added the Theme: rac label obsolete label Apr 1, 2021
@madirey
Copy link
Copy Markdown
Contributor

madirey commented Apr 1, 2021 •
edited
Loading

After playing around in here, I'd suggest a couple of new rule type factories: EventLogRuleType (or PersistenceRuleType? ... which allows for writing multiple signals) and ThresholdRuleType (which encapsulates rule state to persist signals written on overlapping rule intervals so that dupes can be mitigated across buckets... this could mirror how dupe mitigation is performed for the DE threshold rules, except we'd store the state on the rule itself, rather than on the generated signals).

@spong spong mentioned this pull request Apr 1, 2021
Closed
19 tasks
@spong spong force-pushed the reference-rule-playground branch 6 times, most recently from d47bc97 to 1ab31aa Compare April 9, 2021 16:10
@ymao1 ymao1 mentioned this pull request Apr 26, 2021
Closed
@spong spong changed the title [RAC] Reference Rule Playground [RAC][Security Solution] Register Security Detection Rules with Rule Registry Apr 29, 2021
@spong spong added Feature:Event Correlation (EQL) Rule Security Solution Event Correlation (EQL) rule type Feature:Indicator Match Rule Security Solution Indicator Match rule type Feature:ML Rule Security Solution Machine Learning rule type Feature:Rule Value Lists Security Solution Detection Rule Value Lists area Feature:Threshold Rule Security Solution Threshold rule type release_note:skip Skip the PR/issue when compiling release notes Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Team:Detections and Resp Security Detection Response Team v7.14.0 v8.0.0 labels Apr 29, 2021
@spong spong marked this pull request as ready for review April 29, 2021 04:35
@spong spong requested a review from a team as a code owner April 29, 2021 04:35
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Apr 29, 2021

Pinging @elastic/security-solution (Team: SecuritySolution)

@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Apr 29, 2021

Pinging @elastic/security-detections-response (Team:Detections and Resp)

@spong spong mentioned this pull request May 13, 2021
Open
@spong spong requested a review from a team May 14, 2021 14:31
justinkambic
justinkambic approved these changes May 14, 2021
View reviewed changes
Copy link
Copy Markdown
Contributor

@justinkambic justinkambic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unsure why @elastic/uptime was pinged, but I have built this locally and all appears well with our products, and none of the files in the diff are owned by my team.

LGTM

@spong spong removed the request for review from a team May 14, 2021 16:04
smith
smith approved these changes May 14, 2021
View reviewed changes
Copy link
Copy Markdown
Contributor

@smith smith left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍🏻 from me once the APM change is reverted.

spong added 10 commits May 14, 2021 17:16
@spong
Fixes cypress tests, rule create scripts, updates docs, and replaces …
9e213fb 
…DEFAULT_ALERT_INDEX on plugin start
@spong
Fixing the jester
14a0f80
@spong
Merge branch 'master' of github.com:elastic/kibana into reference-rul…
222a52a 
…e-playground
@spong
Merge branch 'master' of github.com:elastic/kibana into reference-rul…
752a7ad 
…e-playground
@spong
Merge branch 'master' of github.com:elastic/kibana into reference-rul…
05c6a3b 
…e-playground
@spong
Merge branch 'master' of github.com:elastic/kibana into reference-rul…
670ce49 
…e-playground
@spong
Merge branch 'master' of github.com:elastic/kibana into reference-rul…
91bede9 
…e-playground
@spong
Scopes ruleDataService to security-solution and updates timeline to q…
5201667 
…uery alerts as data index if configured
@spong
Fixing the jester
5c83a41
@spong
Merge branch 'master' of github.com:elastic/kibana into reference-rul…
2a3cbfd 
…e-playground
@kibanamachine
Copy link
Copy Markdown
Contributor

kibanamachine commented May 28, 2021

💚 Build Succeeded

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id before after diff
ruleRegistry 41 43 +2

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id before after diff
securitySolution 6.9MB 6.9MB +3.5KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id before after diff
securitySolution 60.8KB 61.1KB +264.0B
Unknown metric groups

API count

id before after diff
ruleRegistry 41 43 +2

References to deprecated APIs

id before after diff
canvas 29 25 -4
crossClusterReplication 8 6 -2
fleet 22 20 -2
globalSearch 4 2 -2
lens 67 45 -22
lists 239 236 -3
ml 121 115 -6
monitoring 109 56 -53
securitySolution 342 346 +4
total -90

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @madirey @spong

@kibanamachine
Copy link
Copy Markdown
Contributor

kibanamachine commented May 28, 2021

💔 Backport failed

Status Branch Result
❌ 7.x Commit could not be cherrypicked due to conflicts

To backport manually run:
node scripts/backport --pr 96015

@spong spong mentioned this pull request May 28, 2021
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@smith smith smith approved these changes

@justinkambic justinkambic justinkambic approved these changes

+1 more reviewer

@madirey madirey madirey approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@madirey madirey

@spong spong

Labels

auto-backport Deprecated - use backport:version if exact versions are needed Feature:Event Correlation (EQL) Rule Security Solution Event Correlation (EQL) rule type Feature:Indicator Match Rule Security Solution Indicator Match rule type Feature:ML Rule Security Solution Machine Learning rule type Feature:Rule Value Lists Security Solution Detection Rule Value Lists area Feature:Threshold Rule Security Solution Threshold rule type release_note:skip Skip the PR/issue when compiling release notes Team:APM - DEPRECATED Use Team:obs-ux-infra_services. Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Team:Uptime - DEPRECATED Synthetics & RUM sub-team of Application Observability Theme: rac label obsolete v7.14.0 v8.0.0

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

8 participants

@spong @madirey @elasticmachine @cjcenizal @kibanamachine @smith @justinkambic @dgieselaar