[ML] Adds Metadata and Discovery Analysis Jobs to Security Integration#76023
[ML] Adds Metadata and Discovery Analysis Jobs to Security Integration#76023blaklaybul merged 13 commits intoelastic:masterfrom
Conversation
|
Pinging @elastic/ml-ui (:ml) |
|
@blaklaybul in order to make the api integration tests pass, you have to add the new modules to the list of expected modules. |
|
@pheyos there are no new modules in this PR, it only updates to existing modules. Not sure why I'm getting this error. |
|
@elasticmachine merge upstream |
x-pack/plugins/ml/server/models/data_recognizer/modules/siem_winlogbeat/manifest.json
Outdated
Show resolved
Hide resolved
…nto ml-security-metadata-analysis
| { | ||
| "id": "linux_anomalous_user_name_ecs", | ||
| "file": "linux_anomalous_user_name_ecs.json" | ||
| }, |
There was a problem hiding this comment.
@randomuserid how would you like this to be updated?
There was a problem hiding this comment.
I don't see why that description cannot apply to the new Linux jobs
There was a problem hiding this comment.
great - leaving it as is
x-pack/plugins/ml/server/models/data_recognizer/modules/siem_auditbeat/manifest.json
Show resolved
Hide resolved
| { | ||
| "id": "windows_rare_user_runas_event", | ||
| "file": "windows_rare_user_runas_event.json" | ||
| }, |
There was a problem hiding this comment.
we could remove the words "process and network" because we have branched out to additional event types
There was a problem hiding this comment.
@randomuserid do you want to update this? This is what a user sees when interacting with the module via the ML ui.
There was a problem hiding this comment.
Why don't we leave it alone for now. When we have a decision on multi-index Windows events we can rewrite them then.
.../models/data_recognizer/modules/siem_auditbeat/ml/linux_network_configuration_discovery.json
Outdated
Show resolved
Hide resolved
peteharverson
left a comment
There was a problem hiding this comment.
Added extra comments around the custom URLs.
.../server/models/data_recognizer/modules/siem_winlogbeat/ml/windows_rare_metadata_process.json
Outdated
Show resolved
Hide resolved
.../ml/server/models/data_recognizer/modules/siem_winlogbeat/ml/windows_rare_metadata_user.json
Outdated
Show resolved
Hide resolved
...ver/models/data_recognizer/modules/siem_auditbeat/ml/linux_network_connection_discovery.json
Outdated
Show resolved
Hide resolved
...ver/models/data_recognizer/modules/siem_auditbeat/ml/linux_rare_kernel_module_arguments.json
Outdated
Show resolved
Hide resolved
.../ml/server/models/data_recognizer/modules/siem_auditbeat/ml/linux_rare_metadata_process.json
Outdated
Show resolved
Hide resolved
...plugins/ml/server/models/data_recognizer/modules/siem_auditbeat/ml/linux_rare_sudo_user.json
Outdated
Show resolved
Hide resolved
...ins/ml/server/models/data_recognizer/modules/siem_auditbeat/ml/linux_rare_user_compiler.json
Outdated
Show resolved
Hide resolved
...ver/models/data_recognizer/modules/siem_auditbeat/ml/linux_system_information_discovery.json
Outdated
Show resolved
Hide resolved
.../server/models/data_recognizer/modules/siem_auditbeat/ml/linux_system_process_discovery.json
Outdated
Show resolved
Hide resolved
.../ml/server/models/data_recognizer/modules/siem_auditbeat/ml/linux_system_user_discovery.json
Outdated
Show resolved
Hide resolved
.../server/models/data_recognizer/modules/siem_winlogbeat/ml/windows_rare_metadata_process.json
Show resolved
Hide resolved
|
@elasticmachine merge upstream |
peteharverson
left a comment
There was a problem hiding this comment.
Tested latest changes and all looks good. Just added one comment about the influencers for one of the jobs.
| ], | ||
| "influencers": [ | ||
| "process.name", | ||
| "process.working_directory", |
There was a problem hiding this comment.
If we want to limit the job to 4 influencers for this linux_rare_kernel_module_arguments job, is it better to use process.title in place of process.working_directory? The Anomaly Explorer won't list process.title as a swim lane 'view by' option or in the left hand Top Influencers list otherwise.
There was a problem hiding this comment.
It's my understanding that for this analysis, it's valuable to know the location (i.e. process.working_directory) from which the process is being executed. @randomuserid, if we had to choose between swapping either process.name or process.working_directory for process.title as an influencer, which do you think should go? Here's some example data:
"process" : {
"title" : "/bin/sh -e /usr/share/initramfs-tools/hooks/kmod prereqs",
"name" : "kmod",
"working_directory" : "/"
}
There was a problem hiding this comment.
Note that if we choose to swap out process.name then the two custom URLs that use process.name should be deleted (unless they also work with process.title?).
There was a problem hiding this comment.
Process.name is probably the best one to cut because it will tend to show up in the process.title field & the data feed is limited to a few kernel module related processes by name so process name will always be one of those.
There was a problem hiding this comment.
spoke with @randomuserid. process.name is limited to just 4 processes because of the datafeed query, so let's swap it out for process.title in the influencer list. As @peteharverson mentioned, this means we'll need to remove the custom urls that utilize process.name. I'll make the change.
…/kibana into ml-security-metadata-analysis
peteharverson
left a comment
There was a problem hiding this comment.
Latest changes LGTM!
💚 Build SucceededBuild metricsdistributable file count
History
To update your PR or re-run it, just comment with: |
|
Friendly reminder: Looks like this PR hasn’t been backported yet. |
elastic#76023) * adds enhanced winlogbeat module * adds enhanced auditbeat module * splits discovery jobs * fixes winlogbeat manifest * adds process group * adds custom urls * adds by field as influencer * use process.title as influencer * updates custom url Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
|
Looks like this PR has a backport PR but it still hasn't been merged. Please merge it ASAP to keep the branches relatively in sync. |
#76023) (#77383) * adds enhanced winlogbeat module * adds enhanced auditbeat module * splits discovery jobs * fixes winlogbeat manifest * adds process group * adds custom urls * adds by field as influencer * use process.title as influencer * updates custom url Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>


Summary
This PR contains a suite of new ML job for the Security solution's anomaly detection integration that offers new analyses to enable threat detection on metadata services, system and network discovery processes, compiler events, and processes invoked by
sudo.In total, there are 12 new jobs:
siem_auditbeat/ml/linux_rare_kernel_module_argumentssiem_auditbeat/ml/linux_rare_metadata_processsiem_auditbeat/ml/linux_rare_metadata_usersiem_auditbeat/ml/linux_rare_user_compilersiem_auditbeat/ml/linux_rare_sudo_usersiem_auditbeat/ml/linux_network_connection_discoverysiem_auditbeat/ml/linux_network_configuration_discoverysiem_auditbeat/ml/linux_system_information_discoverysiem_auditbeat/ml/linux_system_process_discoverysiem_auditbeat/ml/linux_system_user_discoverysiem_winlogbeat/ml/windows_rare_metadata_processsiem_winlogbeat/ml/windows_rare_metadata_usermodules:
siem_auditbeat,siem_winlogbeat