Security Analytics workflows based on log types

Logs based on raw log types

In order to run Detectors on raw logs, the fields of these raw log documents are first mapped to ECS fields using field aliases.

Logs based on ECS schema

Users can also ingest logs in ECS schema format. In this case, Detectors will not require any explicit mappings to match rules with logs.

Logs based on OCSF schema

In future, Security Analytics plan to allow users to ingest data in OCSF schema format. In this case, the fields of these log documents in OCSF format are mapped back to ECS fields again using field aliases.

ECS remains the base format for all log types for which detectors run & matches rules with logs.

Proposed Data Structure to store mappings

A flat table is used to store field mappings from raw log fields to ECS fields to OCSF fields. The Raw Log Types field serve as the primary key for this flat table.

This will help in switching the base log type from ECS to OCSF in future easily without any backward compatibility issues.

Support for custom log types

Security Analytics today only support 13 pre-defined log types. But in future, we extend Security Analytics to support custom log types.

Users can define their own custom mappings, custom Sigma Rules & use them to run Detectors on logs from a custom datasource.

Design

The Log Type object will be stored in an OpenSearch index named .opensearch-sap-log-types-config.
The mappings for the index will be defined as follows:

[{
  "name": "Cloudtrail",
  "description": "",
  "mappings": [
    {
      "raw_field": "userIdentity.arn",
      "ecs": "aws-cloudtrail-user_identity-arn",
      "ocsf": "actor.user.uuid"
    },
    {
      "raw_field": "",
      "ecs": "",
      "ocsf": ""
    }
  ]
}]