Skip to content

[filebeat][aws][cloudtrail] accessKeyId should be searchable #18866

Closed
#19121
Closed
[filebeat][aws][cloudtrail] accessKeyId should be searchable#18866
#19121
Assignees
leehinman
Labels
FilebeatFilebeat
@nhnicwaller

Description

@nhnicwaller
nhnicwaller
opened on May 30, 2020

Describe the enhancement:

I'm using Filebeat with the aws module to ingest AWS CloudTrail logs into Elastic Cloud. This works mostly okay, except some of the most interesting pieces of data are not searchable. I am particularly interested in the AssumeRole event emitted by AWS CloudTrail (a full example is shown below). In this event, the field aws.cloudtrail.response_elements contains a stringified object, which unfortunately makes the contents opaque and impossible to search for in Kibana.

What would be super helpful is if credentials.accessKeyId and credentials.sessionToken were made searchable. Maybe this is possible by changing the tokenizer used on this field, but more likely it seems like it would require parsing this JSON and capturing those into new sub-fields.

Here is the line of code where the event body is stringified:

https://github.com/elastic/beats/blob/v7.7.0/x-pack/filebeat/module/aws/cloudtrail/ingest/pipeline.yml#L132

Describe a specific use case for the enhancement or feature:

I am attempting to use Elastic Cloud as a SIEM, and indeed Elastic is positioning itself as a SIEM offering. However, this gap in functionality effectively makes it impossible for me to use Elastic to track down the person that performed a specific action, which makes it substantially less useful as a SIEM.

Example AssumeRole event from aws.cloudtrail:

{
  "_id": "7UVlYnIBw0kYlt9zPYqS",
  "_index": "filebeat-7.6.2-2020.05.07-000001",
  "_score": 1,
  "_source": {
    "@timestamp": "2020-05-29T21:33:22.000Z",
    "agent": {
      "ephemeral_id": "0f5c9338-cf98-4145-8596-f87b6fd640f0",
      "hostname": "c23583bc9eac",
      "id": "87297c4e-bd47-4236-902d-5d4b0dfb5545",
      "type": "filebeat",
      "version": "7.6.2"
    },
    "aws": {
      "cloudtrail": {
        "event_type": "AwsApiCall",
        "event_version": "1.05",
        "recipient_account_id": "REDACTED",
        "request_parameters": "{durationSeconds=900, roleArn=arn:aws:iam::REDACTED:role/myrole, roleSessionName=1590788002139749100}",
        "response_elements": "{assumedRoleUser={arn=arn:aws:sts::REDACTED:assumed-role/myrole/1590788002139749100, assumedRoleId=AROACDRML13PHK3X7J1UL:1590788002139749100}, credentials={accessKeyId=ASIAREDACTED, sessionToken=REDACTED}}",
        "user_identity": {
          "access_key_id": "AKIAREDACTED",
          "arn": "arn:aws:iam::REDACTED:user/REDACTED@REDACTED",
          "type": "IAMUser"
        }
      },
      "s3": {
        "bucket": {
          "arn": "arn:aws:s3:::REDACTED-bucket",
          "name": "REDACTED-bucket"
        },
        "object.key": "AWSLogs/REDACTED/CloudTrail/us-east-1/2020/05/29/REDACTED_CloudTrail_us-east-1_20200529T2135Z_0p9TAh8rnLNQ28t0.json.gz"
      }
    },
    "cloud": {
      "account": {
        "id": "REDACTED"
      },
      "provider": "aws",
      "region": "ca-central-1"
    },
    "ecs": {
      "version": "1.4.0"
    },
    "event": {
      "action": "AssumeRole",
      "dataset": "aws.cloudtrail",
      "id": "fededed9-2b0a-4bbb-bd1b-40270e6a3a19",
      "kind": "event",
      "module": "aws",
      "original": "{\"awsRegion\":\"us-east-1\",\"eventID\":\"b4a0e081-42de-4118-b694-985167c867e4\",\"eventName\":\"AssumeRole\",\"eventSource\":\"sts.amazonaws.com\",\"eventTime\":\"2020-05-29T21:33:22Z\",\"eventType\":\"AwsApiCall\",\"eventVersion\":\"1.05\",\"recipientAccountId\":\"REDACTED\",\"requestID\":\"5c53445c-6665-4519-8ace-6c8f444c654a\",\"requestParameters\":{\"durationSeconds\":900,\"roleArn\":\"arn:aws:iam::REDACTED:role/myrole\",\"roleSessionName\":\"1590788002139749100\"},\"resources\":[{\"ARN\":\"arn:aws:iam::REDACTED:role/myrole\",\"accountId\":\"REDACTED\",\"type\":\"AWS::IAM::Role\"}],\"responseElements\":{\"assumedRoleUser\":{\"arn\":\"arn:aws:sts::REDACTED:assumed-role/myrole/1590788002139749100\",\"assumedRoleId\":\"AROACDRML13PHK3X7J1UL:1590788002139749100\"},\"credentials\":{\"accessKeyId\":\"ASIAREDACTED\",\"expiration\":\"May 29, 2020 9:48:22 PM\",\"sessionToken\":\"FwoGZXIvYXdzEK///////////wEaDM6q1RJMI+laZ3P+1yK3AaSVw/zm5hdBIsnYUgebG5oCISqrQJ+/X8rbqwuUj71MWgBf1vv9nDzv39QhMskyCdDCEsTaXKrpblVxmpOSPotfLYwSgYzY+PIiOnBpZPd7mVQchhNdwqAO8iANQS8Aly7ypUUEA59Wpp2AY+RiEVVMFYeXPpqWTquCUoToSwY/KlhuJeawVnTAeNNrfipaEBolBPLqo2CEe3Uq6WMBekcByWdfZM3bZC7qTaYhzdctCfsIg7+5Myii/8X2BTItWLnqFcxPBxORuDp/9l1/2kREqZmESvJtABso9VtPCJ0gH6oCig0g/65Iz6qk\"}},\"sourceIPAddress\":\"207.6.233.58\",\"userAgent\":\"aws-sdk-go/1.25.38 (go1.13.4; linux; amd64)\",\"userIdentity\":{\"accessKeyId\":\"AKIAREDACTED\",\"accountId\":\"REDACTED\",\"arn\":\"arn:aws:iam::REDACTED:user/REDACTED@REDACTED\",\"principalId\":\"AIDAREDACTED\",\"type\":\"IAMUser\",\"userName\":\"REDACTED@REDACTED\"}}",
      "outcome": "success",
      "provider": "sts.amazonaws.com",
      "type": "info"
    },
    "fileset": {
      "name": "cloudtrail"
    },
    "host": {
      "name": "c23583bc9eac"
    },
    "input": {
      "type": "s3"
    },
    "log": {
      "file.path": "https://REDACTED-bucket.s3-ca-central-1.amazonaws.com/AWSLogs/REDACTED/CloudTrail/us-east-1/2020/05/29/REDACTED_CloudTrail_us-east-1_20200529T2135Z_0p9TAh8rnLNQ28t0.json.gz",
      "offset": 1593
    },
    "service": {
      "type": "aws"
    },
    "source": {
      "address": "555.555.555.555",
      "geo": {
        "city_name": "Vancouver",
        "continent_name": "North America",
        "country_iso_code": "CA",
        "location": {
          "lat": 49.2824,
          "lon": -123.0399
        },
        "region_iso_code": "CA-BC",
        "region_name": "British Columbia"
      }
    },
    "user": {
      "id": "AIDAREDACTED",
      "name": "REDACTED@REDACTED"
    },
    "user_agent": {
      "device": {
        "name": "Other"
      },
      "name": "aws-sdk-go",
      "original": "aws-sdk-go/1.25.38 (go1.13.4; linux; amd64)",
      "version": "1.25.38"
    }
  },
  "_type": "_doc",
  "fields": {
    "@timestamp": [
      "2020-05-29T21:33:22.000Z"
    ],
    "suricata.eve.timestamp": [
      "2020-05-29T21:33:22.000Z"
    ]
  }
}

Metadata

Metadata

Assignees

Labels

FilebeatFilebeat

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions