User uses Elastic Agent (v7.11.2) with the AWS integration (v0.5.0) to process an AWS SQS of CloudTrail logs.
Their S3 source of the logs also include CloudTrail-Digest & Config logs. Since CloudTrail-Digest logs do not have key ‘Records’, there are a lot of following error messages. Filebeat has an option file_selectors to limit the files that are downloaded. For elastic agent, can we have this option to exclude CloudTrail-Digest logs?
{"log.level":"error","@timestamp":"2021-05-25T07:21:52.770+1000","log.logger":"input.s3","log.origin":{"file.name":"s3/collector.go","file.line":304},"message":"createEventsFromS3Info failed processing file from s3 bucket \"aws-controltower-logs\" with name \"x-xxxxxxxxxx/AWSLogs/999999999999/CloudTrail-Digest/ap-northeast-1/2021/05/21/999999999999_CloudTrail-Digest_ap-northeast-1_aws-controltower-BaselineCloudTrail_ap-northeast-2_20210521T064109Z.json.gz\": key 'Records' not found","queue_url":"https://******.amazonaws.com/999999999999/CloudTrail_2_Elastic","region":"ap-northeast-2","ecs.version":"1.6.0"}
User uses Elastic Agent (v7.11.2) with the AWS integration (v0.5.0) to process an AWS SQS of CloudTrail logs.
Their S3 source of the logs also include CloudTrail-Digest & Config logs. Since CloudTrail-Digest logs do not have key ‘Records’, there are a lot of following error messages. Filebeat has an option file_selectors to limit the files that are downloaded. For elastic agent, can we have this option to exclude CloudTrail-Digest logs?