Skip to content

Commit 9013d07

Browse files
author
kaiyan-sheng
authored
Add awsfargate module to collect container logs from Amazon ECS on Fargate (#25041)
1 parent 226485b commit 9013d07

23 files changed

Lines changed: 813 additions & 3 deletions

‎CHANGELOG.next.asciidoc‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -839,6 +839,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
839839
- Add `fail_on_template_error` option for httpjson input. {pull}24784[24784]
840840
- Change `okta.target` to `flattened` field type. {issue}24354[24354] {pull}24636[24636]
841841
- Added `http.request.id` to `nginx/ingress_controller` and `elasticsearch/audit`. {pull}24994[24994]
842+
- Add `awsfargate` module to collect container logs from Amazon ECS on Fargate. {pull}25041[25041]
842843
- New module `cyberarkpas` for CyberArk Privileged Access Security audit logs. {pull}24803[24803]
843844

844845
*Heartbeat*

‎filebeat/docs/fields.asciidoc‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@ grouped in the following categories:
1717
* <<exported-fields-auditd>>
1818
* <<exported-fields-aws>>
1919
* <<exported-fields-aws-cloudwatch>>
20+
* <<exported-fields-awsfargate>>
2021
* <<exported-fields-azure>>
2122
* <<exported-fields-barracuda>>
2223
* <<exported-fields-beat-common>>
@@ -2318,6 +2319,26 @@ type: keyword
23182319

23192320
--
23202321

2322+
[[exported-fields-awsfargate]]
2323+
== AWS Fargate fields
2324+
2325+
Module for collecting container logs from Amazon ECS Fargate.
2326+
2327+
2328+
2329+
[float]
2330+
=== awsfargate
2331+
2332+
Fields from Amazon ECS Fargate logs.
2333+
2334+
2335+
2336+
[float]
2337+
=== log
2338+
2339+
Fields for Amazon Fargate container logs.
2340+
2341+
23212342
[[exported-fields-azure]]
23222343
== Azure fields
23232344

‎filebeat/docs/modules/awsfargate.asciidoc‎

Lines changed: 139 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,139 @@
1+
////
2+
This file is generated! See scripts/docs_collector.py
3+
////
4+
5+
[[filebeat-module-awsfargate]]
6+
[role="xpack"]
7+
8+
:libbeat-xpack-dir: ../../../x-pack/libbeat
9+
10+
:modulename: awsfargate
11+
:has-dashboards: false
12+
13+
== AWS Fargate module
14+
15+
beta[]
16+
17+
This module can be used to collect container logs from Amazon ECS on Fargate.
18+
It uses filebeat `awscloudwatch` input to get log files from one or more log
19+
streams in AWS CloudWatch. Logs from all containers in Fargate launch type tasks
20+
can be sent to CloudWatch by adding the `awslogs` log driver under `logConfiguration`
21+
section in the task definition. For example, `logConfiguration` can be added into
22+
the task definition by adding this section into the `containerDefinitions`:
23+
24+
[source,json]
25+
----
26+
{
27+
"logDriver":"awslogs",
28+
"options":{
29+
"awslogs-group":"awslogs-wordpress",
30+
"awslogs-region":"us-west-2",
31+
"awslogs-stream-prefix":"awslogs-example"
32+
}
33+
}
34+
----
35+
36+
The `awsfargate` module requires AWS credentials configuration in order to make AWS API calls.
37+
Users can either use `access_key_id`, `secret_access_key` and/or
38+
`session_token`, or use `role_arn` AWS IAM role, or use shared AWS credentials file.
39+
40+
Please see <<awsfargate-credentials,AWS credentials options>> for more details.
41+
42+
[float]
43+
=== Module configuration
44+
45+
Example config:
46+
47+
[source,yaml]
48+
----
49+
- module: aws
50+
fargate:
51+
enabled: true
52+
var.credential_profile_name: test-filebeat
53+
var.log_group_arn: arn:aws:logs:us-east-1:1234567890:log-group:/ecs/test-log-group:*
54+
----
55+
56+
*`var.log_group_arn`*::
57+
58+
ARN of the log group to collect logs from.
59+
60+
*`var.log_group_name`*::
61+
62+
Name of the log group to collect logs from. Note: region_name is required when
63+
log_group_name is given.
64+
65+
*`var.region_name`*::
66+
67+
Region that the specified log group belongs to.
68+
69+
*`var.log_streams`*::
70+
71+
A list of strings of log streams names that Filebeat collect log events from.
72+
73+
*`var.log_stream_prefix`*::
74+
75+
A string to filter the results to include only log events from log streams
76+
that have names starting with this prefix.
77+
78+
*`var.start_position`*::
79+
80+
`start_position` allows user to specify if this input should read log files from
81+
the `beginning` or from the `end`.
82+
83+
* `beginning`: reads from the beginning of the log group (default).
84+
* `end`: read only new messages from current time minus `scan_frequency` going forward
85+
86+
*`var.scan_frequency`*::
87+
88+
This config parameter sets how often Filebeat checks for new log events from the
89+
specified log group. Default `scan_frequency` is 1 minute, which means Filebeat
90+
will sleep for 1 minute before querying for new logs again.
91+
92+
*`var.api_timeout`*::
93+
94+
The maximum duration of AWS API can take. If it exceeds the timeout, AWS API
95+
will be interrupted. The default AWS API timeout for a message is 120 seconds.
96+
The minimum is 0 seconds.
97+
98+
*`var.api_sleep`*::
99+
100+
This is used to sleep between AWS `FilterLogEvents` API calls inside the same
101+
collection period. `FilterLogEvents` API has a quota of 5 transactions per
102+
second (TPS)/account/Region. By default, `api_sleep` is 200 ms. This value should
103+
only be adjusted when there are multiple Filebeats or multiple Filebeat inputs
104+
collecting logs from the same region and AWS account.
105+
106+
*`var.shared_credential_file`*::
107+
108+
Filename of AWS credential file.
109+
110+
*`var.credential_profile_name`*::
111+
112+
AWS credential profile name.
113+
114+
*`var.access_key_id`*::
115+
First part of access key.
116+
117+
*`var.secret_access_key`*::
118+
Second part of access key.
119+
120+
*`var.session_token`*::
121+
Required when using temporary security credentials.
122+
123+
*`var.role_arn`*::
124+
AWS IAM Role to assume.
125+
126+
*`var.endpoint`*::
127+
128+
The custom endpoint used to access AWS APIs.
129+
130+
[id="awsfargate-credentials"]
131+
include::{libbeat-xpack-dir}/docs/aws-credentials-config.asciidoc[]
132+
133+
134+
[float]
135+
=== Fields
136+
137+
For a description of each field in the module, see the
138+
<<exported-fields-awsfargate,exported fields>> section.
139+

‎filebeat/docs/modules_list.asciidoc‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@ This file is generated! See scripts/docs_collector.py
77
* <<filebeat-module-apache>>
88
* <<filebeat-module-auditd>>
99
* <<filebeat-module-aws>>
10+
* <<filebeat-module-awsfargate>>
1011
* <<filebeat-module-azure>>
1112
* <<filebeat-module-barracuda>>
1213
* <<filebeat-module-bluecoat>>
@@ -80,6 +81,7 @@ include::modules/activemq.asciidoc[]
8081
include::modules/apache.asciidoc[]
8182
include::modules/auditd.asciidoc[]
8283
include::modules/aws.asciidoc[]
84+
include::modules/awsfargate.asciidoc[]
8385
include::modules/azure.asciidoc[]
8486
include::modules/barracuda.asciidoc[]
8587
include::modules/bluecoat.asciidoc[]

‎filebeat/tests/system/test_modules.py‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -274,7 +274,8 @@ def clean_keys(obj):
274274
"threatintel.abuseurl",
275275
"threatintel.abusemalware",
276276
"threatintel.anomali",
277-
"snyk.vulnerabilities"
277+
"snyk.vulnerabilities",
278+
"awsfargate.log"
278279
}
279280
# dataset + log file pairs for which @timestamp is kept as an exception from above
280281
remove_timestamp_exception = {

‎x-pack/filebeat/docs/inputs/input-aws-cloudwatch.asciidoc‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -43,10 +43,12 @@ The `aws-cloudwatch` input supports the following configuration options plus the
4343
==== `log_group_arn`
4444
ARN of the log group to collect logs from.
4545

46+
[float]
4647
==== `log_group_name`
4748
Name of the log group to collect logs from. Note: region_name is required when
4849
log_group_name is given.
4950

51+
[float]
5052
==== `region_name`
5153
Region that the specified log group belongs to.
5254

‎x-pack/filebeat/filebeat.reference.yml‎

Lines changed: 115 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -348,6 +348,121 @@ filebeat.modules:
348348
# The maximum number of messages to return from SQS. Valid values: 1 to 10.
349349
#var.max_number_of_messages: 5
350350

351+
- module: aws
352+
fargate:
353+
enabled: false
354+
355+
# Filename of AWS credential file
356+
# If not set "$HOME/.aws/credentials" is used on Linux/Mac
357+
# "%UserProfile%\.aws\credentials" is used on Windows
358+
#var.shared_credential_file: /etc/filebeat/aws_credentials
359+
360+
# Profile name for aws credential
361+
# If not set the default profile is used
362+
#var.credential_profile_name: fb-aws
363+
364+
# Use access_key_id, secret_access_key and/or session_token instead of shared credential file
365+
#var.access_key_id: access_key_id
366+
#var.secret_access_key: secret_access_key
367+
#var.session_token: session_token
368+
369+
# The duration that the received messages are hidden from ReceiveMessage request
370+
# Default to be 300s
371+
#var.visibility_timeout: 300s
372+
373+
# Maximum duration before AWS API request will be interrupted
374+
# Default to be 120s
375+
#var.api_timeout: 120s
376+
377+
# Custom endpoint used to access AWS APIs
378+
#var.endpoint: amazonaws.com
379+
380+
# AWS IAM Role to assume
381+
#var.role_arn: arn:aws:iam::123456789012:role/test-mb
382+
383+
# ARN of the log group to collect logs from
384+
#var.log_group_arn: arn:aws:logs:us-east-1:123456789012:log-group:test:*
385+
386+
# Name of the log group to collect logs from
387+
#var.log_group_name: /ecs/test-log-group
388+
389+
# Region that the specified log group belongs to
390+
#var.region_name: us-east-1
391+
392+
# A list of strings of log streams names that Filebeat collect log events from
393+
#var.log_streams: ["/ecs/test-log-group/test-log-stream"]
394+
395+
# A string to filter the results to include only log events from log streams that have names starting with this prefix
396+
#var.log_stream_prefix: /ecs/test-log-group/
397+
398+
# Specify if Filebeat should read log files from the beginning or from the end
399+
# Default start_position is beginning
400+
#var.start_position: beginning
401+
402+
# How often Filebeat checks for new log events from the specified log group
403+
# Default scan_frequency is 1 minute
404+
#var.scan_frequency: 1m
405+
406+
# Time used to sleep between AWS FilterLogEvents API calls inside the same collection period
407+
# Default api_sleep is 200 ms
408+
#var.api_sleep: 200ms
409+
410+
#----------------------------- AWS Fargate Module -----------------------------
411+
- module: awsfargate
412+
log:
413+
enabled: false
414+
415+
# Filename of AWS credential file
416+
# If not set "$HOME/.aws/credentials" is used on Linux/Mac
417+
# "%UserProfile%\.aws\credentials" is used on Windows
418+
#var.shared_credential_file: /etc/filebeat/aws_credentials
419+
420+
# Profile name for aws credential
421+
# If not set the default profile is used
422+
#var.credential_profile_name: fb-aws
423+
424+
# Use access_key_id, secret_access_key and/or session_token instead of shared credential file
425+
#var.access_key_id: access_key_id
426+
#var.secret_access_key: secret_access_key
427+
#var.session_token: session_token
428+
429+
# Maximum duration before AWS API request will be interrupted
430+
# Default to be 120s
431+
#var.api_timeout: 120s
432+
433+
# Custom endpoint used to access AWS APIs
434+
#var.endpoint: amazonaws.com
435+
436+
# AWS IAM Role to assume
437+
#var.role_arn: arn:aws:iam::123456789012:role/test-mb
438+
439+
# ARN of the log group to collect logs from
440+
#var.log_group_arn: arn:aws:logs:us-east-1:123456789012:log-group:test:*
441+
442+
# Name of the log group to collect logs from
443+
#var.log_group_name: /ecs/test-log-group
444+
445+
# Region that the specified log group belongs to
446+
#var.region_name: us-east-1
447+
448+
# A list of strings of log streams names that Filebeat collect log events from
449+
#var.log_streams: ["/ecs/test-log-group/test-log-stream"]
450+
451+
# A string to filter the results to include only log events from log streams that have names starting with this prefix
452+
#var.log_stream_prefix: /ecs/test-log-group/
453+
454+
# Specify if Filebeat should read log files from the beginning or from the end
455+
# Default start_position is beginning
456+
#var.start_position: beginning
457+
458+
# How often Filebeat checks for new log events from the specified log group
459+
# Default scan_frequency is 1 minute
460+
#var.scan_frequency: 1m
461+
462+
# Time used to sleep between AWS FilterLogEvents API calls inside the same collection period
463+
# Default api_sleep is 200 ms
464+
#var.api_sleep: 200ms
465+
351466
#-------------------------------- Azure Module --------------------------------
352467
- module: azure
353468
# All logs

‎x-pack/filebeat/include/list.go‎

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)