Skip to content

Commit 766cd12

Browse files
authored
Merge branch '7.x' into mergify/bp/7.x/pr-24803
2 parents 3b920f6 + f235d33 commit 766cd12

39 files changed

Lines changed: 1215 additions & 27 deletions

CHANGELOG.next.asciidoc

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -371,6 +371,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
371371
- Add new ECS 1.9 field `cloud.service.name` to `add_cloud_metadata` processor. {pull}24993[24993]
372372
- Libbeat: report queue capacity, output batch size, and output client count to monitoring. {pull}24700[24700]
373373
- Add kubernetes.pod.ip field in kubernetes metadata. {pull}25037[25037]
374+
- Discover changes in Kubernetes namespace metadata as soon as they happen. {pull}25117[25117]
374375

375376
*Auditbeat*
376377

@@ -539,6 +540,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
539540
- Add `fail_on_template_error` option for httpjson input. {pull}24784[24784]
540541
- Change `okta.target` to `flattened` field type. {issue}24354[24354] {pull}24636[24636]
541542
- Added `http.request.id` to `nginx/ingress_controller` and `elasticsearch/audit`. {pull}24994[24994]
543+
- Add `awsfargate` module to collect container logs from Amazon ECS on Fargate. {pull}25041[25041]
542544
- New module `cyberarkpas` for CyberArk Privileged Access Security audit logs. {pull}24803[24803]
543545

544546
*Heartbeat*

deploy/kubernetes/elastic-agent-standalone-kubernetes.yaml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -561,6 +561,7 @@ rules:
561561
- namespaces
562562
- events
563563
- pods
564+
- services
564565
verbs: ["get", "list", "watch"]
565566
# Enable this rule only if planing to use kubernetes_secrets provider
566567
#- apiGroups: [""]

deploy/kubernetes/elastic-agent-standalone/elastic-agent-standalone-role.yaml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@ rules:
1111
- namespaces
1212
- events
1313
- pods
14+
- services
1415
verbs: ["get", "list", "watch"]
1516
# Enable this rule only if planing to use kubernetes_secrets provider
1617
#- apiGroups: [""]

filebeat/docs/fields.asciidoc

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@ grouped in the following categories:
1717
* <<exported-fields-auditd>>
1818
* <<exported-fields-aws>>
1919
* <<exported-fields-aws-cloudwatch>>
20+
* <<exported-fields-awsfargate>>
2021
* <<exported-fields-azure>>
2122
* <<exported-fields-barracuda>>
2223
* <<exported-fields-beat-common>>
@@ -2318,6 +2319,26 @@ type: keyword
23182319

23192320
--
23202321

2322+
[[exported-fields-awsfargate]]
2323+
== AWS Fargate fields
2324+
2325+
Module for collecting container logs from Amazon ECS Fargate.
2326+
2327+
2328+
2329+
[float]
2330+
=== awsfargate
2331+
2332+
Fields from Amazon ECS Fargate logs.
2333+
2334+
2335+
2336+
[float]
2337+
=== log
2338+
2339+
Fields for Amazon Fargate container logs.
2340+
2341+
23212342
[[exported-fields-azure]]
23222343
== Azure fields
23232344

Lines changed: 139 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,139 @@
1+
////
2+
This file is generated! See scripts/docs_collector.py
3+
////
4+
5+
[[filebeat-module-awsfargate]]
6+
[role="xpack"]
7+
8+
:libbeat-xpack-dir: ../../../x-pack/libbeat
9+
10+
:modulename: awsfargate
11+
:has-dashboards: false
12+
13+
== AWS Fargate module
14+
15+
beta[]
16+
17+
This module can be used to collect container logs from Amazon ECS on Fargate.
18+
It uses filebeat `awscloudwatch` input to get log files from one or more log
19+
streams in AWS CloudWatch. Logs from all containers in Fargate launch type tasks
20+
can be sent to CloudWatch by adding the `awslogs` log driver under `logConfiguration`
21+
section in the task definition. For example, `logConfiguration` can be added into
22+
the task definition by adding this section into the `containerDefinitions`:
23+
24+
[source,json]
25+
----
26+
{
27+
"logDriver":"awslogs",
28+
"options":{
29+
"awslogs-group":"awslogs-wordpress",
30+
"awslogs-region":"us-west-2",
31+
"awslogs-stream-prefix":"awslogs-example"
32+
}
33+
}
34+
----
35+
36+
The `awsfargate` module requires AWS credentials configuration in order to make AWS API calls.
37+
Users can either use `access_key_id`, `secret_access_key` and/or
38+
`session_token`, or use `role_arn` AWS IAM role, or use shared AWS credentials file.
39+
40+
Please see <<awsfargate-credentials,AWS credentials options>> for more details.
41+
42+
[float]
43+
=== Module configuration
44+
45+
Example config:
46+
47+
[source,yaml]
48+
----
49+
- module: aws
50+
fargate:
51+
enabled: true
52+
var.credential_profile_name: test-filebeat
53+
var.log_group_arn: arn:aws:logs:us-east-1:1234567890:log-group:/ecs/test-log-group:*
54+
----
55+
56+
*`var.log_group_arn`*::
57+
58+
ARN of the log group to collect logs from.
59+
60+
*`var.log_group_name`*::
61+
62+
Name of the log group to collect logs from. Note: region_name is required when
63+
log_group_name is given.
64+
65+
*`var.region_name`*::
66+
67+
Region that the specified log group belongs to.
68+
69+
*`var.log_streams`*::
70+
71+
A list of strings of log streams names that Filebeat collect log events from.
72+
73+
*`var.log_stream_prefix`*::
74+
75+
A string to filter the results to include only log events from log streams
76+
that have names starting with this prefix.
77+
78+
*`var.start_position`*::
79+
80+
`start_position` allows user to specify if this input should read log files from
81+
the `beginning` or from the `end`.
82+
83+
* `beginning`: reads from the beginning of the log group (default).
84+
* `end`: read only new messages from current time minus `scan_frequency` going forward
85+
86+
*`var.scan_frequency`*::
87+
88+
This config parameter sets how often Filebeat checks for new log events from the
89+
specified log group. Default `scan_frequency` is 1 minute, which means Filebeat
90+
will sleep for 1 minute before querying for new logs again.
91+
92+
*`var.api_timeout`*::
93+
94+
The maximum duration of AWS API can take. If it exceeds the timeout, AWS API
95+
will be interrupted. The default AWS API timeout for a message is 120 seconds.
96+
The minimum is 0 seconds.
97+
98+
*`var.api_sleep`*::
99+
100+
This is used to sleep between AWS `FilterLogEvents` API calls inside the same
101+
collection period. `FilterLogEvents` API has a quota of 5 transactions per
102+
second (TPS)/account/Region. By default, `api_sleep` is 200 ms. This value should
103+
only be adjusted when there are multiple Filebeats or multiple Filebeat inputs
104+
collecting logs from the same region and AWS account.
105+
106+
*`var.shared_credential_file`*::
107+
108+
Filename of AWS credential file.
109+
110+
*`var.credential_profile_name`*::
111+
112+
AWS credential profile name.
113+
114+
*`var.access_key_id`*::
115+
First part of access key.
116+
117+
*`var.secret_access_key`*::
118+
Second part of access key.
119+
120+
*`var.session_token`*::
121+
Required when using temporary security credentials.
122+
123+
*`var.role_arn`*::
124+
AWS IAM Role to assume.
125+
126+
*`var.endpoint`*::
127+
128+
The custom endpoint used to access AWS APIs.
129+
130+
[id="awsfargate-credentials"]
131+
include::{libbeat-xpack-dir}/docs/aws-credentials-config.asciidoc[]
132+
133+
134+
[float]
135+
=== Fields
136+
137+
For a description of each field in the module, see the
138+
<<exported-fields-awsfargate,exported fields>> section.
139+

filebeat/docs/modules_list.asciidoc

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@ This file is generated! See scripts/docs_collector.py
77
* <<filebeat-module-apache>>
88
* <<filebeat-module-auditd>>
99
* <<filebeat-module-aws>>
10+
* <<filebeat-module-awsfargate>>
1011
* <<filebeat-module-azure>>
1112
* <<filebeat-module-barracuda>>
1213
* <<filebeat-module-bluecoat>>
@@ -80,6 +81,7 @@ include::modules/activemq.asciidoc[]
8081
include::modules/apache.asciidoc[]
8182
include::modules/auditd.asciidoc[]
8283
include::modules/aws.asciidoc[]
84+
include::modules/awsfargate.asciidoc[]
8385
include::modules/azure.asciidoc[]
8486
include::modules/barracuda.asciidoc[]
8587
include::modules/bluecoat.asciidoc[]

filebeat/tests/system/test_modules.py

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -282,7 +282,8 @@ def clean_keys(obj):
282282
"threatintel.abuseurl",
283283
"threatintel.abusemalware",
284284
"threatintel.anomali",
285-
"snyk.vulnerabilities"
285+
"snyk.vulnerabilities",
286+
"awsfargate.log"
286287
}
287288
# dataset + log file pairs for which @timestamp is kept as an exception from above
288289
remove_timestamp_exception = {

libbeat/autodiscover/providers/kubernetes/pod.go

Lines changed: 82 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -19,11 +19,11 @@ package kubernetes
1919

2020
import (
2121
"fmt"
22+
"sync"
2223
"time"
2324

2425
"github.com/gofrs/uuid"
2526
k8s "k8s.io/client-go/kubernetes"
26-
"k8s.io/client-go/tools/cache"
2727

2828
"github.com/elastic/beats/v7/libbeat/autodiscover/builder"
2929
"github.com/elastic/beats/v7/libbeat/common"
@@ -43,7 +43,11 @@ type pod struct {
4343
watcher kubernetes.Watcher
4444
nodeWatcher kubernetes.Watcher
4545
namespaceWatcher kubernetes.Watcher
46-
namespaceStore cache.Store
46+
47+
// Mutex used by configuration updates not triggered by the main watcher,
48+
// to avoid race conditions between cross updates and deletions.
49+
// Other updaters must use a write lock.
50+
crossUpdate sync.RWMutex
4751
}
4852

4953
// NewPodEventer creates an eventer that can discover and process pod objects
@@ -111,11 +115,20 @@ func NewPodEventer(uuid uuid.UUID, cfg *common.Config, client k8s.Interface, pub
111115
}
112116

113117
watcher.AddEventHandler(p)
118+
119+
if namespaceWatcher != nil && (config.Hints.Enabled() || metaConf.Namespace.Enabled()) {
120+
updater := newNamespacePodUpdater(p.unlockedUpdate, watcher.Store(), &p.crossUpdate)
121+
namespaceWatcher.AddEventHandler(updater)
122+
}
123+
114124
return p, nil
115125
}
116126

117127
// OnAdd ensures processing of pod objects that are newly added
118128
func (p *pod) OnAdd(obj interface{}) {
129+
p.crossUpdate.RLock()
130+
defer p.crossUpdate.RUnlock()
131+
119132
p.logger.Debugf("Watcher Pod add: %+v", obj)
120133
p.emit(obj.(*kubernetes.Pod), "start")
121134
}
@@ -124,6 +137,13 @@ func (p *pod) OnAdd(obj interface{}) {
124137
// if it is terminating, a stop event is scheduled, if not, a stop and a start
125138
// events are sent sequentially to recreate the resources assotiated to the pod.
126139
func (p *pod) OnUpdate(obj interface{}) {
140+
p.crossUpdate.RLock()
141+
defer p.crossUpdate.RUnlock()
142+
143+
p.unlockedUpdate(obj)
144+
}
145+
146+
func (p *pod) unlockedUpdate(obj interface{}) {
127147
pod := obj.(*kubernetes.Pod)
128148

129149
p.logger.Debugf("Watcher Pod update for pod: %+v, status: %+v", pod.Name, pod.Status.Phase)
@@ -162,6 +182,9 @@ func (p *pod) OnUpdate(obj interface{}) {
162182

163183
// OnDelete stops pod objects that are deleted
164184
func (p *pod) OnDelete(obj interface{}) {
185+
p.crossUpdate.RLock()
186+
defer p.crossUpdate.RUnlock()
187+
165188
p.logger.Debugf("Watcher Pod delete: %+v", obj)
166189
time.AfterFunc(p.config.CleanupTimeout, func() { p.emit(obj.(*kubernetes.Pod), "stop") })
167190
}
@@ -448,3 +471,60 @@ func (p *pod) emitEvents(pod *kubernetes.Pod, flag string, containers []kubernet
448471
p.publish(events)
449472
}
450473
}
474+
475+
// podUpdaterHandlerFunc is a function that handles pod updater notifications.
476+
type podUpdaterHandlerFunc func(interface{})
477+
478+
// podUpdaterStore is the interface that an object needs to implement to be
479+
// used as a pod updater store.
480+
type podUpdaterStore interface {
481+
List() []interface{}
482+
}
483+
484+
// namespacePodUpdater notifies updates on pods when their namespaces are updated.
485+
type namespacePodUpdater struct {
486+
handler podUpdaterHandlerFunc
487+
store podUpdaterStore
488+
locker sync.Locker
489+
}
490+
491+
// newNamespacePodUpdater creates a namespacePodUpdater
492+
func newNamespacePodUpdater(handler podUpdaterHandlerFunc, store podUpdaterStore, locker sync.Locker) *namespacePodUpdater {
493+
return &namespacePodUpdater{
494+
handler: handler,
495+
store: store,
496+
locker: locker,
497+
}
498+
}
499+
500+
// OnUpdate handles update events on namespaces.
501+
func (n *namespacePodUpdater) OnUpdate(obj interface{}) {
502+
ns, ok := obj.(*kubernetes.Namespace)
503+
if !ok {
504+
return
505+
}
506+
507+
// n.store.List() returns a snapshot at this point. If a delete is received
508+
// from the main watcher, this loop may generate an update event after the
509+
// delete is processed, leaving configurations that would never be deleted.
510+
// Also this loop can miss updates, what could leave outdated configurations.
511+
// Avoid these issues by locking the processing of events from the main watcher.
512+
if n.locker != nil {
513+
n.locker.Lock()
514+
defer n.locker.Unlock()
515+
}
516+
for _, pod := range n.store.List() {
517+
pod, ok := pod.(*kubernetes.Pod)
518+
if ok && pod.Namespace == ns.Name {
519+
n.handler(pod)
520+
}
521+
}
522+
}
523+
524+
// OnAdd handles add events on namespaces. Nothing to do, if pods are added to this
525+
// namespace they will generate their own add events.
526+
func (*namespacePodUpdater) OnAdd(interface{}) {}
527+
528+
// OnDelete handles delete events on namespaces. Nothing to do, if pods are deleted from this
529+
// namespace they will generate their own delete events.
530+
func (*namespacePodUpdater) OnDelete(interface{}) {}

0 commit comments

Comments
 (0)