Skip to content

Commit 4194408

Browse files
earcurimarc-grandrewkroh
authored
[Filebeat] Add Pensando DFW Module (#21063)
* Add Pensando module init * explicitly define the ECS version per testing * updates to docs from make update * updates for pensando module * updates to documentation and db screenshot * add dashboard export to repo * update to add pensando beat * Update filebeat/module/pensando/dfw/config/dfw.yml Co-authored-by: Marc Guasch <marc-gr@users.noreply.github.com> * Update pipeline.yml Condensed all "remove" fields to 1 list of fields. * Update pipeline.yml Do not remove the payload_raw field. * Update filebeat/module/pensando/_meta/docs.asciidoc Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co> * Update config.yml Added syslog_host and syslog_port values as suggested. * Update docs.asciidoc Added documentation for syslog_host and syslog_port as suggested. * Update pipeline.yml Removing payload_raw - this and json are, essentially, the same field and no longer needed after parsing. * Update pipeline.yml Changed checks if values are != null to use the filebeat specific ignore_empty_value: true instead. * Remove set of event.module Remove the set param for event.module. Filebeat should add this automatically. * Apply suggestions from code review Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co> * Update test.log * Use convert instead of set for some fields Changed ECS sets for IP addresses and ports to converts of type ip and integer respectively. * Updates for geoip and autonomous system * add pensando dfw fields * fixes from make -C filebeat update * fixes for filebeat check * make update changes * Update filebeat/module/pensando/dfw/config/dfw.yml Co-authored-by: Marc Guasch <marc-gr@users.noreply.github.com> * Update filebeat/module/pensando/dfw/ingest/pipeline.yml Co-authored-by: Marc Guasch <marc-gr@users.noreply.github.com> * Update filebeat/module/pensando/dfw/ingest/pipeline.yml Co-authored-by: Marc Guasch <marc-gr@users.noreply.github.com> * Update filebeat/module/pensando/dfw/ingest/pipeline.yml Co-authored-by: Marc Guasch <marc-gr@users.noreply.github.com> * Update filebeat/module/pensando/dfw/ingest/pipeline.yml Co-authored-by: Marc Guasch <marc-gr@users.noreply.github.com> * remove old json file * ran tests * Update filebeat/module/pensando/dfw/ingest/pipeline.yml Co-authored-by: Marc Guasch <marc-gr@users.noreply.github.com> * gen after run of 'mage -v pythonIntegTest' * Update fields.yml * mage fmt update request Co-authored-by: Marc Guasch <marc-gr@users.noreply.github.com> Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>
1 parent 9dbfd44 commit 4194408

21 files changed

Lines changed: 2152 additions & 0 deletions

‎CHANGELOG-developer.next.asciidoc‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -104,6 +104,7 @@ The list below covers the major changes between 7.0.0-rc2 and master only.
104104
- Update Go version to 1.14.7. {pull}20508[20508]
105105
- Add packaging for docker image based on UBI minimal 8. {pull}20576[20576]
106106
- Make the mage binary used by the build process in the docker container to be statically compiled. {pull}20827[20827]
107+
- Add Pensando distributed firewall module. {pull}21063[21063]
107108
- Update ecszap to v0.3.0 for using ECS 1.6.0 in logs {pull}22267[22267]
108109
- Add support for customized monitoring API. {pull}22605[22605]
109110
- Update Go version to 1.15.7. {pull}22495[22495]

‎filebeat/docs/fields.asciidoc‎

Lines changed: 142 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -69,6 +69,7 @@ grouped in the following categories:
6969
* <<exported-fields-oracle>>
7070
* <<exported-fields-osquery>>
7171
* <<exported-fields-panw>>
72+
* <<exported-fields-pensando>>
7273
* <<exported-fields-postgresql>>
7374
* <<exported-fields-process>>
7475
* <<exported-fields-proofpoint>>
@@ -105827,6 +105828,147 @@ Specifies the sub type of the log
105827105828

105828105829
--
105829105830

105831+
[[exported-fields-pensando]]
105832+
== Pensando fields
105833+
105834+
pensando Module
105835+
105836+
105837+
105838+
[float]
105839+
=== pensando
105840+
105841+
Fields from Pensando logs.
105842+
105843+
105844+
105845+
[float]
105846+
=== dfw
105847+
105848+
Fields for Pensando DFW
105849+
105850+
105851+
105852+
*`pensando.dfw.action`*::
105853+
+
105854+
--
105855+
Action on the flow.
105856+
105857+
105858+
type: keyword
105859+
105860+
--
105861+
105862+
*`pensando.dfw.app_id`*::
105863+
+
105864+
--
105865+
Application ID
105866+
105867+
105868+
type: integer
105869+
105870+
--
105871+
105872+
*`pensando.dfw.destination_address`*::
105873+
+
105874+
--
105875+
Address of destination.
105876+
105877+
105878+
type: keyword
105879+
105880+
--
105881+
105882+
*`pensando.dfw.destination_port`*::
105883+
+
105884+
--
105885+
Port of destination.
105886+
105887+
105888+
type: integer
105889+
105890+
--
105891+
105892+
*`pensando.dfw.direction`*::
105893+
+
105894+
--
105895+
Direction of the flow
105896+
105897+
105898+
type: keyword
105899+
105900+
--
105901+
105902+
*`pensando.dfw.protocol`*::
105903+
+
105904+
--
105905+
Protocol of the flow
105906+
105907+
105908+
type: keyword
105909+
105910+
--
105911+
105912+
*`pensando.dfw.rule_id`*::
105913+
+
105914+
--
105915+
Rule ID that was matched.
105916+
105917+
105918+
type: keyword
105919+
105920+
--
105921+
105922+
*`pensando.dfw.session_id`*::
105923+
+
105924+
--
105925+
Session ID of the flow
105926+
105927+
105928+
type: integer
105929+
105930+
--
105931+
105932+
*`pensando.dfw.session_state`*::
105933+
+
105934+
--
105935+
Session state of the flow.
105936+
105937+
105938+
type: keyword
105939+
105940+
--
105941+
105942+
*`pensando.dfw.source_address`*::
105943+
+
105944+
--
105945+
Source address of the flow.
105946+
105947+
105948+
type: keyword
105949+
105950+
--
105951+
105952+
*`pensando.dfw.source_port`*::
105953+
+
105954+
--
105955+
Source port of the flow.
105956+
105957+
105958+
type: integer
105959+
105960+
--
105961+
105962+
*`pensando.dfw.timestamp`*::
105963+
+
105964+
--
105965+
Timestamp of the log.
105966+
105967+
105968+
type: date
105969+
105970+
--
105971+
105830105972
[[exported-fields-postgresql]]
105831105973
== PostgreSQL fields
105832105974

‎filebeat/docs/images/filebeat-pensando-dfw.png‎

170 KB
Loading

‎filebeat/docs/modules/pensando.asciidoc‎

Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
////
2+
This file is generated! See scripts/docs_collector.py
3+
////
4+
5+
[[filebeat-module-pensando]]
6+
:modulename: pensando
7+
:has-dashboards: true
8+
9+
== pensando module
10+
11+
The +{modulename}+ module parses distributed firewall logs created by the
12+
http://pensando.io/[Pensando] distributed services card (DSC).
13+
14+
15+
include::../include/what-happens.asciidoc[]
16+
17+
include::../include/gs-link.asciidoc[]
18+
19+
[float]
20+
=== Compatibility
21+
22+
The Pensando module has been tested with 1.12.0-E-54 and later.
23+
24+
include::../include/configuring-intro.asciidoc[]
25+
The following example shows how to set parameters in the +modules.d/{modulename}.yml+
26+
file to listen for firewall logs sent from the Pensando DSC(s) on port 5514 (default is 9001):
27+
28+
["source","yaml",subs="attributes"]
29+
-----
30+
- module: pensando
31+
access:
32+
enabled: true
33+
var.syslog_host: 0.0.0.0
34+
var.syslog_port: [9001]
35+
-----
36+
:fileset_ex: dfw
37+
38+
include::../include/config-option-intro.asciidoc[]
39+
40+
TODO: document the variables from each fileset. If you're describing a variable
41+
that's common to other modules, you can reuse shared descriptions by including
42+
the relevant file. For example:
43+
44+
[float]
45+
==== `dfw` log fileset settings
46+
47+
include::../include/var-paths.asciidoc[]
48+
49+
[float]
50+
=== Example dashboard
51+
52+
This module comes with a sample dashboard. For example:
53+
54+
[role="screenshot"]
55+
image::./images/filebeat-pensando-dfw.png[]
56+
57+
:has-dashboards!:
58+
59+
:fileset_ex!:
60+
61+
:modulename!:
62+
63+
64+
[float]
65+
=== Fields
66+
67+
For a description of each field in the module, see the
68+
<<exported-fields-pensando,exported fields>> section.
69+

‎filebeat/docs/modules_list.asciidoc‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -50,6 +50,7 @@ This file is generated! See scripts/docs_collector.py
5050
* <<filebeat-module-oracle>>
5151
* <<filebeat-module-osquery>>
5252
* <<filebeat-module-panw>>
53+
* <<filebeat-module-pensando>>
5354
* <<filebeat-module-postgresql>>
5455
* <<filebeat-module-proofpoint>>
5556
* <<filebeat-module-rabbitmq>>
@@ -121,6 +122,7 @@ include::modules/okta.asciidoc[]
121122
include::modules/oracle.asciidoc[]
122123
include::modules/osquery.asciidoc[]
123124
include::modules/panw.asciidoc[]
125+
include::modules/pensando.asciidoc[]
124126
include::modules/postgresql.asciidoc[]
125127
include::modules/proofpoint.asciidoc[]
126128
include::modules/rabbitmq.asciidoc[]

‎filebeat/filebeat.reference.yml‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -335,6 +335,18 @@ filebeat.modules:
335335
# of the document. The default is true.
336336
#var.use_namespace: true
337337

338+
#------------------------------- Pensando Module -------------------------------
339+
- module: pensando
340+
# Firewall logs
341+
dfw:
342+
enabled: true
343+
var.syslog_host: 0.0.0.0
344+
var.syslog_port: 9001
345+
346+
# Set custom paths for the log files. If left empty,
347+
# Filebeat will choose the paths depending on your OS.
348+
# var.paths:
349+
338350
#------------------------------ PostgreSQL Module ------------------------------
339351
#- module: postgresql
340352
# Logs

‎filebeat/include/list.go‎

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎filebeat/module/pensando/_meta/config.yml‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
- module: pensando
2+
# Firewall logs
3+
dfw:
4+
enabled: true
5+
var.syslog_host: 0.0.0.0
6+
var.syslog_port: 9001
7+
8+
# Set custom paths for the log files. If left empty,
9+
# Filebeat will choose the paths depending on your OS.
10+
# var.paths:

‎filebeat/module/pensando/_meta/docs.asciidoc‎

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
:modulename: pensando
2+
:has-dashboards: true
3+
4+
== pensando module
5+
6+
The +{modulename}+ module parses distributed firewall logs created by the
7+
http://pensando.io/[Pensando] distributed services card (DSC).
8+
9+
10+
include::../include/what-happens.asciidoc[]
11+
12+
include::../include/gs-link.asciidoc[]
13+
14+
[float]
15+
=== Compatibility
16+
17+
The Pensando module has been tested with 1.12.0-E-54 and later.
18+
19+
include::../include/configuring-intro.asciidoc[]
20+
The following example shows how to set parameters in the +modules.d/{modulename}.yml+
21+
file to listen for firewall logs sent from the Pensando DSC(s) on port 5514 (default is 9001):
22+
23+
["source","yaml",subs="attributes"]
24+
-----
25+
- module: pensando
26+
access:
27+
enabled: true
28+
var.syslog_host: 0.0.0.0
29+
var.syslog_port: [9001]
30+
-----
31+
:fileset_ex: dfw
32+
33+
include::../include/config-option-intro.asciidoc[]
34+
35+
TODO: document the variables from each fileset. If you're describing a variable
36+
that's common to other modules, you can reuse shared descriptions by including
37+
the relevant file. For example:
38+
39+
[float]
40+
==== `dfw` log fileset settings
41+
42+
include::../include/var-paths.asciidoc[]
43+
44+
[float]
45+
=== Example dashboard
46+
47+
This module comes with a sample dashboard. For example:
48+
49+
[role="screenshot"]
50+
image::./images/filebeat-pensando-dfw.png[]
51+
52+
:has-dashboards!:
53+
54+
:fileset_ex!:
55+
56+
:modulename!:

‎filebeat/module/pensando/_meta/fields.yml‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
- key: pensando
2+
title: Pensando
3+
description: >
4+
pensando Module
5+
fields:
6+
- name: pensando
7+
type: group
8+
description: >
9+
Fields from Pensando logs.
10+
fields:

0 commit comments

Comments
 (0)