Skip to content

Commit 969f495

Browse files
legoguy1000mergify-bot
legoguy1000
authored and
mergify-bot
committed
[Filebeat] Update Fortinet Ingest Pipeline (#24816)
* 22136: Update Fortinet Ingest Pipeline * Update Pipelines * Additional updates * Set virus/ips subtypes to event.kind: alert * update fields * Consolidate processors to script * Update event.outcome logic * replace hashmap * update event.outcome * cleanup * Added Changes for #25254 * regenerate data * update changelog * remove extra items in changelog (cherry picked from commit 890e473)
1 parent 5ded62f commit 969f495

20 files changed

Lines changed: 4630 additions & 2435 deletions

‎CHANGELOG.next.asciidoc‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -76,6 +76,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
7676
- Change logging in logs input to structure logging. Some log message formats have changed. {pull}25299[25299]
7777
- All url.* fields apart from url.original in the Apache, Nginx, IIS, Traefik, S3Access, Cisco, F5, Fortinet, Google Workspace, Imperva, Microsoft, Netscout, O365, Sophos, Squid, Suricata, Zeek, Zia, Zoom, and ZScaler modules are now url unescaped due to using the Elasticsearch uri_parts processor. {pull}24699[24699]
7878
- Deprecated the cyberark module (replaced by cyberarkpas). {issue}25261[25261] {pull}25505[25505]
79+
- Change source field for `event.action` in `fortinet.firewall` module to `fortinet.firewall.action` instead of `fortinet.firewall.eventtype`. {pull}24816[24816]
7980

8081
*Heartbeat*
8182
- Add support for screenshot blocks and use newer synthetics flags that only works in newer synthetics betas. {pull}25808[25808]
@@ -601,6 +602,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
601602
- Add support for `copytruncate` method when rotating input logs with an external tool in `filestream` input. {pull}23457[23457]
602603
- Add `uri_parts` and `user_agent` ingest processors to `aws.elb` module. {issue}26435[26435] {pull}26441[26441]
603604
- Added dataset `recordedfuture` to the `threatintel` module to ingest indicators from Recorded Future Connect API {pull}26481[26481]
605+
- Update `fortinet` ingest pipelines. {issue}22136[22136] {issue}25254[25254] {pull}24816[24816]
604606

605607
*Heartbeat*
606608

‎filebeat/docs/fields.asciidoc‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -62763,6 +62763,16 @@ type: keyword
6276362763
ESP Transform
6276462764

6276562765

62766+
type: keyword
62767+
62768+
--
62769+
62770+
*`fortinet.firewall.eventtype`*::
62771+
+
62772+
--
62773+
UTM Event Type
62774+
62775+
6276662776
type: keyword
6276762777

6276862778
--
@@ -65243,6 +65253,16 @@ type: integer
6524365253
Security action performed by UTM
6524465254

6524565255

65256+
type: keyword
65257+
65258+
--
65259+
65260+
*`fortinet.firewall.utmref`*::
65261+
+
65262+
--
65263+
Reference to UTM
65264+
65265+
6524665266
type: keyword
6524765267

6524865268
--

‎x-pack/filebeat/module/fortinet/fields.go‎

Lines changed: 1 addition & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎x-pack/filebeat/module/fortinet/firewall/_meta/fields.yml‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -766,6 +766,11 @@
766766
description: >
767767
ESP Transform
768768
769+
- name: eventtype
770+
type: keyword
771+
description: >
772+
UTM Event Type
773+
769774
- name: exch
770775
type: keyword
771776
description: >
@@ -2006,6 +2011,11 @@
20062011
description: >
20072012
Security action performed by UTM
20082013
2014+
- name: utmref
2015+
type: keyword
2016+
description: >
2017+
Reference to UTM
2018+
20092019
- name: vap
20102020
type: keyword
20112021
description: >

‎x-pack/filebeat/module/fortinet/firewall/ingest/event.yml‎

Lines changed: 12 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,10 @@ processors:
33
- set:
44
field: event.kind
55
value: event
6+
- set:
7+
field: event.action
8+
value: "{{fortinet.firewall.action}}"
9+
ignore_empty_value: true
610
- set:
711
field: event.outcome
812
value: failure
@@ -16,24 +20,29 @@ processors:
1620
value:
1721
- user
1822
- start
23+
allow_duplicates: false
1924
if: "['FSSO-logon', 'auth-logon'].contains(ctx.fortinet?.firewall?.action)"
2025
- append:
2126
field: event.type
2227
value:
2328
- user
2429
- end
30+
allow_duplicates: false
2531
if: "['FSSO-logoff', 'auth-logout'].contains(ctx.fortinet?.firewall?.action)"
2632
- append:
2733
field: event.type
2834
value: connection
35+
allow_duplicates: false
2936
if: "ctx.fortinet?.firewall?.subtype == 'vpn'"
3037
- append:
3138
field: event.category
3239
value: network
40+
allow_duplicates: false
3341
if: "ctx.fortinet?.firewall?.subtype == 'vpn'"
3442
- append:
3543
field: event.type
3644
value: info
45+
allow_duplicates: false
3746
if: "ctx.fortinet?.firewall?.action == 'perf-stats'"
3847
- append:
3948
field: event.category
@@ -42,16 +51,19 @@ processors:
4251
- append:
4352
field: event.type
4453
value: info
54+
allow_duplicates: false
4555
if: "ctx.fortinet?.firewall?.subtype == 'update'"
4656
- append:
4757
field: event.category
4858
value:
4959
- host
5060
- malware
61+
allow_duplicates: false
5162
if: "ctx.fortinet?.firewall?.subtype == 'update'"
5263
- append:
5364
field: event.category
5465
value: authentication
66+
allow_duplicates: false
5567
if: "ctx.fortinet?.firewall?.subtype == 'user'"
5668
- rename:
5769
field: fortinet.firewall.dstip
@@ -95,10 +107,6 @@ processors:
95107
target_field: destination.domain
96108
ignore_missing: true
97109
if: "ctx.destination?.address == null"
98-
- rename:
99-
field: fortinet.firewall.group
100-
target_field: source.user.group.name
101-
ignore_missing: true
102110
- convert:
103111
field: fortinet.firewall.sentbyte
104112
target_field: source.bytes
@@ -144,10 +152,6 @@ processors:
144152
field: fortinet.firewall.saddr
145153
target_field: source.address
146154
ignore_missing: true
147-
- rename:
148-
field: fortinet.firewall.agent
149-
target_field: user_agent.original
150-
ignore_missing: true
151155
- rename:
152156
field: fortinet.firewall.file
153157
target_field: file.name
@@ -167,18 +171,10 @@ processors:
167171
target_field: event.code
168172
ignore_missing: true
169173
if: "ctx.event?.code == null"
170-
- rename:
171-
field: fortinet.firewall.msg
172-
target_field: message
173-
ignore_missing: true
174174
- rename:
175175
field: fortinet.firewall.policyid
176176
target_field: rule.id
177177
ignore_missing: true
178-
- rename:
179-
field: fortinet.firewall.proto
180-
target_field: network.iana_number
181-
ignore_missing: true
182178
- rename:
183179
field: fortinet.firewall.dir
184180
target_field: network.direction
@@ -207,21 +203,10 @@ processors:
207203
return
208204
}
209205
ctx.network.direction = k;
210-
- rename:
211-
field: fortinet.firewall.service
212-
target_field: network.protocol
213-
ignore_missing: true
214-
- lowercase:
215-
field: network.protocol
216-
ignore_missing: true
217206
- rename:
218207
field: fortinet.firewall.error_num
219208
target_field: error.code
220209
ignore_missing: true
221-
- rename:
222-
field: fortinet.firewall.hostname
223-
target_field: url.domain
224-
ignore_missing: true
225210
- rename:
226211
field: fortinet.firewall.logdesc
227212
target_field: rule.description

‎x-pack/filebeat/module/fortinet/firewall/ingest/pipeline.yml‎

Lines changed: 114 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@ processors:
2424
source: |
2525
def fw = ctx?.fortinet?.firewall;
2626
if (fw != null) {
27-
fw.entrySet().removeIf(entry -> entry.getValue() == "N/A");
27+
fw.entrySet().removeIf(entry -> entry.getValue() == "N/A" || entry.getValue() == "undefined");
2828
}
2929
- set:
3030
field: observer.vendor
@@ -35,12 +35,6 @@ processors:
3535
- set:
3636
field: observer.type
3737
value: firewall
38-
- set:
39-
field: event.module
40-
value: fortinet
41-
- set:
42-
field: event.dataset
43-
value: fortinet.firewall
4438
- set:
4539
field: event.timezone
4640
value: "{{fortinet.firewall.tz}}"
@@ -198,18 +192,7 @@ processors:
198192
)
199193
- remove:
200194
field:
201-
- _temp.time
202-
- _temp
203195
- message
204-
- syslog5424_sd
205-
- syslog5424_pri
206-
- fortinet.firewall.tz
207-
- fortinet.firewall.date
208-
- fortinet.firewall.devid
209-
- fortinet.firewall.eventtime
210-
- fortinet.firewall.time
211-
- fortinet.firewall.duration
212-
- host
213196
ignore_missing: true
214197
- pipeline:
215198
name: '{< IngestPipeline "event" >}'
@@ -220,6 +203,73 @@ processors:
220203
- pipeline:
221204
name: '{< IngestPipeline "utm" >}'
222205
if: "ctx.fortinet?.firewall?.type == 'utm' || ctx.fortinet?.firewall?.type == 'dns'"
206+
- rename:
207+
field: fortinet.firewall.reason
208+
target_field: event.reason
209+
ignore_missing: true
210+
- rename:
211+
field: fortinet.firewall.msg
212+
target_field: message
213+
ignore_missing: true
214+
- rename:
215+
field: fortinet.firewall.proto
216+
target_field: network.iana_number
217+
ignore_missing: true
218+
- script:
219+
lang: painless
220+
ignore_failure: true
221+
if: ctx?.network?.iana_number != null
222+
source: |
223+
if (ctx?.network == null) {
224+
ctx.network = new HashMap();
225+
}
226+
def iana_number = ctx.network.iana_number;
227+
if (iana_number == '1') {
228+
ctx.network.transport = 'icmp';
229+
} else if (iana_number == '2') {
230+
ctx.network.transport = 'igmp';
231+
} else if (iana_number == '6') {
232+
ctx.network.transport = 'tcp';
233+
} else if (iana_number == '17') {
234+
ctx.network.transport = 'udp';
235+
} else if (iana_number == '58') {
236+
ctx.network.transport = 'ipv6-icmp';
237+
}
238+
- rename:
239+
field: fortinet.firewall.group
240+
target_field: source.user.group.name
241+
ignore_missing: true
242+
- uri_parts:
243+
field: fortinet.firewall.url
244+
remove_if_successful: true
245+
ignore_failure: true
246+
if: "ctx.fortinet?.firewall?.url != null"
247+
- set:
248+
field: url.domain
249+
value: "{{fortinet.firewall.hostname}}"
250+
ignore_empty_value: true
251+
if: "ctx?.url?.domain == null"
252+
- rename:
253+
field: fortinet.firewall.service
254+
target_field: network.protocol
255+
ignore_missing: true
256+
- lowercase:
257+
field: network.protocol
258+
ignore_missing: true
259+
- set:
260+
field: network.type
261+
value: ipv4
262+
if: (ctx.source?.ip != null && ctx.source?.ip.contains('.')) || (ctx.destination?.ip != null && ctx.destination?.ip.contains('.'))
263+
- set:
264+
field: network.type
265+
value: ipv6
266+
if: ctx.source?.ip != null && ctx.source?.ip.contains(':') || (ctx.destination?.ip != null && ctx.destination?.ip.contains(':'))
267+
- community_id:
268+
ignore_missing: true
269+
ignore_failure: true
270+
- user_agent:
271+
field: fortinet.firewall.agent
272+
ignore_missing: true
223273
- convert:
224274
field: fortinet.firewall.quotamax
225275
type: long
@@ -315,30 +365,37 @@ processors:
315365
- append:
316366
field: related.ip
317367
value: "{{source.ip}}"
368+
allow_duplicates: false
318369
if: "ctx.source?.ip != null"
319370
- append:
320371
field: related.ip
321372
value: "{{destination.ip}}"
373+
allow_duplicates: false
322374
if: "ctx.destination?.ip != null"
323375
- append:
324376
field: related.user
325377
value: "{{source.user.name}}"
378+
allow_duplicates: false
326379
if: "ctx.source?.user?.name != null"
327380
- append:
328381
field: related.user
329382
value: "{{destination.user.name}}"
383+
allow_duplicates: false
330384
if: "ctx.destination?.user?.name != null"
331385
- append:
332386
field: related.hosts
333387
value: "{{destination.address}}"
388+
allow_duplicates: false
334389
if: "ctx.destination?.address != null"
335390
- append:
336391
field: related.hosts
337392
value: "{{source.address}}"
393+
allow_duplicates: false
338394
if: "ctx.source?.address != null"
339395
- append:
340396
field: related.hosts
341397
value: "{{dns.question.name}}"
398+
allow_duplicates: false
342399
if: "ctx.dns?.question?.name != null"
343400
- script:
344401
lang: painless
@@ -354,6 +411,45 @@ processors:
354411
}
355412
}
356413
}
414+
- remove:
415+
field:
416+
- _temp
417+
- syslog5424_sd
418+
- syslog5424_pri
419+
- fortinet.firewall.tz
420+
- fortinet.firewall.date
421+
- fortinet.firewall.devid
422+
- fortinet.firewall.eventtime
423+
- fortinet.firewall.time
424+
- fortinet.firewall.duration
425+
- host
426+
- fortinet.firewall.hostname
427+
- fortinet.firewall.agent
428+
ignore_missing: true
429+
- script:
430+
lang: painless
431+
description: This script processor iterates over the whole document to remove fields with null values.
432+
source: |
433+
void handleMap(Map map) {
434+
for (def x : map.values()) {
435+
if (x instanceof Map) {
436+
handleMap(x);
437+
} else if (x instanceof List) {
438+
handleList(x);
439+
}
440+
}
441+
map.values().removeIf(v -> v == null);
442+
}
443+
void handleList(List list) {
444+
for (def x : list) {
445+
if (x instanceof Map) {
446+
handleMap(x);
447+
} else if (x instanceof List) {
448+
handleList(x);
449+
}
450+
}
451+
}
452+
handleMap(ctx);
357453
on_failure:
358454
- set:
359455
field: error.message

0 commit comments

Comments
 (0)