Skip to content

Commit 7d74ec3

Browse files
P1llusP1llus
committed
[Filebeat] Adding fixes to the TI module (#24133)
* cleaning up TI module, adding safer config options, updating docs and fixing the MISP tag copy painless script * updating otx pipeline to remove specific null value * fixing grok pattern in MISP to fetch hash values (cherry picked from commit f394755)
1 parent 44efc6f commit 7d74ec3

19 files changed

Lines changed: 500 additions & 223 deletions

File tree

‎filebeat/docs/modules/threatintel.asciidoc‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -12,8 +12,8 @@ This file is generated! See scripts/docs_collector.py
1212
== Threat Intel module
1313
beta[]
1414

15-
This module is a collection of different threat intelligence sources. The ingested data is meant to be used with [Indicator Match rules]https://www.elastic.co/guide/en/security/7.11/rules-ui-create.html#create-indicator-rule, but is also
16-
compatible with other features like [Enrich Processors]https://www.elastic.co/guide/en/elasticsearch/reference/current/enrich-processor.html.
15+
This module is a collection of different threat intelligence sources. The ingested data is meant to be used with https://www.elastic.co/guide/en/security/7.11/rules-ui-create.html#create-indicator-rule[Indicator Match rules], but is also
16+
compatible with other features like https://www.elastic.co/guide/en/elasticsearch/reference/current/enrich-processor.html[Enrich Processors].
1717
The related threat intel attribute that is meant to be used for matching incoming source data is stored under the `threatintel.indicator.*` fields.
1818

1919
Currently supporting:

‎x-pack/filebeat/filebeat.reference.yml‎

Lines changed: 18 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1984,7 +1984,7 @@ filebeat.modules:
19841984
var.url: https://urlhaus-api.abuse.ch/v1/urls/recent/
19851985

19861986
# The interval to poll the API for updates.
1987-
var.interval: 60m
1987+
var.interval: 10m
19881988

19891989
abusemalware:
19901990
enabled: true
@@ -1996,7 +1996,7 @@ filebeat.modules:
19961996
var.url: https://urlhaus-api.abuse.ch/v1/payloads/recent/
19971997

19981998
# The interval to poll the API for updates.
1999-
var.interval: 60m
1999+
var.interval: 10m
20002000

20012001
misp:
20022002
enabled: true
@@ -2010,6 +2010,10 @@ filebeat.modules:
20102010
# The authentication token used to contact the MISP API. Found when looking at user account in the MISP UI.
20112011
var.api_token: API_KEY
20122012

2013+
# Configures the type of SSL verification done, if MISP is running on self signed certificates
2014+
# then the certificate would either need to be trusted, or verification_mode set to none.
2015+
#var.ssl.verification_mode: none
2016+
20132017
# Optional filters that can be applied to the API for filtering out results. This should support the majority of fields in a MISP context.
20142018
# For examples please reference the filebeat module documentation.
20152019
#var.filters:
@@ -2018,10 +2022,10 @@ filebeat.modules:
20182022

20192023
# How far back to look once the beat starts up for the first time, the value has to be in hours. Each request afterwards will filter on any event newer
20202024
# than the last event that was already ingested.
2021-
var.first_interval: 24h
2025+
var.first_interval: 300h
20222026

20232027
# The interval to poll the API for updates.
2024-
var.interval: 60m
2028+
var.interval: 5m
20252029

20262030
otx:
20272031
enabled: true
@@ -2038,22 +2042,26 @@ filebeat.modules:
20382042
# Optional filters that can be applied to retrieve only specific indicators.
20392043
#var.types: "domain,IPv4,hostname,url,FileHash-SHA256"
20402044

2045+
# The timeout of the HTTP client connecting to the OTX API
2046+
#var.http_client_timeout: 120s
2047+
20412048
# How many hours to look back for each request, should be close to the configured interval. Deduplication of events is handled by the module.
2042-
var.lookback_range: 2h
2049+
var.lookback_range: 1h
20432050

20442051
# How far back to look once the beat starts up for the first time, the value has to be in hours.
2045-
var.first_interval: 24h
2052+
var.first_interval: 400h
20462053

20472054
# The interval to poll the API for updates
2048-
var.interval: 60m
2055+
var.interval: 5m
20492056

20502057
anomali:
20512058
enabled: true
20522059

20532060
# Input used for ingesting threat intel data
20542061
var.input: httpjson
20552062

2056-
# The URL used for Threat Intel API calls.
2063+
# The URL used for Threat Intel API calls. Limo has multiple different possibilities for URL's depending
2064+
# on the type of threat intel source that is needed.
20572065
var.url: https://limo.anomali.com/api/v1/taxii2/feeds/collections/41/objects
20582066

20592067
# The Username used by anomali Limo, defaults to guest.
@@ -2063,10 +2071,10 @@ filebeat.modules:
20632071
#var.password: guest
20642072

20652073
# How far back to look once the beat starts up for the first time, the value has to be in hours.
2066-
var.first_interval: 24h
2074+
var.first_interval: 400h
20672075

20682076
# The interval to poll the API for updates
2069-
var.interval: 60m
2077+
var.interval: 5m
20702078

20712079
#---------------------------- Apache Tomcat Module ----------------------------
20722080
- module: tomcat

‎x-pack/filebeat/module/threatintel/_meta/config.yml‎

Lines changed: 18 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@
99
var.url: https://urlhaus-api.abuse.ch/v1/urls/recent/
1010

1111
# The interval to poll the API for updates.
12-
var.interval: 60m
12+
var.interval: 10m
1313

1414
abusemalware:
1515
enabled: true
@@ -21,7 +21,7 @@
2121
var.url: https://urlhaus-api.abuse.ch/v1/payloads/recent/
2222

2323
# The interval to poll the API for updates.
24-
var.interval: 60m
24+
var.interval: 10m
2525

2626
misp:
2727
enabled: true
@@ -35,6 +35,10 @@
3535
# The authentication token used to contact the MISP API. Found when looking at user account in the MISP UI.
3636
var.api_token: API_KEY
3737

38+
# Configures the type of SSL verification done, if MISP is running on self signed certificates
39+
# then the certificate would either need to be trusted, or verification_mode set to none.
40+
#var.ssl.verification_mode: none
41+
3842
# Optional filters that can be applied to the API for filtering out results. This should support the majority of fields in a MISP context.
3943
# For examples please reference the filebeat module documentation.
4044
#var.filters:
@@ -43,10 +47,10 @@
4347

4448
# How far back to look once the beat starts up for the first time, the value has to be in hours. Each request afterwards will filter on any event newer
4549
# than the last event that was already ingested.
46-
var.first_interval: 24h
50+
var.first_interval: 300h
4751

4852
# The interval to poll the API for updates.
49-
var.interval: 60m
53+
var.interval: 5m
5054

5155
otx:
5256
enabled: true
@@ -63,22 +67,26 @@
6367
# Optional filters that can be applied to retrieve only specific indicators.
6468
#var.types: "domain,IPv4,hostname,url,FileHash-SHA256"
6569

70+
# The timeout of the HTTP client connecting to the OTX API
71+
#var.http_client_timeout: 120s
72+
6673
# How many hours to look back for each request, should be close to the configured interval. Deduplication of events is handled by the module.
67-
var.lookback_range: 2h
74+
var.lookback_range: 1h
6875

6976
# How far back to look once the beat starts up for the first time, the value has to be in hours.
70-
var.first_interval: 24h
77+
var.first_interval: 400h
7178

7279
# The interval to poll the API for updates
73-
var.interval: 60m
80+
var.interval: 5m
7481

7582
anomali:
7683
enabled: true
7784

7885
# Input used for ingesting threat intel data
7986
var.input: httpjson
8087

81-
# The URL used for Threat Intel API calls.
88+
# The URL used for Threat Intel API calls. Limo has multiple different possibilities for URL's depending
89+
# on the type of threat intel source that is needed.
8290
var.url: https://limo.anomali.com/api/v1/taxii2/feeds/collections/41/objects
8391

8492
# The Username used by anomali Limo, defaults to guest.
@@ -88,7 +96,7 @@
8896
#var.password: guest
8997

9098
# How far back to look once the beat starts up for the first time, the value has to be in hours.
91-
var.first_interval: 24h
99+
var.first_interval: 400h
92100

93101
# The interval to poll the API for updates
94-
var.interval: 60m
102+
var.interval: 5m

‎x-pack/filebeat/module/threatintel/_meta/docs.asciidoc‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7,8 +7,8 @@
77
== Threat Intel module
88
beta[]
99

10-
This module is a collection of different threat intelligence sources. The ingested data is meant to be used with [Indicator Match rules]https://www.elastic.co/guide/en/security/7.11/rules-ui-create.html#create-indicator-rule, but is also
11-
compatible with other features like [Enrich Processors]https://www.elastic.co/guide/en/elasticsearch/reference/current/enrich-processor.html.
10+
This module is a collection of different threat intelligence sources. The ingested data is meant to be used with https://www.elastic.co/guide/en/security/7.11/rules-ui-create.html#create-indicator-rule[Indicator Match rules], but is also
11+
compatible with other features like https://www.elastic.co/guide/en/elasticsearch/reference/current/enrich-processor.html[Enrich Processors].
1212
The related threat intel attribute that is meant to be used for matching incoming source data is stored under the `threatintel.indicator.*` fields.
1313

1414
Currently supporting:

‎x-pack/filebeat/module/threatintel/abusemalware/config/config.yml‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ interval: {{ .interval }}
66

77
request.method: GET
88
{{ if .ssl }}
9-
- request.ssl: {{ .ssl | tojson }}
9+
request.ssl: {{ .ssl | tojson }}
1010
{{ end }}
1111
request.url: {{ .url }}
1212
request.transforms:
@@ -33,9 +33,11 @@ publisher_pipeline.disable_host: {{ inList .tags "forwarded" }}
3333

3434
processors:
3535
- decode_json_fields:
36-
document_id: "md5_hash"
3736
fields: [message]
3837
target: json
38+
- fingerprint:
39+
fields: ["json.md5_hash"]
40+
target_field: "@metadata._id"
3941
- add_fields:
4042
target: ''
4143
fields:

0 commit comments

Comments
 (0)