Skip to content

Commit a6d8cdb

Browse files
adriansradriansr
authored
Add Anomali ThreatStream support to threatintel module (#26350)
* Add Anomali ThreatStream support to threatintel module This adds a new dataset, `anomalithreatstream` to the threatintel module. It allows to ingest indicators from Anomali ThreatStream Integrator via a custom SDK output.
1 parent f429a82 commit a6d8cdb

15 files changed

Lines changed: 5261 additions & 13 deletions

File tree

‎CHANGELOG.next.asciidoc‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -826,6 +826,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
826826
- Add `include_s3_metadata` config option to the `aws-s3` input for including object metadata in events. {pull}26267[26267]
827827
- RFC 5424 and UNIX socket support in the Syslog input are now GA {pull}26293[26293]
828828
- Update grok patterns for HA Proxy module {issue}25827[25827] {pull}25835[25835]
829+
- Added dataset `anomalithreatstream` to the `threatintel` module to ingest indicators from Anomali ThreatStream {pull}26350[26350]
829830

830831
*Heartbeat*
831832

‎filebeat/docs/fields.asciidoc‎

Lines changed: 185 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -152754,6 +152754,191 @@ type: keyword
152754152754
The STIX reference object.
152755152755

152756152756

152757+
type: keyword
152758+
152759+
--
152760+
152761+
[float]
152762+
=== anomalithreatstream
152763+
152764+
Fields for Anomali ThreatStream
152765+
152766+
152767+
152768+
*`threatintel.anomalithreatstream.classification`*::
152769+
+
152770+
--
152771+
Indicates whether an indicator is private or from a public feed and available publicly. Possible values: private, public.
152772+
152773+
152774+
type: keyword
152775+
152776+
example: private
152777+
152778+
--
152779+
152780+
*`threatintel.anomalithreatstream.confidence`*::
152781+
+
152782+
--
152783+
The measure of the accuracy (from 0 to 100) assigned by ThreatStream's predictive analytics technology to indicators.
152784+
152785+
152786+
type: short
152787+
152788+
--
152789+
152790+
*`threatintel.anomalithreatstream.detail2`*::
152791+
+
152792+
--
152793+
Detail text for indicator.
152794+
152795+
152796+
type: text
152797+
152798+
example: Imported by user 42.
152799+
152800+
--
152801+
152802+
*`threatintel.anomalithreatstream.id`*::
152803+
+
152804+
--
152805+
The ID of the indicator.
152806+
152807+
152808+
type: keyword
152809+
152810+
--
152811+
152812+
*`threatintel.anomalithreatstream.import_session_id`*::
152813+
+
152814+
--
152815+
ID of the import session that created the indicator on ThreatStream.
152816+
152817+
152818+
type: keyword
152819+
152820+
--
152821+
152822+
*`threatintel.anomalithreatstream.itype`*::
152823+
+
152824+
--
152825+
Indicator type. Possible values: "apt_domain", "apt_email", "apt_ip", "apt_url", "bot_ip", "c2_domain", "c2_ip", "c2_url", "i2p_ip", "mal_domain", "mal_email", "mal_ip", "mal_md5", "mal_url", "parked_ip", "phish_email", "phish_ip", "phish_url", "scan_ip", "spam_domain", "ssh_ip", "suspicious_domain", "tor_ip" and "torrent_tracker_url".
152826+
152827+
152828+
type: keyword
152829+
152830+
--
152831+
152832+
*`threatintel.anomalithreatstream.maltype`*::
152833+
+
152834+
--
152835+
Information regarding a malware family, a CVE ID, or another attack or threat, associated with the indicator.
152836+
152837+
152838+
type: wildcard
152839+
152840+
--
152841+
152842+
*`threatintel.anomalithreatstream.md5`*::
152843+
+
152844+
--
152845+
Hash for the indicator.
152846+
152847+
152848+
type: keyword
152849+
152850+
--
152851+
152852+
*`threatintel.anomalithreatstream.resource_uri`*::
152853+
+
152854+
--
152855+
Relative URI for the indicator details.
152856+
152857+
152858+
type: keyword
152859+
152860+
--
152861+
152862+
*`threatintel.anomalithreatstream.severity`*::
152863+
+
152864+
--
152865+
Criticality associated with the threat feed that supplied the indicator. Possible values: low, medium, high, very-high.
152866+
152867+
152868+
type: keyword
152869+
152870+
--
152871+
152872+
*`threatintel.anomalithreatstream.source`*::
152873+
+
152874+
--
152875+
Source for the indicator.
152876+
152877+
152878+
type: keyword
152879+
152880+
example: Analyst
152881+
152882+
--
152883+
152884+
*`threatintel.anomalithreatstream.source_feed_id`*::
152885+
+
152886+
--
152887+
ID for the integrator source.
152888+
152889+
152890+
type: keyword
152891+
152892+
--
152893+
152894+
*`threatintel.anomalithreatstream.state`*::
152895+
+
152896+
--
152897+
State for this indicator.
152898+
152899+
152900+
type: keyword
152901+
152902+
example: active
152903+
152904+
--
152905+
152906+
*`threatintel.anomalithreatstream.trusted_circle_ids`*::
152907+
+
152908+
--
152909+
ID of the trusted circle that imported the indicator.
152910+
152911+
152912+
type: keyword
152913+
152914+
--
152915+
152916+
*`threatintel.anomalithreatstream.update_id`*::
152917+
+
152918+
--
152919+
Update ID.
152920+
152921+
152922+
type: keyword
152923+
152924+
--
152925+
152926+
*`threatintel.anomalithreatstream.url`*::
152927+
+
152928+
--
152929+
URL for the indicator.
152930+
152931+
152932+
type: keyword
152933+
152934+
--
152935+
152936+
*`threatintel.anomalithreatstream.value_type`*::
152937+
+
152938+
--
152939+
Data type of the indicator. Possible values: ip, domain, url, email, md5.
152940+
152941+
152757152942
type: keyword
152758152943

152759152944
--

‎filebeat/docs/modules/threatintel.asciidoc‎

Lines changed: 98 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -22,15 +22,17 @@ fields.
2222

2323
The available filesets are:
2424

25-
* `abuseurl`: Supports gathering URL entities from Abuse.ch.
26-
* `abusemalware`: Supports gathering Malware/Payload entities from Abuse.ch.
27-
* `misp`: Supports gathering threat intel attributes from MISP (replaces MISP module).
28-
* `malwarebazaar`: Supports gathering Malware/Payload entities from Malware Bazaar.
29-
* `otx`: Supports gathering threat intel attributes from AlientVault OTX.
30-
* `anomali`: Supports gathering threat intel attributes from Anomali.
25+
* <<abuseurl,abuseurl>>: Supports gathering URL entities from Abuse.ch.
26+
* <<abusemalware,abusemalware>>: Supports gathering Malware/Payload entities from Abuse.ch.
27+
* <<misp,misp>>: Supports gathering threat intel attributes from MISP (replaces MISP module).
28+
* <<malwarebazaar,malwarebazaar>>: Supports gathering Malware/Payload entities from Malware Bazaar.
29+
* <<otx,otx>>: Supports gathering threat intel attributes from AlientVault OTX.
30+
* <<anomali,anomali>>: Supports gathering threat intel attributes from Anomali Limo.
31+
* <<anomalithreatstream,anomalithreatstream>>: Supports gathering threat intel attributes from Anomali ThreatStream.
3132

3233
include::../include/gs-link.asciidoc[]
3334

35+
[[abuseurl]]
3436
[float]
3537
==== `abuseurl` fileset settings
3638

@@ -70,6 +72,7 @@ Abuse.ch URL Threat Intel is mapped to the following ECS fields.
7072
| host | threatintel.indicator.ip/domain
7173
|==============================================================
7274

75+
[[abusemalware]]
7376
[float]
7477
==== `abusemalware` fileset settings
7578

@@ -109,6 +112,7 @@ Abuse.ch Malware Threat Intel is mapped to the following ECS fields.
109112
| file_size | threatintel.indicator.file.size
110113
|================================================================
111114

115+
[[malwarebazaar]]
112116
[float]
113117
==== `malwarebazaar` fileset settings
114118

@@ -163,6 +167,7 @@ Malware Bazaar Threat Intel is mapped to the following ECS fields.
163167
| code_sign.serial_number | threatintel.indicator.file.x509.serial_number
164168
|================================================================
165169

170+
[[misp]]
166171
[float]
167172
==== `misp` fileset settings
168173

@@ -240,6 +245,7 @@ MISP Threat Intel is mapped to the following ECS fields.
240245

241246
`misp.value` is mapped to the appropriate field dependent on attribute type.
242247

248+
[[otx]]
243249
[float]
244250
==== `otx` fileset settings
245251

@@ -315,6 +321,7 @@ OTX Threat Intel is mapped to the following ECS fields.
315321

316322
`otx.indicator` is mapped to the appropriate field dependent on attribute type.
317323

324+
[[anomali]]
318325
[float]
319326
==== `anomali` fileset settings
320327

@@ -396,6 +403,91 @@ Anomali Threat Intel is mapped to the following ECS fields.
396403

397404
`anomali.pattern` is mapped to the appropriate field dependent on attribute type.
398405

406+
[[anomalithreatstream]]
407+
[float]
408+
==== `anomalithreatstream` fileset settings
409+
410+
To configure the ThreatStream integration you first need to define an output
411+
in the Anomali ThreatStream Integrator using the Elastic SDK provided by Anomali.
412+
It will deliver indicators via HTTP or HTTPS to a Filebeat instance running as
413+
a server.
414+
415+
Configure an Integrator output with the following settings:
416+
417+
* Indicator Filter: `*` (or use any desired filter).
418+
* SDK Executable Command: `/path/to/python /path/to/anomali-sdk/main.py`.
419+
Adjust the paths to the python executable and the directory where the Elastic SDK
420+
has been unpacked.
421+
* Metadata in JSON Format: `{"url": "https://filebeat:8080/", "server_certificate": "/path/to/cert.pem", "secret": "my secret"}`.
422+
- `url`: Use the host and port where Filebeat will be running, and `http` or `https` accordingly.
423+
- `server_certificate`: If using HTTPS, absolute path to the server certificate. Otherwise don't set
424+
this field.
425+
- `secret`: A shared secret string to authenticate messages between the SDK and Filebeat.
426+
427+
Then configure the `anomalithreatstream` fileset in Filebeat accordingly:
428+
[source,yaml]
429+
----
430+
- module: threatintel
431+
anomalithreatstream:
432+
enabled: true
433+
var.input: http_endpoint
434+
var.listen_address: 0.0.0.0 # Listen on all interfaces.
435+
var.listen_port: 8080
436+
var.secret: 'my secret'
437+
var.ssl_certificate: path/to/server_ssl_cert.pem
438+
var.ssl_key: path/to/ssl_key.pem
439+
----
440+
441+
*`var.listen_address`*::
442+
443+
Local address to bind the HTTP server to. Use `0.0.0.0` to accept connections
444+
from all interfaces.
445+
446+
*`var.listen_port`*::
447+
448+
Port number to use for the HTTP server.
449+
450+
*`var.secret`*::
451+
452+
Shared secret between the SDK and Filebeat, used to authenticate messages.
453+
454+
*`var.ssl_certificate`*::
455+
456+
Path to the public SSL certificate for the HTTPS server. If unset, Filebeat
457+
will use unsecure HTTP connections.
458+
459+
*`var.ssl_key`*::
460+
461+
Path to the certificate's private key.
462+
463+
Anomali ThreatStream fields are mapped to the following ECS fields:
464+
465+
[options="header"]
466+
|=============================================================
467+
| ThreatStream fields | ECS Fields
468+
| asn | threatintel.indicator.as.number
469+
| classification<<a,[1]>> | threatintel.indicator.marking.tlp
470+
| confidence<<a,[1]>> | threatintel.indicator.confidence
471+
| country | threatintel.indicator.geo.country_iso_code
472+
| date_first | threatintel.indicator.first_seen
473+
| date_last | threatintel.indicator.last_seen
474+
| detail | tags
475+
| domain | threatintel.indicator.domain
476+
| email | threatintel.indicator.email.address
477+
| itype<<a,[1]>> | threatintel.indicator.type
478+
| lat | threatintel.indicator.geo.location.lat
479+
| lon | threatintel.indicator.geo.location.lon
480+
| md5 | threatintel.indicator.file.hash
481+
| org | threatintel.indicator.as.organization.name
482+
| severity<<a,[1]>> | event.severity
483+
| source | threatintel.indicator.provider
484+
| srcip | threatintel.indicator.ip
485+
| url | threatintel.indicator.url.original
486+
|=============================================================
487+
488+
[[a]]
489+
[small]#[1]: Field is used to derive a value for the ECS field but its original value is kept under `threatintel.anomalithreatstream`.#
490+
399491
:has-dashboards!:
400492

401493
[float]

‎filebeat/tests/system/test_modules.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -277,6 +277,7 @@ def clean_keys(obj):
277277
"threatintel.abuseurl",
278278
"threatintel.abusemalware",
279279
"threatintel.anomali",
280+
"threatintel.anomalithreatstream",
280281
"threatintel.malwarebazaar",
281282
"snyk.vulnerabilities",
282283
"snyk.audit",

‎x-pack/filebeat/filebeat.reference.yml‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2249,6 +2249,31 @@ filebeat.modules:
22492249
# The interval to poll the API for updates
22502250
var.interval: 5m
22512251

2252+
anomalithreatstream:
2253+
enabled: true
2254+
2255+
# Input used for ingesting threat intel data
2256+
var.input: http_endpoint
2257+
2258+
# Address to bind to in order to receive HTTP requests
2259+
# from the Integrator SDK. Use 0.0.0.0 to bind to all
2260+
# existing interfaces.
2261+
var.listen_address: localhost
2262+
2263+
# Port to use to receive HTTP requests from the
2264+
# Integrator SDK.
2265+
var.listen_port: 8080
2266+
2267+
# Secret key to authenticate requests from the SDK.
2268+
var.secret: '<Add your secret here>'
2269+
2270+
# Uncomment the following and set the absolute paths
2271+
# to the server SSL certificate and private key to
2272+
# enable HTTPS secure connections.
2273+
#
2274+
# var.ssl_certificate: path/to/server_ssl_cert.pem
2275+
# var.ssl_key: path/to/ssl_key.pem
2276+
22522277
#---------------------------- Apache Tomcat Module ----------------------------
22532278
- module: tomcat
22542279
log:

0 commit comments

Comments
 (0)