Skip to content

Commit ea252df

Browse files
adriansrmergify-bot
adriansr
authored and
mergify-bot
committed
Add Anomali ThreatStream support to threatintel module (#26350)
* Add Anomali ThreatStream support to threatintel module This adds a new dataset, `anomalithreatstream` to the threatintel module. It allows to ingest indicators from Anomali ThreatStream Integrator via a custom SDK output. (cherry picked from commit a6d8cdb)
1 parent d7f1d6c commit ea252df

15 files changed

Lines changed: 5261 additions & 13 deletions

File tree

‎CHANGELOG.next.asciidoc‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -586,6 +586,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
586586
- Add `include_s3_metadata` config option to the `aws-s3` input for including object metadata in events. {pull}26267[26267]
587587
- RFC 5424 and UNIX socket support in the Syslog input are now GA {pull}26293[26293]
588588
- Update grok patterns for HA Proxy module {issue}25827[25827] {pull}25835[25835]
589+
- Added dataset `anomalithreatstream` to the `threatintel` module to ingest indicators from Anomali ThreatStream {pull}26350[26350]
589590

590591
*Heartbeat*
591592

‎filebeat/docs/fields.asciidoc‎

Lines changed: 185 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -146191,6 +146191,191 @@ type: keyword
146191146191
The STIX reference object.
146192146192

146193146193

146194+
type: keyword
146195+
146196+
--
146197+
146198+
[float]
146199+
=== anomalithreatstream
146200+
146201+
Fields for Anomali ThreatStream
146202+
146203+
146204+
146205+
*`threatintel.anomalithreatstream.classification`*::
146206+
+
146207+
--
146208+
Indicates whether an indicator is private or from a public feed and available publicly. Possible values: private, public.
146209+
146210+
146211+
type: keyword
146212+
146213+
example: private
146214+
146215+
--
146216+
146217+
*`threatintel.anomalithreatstream.confidence`*::
146218+
+
146219+
--
146220+
The measure of the accuracy (from 0 to 100) assigned by ThreatStream's predictive analytics technology to indicators.
146221+
146222+
146223+
type: short
146224+
146225+
--
146226+
146227+
*`threatintel.anomalithreatstream.detail2`*::
146228+
+
146229+
--
146230+
Detail text for indicator.
146231+
146232+
146233+
type: text
146234+
146235+
example: Imported by user 42.
146236+
146237+
--
146238+
146239+
*`threatintel.anomalithreatstream.id`*::
146240+
+
146241+
--
146242+
The ID of the indicator.
146243+
146244+
146245+
type: keyword
146246+
146247+
--
146248+
146249+
*`threatintel.anomalithreatstream.import_session_id`*::
146250+
+
146251+
--
146252+
ID of the import session that created the indicator on ThreatStream.
146253+
146254+
146255+
type: keyword
146256+
146257+
--
146258+
146259+
*`threatintel.anomalithreatstream.itype`*::
146260+
+
146261+
--
146262+
Indicator type. Possible values: "apt_domain", "apt_email", "apt_ip", "apt_url", "bot_ip", "c2_domain", "c2_ip", "c2_url", "i2p_ip", "mal_domain", "mal_email", "mal_ip", "mal_md5", "mal_url", "parked_ip", "phish_email", "phish_ip", "phish_url", "scan_ip", "spam_domain", "ssh_ip", "suspicious_domain", "tor_ip" and "torrent_tracker_url".
146263+
146264+
146265+
type: keyword
146266+
146267+
--
146268+
146269+
*`threatintel.anomalithreatstream.maltype`*::
146270+
+
146271+
--
146272+
Information regarding a malware family, a CVE ID, or another attack or threat, associated with the indicator.
146273+
146274+
146275+
type: wildcard
146276+
146277+
--
146278+
146279+
*`threatintel.anomalithreatstream.md5`*::
146280+
+
146281+
--
146282+
Hash for the indicator.
146283+
146284+
146285+
type: keyword
146286+
146287+
--
146288+
146289+
*`threatintel.anomalithreatstream.resource_uri`*::
146290+
+
146291+
--
146292+
Relative URI for the indicator details.
146293+
146294+
146295+
type: keyword
146296+
146297+
--
146298+
146299+
*`threatintel.anomalithreatstream.severity`*::
146300+
+
146301+
--
146302+
Criticality associated with the threat feed that supplied the indicator. Possible values: low, medium, high, very-high.
146303+
146304+
146305+
type: keyword
146306+
146307+
--
146308+
146309+
*`threatintel.anomalithreatstream.source`*::
146310+
+
146311+
--
146312+
Source for the indicator.
146313+
146314+
146315+
type: keyword
146316+
146317+
example: Analyst
146318+
146319+
--
146320+
146321+
*`threatintel.anomalithreatstream.source_feed_id`*::
146322+
+
146323+
--
146324+
ID for the integrator source.
146325+
146326+
146327+
type: keyword
146328+
146329+
--
146330+
146331+
*`threatintel.anomalithreatstream.state`*::
146332+
+
146333+
--
146334+
State for this indicator.
146335+
146336+
146337+
type: keyword
146338+
146339+
example: active
146340+
146341+
--
146342+
146343+
*`threatintel.anomalithreatstream.trusted_circle_ids`*::
146344+
+
146345+
--
146346+
ID of the trusted circle that imported the indicator.
146347+
146348+
146349+
type: keyword
146350+
146351+
--
146352+
146353+
*`threatintel.anomalithreatstream.update_id`*::
146354+
+
146355+
--
146356+
Update ID.
146357+
146358+
146359+
type: keyword
146360+
146361+
--
146362+
146363+
*`threatintel.anomalithreatstream.url`*::
146364+
+
146365+
--
146366+
URL for the indicator.
146367+
146368+
146369+
type: keyword
146370+
146371+
--
146372+
146373+
*`threatintel.anomalithreatstream.value_type`*::
146374+
+
146375+
--
146376+
Data type of the indicator. Possible values: ip, domain, url, email, md5.
146377+
146378+
146194146379
type: keyword
146195146380

146196146381
--

‎filebeat/docs/modules/threatintel.asciidoc‎

Lines changed: 98 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -23,15 +23,17 @@ fields.
2323
[float]
2424
=== The available filesets are:
2525

26-
* `abuseurl`: Supports gathering URL entities from Abuse.ch.
27-
* `abusemalware`: Supports gathering Malware/Payload entities from Abuse.ch.
28-
* `misp`: Supports gathering threat intel attributes from MISP (replaces MISP module).
29-
* `malwarebazaar`: Supports gathering Malware/Payload entities from Malware Bazaar.
30-
* `otx`: Supports gathering threat intel attributes from AlientVault OTX.
31-
* `anomali`: Supports gathering threat intel attributes from Anomali.
26+
* <<abuseurl,abuseurl>>: Supports gathering URL entities from Abuse.ch.
27+
* <<abusemalware,abusemalware>>: Supports gathering Malware/Payload entities from Abuse.ch.
28+
* <<misp,misp>>: Supports gathering threat intel attributes from MISP (replaces MISP module).
29+
* <<malwarebazaar,malwarebazaar>>: Supports gathering Malware/Payload entities from Malware Bazaar.
30+
* <<otx,otx>>: Supports gathering threat intel attributes from AlientVault OTX.
31+
* <<anomali,anomali>>: Supports gathering threat intel attributes from Anomali Limo.
32+
* <<anomalithreatstream,anomalithreatstream>>: Supports gathering threat intel attributes from Anomali ThreatStream.
3233

3334
include::../include/gs-link.asciidoc[]
3435

36+
[[abuseurl]]
3537
[float]
3638
==== `abuseurl` fileset settings
3739

@@ -71,6 +73,7 @@ Abuse.ch URL Threat Intel is mapped to the following ECS fields.
7173
| host | threatintel.indicator.ip/domain
7274
|==============================================================
7375

76+
[[abusemalware]]
7477
[float]
7578
==== `abusemalware` fileset settings
7679

@@ -110,6 +113,7 @@ Abuse.ch Malware Threat Intel is mapped to the following ECS fields.
110113
| file_size | threatintel.indicator.file.size
111114
|================================================================
112115

116+
[[malwarebazaar]]
113117
[float]
114118
==== `malwarebazaar` fileset settings
115119

@@ -164,6 +168,7 @@ Malware Bazaar Threat Intel is mapped to the following ECS fields.
164168
| code_sign.serial_number | threatintel.indicator.file.x509.serial_number
165169
|================================================================
166170

171+
[[misp]]
167172
[float]
168173
==== `misp` fileset settings
169174

@@ -241,6 +246,7 @@ MISP Threat Intel is mapped to the following ECS fields.
241246

242247
`misp.value` is mapped to the appropriate field dependent on attribute type.
243248

249+
[[otx]]
244250
[float]
245251
==== `otx` fileset settings
246252

@@ -316,6 +322,7 @@ OTX Threat Intel is mapped to the following ECS fields.
316322

317323
`otx.indicator` is mapped to the appropriate field dependent on attribute type.
318324

325+
[[anomali]]
319326
[float]
320327
==== `anomali` fileset settings
321328

@@ -397,6 +404,91 @@ Anomali Threat Intel is mapped to the following ECS fields.
397404

398405
`anomali.pattern` is mapped to the appropriate field dependent on attribute type.
399406

407+
[[anomalithreatstream]]
408+
[float]
409+
==== `anomalithreatstream` fileset settings
410+
411+
To configure the ThreatStream integration you first need to define an output
412+
in the Anomali ThreatStream Integrator using the Elastic SDK provided by Anomali.
413+
It will deliver indicators via HTTP or HTTPS to a Filebeat instance running as
414+
a server.
415+
416+
Configure an Integrator output with the following settings:
417+
418+
* Indicator Filter: `*` (or use any desired filter).
419+
* SDK Executable Command: `/path/to/python /path/to/anomali-sdk/main.py`.
420+
Adjust the paths to the python executable and the directory where the Elastic SDK
421+
has been unpacked.
422+
* Metadata in JSON Format: `{"url": "https://filebeat:8080/", "server_certificate": "/path/to/cert.pem", "secret": "my secret"}`.
423+
- `url`: Use the host and port where Filebeat will be running, and `http` or `https` accordingly.
424+
- `server_certificate`: If using HTTPS, absolute path to the server certificate. Otherwise don't set
425+
this field.
426+
- `secret`: A shared secret string to authenticate messages between the SDK and Filebeat.
427+
428+
Then configure the `anomalithreatstream` fileset in Filebeat accordingly:
429+
[source,yaml]
430+
----
431+
- module: threatintel
432+
anomalithreatstream:
433+
enabled: true
434+
var.input: http_endpoint
435+
var.listen_address: 0.0.0.0 # Listen on all interfaces.
436+
var.listen_port: 8080
437+
var.secret: 'my secret'
438+
var.ssl_certificate: path/to/server_ssl_cert.pem
439+
var.ssl_key: path/to/ssl_key.pem
440+
----
441+
442+
*`var.listen_address`*::
443+
444+
Local address to bind the HTTP server to. Use `0.0.0.0` to accept connections
445+
from all interfaces.
446+
447+
*`var.listen_port`*::
448+
449+
Port number to use for the HTTP server.
450+
451+
*`var.secret`*::
452+
453+
Shared secret between the SDK and Filebeat, used to authenticate messages.
454+
455+
*`var.ssl_certificate`*::
456+
457+
Path to the public SSL certificate for the HTTPS server. If unset, Filebeat
458+
will use unsecure HTTP connections.
459+
460+
*`var.ssl_key`*::
461+
462+
Path to the certificate's private key.
463+
464+
Anomali ThreatStream fields are mapped to the following ECS fields:
465+
466+
[options="header"]
467+
|=============================================================
468+
| ThreatStream fields | ECS Fields
469+
| asn | threatintel.indicator.as.number
470+
| classification<<a,[1]>> | threatintel.indicator.marking.tlp
471+
| confidence<<a,[1]>> | threatintel.indicator.confidence
472+
| country | threatintel.indicator.geo.country_iso_code
473+
| date_first | threatintel.indicator.first_seen
474+
| date_last | threatintel.indicator.last_seen
475+
| detail | tags
476+
| domain | threatintel.indicator.domain
477+
| email | threatintel.indicator.email.address
478+
| itype<<a,[1]>> | threatintel.indicator.type
479+
| lat | threatintel.indicator.geo.location.lat
480+
| lon | threatintel.indicator.geo.location.lon
481+
| md5 | threatintel.indicator.file.hash
482+
| org | threatintel.indicator.as.organization.name
483+
| severity<<a,[1]>> | event.severity
484+
| source | threatintel.indicator.provider
485+
| srcip | threatintel.indicator.ip
486+
| url | threatintel.indicator.url.original
487+
|=============================================================
488+
489+
[[a]]
490+
[small]#[1]: Field is used to derive a value for the ECS field but its original value is kept under `threatintel.anomalithreatstream`.#
491+
400492
:has-dashboards!:
401493

402494
[float]

‎filebeat/tests/system/test_modules.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -285,6 +285,7 @@ def clean_keys(obj):
285285
"threatintel.abuseurl",
286286
"threatintel.abusemalware",
287287
"threatintel.anomali",
288+
"threatintel.anomalithreatstream",
288289
"threatintel.malwarebazaar",
289290
"snyk.vulnerabilities",
290291
"snyk.audit",

‎x-pack/filebeat/filebeat.reference.yml‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2249,6 +2249,31 @@ filebeat.modules:
22492249
# The interval to poll the API for updates
22502250
var.interval: 5m
22512251

2252+
anomalithreatstream:
2253+
enabled: true
2254+
2255+
# Input used for ingesting threat intel data
2256+
var.input: http_endpoint
2257+
2258+
# Address to bind to in order to receive HTTP requests
2259+
# from the Integrator SDK. Use 0.0.0.0 to bind to all
2260+
# existing interfaces.
2261+
var.listen_address: localhost
2262+
2263+
# Port to use to receive HTTP requests from the
2264+
# Integrator SDK.
2265+
var.listen_port: 8080
2266+
2267+
# Secret key to authenticate requests from the SDK.
2268+
var.secret: '<Add your secret here>'
2269+
2270+
# Uncomment the following and set the absolute paths
2271+
# to the server SSL certificate and private key to
2272+
# enable HTTPS secure connections.
2273+
#
2274+
# var.ssl_certificate: path/to/server_ssl_cert.pem
2275+
# var.ssl_key: path/to/ssl_key.pem
2276+
22522277
#---------------------------- Apache Tomcat Module ----------------------------
22532278
- module: tomcat
22542279
log:

0 commit comments

Comments
 (0)