Skip to content

Commit 6d89566

Browse files
adriansradriansr
authored
Add Recorded Future support to threatintel module (#26481)
This adds a new fileset, `recordedfuture`, to the treatintel module. It ingests indicators via the Recorded Future Connect API.
1 parent 076e0a6 commit 6d89566

22 files changed

Lines changed: 2567 additions & 17 deletions

‎CHANGELOG.next.asciidoc‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -834,6 +834,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
834834
- Added dataset `anomalithreatstream` to the `threatintel` module to ingest indicators from Anomali ThreatStream {pull}26350[26350]
835835
- Add support for `copytruncate` method when rotating input logs with an external tool in `filestream` input. {pull}23457[23457]
836836
- Add `uri_parts` and `user_agent` ingest processors to `aws.elb` module. {issue}26435[26435] {pull}26441[26441]
837+
- Added dataset `recordedfuture` to the `threatintel` module to ingest indicators from Recorded Future Connect API {pull}26481[26481]
837838

838839
*Heartbeat*
839840

‎filebeat/docs/fields.asciidoc‎

Lines changed: 171 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -152208,6 +152208,17 @@ example: Montreal
152208152208

152209152209
--
152210152210

152211+
*`threatintel.indicator.geo.continent_name`*::
152212+
+
152213+
--
152214+
Name of the continent.
152215+
152216+
type: keyword
152217+
152218+
example: North America
152219+
152220+
--
152221+
152211152222
*`threatintel.indicator.geo.country_iso_code`*::
152212152223
+
152213152224
--
@@ -153643,6 +153654,166 @@ type: keyword
153643153654

153644153655
--
153645153656

153657+
[float]
153658+
=== recordedfuture
153659+
153660+
Fields for Recorded Future Threat Intel
153661+
153662+
153663+
153664+
[float]
153665+
=== entity
153666+
153667+
Entity that represents a threat.
153668+
153669+
153670+
153671+
*`threatintel.recordedfuture.entity.id`*::
153672+
+
153673+
--
153674+
Entity ID.
153675+
153676+
153677+
type: keyword
153678+
153679+
example: ip:192.0.2.13
153680+
153681+
--
153682+
153683+
*`threatintel.recordedfuture.entity.name`*::
153684+
+
153685+
--
153686+
Entity name. Value for the entity.
153687+
153688+
153689+
type: keyword
153690+
153691+
example: 192.0.2.13
153692+
153693+
--
153694+
153695+
*`threatintel.recordedfuture.entity.type`*::
153696+
+
153697+
--
153698+
Entity type.
153699+
153700+
153701+
type: keyword
153702+
153703+
example: IpAddress
153704+
153705+
--
153706+
153707+
*`threatintel.recordedfuture.intelCard`*::
153708+
+
153709+
--
153710+
Link to the Recorded Future Intelligence Card for to this indicator.
153711+
153712+
153713+
type: keyword
153714+
153715+
--
153716+
153717+
*`threatintel.recordedfuture.ip_range`*::
153718+
+
153719+
--
153720+
Range of IPs for this indicator.
153721+
153722+
153723+
type: ip_range
153724+
153725+
example: 192.0.2.0/16
153726+
153727+
--
153728+
153729+
[float]
153730+
=== risk
153731+
153732+
Risk fields.
153733+
153734+
153735+
153736+
*`threatintel.recordedfuture.risk.criticality`*::
153737+
+
153738+
--
153739+
Risk criticality (0-4).
153740+
153741+
153742+
type: byte
153743+
153744+
--
153745+
153746+
*`threatintel.recordedfuture.risk.criticalityLabel`*::
153747+
+
153748+
--
153749+
Risk criticality label. One of None, Unusual, Suspicious, Malicious, Very Malicious.
153750+
153751+
153752+
type: keyword
153753+
153754+
--
153755+
153756+
*`threatintel.recordedfuture.risk.evidenceDetails`*::
153757+
+
153758+
--
153759+
Risk's evidence details.
153760+
153761+
153762+
type: flattened
153763+
153764+
--
153765+
153766+
*`threatintel.recordedfuture.risk.score`*::
153767+
+
153768+
--
153769+
Risk score (0-99).
153770+
153771+
153772+
type: short
153773+
153774+
--
153775+
153776+
*`threatintel.recordedfuture.risk.riskString`*::
153777+
+
153778+
--
153779+
Number of Risk Rules observed as a factor of total number of rules.
153780+
153781+
153782+
type: keyword
153783+
153784+
example: 1/54
153785+
153786+
--
153787+
153788+
*`threatintel.recordedfuture.risk.riskSummary`*::
153789+
+
153790+
--
153791+
Risk summary.
153792+
153793+
153794+
type: keyword
153795+
153796+
example: 1 of 54 Risk Rules currently observed.
153797+
153798+
--
153799+
153800+
*`threatintel.recordedfuture.risk.riskSummary.text`*::
153801+
+
153802+
--
153803+
type: text
153804+
153805+
--
153806+
153807+
*`threatintel.recordedfuture.risk.rules`*::
153808+
+
153809+
--
153810+
Number of rules observed.
153811+
153812+
153813+
type: long
153814+
153815+
--
153816+
153646153817
[[exported-fields-tomcat]]
153647153818
== Apache Tomcat fields
153648153819

‎filebeat/docs/modules/threatintel.asciidoc‎

Lines changed: 97 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,7 @@ The available filesets are:
2929
* <<otx,otx>>: Supports gathering threat intel attributes from AlientVault OTX.
3030
* <<anomali,anomali>>: Supports gathering threat intel attributes from Anomali Limo.
3131
* <<anomalithreatstream,anomalithreatstream>>: Supports gathering threat intel attributes from Anomali ThreatStream.
32+
* <<recordedfuture,recordedfuture>>: Supports gathering threat intel attributes from Recorded Future.
3233

3334
include::../include/gs-link.asciidoc[]
3435

@@ -223,7 +224,7 @@ How often the API is polled for updated information.
223224

224225
*`var.first_interval`*::
225226

226-
How far back to search when retrieving events the first time the beat starts up.
227+
How far back to search when retrieving events the first time {beatname_uc} starts up.
227228
After the first interval has passed the module itself will use the timestamp
228229
from the last response as the filter when retrieving new events.
229230

@@ -297,7 +298,7 @@ How often the API is polled for updated information.
297298

298299
*`var.first_interval`*::
299300

300-
How far back to search when retrieving events the first time the beat starts up.
301+
How far back to search when retrieving events the first time the {beatname_uc} starts up.
301302
After the first interval has passed the module itself will use the timestamp
302303
from the last response as the filter when retrieving new events.
303304

@@ -409,7 +410,7 @@ Anomali Threat Intel is mapped to the following ECS fields.
409410

410411
To configure the ThreatStream integration you first need to define an output
411412
in the Anomali ThreatStream Integrator using the Elastic SDK provided by Anomali.
412-
It will deliver indicators via HTTP or HTTPS to a Filebeat instance running as
413+
It will deliver indicators via HTTP or HTTPS to a {beatname_uc} instance running as
413414
a server.
414415

415416
Configure an Integrator output with the following settings:
@@ -419,12 +420,12 @@ Configure an Integrator output with the following settings:
419420
Adjust the paths to the python executable and the directory where the Elastic SDK
420421
has been unpacked.
421422
* Metadata in JSON Format: `{"url": "https://filebeat:8080/", "server_certificate": "/path/to/cert.pem", "secret": "my secret"}`.
422-
- `url`: Use the host and port where Filebeat will be running, and `http` or `https` accordingly.
423+
- `url`: Use the host and port where {beatname_uc} will be running, and `http` or `https` accordingly.
423424
- `server_certificate`: If using HTTPS, absolute path to the server certificate. Otherwise don't set
424425
this field.
425-
- `secret`: A shared secret string to authenticate messages between the SDK and Filebeat.
426+
- `secret`: A shared secret string to authenticate messages between the SDK and {beatname_uc}.
426427

427-
Then configure the `anomalithreatstream` fileset in Filebeat accordingly:
428+
Then configure the `anomalithreatstream` fileset in {beatname_uc} accordingly:
428429
[source,yaml]
429430
----
430431
- module: threatintel
@@ -449,11 +450,11 @@ Port number to use for the HTTP server.
449450

450451
*`var.secret`*::
451452

452-
Shared secret between the SDK and Filebeat, used to authenticate messages.
453+
Shared secret between the SDK and {beatname_uc}, used to authenticate messages.
453454

454455
*`var.ssl_certificate`*::
455456

456-
Path to the public SSL certificate for the HTTPS server. If unset, Filebeat
457+
Path to the public SSL certificate for the HTTPS server. If unset, {beatname_uc}
457458
will use unsecure HTTP connections.
458459

459460
*`var.ssl_key`*::
@@ -488,6 +489,94 @@ Anomali ThreatStream fields are mapped to the following ECS fields:
488489
[[a]]
489490
[small]#[1]: Field is used to derive a value for the ECS field but its original value is kept under `threatintel.anomalithreatstream`.#
490491

492+
[[recordedfuture]]
493+
[float]
494+
==== `recordedfuture` fileset settings
495+
496+
The `recordedfuture` fileset fetches intelligence from the Recorded Future Connect API.
497+
It supports `domain`, `hash`, `ip` and `url` data types.
498+
499+
To enable it you need to define the URL to fetch data from. You can construct this URL
500+
using the https://api.recordedfuture.com/index.html[Recorded Future API Explorer.] The URL
501+
must point to the `/search` endpoint and contain a suitable `limit`
502+
(how many records to return from a single request) and `fields` parameters.
503+
The `entity` and `timestamps` fields are required.
504+
505+
Sample configuration:
506+
[source,yaml]
507+
----
508+
- module: threatintel
509+
recordedfuture:
510+
enabled: true
511+
var.input: httpjson
512+
var.interval: 5m
513+
var.first_interval: 168h
514+
var.url: "https://api.recordedfuture.com/v2/ip/search?limit=200&fields=entity,timestamps,risk,intelCard,location&metadata=false"
515+
var.api_token: "<RF_TOKEN>"
516+
----
517+
518+
To fetch threat intelligence from multiple data types, you must define more than
519+
one instance of the module:
520+
[source,yaml]
521+
----
522+
- module: threatintel
523+
recordedfuture:
524+
enabled: true
525+
var.input: httpjson
526+
var.interval: 5m
527+
var.first_interval: 168h
528+
var.url: "https://api.recordedfuture.com/v2/ip/search?limit=200&fields=entity,timestamps,risk,intelCard,location&metadata=false"
529+
var.api_token: "<RF_TOKEN>"
530+
- module: threatintel
531+
recordedfuture:
532+
enabled: true
533+
var.input: httpjson
534+
var.interval: 1m
535+
var.first_interval: 168h
536+
var.url: "https://api.recordedfuture.com/v2/hash/search?limit=200&fields=entity,fileHashes,timestamps,risk,intelCard,location&metadata=false"
537+
var.api_token: "<RF_TOKEN>"
538+
----
539+
540+
*`var.url`*::
541+
542+
The URL of the API endpoint to connect with.
543+
544+
*`var.api_token`*::
545+
546+
The API token used to access Recorded Future API.
547+
548+
*`var.interval`*::
549+
550+
How often the API is polled for updated information.
551+
552+
*`var.first_interval`*::
553+
554+
How far back to search when retrieving events the first time {beatname_uc} starts up.
555+
After the first interval has passed the module itself will use the timestamp
556+
from the last response as the filter when retrieving new events.
557+
558+
*`var.proxy_url`*::
559+
560+
Optional URL to use as HTTP proxy.
561+
562+
563+
Recorded Future fields are mapped to the following ECS fields:
564+
565+
[options="header"]
566+
|=============================================================
567+
| Recorded Future fields | ECS Fields
568+
| entity.name | threatintel.indicator.{url,ip,domain,file.hash}
569+
| entity.type | threatintel.indicator.type
570+
| fileHashes | threatintel.indicator.file.hash
571+
| intelCard | event.reference
572+
| location.asn | threatintel.indicator.as.number
573+
| location.location | threatintel.indicator.geo
574+
| location.organization | threatintel.indicator.as.organization.name
575+
| risk.score | event.risk_score
576+
| timestamps.firstSeen | threatintel.indicator.first_seen
577+
| timestamps.lastSeen | threatintel.indicator.last_seen
578+
|=============================================================
579+
491580
:has-dashboards!:
492581

493582
[float]

‎filebeat/tests/system/test_modules.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -279,6 +279,7 @@ def clean_keys(obj):
279279
"threatintel.anomali",
280280
"threatintel.anomalithreatstream",
281281
"threatintel.malwarebazaar",
282+
"threatintel.recordedfuture",
282283
"snyk.vulnerabilities",
283284
"snyk.audit",
284285
"awsfargate.log",

‎x-pack/filebeat/filebeat.reference.yml‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2274,6 +2274,38 @@ filebeat.modules:
22742274
# var.ssl_certificate: path/to/server_ssl_cert.pem
22752275
# var.ssl_key: path/to/ssl_key.pem
22762276

2277+
recordedfuture:
2278+
enabled: true
2279+
2280+
# Input used for ingesting threat intel data
2281+
var.input: httpjson
2282+
2283+
# The interval to poll the API for updates
2284+
var.interval: 5m
2285+
2286+
# How far back in time to start fetching intelligence when run for the
2287+
# first time. Value must be in hours. Default: 168h (1 week).
2288+
var.first_interval: 168h
2289+
2290+
# The URL used for Threat Intel API calls.
2291+
# Must include the `limit` parameter and at least `entity` and `timestamps` fields.
2292+
# See the Connect API Explorer for a list of possible parameters.
2293+
#
2294+
# For `ip` entities:
2295+
var.url: "https://api.recordedfuture.com/v2/ip/search?limit=200&fields=entity,timestamps,risk,intelCard,location&metadata=false"
2296+
2297+
# For `domain` entities:
2298+
# var.url: "https://api.recordedfuture.com/v2/domain/search?limit=200&fields=entity,timestamps,risk,intelCard,location&metadata=false"
2299+
2300+
# For `hash` entities:
2301+
# var.url: "https://api.recordedfuture.com/v2/hash/search?limit=200&fields=entity,fileHashes,timestamps,risk,intelCard,location&metadata=false"
2302+
2303+
# For `url` entities:
2304+
# var.url: "https://api.recordedfuture.com/v2/url/search?limit=200&fields=entity,timestamps,risk&metadata=false"
2305+
2306+
# Set your API Token.
2307+
var.api_token: "<RF_TOKEN>"
2308+
22772309
#---------------------------- Apache Tomcat Module ----------------------------
22782310
- module: tomcat
22792311
log:

0 commit comments

Comments
 (0)