[Filebeat] Threatintel compatibility updates by rylnd · Pull Request #27323 · elastic/beats

rylnd · 2021-08-11T19:32:19Z

What does this PR do?

This is a draft of the work discussed between @P1llus and myself. While the plan is to migrate most of this functionality to integration packages, there are some incompatibilities/logical errors that can/should be cleaned up in these modules. At a high level, we've outlined the following changes:

removing indicator.domain, which has been deprecated in favor of indicator.url.domain
moving from event.reference to indicator.reference, as it's useful for investigation and event.* fields aren't copied as part of enrichment
~~moving from threatintel.indicator to threat.indicator~~
~~moving any non-ECS threatintel.[MODULE] fieldsets to not be nested under threatintel~~

Why is it important?

These changes will allow filebeat 7.15 users to ingest CTI data compatible with ECS 1.11.

Remaining work

verify that our geo_point fields are being ingested correctly
updating of this field to address https://github.com/elastic/security-team/issues/1494
- the abuseurl module uses urlhaus_reference to populate its analogous reference field. The test data doesn't indicate so, but if that exists for abusemalware as well, that's a simple fix. If that's not present, the best solution is likely to drop that field from the module.
verification from a beats developer 😬

Work determined to be unnecessary for 7.15

updating all threatintel modules to populate threat.indicator instead of threatintel.indicator
moving threatintel.[MODULE] fieldsets to the root level

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

This only covers modules starting with an a; the rest will follow shortly. In general, these changes address the following goals: * preference for indicator.url.domain, and deprecation of indicator.domain * moving from event.reference to indicator.reference

Along with conditional checks to ensure we're not overwriting the relevant uri_parts data from earlier in the pipeline.

elasticmachine · 2021-08-11T19:36:12Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2021-09-07T13:53:40.492+0000
Duration: 101 min 47 sec
Commit: 5d7443c

Test stats 🧪

Test	Results
Failed	0
Passed	15220
Skipped	2314
Total	17534

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	15220
Skipped	2314
Total	17534

P1llus · 2021-08-31T08:05:38Z

Adding a comment here, some of the remaining work will be in a separate PR. We wont be changing threatintel.* to threat.*, or removing the nested fields before 7.16.

Will work with @rylnd to get this merged this week

elasticmachine · 2021-09-02T14:41:27Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

P1llus · 2021-09-07T06:14:16Z

/test

* First pass on updating filebeat threatintel logic for ECS 1.11 This only covers modules starting with an a; the rest will follow shortly. In general, these changes address the following goals: * preference for indicator.url.domain, and deprecation of indicator.domain * moving from event.reference to indicator.reference * Move remaining modules from indicator.domain -> indicator.url.domain Along with conditional checks to ensure we're not overwriting the relevant uri_parts data from earlier in the pipeline. * Update indicator.reference in relevant modules * Fix missing prefix in target field * linting and apply new testfiles * Run `make update` in filebeat * fixing duplicate fields * mage fmt update * linting Co-authored-by: Marius Iversen <marius.iversen@elastic.co> (cherry picked from commit 4be2694) # Conflicts: # x-pack/filebeat/module/threatintel/abuseurl/ingest/pipeline.yml # x-pack/filebeat/module/threatintel/anomali/ingest/pipeline.yml

…27777) * [Filebeat] Threatintel compatibility updates (#27323) * First pass on updating filebeat threatintel logic for ECS 1.11 This only covers modules starting with an a; the rest will follow shortly. In general, these changes address the following goals: * preference for indicator.url.domain, and deprecation of indicator.domain * moving from event.reference to indicator.reference * Move remaining modules from indicator.domain -> indicator.url.domain Along with conditional checks to ensure we're not overwriting the relevant uri_parts data from earlier in the pipeline. * Update indicator.reference in relevant modules * Fix missing prefix in target field * linting and apply new testfiles * Run `make update` in filebeat * fixing duplicate fields * mage fmt update * linting Co-authored-by: Marius Iversen <marius.iversen@elastic.co> (cherry picked from commit 4be2694) # Conflicts: # x-pack/filebeat/module/threatintel/abuseurl/ingest/pipeline.yml # x-pack/filebeat/module/threatintel/anomali/ingest/pipeline.yml * fixing mergify conflicts Co-authored-by: Ryland Herrick <ryalnd@gmail.com> Co-authored-by: Marius Iversen <marius.iversen@elastic.co>

…27778) * [Filebeat] Threatintel compatibility updates (#27323) * First pass on updating filebeat threatintel logic for ECS 1.11 This only covers modules starting with an a; the rest will follow shortly. In general, these changes address the following goals: * preference for indicator.url.domain, and deprecation of indicator.domain * moving from event.reference to indicator.reference * Move remaining modules from indicator.domain -> indicator.url.domain Along with conditional checks to ensure we're not overwriting the relevant uri_parts data from earlier in the pipeline. * Update indicator.reference in relevant modules * Fix missing prefix in target field * linting and apply new testfiles * Run `make update` in filebeat * fixing duplicate fields * mage fmt update * linting Co-authored-by: Marius Iversen <marius.iversen@elastic.co> (cherry picked from commit 4be2694) # Conflicts: # x-pack/filebeat/module/threatintel/abuseurl/ingest/pipeline.yml # x-pack/filebeat/module/threatintel/anomali/ingest/pipeline.yml * fixing mergify conflicts Co-authored-by: Ryland Herrick <ryalnd@gmail.com> Co-authored-by: Marius Iversen <marius.iversen@elastic.co>

* master: (39 commits) [Heartbeat] Move JSON tests from python->go (elastic#27816) docs: simplify permissions for Dockerfile COPY (elastic#27754) Osquerybeat: Fix osquery logger plugin severy levels mapping (elastic#27789) [Filebeat] Update compatibility function to remove processor description on ES < 7.9.0 (elastic#27774) warn log entry and no validation failure when both queue_url and buck… (elastic#27612) libbeat/cmd/instance: ensure test config file has appropriate permissions (elastic#27178) [Heartbeat] Add httpcommon options to ZipURL (elastic#27699) Add a header round tripper option to httpcommon (elastic#27509) [Elastic Agent] Add validation to ensure certificate paths are absolute. (elastic#27779) Rename dashboards according to module.yml files for master (elastic#27749) Refactor vagrantfile, add scripts for provisioning with docker/kind (elastic#27726) Accept syslog dates with leading 0 (elastic#27775) [Filebeat] Add timezone config option to decode_cef and syslog input (elastic#27727) [Filebeat] Threatintel compatibility updates (elastic#27323) Add support for ephemeral containers in elastic agent dynamic provider (elastic#27707) [Filebeat] Integration tests in CI for AWS-S3 input (elastic#27491) Fix flakyness of TestFilestreamEmptyLine (elastic#27705) [Filebeat] kafka v2 using parsers (elastic#27335) Update Kafka version parsing / supported range (elastic#27720) Update Sarama to 1.29.1 (elastic#27717) ...

* First pass on updating filebeat threatintel logic for ECS 1.11 This only covers modules starting with an a; the rest will follow shortly. In general, these changes address the following goals: * preference for indicator.url.domain, and deprecation of indicator.domain * moving from event.reference to indicator.reference * Move remaining modules from indicator.domain -> indicator.url.domain Along with conditional checks to ensure we're not overwriting the relevant uri_parts data from earlier in the pipeline. * Update indicator.reference in relevant modules * Fix missing prefix in target field * linting and apply new testfiles * Run `make update` in filebeat * fixing duplicate fields * mage fmt update * linting Co-authored-by: Marius Iversen <marius.iversen@elastic.co>

rylnd added 3 commits August 11, 2021 14:01

Move remaining modules from indicator.domain -> indicator.url.domain

a65bb59

Along with conditional checks to ensure we're not overwriting the relevant uri_parts data from earlier in the pipeline.

Update indicator.reference in relevant modules

9c44a14

rylnd added enhancement v7.15.0 labels Aug 11, 2021

rylnd requested a review from P1llus August 11, 2021 19:32

rylnd assigned P1llus Aug 11, 2021

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Aug 11, 2021

Fix missing prefix in target field

2b82431

jsoriano added the Team:Security-External Integrations label Aug 20, 2021

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Aug 20, 2021

linting and apply new testfiles

d36ae86

P1llus marked this pull request as ready for review September 2, 2021 14:41

Run make update in filebeat

221da56

P1llus added 3 commits September 7, 2021 10:23

fixing duplicate fields

b1bbd3e

mage fmt update

4e03962

linting

5d7443c

marc-gr approved these changes Sep 7, 2021

View reviewed changes

P1llus removed their request for review September 7, 2021 14:16

P1llus merged commit 4be2694 into master Sep 7, 2021

P1llus added backport-v7.15.0 Automated backport with mergify backport-v7.16.0 Automated backport with mergify labels Sep 7, 2021

mergify bot mentioned this pull request Sep 7, 2021

[7.x](backport #27323) [Filebeat] Threatintel compatibility updates #27777

Merged

mergify bot mentioned this pull request Sep 7, 2021

[7.15](backport #27323) [Filebeat] Threatintel compatibility updates #27778

Merged

rylnd deleted the threatintel-compatibility-updates branch September 7, 2021 18:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Filebeat] Threatintel compatibility updates#27323

[Filebeat] Threatintel compatibility updates#27323
P1llus merged 9 commits intomasterfrom
threatintel-compatibility-updates

rylnd commented Aug 11, 2021 •

edited

Loading

Uh oh!

elasticmachine commented Aug 11, 2021 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

Trends 🧪

Test stats 🧪

Uh oh!

P1llus commented Aug 31, 2021

Uh oh!

elasticmachine commented Sep 2, 2021

Uh oh!

P1llus commented Sep 7, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

rylnd commented Aug 11, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Why is it important?

Remaining work

Work determined to be unnecessary for 7.15

Checklist

Related issues

Uh oh!

elasticmachine commented Aug 11, 2021 • edited by jenkins-beats-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

Trends 🧪

💚 Flaky test report

Test stats 🧪

Uh oh!

P1llus commented Aug 31, 2021

Uh oh!

elasticmachine commented Sep 2, 2021

Uh oh!

P1llus commented Sep 7, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

rylnd commented Aug 11, 2021 •

edited

Loading

elasticmachine commented Aug 11, 2021 •

edited by jenkins-beats-ci bot

Loading