Skip to content

[Filebeat] Fix threatintel.indicator.url.full field not populating#26508

Merged
P1llus merged 4 commits intoelastic:masterfrom
legoguy1000:26351-threatintel-url
Jun 29, 2021
Merged

[Filebeat] Fix threatintel.indicator.url.full field not populating#26508
P1llus merged 4 commits intoelastic:masterfrom
legoguy1000:26351-threatintel-url

Conversation

@legoguy1000
Copy link
Copy Markdown
Contributor

@legoguy1000 legoguy1000 commented Jun 27, 2021 •
edited
Loading

What does this PR do?

Properly sets the threatintel.indicator.url.full field for URL events for threatintel.abuseurl,misp,anomali,anomalithreatstream,otx

Why is it important?

url.full field wasn't being properly set.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

  • [ ]

How to test this PR locally

cd beats/x-pack/filebeat
TESTING_FILEBEAT_MODULES=threatintel mage -v pythonIntegTest

Related issues

Screenshots

Logs

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jun 27, 2021
@legoguy1000 legoguy1000 marked this pull request as ready for review June 27, 2021 16:37
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Jun 27, 2021 •
edited by jenkins-beats-ci bot
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: P1llus commented: /test

  • Start Time: 2021-06-29T12:21:36.503+0000

  • Duration: 109 min 21 sec

  • Commit: f02b12a

Test stats 🧪

Test Results
Failed 0
Passed 7400
Skipped 1201
Total 8601

Trends 🧪

Image of Build Times

Image of Tests

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test Results
Failed 0
Passed 7400
Skipped 1201
Total 8601

@legoguy1000 legoguy1000 force-pushed the 26351-threatintel-url branch from a115ff0 to 147cded Compare June 29, 2021 10:44
@P1llus
Copy link
Copy Markdown
Member

P1llus commented Jun 29, 2021 •
edited
Loading

I think there is a few different ways to approach the same issue, but I would rather we depend more on the fields generated by the uri_parts processor. Instead of removing the remove_if_sucessfull setting, we should copy the value from url.original, so all the threatintel filesets should use uri_parts processor in combination with a set processor to copy_from the threatintel.indicator.url.original field. Do you think we could make that consistent over all the TI filesets?

- uri_parts:
    field: json.url
    target_field: threatintel.indicator.url
    keep_original: true
    remove_if_sucessful: true
    if: 'ctx.json.url != null'

- set:
    field: threatintel.indicator.url.full
    copy_from: threatintel.indicator.url.original
    ignore_empty_value: true

@P1llus
Copy link
Copy Markdown
Member

P1llus commented Jun 29, 2021

/test

@P1llus P1llus self-requested a review June 29, 2021 12:33
@P1llus P1llus self-assigned this Jun 29, 2021
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Jun 29, 2021

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jun 29, 2021
@P1llus P1llus added Filebeat Filebeat needs_integration_sync Changes in this PR need synced to elastic/integrations. and removed needs_integration_sync Changes in this PR need synced to elastic/integrations. labels Jun 29, 2021
@P1llus P1llus merged commit c45aba5 into elastic:master Jun 29, 2021
@P1llus P1llus added the backport-v7.14.0 Automated backport with mergify label Jun 29, 2021
mergify bot pushed a commit that referenced this pull request Jun 29, 2021
@legoguy1000
[Filebeat] Fix threatintel.indicator.url.full field not populating (#…
ba60688 
…26508)

* #26351: Fix Threat Intel Full URL field

* update changelog

* remove commented items

* updated pipelines per comments

(cherry picked from commit c45aba5)
@mergify mergify bot mentioned this pull request Jun 29, 2021
Merged
@legoguy1000 legoguy1000 deleted the 26351-threatintel-url branch June 29, 2021 14:14
P1llus pushed a commit that referenced this pull request Jun 29, 2021
@mergify @legoguy1000
[Filebeat] Fix threatintel.indicator.url.full field not populating (#…
4c8c1d0 
…26508) (#26569)

* #26351: Fix Threat Intel Full URL field

* update changelog

* remove commented items

* updated pipelines per comments

(cherry picked from commit c45aba5)

Co-authored-by: Alex Resnick <adr8292@gmail.com>
v1v added a commit to v1v/beats that referenced this pull request Jun 29, 2021
@v1v
Merge remote-tracking branch 'upstream/master' into feature/support-d…
2c9f25c 
…arwin-arm64

* upstream/master: (295 commits)
  Update urllib to 1.26.5. (elastic#26380)
  Update golang.org/x/crypto (elastic#26448)
  [Filebeat] Update Fortinet Ingest Pipeline (elastic#24816)
  Move parsers outside of filestream input so others can use them as well (elastic#26541)
  [Filebeat] Fix `threatintel.indicator.url.full` field not populating (elastic#26508)
  [Filebeat] Add network direction processor to Zeek and Suricata modules (elastic#24620)
  Logging code cleanup related to Nomad auto-discovery (elastic#26498)
  [Metricbeat] Add Couchbase's Sync Gateway module (elastic#25599)
  Refactor add_cloud_metadata to handle ECS fields easier (elastic#26438)
  [Elastic Agent] Improper casting of int64 (elastic#26520)
  [Elastic Agent] Enable configuring monitoring namespace (elastic#26439)
  [Heartbeat] configure permissions for synthetics config (elastic#26393)
  Osquerybeat: set the raw index name to supress the timestamp suffix (elastic#26545)
  [Heartbeat] add screenshots config to synthetics (elastic#26455)
  [Elastic Agent] Use http2 to connect to Fleet Server. (elastic#26474)
  Remove all docs about  Beats central management (elastic#26399)
  update data.json for gcp billing (elastic#26506)
  Skip x-pack metricbeat tests (elastic#26537)
  [Elastic Agent] Fix issue with FLEET_CA not being used with Fleet Server in container (elastic#26529)
  Add changelog entry for  elastic#26224 (elastic#26531)
  ...
@legoguy1000 legoguy1000 mentioned this pull request Jun 30, 2021
Closed
mdelapenya added a commit to mdelapenya/beats that referenced this pull request Jun 30, 2021
@mdelapenya
Merge branch 'master' into archive-system-tests
218ee9a 
* master: (25 commits)
  fix: Force PLATFORMS environment variable when we build Elastic Agent dependencies on arm64 (elastic#26415)
  macos for metricbeat to run in the extended meta-stage (elastic#26573)
  Packaging: add arm7 platform in the main pipeline (elastic#26575)
  [Heartbeat] Skip flakey timer queue test (elastic#26592)
  Update to "read_pipeline" permission (elastic#26465) (elastic#26580)
  API keys do not reflect the need for read_pipeline (elastic#26466) (elastic#26582)
  Add Fleet agent.id to Agent monitoring data (elastic#26548)
  Add kinesis metricset (elastic#25989)
  Refactor of system/memory metricset (elastic#26334)
  Introduce httpcommon package in libbeat (add support for Proxy) (elastic#25219)
  [Filebeat] change multiline configuration in awss3 input to parsers (elastic#25873)
  docs: Hint for the error "Error extracting container id" (elastic#25824)
  [Docs] Fixed metricbeat redis exported field CPU descriptions (elastic#25846) (elastic#26496)
  Update urllib to 1.26.5. (elastic#26380)
  Update golang.org/x/crypto (elastic#26448)
  [Filebeat] Update Fortinet Ingest Pipeline (elastic#24816)
  Move parsers outside of filestream input so others can use them as well (elastic#26541)
  [Filebeat] Fix `threatintel.indicator.url.full` field not populating (elastic#26508)
  [Filebeat] Add network direction processor to Zeek and Suricata modules (elastic#24620)
  Logging code cleanup related to Nomad auto-discovery (elastic#26498)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@P1llus P1llus P1llus approved these changes

Assignees

@P1llus P1llus

Labels

backport-v7.14.0 Automated backport with mergify Filebeat Filebeat

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@legoguy1000 @elasticmachine @P1llus @jamiehynds