[Filebeat] Add network direction processor to Zeek and Suricata modules by legoguy1000 · Pull Request #24620 · elastic/beats

legoguy1000 · 2021-03-18T00:05:25Z

What does this PR do?

Adds the add_network_direction processor to the Zeek & Suricata module filesets using the internal_networks variable. The internal_networks variable is set to default to [ private ] and the add_network_direction will only run if that varaible is defined.

Why is it important?

The add_network_direction process adds the network.direction to documents which allows users to easily filter on traffic based on direction without needing to know specific IP subnets.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

How to test this PR locally

cd beats/x-pack/filebeat
GENERATE=true TESTING_FILEBEAT_MODULES=suricata,zeek mage -v pythonIntegTest

Related issues

N/A

Use cases

Screenshots

Logs

elasticmachine · 2021-03-18T00:10:11Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: P1llus commented: /test
Start Time: 2021-06-29T11:55:40.436+0000
Duration: 115 min 44 sec
Commit: 28e08cc

Test stats 🧪

Test	Results
Failed	0
Passed	14229
Skipped	2311
Total	16540

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	14229
Skipped	2311
Total	16540

elasticmachine · 2021-03-18T15:26:50Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

legoguy1000 · 2021-03-28T17:10:09Z

@andrewkroh I think I'm almost done with this PR but I think I found a bug and want to check with you first. I noticed on the Snort and Sonicwall JS scripts, for source and dest IP it was using the fld_append function. This causes the field to be an array/list even if its a single value. When I tried to add the add_network_direction processor, it didn't work until I change the JS function to fld_set so its just a single value. My thought is that if it is an array/list, the processor should just use the first value. Thoughts??

mergify · 2021-04-07T16:28:16Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b add-network-directions upstream/add-network-directions
git merge upstream/master
git push upstream add-network-directions

mergify · 2021-04-08T19:19:11Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b add-network-directions upstream/add-network-directions
git merge upstream/master
git push upstream add-network-directions

This reverts commit 1c5722d9f1f6cd5370ef3f3d7a882e49731a4e3d.

P1llus · 2021-05-19T07:43:42Z

run tests

mergify · 2021-05-19T12:01:40Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b add-network-directions upstream/add-network-directions
git merge upstream/master
git push upstream add-network-directions

P1llus · 2021-06-18T15:50:07Z

/test

mergify · 2021-06-23T15:28:15Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b add-network-directions upstream/add-network-directions
git merge upstream/master
git push upstream add-network-directions

P1llus · 2021-06-24T12:22:44Z

/test

P1llus · 2021-06-24T13:08:38Z

Is this okay to merge now @andrewkroh? We moved everything out of ingest pipelines and into beats, so that they offer backwards compatibility as well.

andrewkroh

Modifying snort and sonicwall will cause issues in the future since those are generated modules from https://github.com/adriansr/nwdevice2filebeat. This feature could be added to the generator then all of the RSA would get this processor. IMO I think we should remove the changes from those two modules and proceed with this PR.

andrewkroh · 2021-06-24T13:40:55Z

x-pack/filebeat/module/sonicwall/firewall/config/liblogparser.js

    "city.src": {to:[{field: "source.geo.city_name", setter: fld_set}]},
-    "daddr": {convert: to_ip, to:[{field: "destination.ip", setter: fld_append},{field: "related.ip", setter: fld_append}]},
-    "daddr_v6": {convert: to_ip, to:[{field: "destination.ip", setter: fld_append},{field: "related.ip", setter: fld_append}]},
+    "daddr": {convert: to_ip, to:[{field: "destination.ip", setter: fld_set},{field: "related.ip", setter: fld_append}]},


Why set and not append? Normally related.ip is an array. This seems unrelated to adding network direction.

i don't recall removing "daddr_v6": {convert: to_ip, to:[{field: "destination.ip", setter: fld_append},{field: "related.ip", setter: fld_append}]},. That is probably an accident. As for the change from {field: "destination.ip", setter: fld_append} to {field: "destination.ip", setter: fld_set}, is because the network_direction processor requires a string field not an array.

The append for related.ip is unchanged.

I am reverting the changes for Snort and Sonicwall.

Thanks for pointing out at source.ip and destination.ip should be scalars. I think we should make that change just not in this PR. Would you please open an issue for this? It does look like you reverted most of the snort/sonicwall changes, but a small part of snort is still changed.

P1llus · 2021-06-29T11:55:26Z

/test

…es (#24620) * Add network direction processor to zeek and suricata module * Add Snort & Sonicwall * update changelog * use ES network_direction processor * Revert "use ES network_direction processor" This reverts commit 1c5722d9f1f6cd5370ef3f3d7a882e49731a4e3d. * update docs with new variable * Removed Snort and Sonicwall * update docs * missed one Co-authored-by: Marius Iversen <marius.iversen@elastic.co> (cherry picked from commit 9e670f7)

…arwin-arm64 * upstream/master: (295 commits) Update urllib to 1.26.5. (elastic#26380) Update golang.org/x/crypto (elastic#26448) [Filebeat] Update Fortinet Ingest Pipeline (elastic#24816) Move parsers outside of filestream input so others can use them as well (elastic#26541) [Filebeat] Fix `threatintel.indicator.url.full` field not populating (elastic#26508) [Filebeat] Add network direction processor to Zeek and Suricata modules (elastic#24620) Logging code cleanup related to Nomad auto-discovery (elastic#26498) [Metricbeat] Add Couchbase's Sync Gateway module (elastic#25599) Refactor add_cloud_metadata to handle ECS fields easier (elastic#26438) [Elastic Agent] Improper casting of int64 (elastic#26520) [Elastic Agent] Enable configuring monitoring namespace (elastic#26439) [Heartbeat] configure permissions for synthetics config (elastic#26393) Osquerybeat: set the raw index name to supress the timestamp suffix (elastic#26545) [Heartbeat] add screenshots config to synthetics (elastic#26455) [Elastic Agent] Use http2 to connect to Fleet Server. (elastic#26474) Remove all docs about Beats central management (elastic#26399) update data.json for gcp billing (elastic#26506) Skip x-pack metricbeat tests (elastic#26537) [Elastic Agent] Fix issue with FLEET_CA not being used with Fleet Server in container (elastic#26529) Add changelog entry for elastic#26224 (elastic#26531) ...

…es (#24620) (#26568) * Add network direction processor to zeek and suricata module * Add Snort & Sonicwall * update changelog * use ES network_direction processor * Revert "use ES network_direction processor" This reverts commit 1c5722d9f1f6cd5370ef3f3d7a882e49731a4e3d. * update docs with new variable * Removed Snort and Sonicwall * update docs * missed one Co-authored-by: Marius Iversen <marius.iversen@elastic.co> (cherry picked from commit 9e670f7) Co-authored-by: Alex Resnick <adr8292@gmail.com>

* master: (25 commits) fix: Force PLATFORMS environment variable when we build Elastic Agent dependencies on arm64 (elastic#26415) macos for metricbeat to run in the extended meta-stage (elastic#26573) Packaging: add arm7 platform in the main pipeline (elastic#26575) [Heartbeat] Skip flakey timer queue test (elastic#26592) Update to "read_pipeline" permission (elastic#26465) (elastic#26580) API keys do not reflect the need for read_pipeline (elastic#26466) (elastic#26582) Add Fleet agent.id to Agent monitoring data (elastic#26548) Add kinesis metricset (elastic#25989) Refactor of system/memory metricset (elastic#26334) Introduce httpcommon package in libbeat (add support for Proxy) (elastic#25219) [Filebeat] change multiline configuration in awss3 input to parsers (elastic#25873) docs: Hint for the error "Error extracting container id" (elastic#25824) [Docs] Fixed metricbeat redis exported field CPU descriptions (elastic#25846) (elastic#26496) Update urllib to 1.26.5. (elastic#26380) Update golang.org/x/crypto (elastic#26448) [Filebeat] Update Fortinet Ingest Pipeline (elastic#24816) Move parsers outside of filestream input so others can use them as well (elastic#26541) [Filebeat] Fix `threatintel.indicator.url.full` field not populating (elastic#26508) [Filebeat] Add network direction processor to Zeek and Suricata modules (elastic#24620) Logging code cleanup related to Nomad auto-discovery (elastic#26498) ...

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Mar 18, 2021

legoguy1000 force-pushed the add-network-directions branch from 3d36054 to 415e657 Compare March 18, 2021 00:11

legoguy1000 changed the title ~~[Filebeat] Add network direction processor to Zeek, Sonicwall (Firewall), Suricata modules~~ [Filebeat] Add network direction processor to Zeek and Suricata modules Mar 18, 2021

andresrc added the Team:Security-External Integrations label Mar 18, 2021

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Mar 18, 2021

legoguy1000 force-pushed the add-network-directions branch from 415e657 to 7095b42 Compare March 18, 2021 15:25

legoguy1000 marked this pull request as ready for review March 18, 2021 15:26

legoguy1000 marked this pull request as draft March 18, 2021 15:27

legoguy1000 force-pushed the add-network-directions branch from 7095b42 to 3e7aad7 Compare March 19, 2021 13:23

legoguy1000 mentioned this pull request Mar 22, 2021

Add network.direction classification to rsa2elk modules #23114

Closed

legoguy1000 force-pushed the add-network-directions branch 3 times, most recently from 12e30f0 to 48e6433 Compare March 24, 2021 00:55

legoguy1000 force-pushed the add-network-directions branch 2 times, most recently from 75e604e to 961dc62 Compare April 2, 2021 12:21

legoguy1000 marked this pull request as ready for review April 2, 2021 12:21

legoguy1000 changed the title ~~[Filebeat] Add network direction processor to Zeek and Suricata modules~~ [Filebeat] Add network direction processor to Zeek, Suricata, Snort, Sonicwall modules Apr 2, 2021

legoguy1000 force-pushed the add-network-directions branch from 3eab959 to 9471a69 Compare April 7, 2021 16:58

legoguy1000 marked this pull request as draft April 28, 2021 15:57

legoguy1000 force-pushed the add-network-directions branch 3 times, most recently from 6ceaf3e to 1c5722d Compare April 28, 2021 20:10

legoguy1000 marked this pull request as ready for review April 28, 2021 20:10

legoguy1000 mentioned this pull request Apr 28, 2021

Fix community_id processor so that ports greater than 65535 aren't valid #25409

Merged

6 tasks

andrewkroh assigned P1llus May 3, 2021

legoguy1000 added 2 commits May 18, 2021 01:59

use ES network_direction processor

62c7a6f

Revert "use ES network_direction processor"

0966bf5

This reverts commit 1c5722d9f1f6cd5370ef3f3d7a882e49731a4e3d.

legoguy1000 force-pushed the add-network-directions branch from 2fcdd43 to 0cd2824 Compare May 18, 2021 02:00

update docs with new variable

7a4381c

legoguy1000 force-pushed the add-network-directions branch from 0cd2824 to 7a4381c Compare May 18, 2021 02:03

Merge branch 'master' into add-network-directions

f39206f

Merge branch 'master' into add-network-directions

854d181

andrewkroh reviewed Jun 24, 2021

View reviewed changes

legoguy1000 changed the title ~~[Filebeat] Add network direction processor to Zeek, Suricata, Snort, Sonicwall modules~~ [Filebeat] Add network direction processor to Zeek and Suricata modules Jun 25, 2021

legoguy1000 added 2 commits June 25, 2021 11:54

Removed Snort and Sonicwall

0ae35f0

update docs

6b8ad36

legoguy1000 mentioned this pull request Jun 25, 2021

Modules using RSA converter are setting Scalar fields to Arrays #26504

Closed

missed one

28e08cc

legoguy1000 force-pushed the add-network-directions branch from e5086b0 to 28e08cc Compare June 25, 2021 14:18

andrewkroh approved these changes Jun 29, 2021

View reviewed changes

P1llus merged commit 9e670f7 into elastic:master Jun 29, 2021

P1llus added the backport-v7.14.0 Automated backport with mergify label Jun 29, 2021

mergify bot mentioned this pull request Jun 29, 2021

[7.x](backport #24620) [Filebeat] Add network direction processor to Zeek and Suricata modules #26568

Merged

legoguy1000 deleted the add-network-directions branch June 29, 2021 14:06

Conversation

legoguy1000 commented Mar 18, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Why is it important?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

Uh oh!

elasticmachine commented Mar 18, 2021 • edited by jenkins-beats-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

Trends 🧪

💚 Flaky test report

Test stats 🧪

Uh oh!

elasticmachine commented Mar 18, 2021

Uh oh!

legoguy1000 commented Mar 28, 2021

Uh oh!

mergify bot commented Apr 7, 2021

Uh oh!

mergify bot commented Apr 8, 2021

Uh oh!

P1llus commented May 19, 2021

Uh oh!

mergify bot commented May 19, 2021

Uh oh!

P1llus commented Jun 18, 2021

Uh oh!

mergify bot commented Jun 23, 2021

Uh oh!

P1llus commented Jun 24, 2021

Uh oh!

P1llus commented Jun 24, 2021

Uh oh!

andrewkroh left a comment

Choose a reason for hiding this comment

Uh oh!

andrewkroh Jun 24, 2021

Choose a reason for hiding this comment

Uh oh!

legoguy1000 Jun 24, 2021

Choose a reason for hiding this comment

Uh oh!

legoguy1000 Jun 24, 2021

Choose a reason for hiding this comment

Uh oh!

legoguy1000 Jun 25, 2021

Choose a reason for hiding this comment

Uh oh!

legoguy1000 Jun 25, 2021

Choose a reason for hiding this comment

Uh oh!

andrewkroh Jun 25, 2021

Choose a reason for hiding this comment

Uh oh!

P1llus commented Jun 29, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

legoguy1000 commented Mar 18, 2021 •

edited

Loading

elasticmachine commented Mar 18, 2021 •

edited by jenkins-beats-ci bot

Loading