Skip to content

Cherry-pick #24570 to 7.x: [Filebeat] Add Malware Bazaar to Threat Intel Module#25177

Merged
P1llus merged 5 commits intoelastic:7.xfrom
P1llus:backport_24570_7.x
Apr 20, 2021
Merged

Cherry-pick #24570 to 7.x: [Filebeat] Add Malware Bazaar to Threat Intel Module#25177
P1llus merged 5 commits intoelastic:7.xfrom
P1llus:backport_24570_7.x

Conversation

@P1llus
Copy link
Copy Markdown
Member

@P1llus P1llus commented Apr 20, 2021 •
edited by zube bot
Loading

Cherry-pick of PR #24570 to 7.x branch. Original message:

Closes https://github.com/elastic/cti-team/issues/33

What does this PR do?

This PR adds the Malware Bazaar threat feed to the threat intel module of Filebeat.

Why is it important?

Malware Bazaar provides rich file metadata about malware that can assist cyber intelligence analysts, threat hunters, and incident responders during incident response and ongoing security operations.

Currently, the threat intel module for Filebeat did not have the data provided by Malware Bazaar.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

  • Validated that ./filebeat setup imports the saved search, visualization, and dashboard
  • Validated that data provided by the Threat Intel Filebeat module populates the dashboards

How to test this PR locally

This can be tested by:

cd beats/x-pack/filebeat
mage update && mage build
./filebeat modules enable threatintel
# if using Elastic Cloud, update filebeat.yml to cloud.id and cloud.auth
./filebeat setup -E setup.dashboards.directory=./build/kibana
./filebeat

Go to KIbana -> dashboards -> apply the threat intel tag

Related issues

Resolves #24569

Use cases

Threat hunting, security operations, and intelligence analysis.

Screenshots

filebeat-threatintel-malware-bazaar

Logs

Original data from source

  {
            "sha256_hash": "ff4ad9465d1312e24dff76c47ddb519bc46f501239e5813d4d70ac6bd7c70638",
            "sha3_384_hash": "40ce04e3e695d10b42d9ca1026476aa6076923a911652c6f263f866c8ce86e9bf2b03e1626f3ce20f6d4f1649ed3f54a",
            "sha1_hash": "eb519d0338c102103d0e8467a5dba55c99a40233",
            "md5_hash": "4f0f5e2c6bc8d65d14ff201d799824ae",
            "first_seen": "2021-03-16 16:07:32",
            "last_seen": null,
            "file_name": "tarifvertrag_chemie_bayern_entgelttabelle.js",
            "file_size": 2939,
            "file_type_mime": "text\/plain",
            "file_type": "js",
            "reporter": "dor0n1",
            "origin_country": "DE",
            "anonymous": 0,
            "signature": null,
            "imphash": null,
            "tlsh": "6E51A6E5B149E15101073734255FE86CB9B3A949D40ED420D749D7DF28B603D4E27A9E",
            "telfhash": null,
            "ssdeep": "48:M27hDQrbyxd8CROAdiQD0GMFJ0QXtAkiQKXJXWcaiMKjTlLJ\/9zhjOquc9+ac03V:5Vwbg8BxHpX4XMAF96qfmaMJ0ncTwy1w",
            "tags": [
                "gootloader"
            ],
            "code_sign": [],
            "intelligence": {
                "clamav": null,
                "downloads": "15",
                "uploads": "1",
                "mail": null
            }
        },

Data from module

{
  "_index": "filebeat-8.0.0-2021.03.16-000001",
  "_type": "_doc",
  "_id": "38de8b728560e8705f707d4607706d73f04b4d50f925608ffed7b873e66e9b67",
  "_version": 1,
  "_score": null,
  "_source": {
    "agent": {
      "name": "c02zc1eklvdr",
      "id": "4aa831e8-28aa-4037-ab92-02cdcdf0e6ae",
      "ephemeral_id": "c6efb3b7-007f-42ad-b2bf-c34fb577a9a7",
      "type": "filebeat",
      "version": "8.0.0"
    },
    "fileset": {
      "name": "malwarebazaar"
    },
    "threatintel": {
      "malwarebazaar": {
        "intelligence": {
          "mail": {
            "CH": "low",
            "Generic": "high"
          },
          "downloads": 22,
          "uploads": 1
        },
        "tags": [
          "doc"
        ],
        "anonymous": 0,
        "code_sign": []
      },
      "indicator": {
        "geo": {
          "country_iso_code": "FR"
        },
        "first_seen": "2021-03-16T16:05:36.000Z",
        "file": {
          "extension": "doc",
          "size": 167472,
          "mime_type": "text/rtf",
          "pe": {},
          "name": "Purchase order.doc",
          "hash": {
            "sha1": "1471a4b801b065b315f130a5d512dc7597c71657",
            "sha384": "c70be406ff428593018f85cc8363635c259506197972371e24f7161011a8a46dca83ac0268ca80d8a92d6ccf93fcc9a1",
            "sha256": "623119afef4f91cf052823dd4d6b1ec9f04dc7fbe76a313035e0e4aa00888454",
            "tlsh": "30F36DA8E991CDD4CBCFC9D94A1E3A952033FA79C6C36C964438F3F50B9267F4A16850",
            "ssdeep": "3072:Cv6q8aN9Zts1unvhfKwGt7D2l8hnoo+ixxpbeBJTJzEmFhThl4O/b+8OGo3VaESF:CvkaN9Zts6vVGt/2l8nxpKJTJznhdDTP",
            "md5": "a24571eb4baea7a5b67480d524668617"
          }
        },
        "provider": "cocaman",
        "type": "file"
      }
    },
    "tags": [
      "threatintel-malwarebazaar",
      "forwarded"
    ],
    "input": {
      "type": "httpjson"
    },
    "@timestamp": "2021-03-16T16:37:32.980Z",
    "ecs": {
      "version": "1.6.0"
    },
    "related": {
      "hash": [
        "a24571eb4baea7a5b67480d524668617",
        "623119afef4f91cf052823dd4d6b1ec9f04dc7fbe76a313035e0e4aa00888454",
        "3072:Cv6q8aN9Zts1unvhfKwGt7D2l8hnoo+ixxpbeBJTJzEmFhThl4O/b+8OGo3VaESF:CvkaN9Zts6vVGt/2l8nxpKJTJznhdDTP"
      ]
    },
    "service": {
      "type": "threatintel"
    },
    "event": {
      "ingested": "2021-03-16T16:37:34.071668028Z",
      "created": "2021-03-16T16:37:32.980Z",
      "kind": "enrichment",
      "module": "threatintel",
      "category": "threat",
      "type": "indicator",
      "dataset": "threatintel.malwarebazaar"
    }
  },
  "fields": {
    "event.ingested": [
      "2021-03-16T16:37:34.071Z"
    ],
    "@timestamp": [
      "2021-03-16T16:37:32.980Z"
    ],
    "event.created": [
      "2021-03-16T16:37:32.980Z"
    ]
  },
  "sort": [
    1615912652980
  ]
},

@peasead @dcode
@P1llus @adriansr @andrewkroh
[Filebeat] Add Malware Bazaar to Threat Intel Module (elastic#24570)
8860a8f 
* fixed

* update

* Set content-type to form encoded

* update config

* dashboard and config work

* test data

* updated docs

* dashboard screenshot

* image location

* ran mage fmt update

* updated changelog

* mage fmt

* ran the tests

* mage fmt after testing

* added timestamp fix

* fixed related.hash and tlsh

* added elf.telfhash

* mage'd everything

* updated dashboard

* Mage update

* update snyk build to ignore timestamps

* fixing test_modules.py timestamp

* Add missing comma to array list item

Co-authored-by: Derek Ditch <derek.ditch@elastic.co>
Co-authored-by: Marius Iversen <marius.iversen@elastic.co>
Co-authored-by: Adrian Serrano <adrisr83@gmail.com>
Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>
(cherry picked from commit 6a0dc39)
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Apr 20, 2021

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Apr 20, 2021
@mergify
Copy link
Copy Markdown
Contributor

mergify bot commented Apr 20, 2021

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b backport_24570_7.x upstream/backport_24570_7.x
git merge upstream/7.x
git push upstream backport_24570_7.x

@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Apr 20, 2021 •
edited by jenkins-beats-ci bot
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: Pull request #25177 updated

  • Start Time: 2021-04-20T18:52:24.063+0000

  • Duration: 75 min 39 sec

  • Commit: 15caa2e

Test stats 🧪

Test Results
Failed 0
Passed 13576
Skipped 2285
Total 15861

Trends 🧪

Image of Build Times

Image of Tests

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test Results
Failed 0
Passed 13576
Skipped 2285
Total 15861

@P1llus
Copy link
Copy Markdown
Member Author

P1llus commented Apr 20, 2021

run elasticsearch-ci/docs

@P1llus P1llus merged commit 9e04a75 into elastic:7.x Apr 20, 2021
@zube zube bot removed the [zube]: Done label Jul 20, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

backport

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@P1llus @elasticmachine @peasead