[BBOT] New Integration - Updated Pull Request by CarsonHrusovsky · Pull Request #9651 · elastic/integrations

CarsonHrusovsky · 2024-04-18T22:08:47Z

Enhancement
Added BBOT Integration
WHAT: BBOT is an OSINT ASM tool that is available to the general public. The link for this tool can be found here: https://github.com/blacklanternsecurity/bbot
WHY: This allows ingest of ASM findings into Elastic, allowing for review and analysis.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Screenshots

CarsonHrusovsky · 2024-04-18T22:10:21Z

@nicpenning @efd6

Updated Pull request to fix CLA agreement issues. Let me know how this one looks.

#9427

nicpenning · 2024-04-18T22:46:29Z

nits - It doesn't appear the logo is fitting within the white background. The bottom of the logo passes the white square box. It is also not centered in the white box. Seems a bit janky for my taste.

cla-checker-service · 2024-04-19T14:14:31Z

💚 CLA has been signed

nicpenning · 2024-04-19T14:40:12Z

@CarsonHrusovsky - Do you have a screenshot of how the logo looks in dark mode now? Perhaps Black Lantern Security could sign off on that.

CarsonHrusovsky · 2024-04-19T14:50:16Z

Sure here you go

nicpenning · 2024-04-19T17:04:06Z

@aconite33, @TheTechromancer - Do you approve of the logo usage above?

aconite33 · 2024-04-19T17:06:47Z

@aconite33, @TheTechromancer - Do you approve of the logo usage above?

This looks great!

CarsonHrusovsky · 2024-04-19T17:17:06Z

Okay we are ready again 😃

packages/bbot/_dev/build/docs/README.md

packages/bbot/changelog.yml

packages/bbot/data_stream/asm_intel/_dev/test/system/test-default-config.yml

packages/bbot/data_stream/asm_intel/fields/fields.yml

packages/bbot/data_stream/asm_intel/sample_event.json

Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>

efd6 · 2024-04-22T20:42:56Z

/test

efd6 · 2024-04-22T20:48:47Z

packages/bbot/data_stream/asm_intel/_dev/test/system/test-default-config.yml

+      - "{{SERVICE_LOGS_DIR}}/*.log"
+    preserve_original_event: true
+assert:
+  hit_count: 7


Final new line.

packages/bbot/data_stream/asm_intel/elasticsearch/ingest_pipeline/default.yml

efd6 · 2024-04-22T20:55:37Z

Please run elastic-package build.

…ine/default.yml Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>

packages/bbot/data_stream/asm_intel/elasticsearch/ingest_pipeline/default.yml

CarsonHrusovsky · 2024-04-22T21:20:09Z

When I build the package, the only file that changes is the

READEME.md

is this correct?

efd6 · 2024-04-22T21:43:01Z

Yes, that's correct.

efd6 · 2024-04-22T21:43:08Z

/test

elasticmachine · 2024-04-22T21:55:54Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

efd6

Do we have additional test cases that could be added for test coverage? Currently we are missing coverage for IP_ADDRESS through to FINDING and TECHNOLOGY to WAF.

CarsonHrusovsky · 2024-04-24T19:34:45Z

These additional tests have shown an error in our fields - however I am unsure if we should be encountering this error. I am going to open an issue post on their github and discuss.

The issue arises with our lowercasing of the fields but with incorrect naming scheme (possibly).

CarsonHrusovsky · 2024-04-24T19:40:50Z

blacklanternsecurity/bbot#1304

CarsonHrusovsky · 2024-04-25T18:39:59Z

The bug has been fixed. I've updated our tests with your recommended adjustments.

efd6 · 2024-04-25T22:33:27Z

/test

elastic-sonarqube · 2024-04-25T22:47:19Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
80.5% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

elasticmachine · 2024-04-25T22:47:42Z

💚 Build Succeeded

Buildkite Build
Commit: 722861a

History

💚 Build #10644 succeeded d9d9d2e
💔 Build #10641 failed 4d5bcac

efd6

Thanks for your patience with this.

efd6 · 2024-04-25T23:10:18Z

I'm still concerned with this comment which was lost because it was marked resolved without conclusion (please don't do this) and then the PR was moved. I can guarantee that this will result in support cases due to customer confusion, so I would like to have it resolved. This can happen in a follow-up PR.

elasticmachine · 2024-04-25T23:29:49Z

Package bbot - 0.1.0 containing this change is available at https://epr.elastic.co/search?package=bbot

CarsonHrusovsky · 2024-04-26T00:26:39Z

😁 this is awesome thank you for the help @efd6. I am sure I will find some changes in the near future as we start using this integration, so I will add a change to the manifest for the path as well then.

commit e2a688fbb1c8712ba0cad243713146867ac2f986 Author: milan-elastic <milan.Parmar@elastic.co> Date: Wed May 1 15:43:52 2024 +0530 Squashed commit of the following: commit a17de73aa84608f67a1baca4c094819b562e42e0 Author: milan-elastic <“milan.parmar@elastic.co”> Date: Wed May 1 15:29:41 2024 +0530 Squashed commit of the following: commit fccdb1f83f0048b07df6ee82fbd91ca432c799b9 Author: milan-elastic <milan.parmar@elastic.co> Date: Wed May 1 14:58:41 2024 +0530 add global filter on dashboard level for hadoop commit 686e49be78dc980b2f12d365580cb800fd7cf330 Merge: 024d864b4 01201a7 Author: “milan-elastic” <“milan.parmar@elastic.co”> Date: Wed May 1 11:38:59 2024 +0530 Merge branch 'main' of github.com:milan-elastic/integrations into mongodb-atlas-database-logs commit 01201a7 Author: Eric Forte <119343520+eric-forte-elastic@users.noreply.github.com> Date: Tue Apr 30 10:46:55 2024 -0400 [Security Rules] Update security rules package to v8.13.5 (elastic#9762) * [Security Rules] Update security rules package to v8.13.5 * Add changelog entry for 8.13.5 --------- Co-authored-by: protectionsmachine <72879786+protectionsmachine@users.noreply.github.com> commit c9d1f1b Author: Eric Forte <119343520+eric-forte-elastic@users.noreply.github.com> Date: Tue Apr 30 09:30:30 2024 -0400 [Security Rules] Update security rules package to v8.13.5-beta.1 (elastic#9758) * [Security Rules] Update security rules package to v8.13.5-beta.1 * Add changelog entry for 8.13.5-beta.1 --------- Co-authored-by: protectionsmachine <72879786+protectionsmachine@users.noreply.github.com> commit a79f813 Author: Tetiana Kravchenko <tetiana.kravchenko@elastic.co> Date: Tue Apr 30 11:32:37 2024 +0200 [kubernetes] Remove deprecated fields, add missing status.last_terminated_reason metric (elastic#9736) * remove deprecated fields Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co> * Update changelog.yml * add missing metric: last_terminated_reason; update description of the status.reason field Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co> --------- Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co> commit b1627a3 Author: ShourieG <105607378+ShourieG@users.noreply.github.com> Date: Tue Apr 30 13:03:29 2024 +0530 [integrations][http_endpoint] - Converted HTTP Endpoint Integration to input type (elastic#9732) * converted http_endpoint to input package type * updated changelog * updated original event in sample event commit 3a9b508 Author: Lalit Satapathy <69236064+lalit-satapathy@users.noreply.github.com> Date: Tue Apr 30 11:49:09 2024 +0530 Remove separate codeowners for system package kibana paths. (elastic#9731) commit c90e817 Author: Krishna Chaitanya Reddy Burri <krishnachaitanyareddy.burri@elastic.co> Date: Tue Apr 30 11:32:17 2024 +0530 [Crowdstrike,Azure] Fix flaky tests with ECS fields (elastic#9738) * Fix flaky pipeline tests. * `azure.graphactivitylogs`: Add missing ECS field definitions. * `crowdstrike.falcon`: Update `geoip` processor to `destination` instead of `source`. commit ace8fb4 Author: Aliabbas Attarwala <124054599+aliabbas-elastic@users.noreply.github.com> Date: Mon Apr 29 16:37:23 2024 +0530 [O11y][AWS] Rally benchmark `aws.cloudtrail` (elastic#9448) commit d4e4aa4 Author: niraj-elastic <124254029+niraj-elastic@users.noreply.github.com> Date: Mon Apr 29 14:45:46 2024 +0530 [Apache] Update grok pattern for accepting user-identity (elastic#9632) * update grok pattern * update changelog * address review comments * address review comments Co-authored-by: muthu-mps <101238137+muthu-mps@users.noreply.github.com> * address review comments * address review comment --------- Co-authored-by: muthu-mps <101238137+muthu-mps@users.noreply.github.com> commit dce5699 Author: Mario Rodriguez Molins <mario.rodriguez@elastic.co> Date: Mon Apr 29 10:33:19 2024 +0200 Enable publishing packages from integrations-publish pipeline (elastic#9712) Enable publishing packages from integrations-publish pipeline, and remove corresponding step from the main pipeline. commit c7bc530 Author: Chema Martínez <chema.martinez@elastic.co> Date: Sat Apr 27 08:57:55 2024 +0200 [zscaler_zia] Fix mapping of source.ip and source.nat.ip (elastic#9727) * Fix mapping of source.ip and source.nat.ip * Update changelog * updated web datastream pipeline tests --------- Co-authored-by: Shourie Ganguly <shourie.ganguly@elastic.co> commit 4750ea8 Author: Mario Rodriguez Molins <mario.rodriguez@elastic.co> Date: Fri Apr 26 13:09:53 2024 +0200 [nginx] Update nginx config to listen in ipv6 too (elastic#9720) commit 25b0988 Author: Mario Rodriguez Molins <mario.rodriguez@elastic.co> Date: Fri Apr 26 10:45:03 2024 +0200 [Buildkite] Update filter to use api source (elastic#9717) commit 45327cf Author: Mario Rodriguez Molins <mario.rodriguez@elastic.co> Date: Fri Apr 26 10:13:22 2024 +0200 [Buildkite] Update filter condition to allow just from webhook source (elastic#9714) commit 024d864b49f1dd333529f96e06de6dec15aac703 Author: milan-elastic <milan.parmar@elastic.co> Date: Fri Apr 26 13:00:47 2024 +0530 add dashboard level filter for apache tomcat commit 1cb5fad Author: Dan Kortschak <dan.kortschak@elastic.co> Date: Fri Apr 26 16:23:35 2024 +0930 entityanalytics_ad: new package for Active Directory user collection (elastic#9485) commit 37c598f Author: CarsonHrusovsky <95260807+CarsonHrusovsky@users.noreply.github.com> Date: Thu Apr 25 18:13:26 2024 -0500 [BBOT] New integration for Black Lantern Security scanner (elastic#9651) commit d13e474 Author: Mario Rodriguez Molins <mario.rodriguez@elastic.co> Date: Thu Apr 25 11:55:39 2024 +0200 [Buildkite] Skip install package command in serverless builds for some packages (elastic#9686) commit 0c2198b Author: Mario Rodriguez Molins <mario.rodriguez@elastic.co> Date: Thu Apr 25 11:41:42 2024 +0200 [Buildkite] Add retry suffix for logs (elastic#9703) commit d932e79 Author: Simon Kötting <145989254+SimonKoetting@users.noreply.github.com> Date: Thu Apr 25 07:35:45 2024 +0200 [Exchange Server] GA of Integration, Add Dashbord Panel Titles & System Tests (elastic#9560) * Add Dashboard Titles * Add Dashboard Titles * Change Version to GA * adjust PR in Changelog * Add System Tests to all datstreams * fix imap system test config * remove Folder structure out of system tests sample logs * Fix mapping * Add convert for inode field * specify numeric_keyword_fields in system tests commit dba2901 Author: Dan Kortschak <dan.kortschak@elastic.co> Date: Thu Apr 25 10:21:30 2024 +0930 rapid7_insightvm: canonicalize host.name to lower case and map subdomain to host.hostname (elastic#9665) commit 4284262 Author: Panos Koutsovasilis <panos.koutsovasilis@elastic.co> Date: Wed Apr 24 20:34:13 2024 +0300 fix(fim): add auto option for backend and make it the default one (elastic#9702) commit c563bb3 Author: Panos Koutsovasilis <panos.koutsovasilis@elastic.co> Date: Wed Apr 24 19:40:04 2024 +0300 [juniper_netscreen]: include log.file.device_id and log.file.inode in base-fields (elastic#9658) * fix(juniper_netscreen): include log.file.device_id and log.file.inode in base-fields.yml * fix(juniper_netscreen): update README.md commit f187d0d Author: Panos Koutsovasilis <panos.koutsovasilis@elastic.co> Date: Wed Apr 24 19:11:28 2024 +0300 [juniper_junos]: include log.file.device_id and log.file.inode in base-fields (elastic#9657) * fix(juniper_junos): include log.file.device_id and log.file.inode in base-fields.yml * fix(juniper_junos): update README.md

CarsonHrusovsky added 2 commits April 18, 2024 16:55

Update CODEOWNERS

e99c605

The Integration is reborn

eb62706

Updated Logo

b78d261

efd6 reviewed Apr 21, 2024

View reviewed changes

CarsonHrusovsky and others added 8 commits April 22, 2024 08:47

Update packages/bbot/_dev/build/docs/README.md

ad77cd5

Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>

Update packages/bbot/_dev/build/docs/README.md

86295a5

Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>

Update packages/bbot/changelog.yml

0da5c39

Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>

Suggested changes to fields.yml

15e4dad

Suggested convention update

ba15ad1

Updated example path in manifest.

ec3878b

On failure handler

b2167dc

Attempt 2 of rebuild of package + system tests

4d5bcac

efd6 reviewed Apr 22, 2024

View reviewed changes

Update packages/bbot/data_stream/asm_intel/elasticsearch/ingest_pipel…

8521491

…ine/default.yml Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>

efd6 reviewed Apr 22, 2024

View reviewed changes

packages/bbot/data_stream/asm_intel/elasticsearch/ingest_pipeline/default.yml Outdated Show resolved Hide resolved

CarsonHrusovsky added 3 commits April 22, 2024 16:15

Added new line

d8ba313

new line

03e8b44

built

d9d9d2e

efd6 reviewed Apr 22, 2024

View reviewed changes

Updated pipeline test.

722861a

efd6 approved these changes Apr 25, 2024

View reviewed changes

efd6 merged commit 37c598f into elastic:main Apr 25, 2024

gizas pushed a commit that referenced this pull request Apr 26, 2024

[BBOT] New integration for Black Lantern Security scanner (#9651)

a1bea00

andrewkroh added the Integration:bbot BBOT (Bighuge BLS OSINT Tool) (Community supported) label Jul 22, 2024

andrewkroh added the New Integration Issue or pull request for creating a new integration package. label Aug 13, 2024

qcorporation pushed a commit that referenced this pull request Feb 3, 2025

[BBOT] New integration for Black Lantern Security scanner (#9651)

7bb19ce

qcorporation pushed a commit that referenced this pull request Feb 4, 2025

[BBOT] New integration for Black Lantern Security scanner (#9651)

5a2be50

Conversation

CarsonHrusovsky commented Apr 18, 2024

Checklist

Screenshots

Uh oh!

CarsonHrusovsky commented Apr 18, 2024

Uh oh!

nicpenning commented Apr 18, 2024

Uh oh!

cla-checker-service bot commented Apr 19, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nicpenning commented Apr 19, 2024

Uh oh!

CarsonHrusovsky commented Apr 19, 2024

Uh oh!

nicpenning commented Apr 19, 2024

Uh oh!

aconite33 commented Apr 19, 2024

Uh oh!

CarsonHrusovsky commented Apr 19, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

efd6 commented Apr 22, 2024

Uh oh!

efd6 Apr 22, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

efd6 commented Apr 22, 2024

Uh oh!

Uh oh!

CarsonHrusovsky commented Apr 22, 2024

Uh oh!

efd6 commented Apr 22, 2024

Uh oh!

efd6 commented Apr 22, 2024

Uh oh!

elasticmachine commented Apr 22, 2024

🚀 Benchmarks report

Uh oh!

efd6 left a comment

Choose a reason for hiding this comment

Uh oh!

CarsonHrusovsky commented Apr 24, 2024

Uh oh!

CarsonHrusovsky commented Apr 24, 2024

Uh oh!

CarsonHrusovsky commented Apr 25, 2024

Uh oh!

efd6 commented Apr 25, 2024

Uh oh!

elastic-sonarqube bot commented Apr 25, 2024

Quality Gate passed

Uh oh!

elasticmachine commented Apr 25, 2024

💚 Build Succeeded

History

Uh oh!

efd6 left a comment

Choose a reason for hiding this comment

Uh oh!

efd6 commented Apr 25, 2024

Uh oh!

elasticmachine commented Apr 25, 2024

Uh oh!

CarsonHrusovsky commented Apr 26, 2024

Uh oh!

Reviewers

Assignees

Labels

cla-checker-service bot commented Apr 19, 2024 •

edited

Loading