[BBOT] New Integration - Bighuge BLS OSINT Tool ASM Data by CarsonHrusovsky · Pull Request #9427 · elastic/integrations

CarsonHrusovsky · 2024-03-22T14:15:17Z

Enhancement
Added BBOT Integration
WHAT: BBOT is an OSINT ASM tool that is available to the general public. The link for this tool can be found here: https://github.com/blacklanternsecurity/bbot
WHY: This allows ingest of ASM findings into Elastic, allowing for review and analysis.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

Currently the pictures uploaded are not the official logo of Black Lantern Security. It would be nice if we could get the official logo on this down the road.

Screenshots

cla-checker-service · 2024-03-22T14:15:20Z

❌ Author of the following commits did not sign a Contributor Agreement:
, , , , , , , , , , , , , , , , , , , , 698ae39, a94f1f9, 2489705,

Please, read and sign the above mentioned agreement if you want to contribute to this project

nicpenning · 2024-03-22T14:23:32Z

@jamiehynds - Can we check with Black Lantern Security for usage of their logo for the integration?

https://www.blacklanternsecurity.com/

CarsonHrusovsky · 2024-03-25T14:58:55Z

This is ready to be reviewed and tested. 👍

elasticmachine · 2024-03-28T03:26:17Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

efd6

Please add a line to .github/CODEOWNERS.

efd6 · 2024-03-28T03:26:58Z

packages/bbot/_dev/build/build.yml

+dependencies:
+  ecs:
+    reference: git@v8.11.0
+    import_mappings: true


Final new line.

efd6 · 2024-03-28T03:27:09Z

packages/bbot/_dev/deploy/docker/docker-compose.yml

+    volumes:
+      - ./sample_logs:/sample_logs:ro
+      - ${SERVICE_LOGS_DIR}:/var/log
+    command: /bin/sh -c "cp /sample_logs/* /var/log/"


packages/bbot/_dev/deploy/docker/sample_logs/log.log

packages/bbot/data_stream/asm_intel/_dev/test/pipeline/test-bbot-ndjson.log

efd6 · 2024-03-28T03:27:32Z

packages/bbot/data_stream/asm_intel/_dev/test/pipeline/test-common-config.yml

@@ -0,0 +1,3 @@
+fields:
+  tags:
+    - preserve_original_event


packages/bbot/data_stream/asm_intel/elasticsearch/ingest_pipeline/default.yml

efd6 · 2024-03-28T03:43:40Z

packages/bbot/data_stream/asm_intel/elasticsearch/ingest_pipeline/default.yml

+# ECS Fields #
+##############
+- rename:
+    field: bbot.data.PROTOCOL.port


Please lowercase the keys.

efd6 · 2024-03-28T03:44:17Z

packages/bbot/data_stream/asm_intel/fields/fields.yml

+  fields:
+    - name: data.DNS_NAME
+      type: keyword
+      description: >


Are these to be filled?

packages/bbot/data_stream/asm_intel/manifest.yml

…descriptions to fields. Updated README.md. Removed READ.me. Added cleanup processor.

CarsonHrusovsky · 2024-04-02T19:31:22Z

This is still being worked on.

CarsonHrusovsky · 2024-04-03T15:19:25Z

Okay this is ready for review again. A few points I will make here as far as changes:

I have left the keys in the format they are in (bbot.data.THISISCAPS.fnding). My reasoning is that within BBOT, the capitalization is a reference for an event type, so having this data is useful for an analyst. If this is something that cannot be ignored and must be lowercased, I will do this however.
Added additional documentation for users to learn more about the tool in the manifest.
Filled out data descriptions in fields.yml.
Updated pipeline with other suggestions.

nicpenning · 2024-04-12T02:47:58Z

@efd6 Any updates on this? Looks like it's ready for another review.

efd6

I have left the keys in the format they are in (bbot.data.THISISCAPS.fnding). My reasoning is that within BBOT, the capitalization is a reference for an event type, so having this data is useful for an analyst. If this is something that cannot be ignored and must be lowercased, I will do this however.

From looking at the BBOT docs and the events here, I can't find any cases where they are lower cased. Given this, there is apparently no information provided by the case. If this is the case, please lower-case them. If it's not the case, please point me to documentation explaining the semantics.

packages/bbot/_dev/build/build.yml

packages/bbot/_dev/deploy/docker/docker-compose.yml

packages/bbot/data_stream/asm_intel/_dev/test/pipeline/test-common-config.yml

packages/bbot/data_stream/asm_intel/fields/beats.yml

packages/bbot/data_stream/asm_intel/fields/ecs.yml

packages/bbot/data_stream/asm_intel/manifest.yml

efd6 · 2024-04-12T03:49:29Z

packages/bbot/validation.yml

@@ -0,0 +1,3 @@
+errors:
+  exclude_checks:
+    - SVR00004 # Added search to dashboard.


nicpenning · 2024-04-12T12:02:15Z

On the file path, would you recommend that in the integration readme it states an example of executing bbot with an example path and use that?

BBOT docs https://www.blacklanternsecurity.com/bbot/scanning/output/

You are correct that the agent is running a different user than the integration which is usually system.

Example in the readme today:
bbot -t elastic.co --strict-scope -f safe passive -c output_modules.json.siem_friendly=true -om json

Perhaps it can be added in docs with the --output to read as:
bbot -t elastic.co --strict-scope -f safe passive -c output_modules.json.siem_friendly=true -om json --output \var\logs\bbot_scans\*.ndjson

Then the path could align with the example?

The problem is in the BBOT docs they drop it by default in the user that ran the scan.

Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>

…mmon-config.yml Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>

Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>

CarsonHrusovsky · 2024-04-16T16:28:49Z

Fields have been lowercased. Marked for review once more :D - Thanks for all the help on this @efd6

efd6 · 2024-04-16T22:02:50Z

/test

efd6 · 2024-04-16T22:03:57Z

Please fix up the cases where there is no final new line in the files (all except json for generated test expectations and sample events).

efd6 · 2024-04-17T00:11:17Z

/test

elasticmachine · 2024-04-17T00:18:16Z

💔 Build Failed

Buildkite Build
Commit: 610da48

Failed CI Steps

Check integrations bbot

History

💔 Build #10504 failed ea3c349

efd6 · 2024-04-17T00:21:10Z

Please run elastic-package build and then check and commit the changes.

aconite33 · 2024-04-17T17:07:42Z

@jamiehynds - Can we check with Black Lantern Security for usage of their logo for the integration?

https://www.blacklanternsecurity.com/

I spoke with Jamie. We're good having our logo used for the integration.

I'm one of the principals behind BLS, we appreciate the integration and team work!

… BBOT svg

CarsonHrusovsky · 2024-04-17T17:13:11Z

Lets give it another go!

TheTechromancer · 2024-04-17T17:15:17Z

Hey guys thanks for your work on this. Since this is an open source effort, you have approval from the owners to use our official logos on this project:

nicpenning · 2024-04-17T18:49:22Z

Awesome, thank you @TheTechromancer !

Do you happen to have SVG versions of your logos? The icon usage on the integrations website and in Kibana requires an SVG according to the package manifest spec. Unless @efd6 knows of a way to use a PNG for the icon (today you see the rising Phoenix populated in the initial screenshot).

nicpenning · 2024-04-17T18:55:00Z

Also, Techro, do y'all have a preference of which logo to use for the integration? It will only be 1 logo that is needed. The Phoenix I referenced earlier is where it will live. Plus it will be here in time: https://www.elastic.co/integrations/data-integrations

TheTechromancer · 2024-04-17T19:38:15Z

Hmm, we don't have a version of the BBOT one that will look good on a white background. The best one to use might be this one:

CarsonHrusovsky · 2024-04-17T20:38:21Z

Thoughts? @TheTechromancer @aconite33

Proof of concept currently.

efd6 · 2024-04-17T21:18:29Z

There is a commit here that has a name that has not signed the CLA. Can all the contributors to this PR please check that they have signed with the email addresses that appear in the author email field of the commit they contributed.

nicpenning · 2024-04-17T21:21:33Z

There is a commit here that has a name that has not signed the CLA. Can all the contributors to this PR please check that they have signed with the email addresses that appear in the author email field of the commit they contributed.

Bummer:

Looks like the local git blew away Carson's user info so it doesn't recognize his email.

CarsonHrusovsky · 2024-04-18T15:30:43Z

@efd6 Is something like this possible?

icons:
  - src: /img/Black_Lantern_Light-Mode.svg
    title: BBOT Logo light-mode
    size: 32x32
    type: image/svg+xml
    dark_mode: false
  - src: /img/Black_Lantern_Dark-Mode.svg
    title: BBOT Logo dark-mode
    size: 32x32
    type: image/svg+xml
    dark_mode: true

Seems like the dark_mode: tag for icons doesn't work / is unpredictable.

This reverts commit 698ae39. This commit broke the PR. Attempting to fix.

nicpenning · 2024-04-18T17:45:10Z

@efd6 Is something like this possible?
icons:
  - src: /img/Black_Lantern_Light-Mode.svg
    title: BBOT Logo light-mode
    size: 32x32
    type: image/svg+xml
    dark_mode: false
  - src: /img/Black_Lantern_Dark-Mode.svg
    title: BBOT Logo dark-mode
    size: 32x32
    type: image/svg+xml
    dark_mode: true
Seems like the dark_mode: tag for icons does work, but I've had trouble testing this as when I make icon changes and build/install the package, it seems unpredictable if these changes are reflected in elastic. If I take the stack down and put it back up it does update.

I think this is not a feature or it is broken. In the Okta Entity Analytics, it is not using the dark mode SVG.

icons:

src: /img/okta-logo.svg
title: Okta Logo
size: 32x32
type: image/svg+xml
src: /img/okta-logo-dark.svg
title: Okta Logo
size: 32x32
type: image/svg+xml
dark_mode: true

The solution is here is applying a white background to the img which seems trivial to do in SVG.

nicpenning · 2024-04-18T17:47:35Z

IE

CarsonHrusovsky · 2024-04-18T18:42:59Z

IE

Implemented.

…official BBOT svg" This reverts commit a94f1f9. This broke CLI

CarsonHrusovsky · 2024-04-18T19:52:08Z

I think this PR is dead. I will open another and mention this one, I am afraid if I keep trying to undo commits I will break something else.

CarsonHrusovsky · 2024-04-18T20:06:25Z

I think this PR is dead. I will open another and mention this one, I am afraid if I keep trying to undo commits I will break something else.

Added BBOT Integration!

6752c82

pull request added to github

9cffe31

CarsonHrusovsky added 3 commits March 22, 2024 08:26

Removed unused pictures.

aa2ce9a

Updated Policy Template

18de4d2

Forwarded.

f4faa67

nicpenning mentioned this pull request Mar 22, 2024

JSON module: option for SIEM-friendly output blacklanternsecurity/bbot#1049

Merged

efd6 added enhancement New feature or request Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] labels Mar 28, 2024

efd6 reviewed Mar 28, 2024

View reviewed changes

Removed Brackets. Edited if statement syntax. Added new lines, added …

24f3911

…descriptions to fields. Updated README.md. Removed READ.me. Added cleanup processor.

CarsonHrusovsky mentioned this pull request Apr 2, 2024

[BBOT] Elastic Integration QOL Update blacklanternsecurity/bbot#1232

Closed

added CODEOWNERS line

77e5f9b

updated expected results file

19ce033

Alphabetical Order

c042730

efd6 reviewed Apr 12, 2024

View reviewed changes

CarsonHrusovsky and others added 7 commits April 12, 2024 08:21

Update packages/bbot/_dev/build/build.yml

02758a3

Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>

Update packages/bbot/_dev/deploy/docker/docker-compose.yml

6e8876f

Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>

Update packages/bbot/data_stream/asm_intel/_dev/test/pipeline/test-co…

2e9d95d

…mmon-config.yml Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>

Update packages/bbot/data_stream/asm_intel/fields/beats.yml

a30c35d

Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>

Update packages/bbot/data_stream/asm_intel/fields/ecs.yml

5db3293

Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>

Update packages/bbot/data_stream/asm_intel/manifest.yml

9887692

Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>

Fixed.

fd95444

lowercased fields.

fd4c56b

Updated for changes.

ea3c349

New Line Commits

610da48

elastic-package build

698ae39

New line fixes - This is ready to go. Will update later with official…

a94f1f9

… BBOT svg

Revert "elastic-package build"

2489705

This reverts commit 698ae39. This commit broke the PR. Attempting to fix.

Revert "New line fixes - This is ready to go. Will update later with …

14d500b

…official BBOT svg" This reverts commit a94f1f9. This broke CLI

CarsonHrusovsky closed this Apr 18, 2024

CarsonHrusovsky mentioned this pull request Apr 18, 2024

[BBOT] New Integration - Updated Pull Request #9651

Merged

4 tasks

@@ @@ -0,0 +1,3 @@ @@
+              fields:
+                tags:
+                  - preserve_original_event

                
                    No newline at end of file

Conversation

CarsonHrusovsky commented Mar 22, 2024

Checklist

Author's Checklist

Screenshots

Uh oh!

cla-checker-service bot commented Mar 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nicpenning commented Mar 22, 2024

Uh oh!

CarsonHrusovsky commented Mar 25, 2024

Uh oh!

elasticmachine commented Mar 28, 2024

Uh oh!

efd6 left a comment

Choose a reason for hiding this comment

Uh oh!

efd6 Mar 28, 2024

Choose a reason for hiding this comment

Uh oh!

efd6 Mar 28, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

efd6 Mar 28, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

efd6 Mar 28, 2024

Choose a reason for hiding this comment

Uh oh!

efd6 Mar 28, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

CarsonHrusovsky commented Apr 2, 2024

Uh oh!

CarsonHrusovsky commented Apr 3, 2024

Uh oh!

nicpenning commented Apr 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

efd6 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

efd6 Apr 12, 2024

Choose a reason for hiding this comment

Uh oh!

nicpenning commented Apr 12, 2024

Uh oh!

CarsonHrusovsky commented Apr 16, 2024

Uh oh!

efd6 commented Apr 16, 2024

Uh oh!

efd6 commented Apr 16, 2024

Uh oh!

efd6 commented Apr 17, 2024

Uh oh!

elasticmachine commented Apr 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💔 Build Failed

Failed CI Steps

History

Uh oh!

efd6 commented Apr 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aconite33 commented Apr 17, 2024

Uh oh!

CarsonHrusovsky commented Apr 17, 2024

cla-checker-service bot commented Mar 22, 2024 •

edited

Loading

nicpenning commented Apr 12, 2024 •

edited

Loading

elasticmachine commented Apr 17, 2024 •

edited

Loading

efd6 commented Apr 17, 2024 •

edited

Loading

nicpenning commented Apr 17, 2024 •

edited

Loading

CarsonHrusovsky commented Apr 18, 2024 •

edited

Loading

CarsonHrusovsky commented Apr 18, 2024 •

edited

Loading