Adds ML supervised model DGA package by alvarezmelissa87 · Pull Request #2352 · elastic/integrations

alvarezmelissa87 · 2021-12-16T23:59:20Z

What does this PR do?

Adds the ML supervised model package for DGA model.

Package includes:

pipelines
ml_module
ml_model
security rules

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

[ ]

How to test this PR locally

Related issues

Screenshots

elasticmachine · 2021-12-17T00:09:29Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-03-29T00:49:11.696+0000
Duration: 13 min 31 sec

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

ajosh0504

Overall looks good if it's tested and working as expected. Minor nit: We should probably consider the model name to be something more understandable? It's currently dga_1611725_2.0. This will also need to be changed in the inference pipeline.

ajosh0504 · 2022-01-31T21:21:44Z

packages/ml_dga/changelog.yml

Should the type here be enhancement? Or could this be something that's descriptive of the model, for example dga?

Updated to 'ml-dga' so it's more descriptive in 7e5dafe824acfedaf66d9e2f4680a41095753495

dishadasgupta · 2022-01-31T22:54:04Z

packages/ml_dga/docs/README.md

Is this supposed to link to something? The link right after it? Usually I see square brackets in docs and think it's supposed to point somewhere

yes, this links to https://www.elastic.co/guide/en/security/7.13/detection-engine-overview.html - we don't have anything about those models in the ML guide, so this is likely the only place that mentions them.

I think there might be an extra space in the doc maybe? When I view the file, it's not hyperlinked if that's what's intended

Should the version of the docs be updated to the 8.0 page - https://www.elastic.co/guide/en/security/8.0/detection-engine-overview.html?

Good call - updated in ff4d691

dishadasgupta · 2022-01-31T22:57:31Z

packages/ml_dga/kibana/ml_module/dga-ml.json

I'm assuming this is leftover from the ProblemChild PR?

Yep - good catch - fixed in 7e5dafe824acfedaf66d9e2f4680a41095753495

dishadasgupta · 2022-01-31T22:58:11Z

packages/ml_dga/kibana/ml_module/dga-ml.json

Nitpicky admittedly but the user sees these descriptions right? Misspelling in "comand"

Same with 'algorithm' in the DGA parenthetical

Thanks! Updated in 7e5dafe824acfedaf66d9e2f4680a41095753495

dishadasgupta · 2022-01-31T23:06:04Z

@alvarezmelissa87 How do tests look for this? If you could provide a quick screen recording of how this looks/is supposed to look, that'd be cool :) Might help clarify maybe any assumptions made in testing that a user may not know (I'm happy to try to replicate that if you need it at all, lmk)

Update: I'm seeing this issue with demos for ProblemChild: https://github.com/elastic/ml-team/issues/628 with work being done on DGA. I'll follow this to see DGA demos, please let me know if that's not the case where I should be looking once that's done

peteharverson · 2022-02-02T15:27:05Z

packages/ml_dga/manifest.yml

+title: "ML DGA"
+version: 0.0.1
+license: platinum
+description: "ML solution package to detect domain generation algorithm (DGA) activity in your network data."


Not this specific description, but can we add a description field for the model - as shown in the Trained Models list in the ML UI.

Updated in ff4d691

packages/ml_dga/changelog.yml

alvarezmelissa87 · 2022-02-02T15:29:49Z

cc @lcawl for package/model description text in the readme and such.

packages/ml_dga/changelog.yml

peteharverson · 2022-02-02T17:32:53Z

packages/ml_dga/docs/README.md

Should the version of the docs be updated to the 8.0 page - https://www.elastic.co/guide/en/security/8.0/detection-engine-overview.html?

packages/ml_dga/kibana/ml_module/dga-ml.json

packages/ml_dga/manifest.yml

packages/ml_dga/kibana/ml_module/dga-ml.json

packages/dga/elasticsearch/ingest_pipeline/ml_dga_ingest_pipeline.yml

mtojek

The file packages/dga/elasticsearch/ml_model/dga_1611725_2.0.json is relatively big (~44MB).

Could you please tell us how many package revisions do you plan to add? Is it only this single one or do you plan to iterate on it in the next PRs? For example, 3 iterations will add an extra ~130MB to the Docker image, which is ~10% of all packages. I'm not sure if we have the capacity for this.

peteharverson · 2022-02-08T11:47:53Z

Could you please tell us how many package revisions do you plan to add? Is it only this single one or do you plan to iterate on it in the next PRs? For example, 3 iterations will add an extra ~130MB to the Docker image, which is ~10% of all packages. I'm not sure if we have the capacity for this.

@mtojek only a single revision for this initial DGA package is planned. We are working with the docs team for example, to make sure the README does not require any further PRs, and with the security team to ensure that the package contains all the required detection rules before promoting the package to production.

Looking further ahead, I do not anticipate that the assets (ML model and anomaly detection job, security detection rules) will need frequent updates. The last update to the model for example was released in the security detection rules repo in May 2021.

Is there any update on the plans to redesign parts of the package registry to provide better support for large files - as discussed in elastic/package-registry#670 and https://github.com/elastic/obs-dc-team/issues/570 for example?

mtojek · 2022-02-08T11:58:52Z

Thanks for the confirmation, Pete.

Is there any update on the plans to redesign parts of the package registry to provide better support for large files - as discussed in elastic/package-registry#670 and https://github.com/elastic/obs-dc-team/issues/570 for example?

We're working on this now, but considering the team capacity, it will take a few months to fully switch.

lcawl

One suggestion, otherwise text LGTM

packages/dga/_dev/build/docs/README.md

lcawl · 2022-03-18T20:24:05Z

packages/dga/_dev/build/docs/README.md

+
+In **Machine Learning > Anomaly Detection**, when you create a job, you should see an option to `Use preconfigured jobs` with a card for Domain Generated Algorithm (DGA) detection. When you select the card, you will see a pre-configured anomaly detection job that you can enable depending on what makes the most sense for your environment. Note this job is only useful for indices that have been enriched by the ingest pipeline.
+
+### (Optional) Enable Security rules


We generally try to stick to sentence-case

Suggested change

### (Optional) Enable Security rules

### (Optional) Enable security rules

packages/dga/_dev/build/docs/README.md

alvarezmelissa87 · 2022-03-29T01:10:41Z

This has been updated and is ready for a final look when you get a chance 🙏 cc @mtojek

alvarezmelissa87 added the enhancement New feature or request label Dec 16, 2021

alvarezmelissa87 self-assigned this Dec 16, 2021

peteharverson mentioned this pull request Dec 22, 2021

[Discuss] Support stored scripts in Fleet packages elastic/package-spec#202

Open

alvarezmelissa87 force-pushed the ml-supervised-model-package branch from 3fd4939 to 109d650 Compare January 20, 2022 22:43

ajosh0504 reviewed Jan 31, 2022

View reviewed changes

dishadasgupta reviewed Jan 31, 2022

View reviewed changes

alvarezmelissa87 force-pushed the ml-supervised-model-package branch from 7e5dafe to bf502fb Compare February 1, 2022 22:25

alvarezmelissa87 marked this pull request as ready for review February 1, 2022 22:26

peteharverson reviewed Feb 2, 2022

View reviewed changes

packages/ml_dga/changelog.yml Outdated Show resolved Hide resolved

peteharverson reviewed Feb 2, 2022

View reviewed changes

packages/ml_dga/kibana/ml_module/dga-ml.json Outdated Show resolved Hide resolved

peteharverson mentioned this pull request Feb 4, 2022

Add ML modules to Apache integration readme #2640

Merged

4 tasks

alvarezmelissa87 requested a review from lcawl February 4, 2022 16:28

peteharverson reviewed Feb 4, 2022

View reviewed changes

packages/dga/elasticsearch/ingest_pipeline/ml_dga_ingest_pipeline.yml Outdated Show resolved Hide resolved

lcawl mentioned this pull request Feb 5, 2022

Add ML modules to DGA readme alvarezmelissa87/integrations#1

Closed

alvarezmelissa87 mentioned this pull request Feb 5, 2022

Add support for platinum license elastic/package-spec#267

Closed

2 tasks

mtojek requested a review from a team February 7, 2022 08:59

mtojek suggested changes Feb 7, 2022

View reviewed changes

alvarezmelissa87 mentioned this pull request Feb 7, 2022

Update ml_model file format pattern elastic/package-spec#268

Merged

2 tasks

lcawl approved these changes Feb 9, 2022

View reviewed changes

packages/dga/_dev/build/docs/README.md Outdated Show resolved Hide resolved

lcawl reviewed Feb 9, 2022

View reviewed changes

packages/dga/_dev/build/docs/README.md Outdated Show resolved Hide resolved

lcawl reviewed Feb 9, 2022

View reviewed changes

packages/dga/_dev/build/docs/README.md Outdated Show resolved Hide resolved

alvarezmelissa87 added 16 commits March 18, 2022 12:03

adds ml_dga package

3fec835

adds ml_problem_child package

77ef661

adds dga pipelines

8fb09b6

update license type requirement to platinum

c091a1c

rename model files to model id

12cead9

fix dga pipeline

5cc60bb

remove problem child package. add module to dga package

279e5e0

adds security rules to dga package

ebe2bbd

update minimum version requirement and deprecated model property

1852d1d

format json files

12e226b

update icon. add groups to ad job

d74dbe8

rename directory

1c1138d

update pipeline description

8d28b2c

update readme with asset info

d3dfb68

add license requirement to card and readme

12c795d

add asset context to readme

3fd49c9

alvarezmelissa87 force-pushed the ml-supervised-model-package branch from ba6d93d to 3fd49c9 Compare March 18, 2022 18:03

add updated subscription language and update codeowners file

b3c955c

lcawl reviewed Mar 18, 2022

View reviewed changes

packages/dga/_dev/build/docs/README.md Outdated Show resolved Hide resolved

lcawl reviewed Mar 18, 2022

View reviewed changes

packages/dga/_dev/build/docs/README.md Outdated Show resolved Hide resolved

alvarezmelissa87 added 2 commits March 18, 2022 15:04

update readme and add security tag

962f946

fix ml-module file id

32759ed

mtojek approved these changes Mar 29, 2022

View reviewed changes

alvarezmelissa87 merged commit 0af0ace into elastic:main Mar 29, 2022

alvarezmelissa87 deleted the ml-supervised-model-package branch March 29, 2022 14:48

alvarezmelissa87 mentioned this pull request Apr 7, 2022

Fix typo in ML supervised model DGA package #3033

Merged

4 tasks

andrewkroh added Integration:dga Domain Generation Algorithm Detection New Integration Issue or pull request for creating a new integration package. labels Aug 13, 2024


		In Machine Learning > Anomaly Detection, when you create a job, you should see an option to `Use preconfigured jobs` with a card for Domain Generated Algorithm (DGA) detection. When you select the card, you will see a pre-configured anomaly detection job that you can enable depending on what makes the most sense for your environment. Note this job is only useful for indices that have been enriched by the ingest pipeline.

		### (Optional) Enable Security rules

Conversation

alvarezmelissa87 commented Dec 16, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

Uh oh!

elasticmachine commented Dec 17, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

🤖 GitHub comments

Uh oh!

ajosh0504 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dishadasgupta commented Jan 31, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

alvarezmelissa87 commented Feb 2, 2022

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mtojek left a comment

Choose a reason for hiding this comment

Uh oh!

peteharverson commented Feb 8, 2022

Uh oh!

mtojek commented Feb 8, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lcawl left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alvarezmelissa87 commented Dec 16, 2021 •

edited

Loading

elasticmachine commented Dec 17, 2021 •

edited

Loading

ajosh0504 left a comment •

edited

Loading

dishadasgupta commented Jan 31, 2022 •

edited

Loading

mtojek commented Feb 8, 2022 •

edited

Loading