Skip to content

[ML] Adding support for saved object based ml modules#92855

Merged
jgowdyelastic merged 18 commits intoelastic:masterfrom
jgowdyelastic:adding-support-for-saved-object-based-ml-modules
Mar 11, 2021
Merged

[ML] Adding support for saved object based ml modules#92855
jgowdyelastic merged 18 commits intoelastic:masterfrom
jgowdyelastic:adding-support-for-saved-object-based-ml-modules

Conversation

@jgowdyelastic
Copy link
Copy Markdown
Member

@jgowdyelastic jgowdyelastic commented Feb 25, 2021 •
edited
Loading

Adds the ability to install ML modules from a new saved object type.

Preempts future work where we move ML modules out of kibana and into integration packages.
As these currently do not exist, it is difficult to test that side of this PR without adding the module to an existing package and running the test package registry service as detailed here.

This is an example nginx package with the ML module added.

image

After installing the package or the assets only, nginx is then available as a module as usual.

Saved object modules do not contain kibana saved objects (dashboards, visualisations etc).

Checklist

Delete any items that are not applicable to this PR.

@jgowdyelastic
Copy link
Copy Markdown
Member Author

jgowdyelastic commented Feb 26, 2021

@elasticmachine merge upstream

@jgowdyelastic
Copy link
Copy Markdown
Member Author

jgowdyelastic commented Mar 1, 2021

@elasticmachine merge upstream

@jgowdyelastic
Copy link
Copy Markdown
Member Author

jgowdyelastic commented Mar 8, 2021

@elasticmachine merge upstream

@jgowdyelastic jgowdyelastic self-assigned this Mar 9, 2021
@jgowdyelastic jgowdyelastic added :ml Feature:Anomaly Detection ML anomaly detection release_note:enhancement review v7.13.0 v8.0.0 auto-backport Deprecated - use backport:version if exact versions are needed labels Mar 9, 2021
@jgowdyelastic jgowdyelastic marked this pull request as ready for review March 9, 2021 12:00
@jgowdyelastic jgowdyelastic requested review from a team as code owners March 9, 2021 12:00
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Mar 9, 2021

Pinging @elastic/ml-ui (:ml)

@botelastic botelastic Bot added the Team:Fleet Team label for Observability Data Collection Fleet team label Mar 9, 2021
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Mar 9, 2021

Pinging @elastic/fleet (Team:Fleet)

@ruflin
Copy link
Copy Markdown
Contributor

ruflin commented Mar 9, 2021

Great to see this moving forward. We normally start with new asset types in the package-spec to make sure the way it should behave / validate etc is defined: https://github.com/elastic/package-spec/blob/master/CONTRIBUTING.md#change-proposals Could you open a issue / PR in the package spec for this? @ycombinator @mtojek For awareness.

For the PR, could it be split up into two PRs? One just adding the SO support for ml modules and the second one around adding it to fleet?

darnautov
darnautov reviewed Mar 9, 2021
View reviewed changes
Comment thread x-pack/plugins/fleet/common/types/models/epm.ts Outdated
Comment thread x-pack/plugins/ml/server/models/data_recognizer/data_recognizer.ts Outdated
Comment thread x-pack/plugins/ml/server/models/data_recognizer/data_recognizer.ts Outdated
Comment thread x-pack/plugins/ml/server/models/data_recognizer/data_recognizer.ts Outdated
Comment thread x-pack/plugins/ml/server/models/data_recognizer/data_recognizer.ts Outdated
Comment thread x-pack/plugins/ml/server/models/data_recognizer/data_recognizer.ts Outdated
Comment thread x-pack/plugins/ml/server/models/data_recognizer/data_recognizer.ts Outdated
Comment thread x-pack/plugins/ml/server/models/data_recognizer/data_recognizer.ts Outdated
Comment thread x-pack/plugins/ml/server/models/data_recognizer/data_recognizer.ts Outdated
Comment thread x-pack/plugins/ml/server/saved_objects/mappings.ts Outdated
@jgowdyelastic
Copy link
Copy Markdown
Member Author

jgowdyelastic commented Mar 9, 2021

For the PR, could it be split up into two PRs? One just adding the SO support for ml modules and the second one around adding it to fleet?

@ruflin Sure yeah, i'll remove the fleet changes from this PR.

alvarezmelissa87
alvarezmelissa87 reviewed Mar 10, 2021
View reviewed changes
Comment thread x-pack/plugins/ml/server/models/data_recognizer/data_recognizer.ts Outdated
};
}

function isModule(arg: any): arg is Module {
Copy link
Copy Markdown
Contributor

@alvarezmelissa87 alvarezmelissa87 Mar 10, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be an 'unknown' instead of an 'any' now?

alvarezmelissa87
alvarezmelissa87 approved these changes Mar 10, 2021
View reviewed changes
Copy link
Copy Markdown
Contributor

@alvarezmelissa87 alvarezmelissa87 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code LGTM ⚡

@jgowdyelastic
Copy link
Copy Markdown
Member Author

jgowdyelastic commented Mar 11, 2021

@elasticmachine merge upstream

darnautov
darnautov approved these changes Mar 11, 2021
View reviewed changes
Copy link
Copy Markdown
Contributor

@darnautov darnautov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code LGTM

walterra
walterra approved these changes Mar 11, 2021
View reviewed changes
Copy link
Copy Markdown
Contributor

@walterra walterra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code LGTM

@kibanamachine
Copy link
Copy Markdown
Contributor

kibanamachine commented Mar 11, 2021

💚 Build Succeeded

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id before after diff
ml 6.4MB 6.4MB +46.0B

Saved Objects .kibana field count

Every field in each saved object type adds overhead to Elasticsearch. Kibana needs to keep the total field count below Elasticsearch's default limit of 1000 fields. Only specify field mappings for the fields you wish to search on or query. See https://www.elastic.co/guide/en/kibana/master/development-plugin-saved-objects.html#_mappings

id before after diff
ml-module - 11 +11

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @jgowdyelastic

@jgowdyelastic jgowdyelastic merged commit dfb4eac into elastic:master Mar 11, 2021
@jgowdyelastic jgowdyelastic deleted the adding-support-for-saved-object-based-ml-modules branch March 11, 2021 14:08
@kibanamachine kibanamachine mentioned this pull request Mar 11, 2021
Merged
kibanamachine added a commit to kibanamachine/kibana that referenced this pull request Mar 11, 2021
@jgowdyelastic @kibanamachine
[ML] Adding support for saved object based ml modules (elastic#92855)
1ecea48 
* [ML] Adding support for saved object based ml modules

* updating icon mapping

* cleaning up code

* missed private variable

* removing mappings json file

* renaming module id

* updating test

* removing unrelated file

* type clean up

* changing logo type

* changes based on review

* removing fleet changes

* updating type guards

* fixing list module return type

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
@kibanamachine
Copy link
Copy Markdown
Contributor

kibanamachine commented Mar 11, 2021

💚 Backport successful

✅ 7.x / #94428

Successful backport PRs will be merged automatically after passing CI.

kibanamachine added a commit that referenced this pull request Mar 11, 2021
@kibanamachine @jgowdyelastic
[ML] Adding support for saved object based ml modules (#92855) (#94428)
b97936d 
* [ML] Adding support for saved object based ml modules

* updating icon mapping

* cleaning up code

* missed private variable

* removing mappings json file

* renaming module id

* updating test

* removing unrelated file

* type clean up

* changing logo type

* changes based on review

* removing fleet changes

* updating type guards

* fixing list module return type

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

Co-authored-by: James Gowdy <jgowdy@elastic.co>
This was referenced Mar 16, 2021
Closed
Merged
@peteharverson peteharverson mentioned this pull request Apr 22, 2021
Closed
6 tasks
@peteharverson peteharverson added release_note:feature Makes this part of the condensed release notes and removed release_note:enhancement labels May 4, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@walterra walterra walterra approved these changes

@darnautov darnautov darnautov approved these changes

@alvarezmelissa87 alvarezmelissa87 alvarezmelissa87 approved these changes

@peteharverson peteharverson Awaiting requested review from peteharverson

Assignees

@jgowdyelastic jgowdyelastic

Labels

auto-backport Deprecated - use backport:version if exact versions are needed Feature:Anomaly Detection ML anomaly detection :ml release_note:feature Makes this part of the condensed release notes review Team:Fleet Team label for Observability Data Collection Fleet team v7.13.0 v8.0.0

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

8 participants

@jgowdyelastic @elasticmachine @ruflin @kibanamachine @walterra @darnautov @alvarezmelissa87 @peteharverson