[ML] Module setup with dynamic model memory estimation#60656
Merged
darnautov merged 17 commits intoelastic:masterfrom Mar 24, 2020
Merged
[ML] Module setup with dynamic model memory estimation#60656darnautov merged 17 commits intoelastic:masterfrom
darnautov merged 17 commits intoelastic:masterfrom
Conversation
Contributor
|
Pinging @elastic/ml-ui (:ml) |
7 tasks
de05507 to
cbc5f0d
Compare
x-pack/plugins/ml/server/models/data_recognizer/data_recognizer.ts
Outdated
Show resolved
Hide resolved
x-pack/plugins/ml/server/models/data_recognizer/data_recognizer.ts
Outdated
Show resolved
Hide resolved
x-pack/plugins/ml/server/models/fields_service/fields_aggs_cache.ts
Outdated
Show resolved
Hide resolved
x-pack/plugins/ml/server/models/fields_service/fields_aggs_cache.ts
Outdated
Show resolved
Hide resolved
x-pack/plugins/ml/server/models/data_recognizer/data_recognizer.ts
Outdated
Show resolved
Hide resolved
x-pack/plugins/ml/server/models/data_recognizer/data_recognizer.ts
Outdated
Show resolved
Hide resolved
x-pack/plugins/ml/server/models/data_recognizer/data_recognizer.ts
Outdated
Show resolved
Hide resolved
x-pack/plugins/ml/server/models/data_recognizer/data_recognizer.ts
Outdated
Show resolved
Hide resolved
x-pack/plugins/ml/server/models/data_recognizer/data_recognizer.ts
Outdated
Show resolved
Hide resolved
Contributor
Author
|
@elasticmachine merge upstream |
jgowdyelastic
approved these changes
Mar 23, 2020
Member
jgowdyelastic
left a comment
There was a problem hiding this comment.
Added some small comments, but generally LGTM
x-pack/plugins/ml/server/models/calculate_model_memory_limit/calculate_model_memory_limit.ts
Outdated
Show resolved
Hide resolved
x-pack/plugins/ml/server/models/data_recognizer/data_recognizer.ts
Outdated
Show resolved
Hide resolved
Contributor
💚 Build SucceededHistory
To update your PR or re-run it, just comment with: |
peteharverson
approved these changes
Mar 23, 2020
Contributor
peteharverson
left a comment
There was a problem hiding this comment.
Tested latest edit and LGTM
darnautov
added a commit
to darnautov/kibana
that referenced
this pull request
Mar 24, 2020
* [ML] add estimateModelMemory to the setup endpoint * [ML] wip caching cardinality checks * [ML] refactor * [ML] fix a fallback time range * [ML] fix typing issue * [ML] fields_aggs_cache.ts as part of fields_service * [ML] fix types, add comments * [ML] check for MML overrides * [ML] disable estimateModelMemory * [ML] fix typing * [ML] check for empty max mml * [ML] refactor, update types, fix jobsForModelMemoryEstimation * [ML] fix override lookup * [ML] resolve nit comments * [ML] init jobsForModelMemoryEstimation
gmmorris
added a commit
to gmmorris/kibana
that referenced
this pull request
Mar 24, 2020
* master: (34 commits) [APM] add service map config options to legacy plugin (elastic#61002) [App Arch] migrate legacy CSS to new platform (core_plugins/kibana_react) (elastic#59882) Migrated styles for "share" plugin to new platform (elastic#59981) [ML] Module setup with dynamic model memory estimation (elastic#60656) Drilldowns (elastic#59632) Upgrade mocha dev-dependency from 6.2.2 to 7.1.1 (elastic#60779) [SIEM] Overview: Recent cases widget (elastic#60993) [ML] Functional tests - stabilize df analytics clone tests (elastic#60497) [SIEM] Updates process and TLS tables to use ECS 1.5 fields (elastic#60854) Migrate doc view part of discover (elastic#58094) Revert "[APM] Collect telemetry about data/API performance (elastic#51612)" fix(NA): log rotation watchers usage (elastic#60956) [SIEM] [CASES] Build lego blocks case details view (elastic#60864) Create Painless Lab app (elastic#57538) [SIEM] Move Timeline Template field to first step of rule creation (elastic#60840) [Reporting/New Platform Migration] Use a new config service on server-side (elastic#55882) [Alerting] allow email action to not require auth (elastic#60839) [Maps] Default ES document layer scaling type to clusters and show scaling UI in the create wizard (elastic#60668) [APM] Collect telemetry about data/API performance (elastic#51612) Implement Kibana Login Selector (elastic#53010) ...
darnautov
added a commit
that referenced
this pull request
Mar 24, 2020
* [ML] add estimateModelMemory to the setup endpoint * [ML] wip caching cardinality checks * [ML] refactor * [ML] fix a fallback time range * [ML] fix typing issue * [ML] fields_aggs_cache.ts as part of fields_service * [ML] fix types, add comments * [ML] check for MML overrides * [ML] disable estimateModelMemory * [ML] fix typing * [ML] check for empty max mml * [ML] refactor, update types, fix jobsForModelMemoryEstimation * [ML] fix override lookup * [ML] resolve nit comments * [ML] init jobsForModelMemoryEstimation
5 tasks
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Part of #60386
/api/ml/modules/setup/{moduleId}kibana endpoint now supports a new parameterestimateModelMemoryto perform an actual model memory estimation instead of using hardcoded values from the JSON configuration of the module.cardinalityCheckProvidernow supports caching of cardinality field values based on the index, time field and provided time range. It still can be improved taking into account query and bucket span, and also having some expiration time, but for now, it suffices to reduce unnecessary cardinality checks.For now recognizer module wizard has
estimateModelMemorydisabled (0e3c9e1) and is going to use hardcoded values from the JSON config until all the required job is done on elasticsearch side.Checklist