[ML] Add runtime fields support by qn895 · Pull Request #78700 · elastic/kibana

qn895 · 2020-09-28T22:22:26Z

Summary

This PR is part of #77462 to add support for runtime fields. Instead of querying for fields using _source, it instead uses the new fields API https://www.elastic.co/guide/en/elasticsearch/reference/master/search-fields.html#search-fields-param.

However, there are some limitations. Because the values are returned as a flat list in the fields section in each hit, we are keeping the results field under source to maintain the shape of the records.

ML

Source data grid table cells are empty for runtime fields defined in the mappings that are not present in _source
Anomaly detection datafeed preview uses _source (advanced wizard, jobs list)
Categorization job wizard prevents use of a runtime field defined in the mappings that is not present in _source, failing at the No examples for this field could be found step
Testing a custom_url which uses a runtime field for a job which has not yet created anomalies looks for field _source

Transforms

Source data grid table cells are empty for runtime fields defined in the mappings that are not present in _source (example below responsetimeplus is a runtime field)
This was checked for cross-browser compatibility

…fined in the mappings that is not present in _source

…fined in the mappings that are not present in _source

…elds-support # Conflicts: # x-pack/plugins/transform/public/app/hooks/use_index_data.ts # x-pack/plugins/transform/public/shared_imports.ts

…elds-support

elasticmachine · 2020-10-01T17:18:22Z

Pinging @elastic/ml-ui (:ml)

qn895 · 2020-10-01T21:11:23Z

@elasticmachine merge upstream

peteharverson

Tested and LGTM

jgowdyelastic · 2020-10-02T12:49:20Z

x-pack/plugins/ml/server/models/job_service/new_job/categorization/validation_results.ts

-          values: { percent: NULL_COUNT_PERCENT_LIMIT * 100 },
-        }),
-      });
+    // add check so message won't be added if all values are null


if 100% of field valuess are null, why do we not want to report this?

Sorry I should have been clearer in the comment.

Currently, if all values are null, the message we are giving back is: 'No examples for this field could be found. Please ensure the selected date range contains data.'

The added check here ensures we only give that one message back instead of both 'More than 75% of field values are null.' and 'No examples for this field could be found.'

ok makes sense.
It think it might worth adding something to this comment for future readers. Something like:

If all values are null, VALIDATION_RESULT.NO_EXAMPLES will be raised. So we don't need to display this warning as well

Thanks. Updated the comment to be more explicit here. 219c661

alvarezmelissa87 · 2020-10-02T15:44:49Z

x-pack/plugins/ml/public/application/data_frame_analytics/common/get_index_data.ts

        }, {} as EsSorting);

      const { pageIndex, pageSize } = pagination;
+      // TODO: remove results_field from `fields` when possible


Is this TODO still necessary?

The todo is necessary for now because we are getting the flattened fields.ml.blah as well as the unflattened _source.ml object. Since the fields API might change in the future to either return the values unflattened or add a way to exclude certain fields, maybe we can do a follow up PR for when the API changes.

alvarezmelissa87

Just left a small comment but LGTM ⚡

qn895 · 2020-10-05T01:42:30Z

@elasticmachine merge upstream

qn895 · 2020-10-05T14:06:10Z

@elasticmachine merge upstream

jgowdyelastic

Code LGTM, added a message about updating a comment.

…heck

kibanamachine · 2020-10-05T17:32:23Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 219c661

Metrics [docs]

async chunks size

id	before	after	diff
`ml`	10.6MB	10.6MB	+70.4KB
`transform`	1.2MB	1.2MB	+106.0B
total			+70.5KB

History

💚 Build #79625 succeeded 4cd2497
💚 Build #79453 succeeded e97e65a
💚 Build #78953 succeeded b94cfb6
💔 Build #78851 failed 544e789
💔 Build #78105 failed 37fa808

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

qn895 added 5 commits September 28, 2020 16:47

[ML] Fix Categorization job wizard prevents use of a runtime field de…

ba98e99

…fined in the mappings that is not present in _source

[ML] Fix Categorization job wizard prevents use of a runtime field de…

38fce03

…fined in the mappings that is not present in _source

[ML] Fix source data grid table cells are empty for runtime fields de…

054a26b

…fined in the mappings that are not present in _source

[ML] Move getProcessedFields to common

c286841

[ML] Update previews for transforms

15064b7

qn895 added :ml Feature:Anomaly Detection ML anomaly detection Feature:Data Frame Analytics ML data frame analytics features labels Sep 28, 2020

qn895 self-assigned this Sep 28, 2020

qn895 changed the title ~~[ML] WIP Add runtime fields support~~ [ML] WIP - Add runtime fields support Sep 28, 2020

qn895 added 5 commits September 29, 2020 15:31

[ML] Add runtime support for search previews & url

d6ecdad

Merge remote-tracking branch 'upstream/master' into ml-add-runtime-fi…

43aee77

…elds-support # Conflicts: # x-pack/plugins/transform/public/app/hooks/use_index_data.ts # x-pack/plugins/transform/public/shared_imports.ts

[ML] Fix import order

37fa808

[ML] Add check for fields for when it's undefined

18bfb9c

Merge remote-tracking branch 'upstream/master' into ml-add-runtime-fi…

544e789

…elds-support

qn895 changed the title ~~[ML] WIP - Add runtime fields support~~ [ML] Add runtime fields support Oct 1, 2020

qn895 marked this pull request as ready for review October 1, 2020 17:18

qn895 requested a review from a team as a code owner October 1, 2020 17:18

qn895 requested review from alvarezmelissa87, darnautov, jgowdyelastic, peteharverson and walterra October 1, 2020 17:19

qn895 added release_note:skip Skip the PR/issue when compiling release notes v7.10.0 v8.0.0 labels Oct 1, 2020

Merge branch 'master' into ml-add-runtime-fields-support

b94cfb6

peteharverson removed the release_note:skip Skip the PR/issue when compiling release notes label Oct 2, 2020

peteharverson added the release_note:enhancement label Oct 2, 2020

peteharverson approved these changes Oct 2, 2020

View reviewed changes

jgowdyelastic reviewed Oct 2, 2020

View reviewed changes

alvarezmelissa87 reviewed Oct 2, 2020

View reviewed changes

alvarezmelissa87 approved these changes Oct 2, 2020

View reviewed changes

Merge branch 'master' into ml-add-runtime-fields-support

e97e65a

Merge branch 'master' into ml-add-runtime-fields-support

4cd2497

jgowdyelastic approved these changes Oct 5, 2020

View reviewed changes

[ML] Add clearer comment on the added nullCount !== examples.length c…

219c661

…heck

qn895 merged commit ad89e6f into elastic:master Oct 5, 2020

qn895 deleted the ml-add-runtime-fields-support branch October 5, 2020 18:01

qn895 mentioned this pull request Oct 5, 2020

[7.x] [ML] Add runtime fields support (#78700) #79528

Merged

qn895 added a commit to qn895/kibana that referenced this pull request Oct 5, 2020

[ML] Add runtime fields support (elastic#78700)

3909a8c

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

peteharverson mentioned this pull request Oct 7, 2020

[ML] Support runtime fields #77462

Closed

12 tasks

Conversation

qn895 commented Sep 28, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

ML

Transforms

Uh oh!

elasticmachine commented Oct 1, 2020

Uh oh!

qn895 commented Oct 1, 2020

Uh oh!

peteharverson left a comment

Choose a reason for hiding this comment

Uh oh!

jgowdyelastic Oct 2, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

qn895 Oct 2, 2020

Choose a reason for hiding this comment

Uh oh!

jgowdyelastic Oct 5, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

qn895 Oct 5, 2020

Choose a reason for hiding this comment

Uh oh!

alvarezmelissa87 Oct 2, 2020

Choose a reason for hiding this comment

Uh oh!

qn895 Oct 2, 2020

Choose a reason for hiding this comment

Uh oh!

alvarezmelissa87 left a comment

Choose a reason for hiding this comment

Uh oh!

qn895 commented Oct 5, 2020

Uh oh!

qn895 commented Oct 5, 2020

Uh oh!

jgowdyelastic left a comment

Choose a reason for hiding this comment

Uh oh!

kibanamachine commented Oct 5, 2020

💚 Build Succeeded

Metrics [docs]

async chunks size

History

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

qn895 commented Sep 28, 2020 •

edited

Loading

jgowdyelastic Oct 2, 2020 •

edited

Loading

jgowdyelastic Oct 5, 2020 •

edited

Loading