[Discover] Add support for unmapped fields using the fields API by majagrubic · Pull Request #89074 · elastic/kibana

majagrubic · 2021-01-23T19:35:15Z

Summary

Fixes #88192.
This PR adds support for unmapped fields using the new fields API. The logic we discussed to implement this is as follows:

There is a switch in the UI with the deprecation notice; the switch is on by default for existing saved searches; off by default for new saved searches
To determine if it's an existing (pre 7.12) saved search, we need a data migration

Current flow:
(notice that for same index pattern unmapped fields don't show if it's a new saved search, but they show up when you load a pre-existing one)

How to test this?

Before pulling the branch, create an index in ES with unmapped fields. Eg:

PUT /my-test-index
{
  "settings": {
    "number_of_shards": 1
  },
  "mappings": {
    "properties": {
      "field1": { "type": "text" }
    }
  }
}

PUT /my-test-index/_doc/1
{
  "field1": "Some blob of text",
  "field2": "Some other text"
}

PUT /my-test-index/_doc/2
{
  "field1": "Some blob of text",
  "field2": "Some other text"
}

PUT /my-test-index/_doc/3
{
  "field1": "Some blob of text",
  "field3": 12
}

PUT /my-test-index-2
{
  "settings": {
    "number_of_shards": 1
  },
  "mappings": {
     "dynamic": false
  }
}

PUT /my-test-index-2/_doc/1
{
  "field1": "Some blob of text",
  "field2": "Some other text"
}

PUT /my-test-index-2/_doc/2
{
  "field1": "Some blob of text",
  "field2": "Some other text"
}

PUT /my-test-index-2/_doc/3
{
  "field4": "Some blob of text",
  "field5": 12
}


PUT /my-test-index-2/_doc/4
{
  "field4": "Some blob of text",
  "field5": 12
}

Create an index pattern from the newly created index
Go to Discover, select the index pattern created under 2
Save the existing saved search
Pull the branch, restart Kibana
Observe how unmapped fields are behaving

Checklist

Delete any items that are not applicable to this PR.

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios
Any UI touched in this PR is usable by keyboard only (learn more about keyboard accessibility)
Any UI touched in this PR does not create any new axe failures (run axe in browser: FF, Chrome)
If a plugin configuration key changed, check if it needs to be whitelisted in the cloud and added to the docker list
This renders correctly on smaller devices using a responsive layout. (You can test this in your browser)
This was checked for cross-browser compatibility

For maintainers

This was checked for breaking API changes and was labeled appropriately

…ibana into discover-ssource-fields

kertal · 2021-02-01T08:58:52Z

@elasticmachine merge upstream

kertal · 2021-02-01T10:12:43Z

test/functional/fixtures/es_archiver/data/mappings.json

@@ -0,0 +1,450 @@
+{


since you're using

test/functional/fixtures/es_archiver/unmapped_fields

for your functionals, you can delete

test/functional/fixtures/es_archiver/data

src/plugins/discover/public/application/helpers/update_search_source.ts

test/functional/apps/discover/_indexpattern_with_unmapped_fields.ts

kertal · 2021-02-01T10:57:41Z

src/plugins/discover/public/application/components/sidebar/discover_sidebar.tsx

+   */
+  unmappedFieldsConfig?: {
+    /**
+     * callback funtction to change the value of `showUnmappedFields` flag


typo: funtction -> function

kertal

It's looking pretty good 👍 (so far just minor remarks) , while testing I've found one case that I think needs adaptation or discussion. when adding a pre712 saved search, unmapped fields are displayed in Discover, but not when using the saved search on a Dashboard:

saved-search-pre712.mp4

majagrubic · 2021-02-02T20:02:52Z

@elasticmachine merge upstream

majagrubic · 2021-02-03T07:01:32Z

@kertal I fixed the embeddable so that it works with the new API now

kertal · 2021-02-03T09:26:10Z

@elasticmachine merge upstream

kibanamachine · 2021-02-03T12:16:27Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 64282c8

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`discover`	413.4KB	417.1KB	+3.7KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`discover`	69.1KB	69.1KB	+47.0B

Saved Objects .kibana field count

Every field in each saved object type adds overhead to Elasticsearch. Kibana needs to keep the total field count below Elasticsearch's default limit of 1000 fields. Only specify field mappings for the fields you wish to search on or query. See https://www.elastic.co/guide/en/kibana/master/development-plugin-saved-objects.html#_mappings

id	before	after	diff
`search`	9	10	+1

History

💚 Build #103326 succeeded 0b8102b
💚 Build #103160 succeeded b16e3fd
💔 Build #103051 failed d7b2fdf
💔 Build #102968 failed abecb21
💚 Build #102497 succeeded fda7725

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

kertal

Code LGTM 👍, embeddable now works. Tested locally using Chrome/Safari, works as expected. One note: using the Angular legacy grid, rendering of records in the table are not updated when there's a refetch when switching the Show unmapped. This is due to the internal Angular caching works. There's no easy workaround here than: switching to EuiDataGrid. There it works, so this should not block this PR.

kertal · 2021-02-03T12:33:10Z

src/plugins/discover/public/application/embeddable/search_embeddable.ts

    );
+    if (useNewFieldsApi) {
+      searchSource.removeField('fieldsFromSource');
+      const fields: Record<string, any> = { field: '*' };


Nit: another any that could bite the dust

Will address this in the next PR

…#89074) (#90184) * [Discover] Add support for unmapped fields using the fields API (#89074) * Add search source to example plugin. * Add uiSetting for fields API. * Update SearchSource to support fields API. * [PoC] reading from the fields API in Discover * Add N fields as a default column * Make fields column non-removeable * Do not add 'fields' to state * Remove fields from app state and read from source when needed * Remove fields column if a new column is added * Add search source to example plugin. * Add uiSetting for fields API. * Update SearchSource to support fields API. * Improve error handling in search examples plugin. * Add unit tests for legacy behavior. * Remove uiSettings feature flag; add fieldsFromSource config. * Rewrite flatten() based on final API design. * Update example app based on final API design. * Update maps app to use legacy fieldsFromSource. * Update Discover to use legacy fieldsFromSource. * Rename source filters to field filters. * Address feedback. * Update generated docs. * Update maps functional test. * Formatting fields column similar to _source * Moving logic for using search API to updating search source * Fix small merge error * Move useSource switch to Discover section of advanced settings * Do not use fields and source at the same time * Remove unmapped fields switch * Add basic support for grouping multifields * Remove output.txt * Fix some merge leftovers * Fix some merge leftovers * Fix merge errors * Fix typescript errors and update nested fields logic * Add a unit test * Fixing field formats * Fix multifield selection logic * Request all fields from source * Fix eslint * Fix default columns when switching between _source and fields * More unit tests * Update API changes * Add unit test for discover field details footer * Remove unused file * Remove fields formatting from index pattern * Remove unnecessary check * Addressing design comments * Fixing fields column display and renaming it to Document * Adding more unit tests * Adding a missing check for useNewFieldsAPI; minor fixes * Fixing typescript error * Remove unnecessary console statement * Add missing prop * Fixing import order * Adding functional test to test fields API * [Functional test] Clean up in after * Fixing context app * Addressing PR comments * Add support for unmapped fields * Add data migration * Add toggle unmapped fields logic * Adding more unit tests * Some cleanup * More unit tests * Fixing failing snapshot * Add tooltip next to unmapped switch * Add functional test for the feature * Fixing a typo in a functional test * Refetch data when unmapped fields value changes * Updating mapping * Support for fields API in search embeddable * Addressing PR comments * Fix failing unit test * Updating the text Co-authored-by: Luke Elmers <luke.elmers@elastic.co> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> * Fixing ES license Co-authored-by: Luke Elmers <luke.elmers@elastic.co> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

lukeelmers and others added 30 commits November 2, 2020 10:05

Add search source to example plugin.

a232550

Add uiSetting for fields API.

ba796e0

Update SearchSource to support fields API.

9fbcc82

[PoC] reading from the fields API in Discover

5afd264

Add N fields as a default column

ac1e273

Make fields column non-removeable

e6f2a22

Merge master

c420e96

Do not add 'fields' to state

8ad5f99

Remove fields from app state and read from source when needed

64f0868

Remove fields column if a new column is added

b843d33

Add search source to example plugin.

1a3196d

Add uiSetting for fields API.

0cf84e4

Update SearchSource to support fields API.

ad23b7f

Improve error handling in search examples plugin.

3681311

Add unit tests for legacy behavior.

5ac33bb

Remove uiSettings feature flag; add fieldsFromSource config.

9e6c89f

Rewrite flatten() based on final API design.

d34ab66

Update example app based on final API design.

4773949

Update maps app to use legacy fieldsFromSource.

d272609

Update Discover to use legacy fieldsFromSource.

30c7c5f

Rename source filters to field filters.

24a5f9d

Address feedback.

9c7a967

Update generated docs.

5c124f8

Update maps functional test.

79e59a8

Formatting fields column similar to _source

f7d304b

Merge upstream

4053bf1

Moving logic for using search API to updating search source

ee33143

Fix small merge error

e3ace69

Merge branch 'master' into feat/ssource-fields

84136f6

Merge branch 'feat/ssource-fields' of https://github.com/lukeelmers/k…

804c5f8

…ibana into discover-ssource-fields

Merge branch 'master' into unmapped-fields

fda7725

kertal reviewed Feb 1, 2021

View reviewed changes

src/plugins/discover/public/application/helpers/update_search_source.ts Show resolved Hide resolved

kertal reviewed Feb 1, 2021

View reviewed changes

test/functional/apps/discover/_indexpattern_with_unmapped_fields.ts Show resolved Hide resolved

kertal reviewed Feb 1, 2021

View reviewed changes

Maja Grubic added 5 commits February 2, 2021 08:26

Merge branch 'master' into unmapped-fields

c2de87d

Updating mapping

6e12b58

Support for fields API in search embeddable

b51c0be

Addressing PR comments

abecb21

Fix failing unit test

d7b2fdf

Merge branch 'master' into unmapped-fields

b16e3fd

majagrubic requested a review from kertal February 3, 2021 07:01

kibanamachine and others added 2 commits February 3, 2021 04:26

Merge branch 'master' into unmapped-fields

0b8102b

Updating the text

64282c8

kertal approved these changes Feb 3, 2021

View reviewed changes

majagrubic requested a review from gchaps February 3, 2021 12:58

majagrubic merged commit c6be748 into elastic:master Feb 3, 2021

majagrubic deleted the unmapped-fields branch February 3, 2021 15:05

This was referenced Feb 3, 2021

[7.x] [Discover] Add support for unmapped fields using the fields API (#89074) #90184

Merged

[Discover] Minor cleanup #90260

Merged

timroes added release_note:enhancement and removed release_note:roadmap labels Mar 10, 2021

kertal mentioned this pull request Apr 26, 2022

[Reporting] CSV Export shows no data when an unmapped field is used as a column #130557

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Discover] Add support for unmapped fields using the fields API#89074

[Discover] Add support for unmapped fields using the fields API#89074
majagrubic merged 94 commits intoelastic:masterfrom
majagrubic:unmapped-fields

majagrubic commented Jan 23, 2021 •

edited

Loading

Uh oh!

kertal commented Feb 1, 2021

Uh oh!

kertal Feb 1, 2021

Uh oh!

Uh oh!

Uh oh!

kertal Feb 1, 2021

Uh oh!

kertal left a comment

Uh oh!

majagrubic commented Feb 2, 2021

Uh oh!

majagrubic commented Feb 3, 2021

Uh oh!

kertal commented Feb 3, 2021

Uh oh!

kibanamachine commented Feb 3, 2021

Uh oh!

kertal left a comment

Uh oh!

kertal Feb 3, 2021

Uh oh!

majagrubic Feb 3, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

majagrubic commented Jan 23, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

How to test this?

Checklist

For maintainers

Uh oh!

kertal commented Feb 1, 2021

Uh oh!

kertal Feb 1, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

kertal Feb 1, 2021

Choose a reason for hiding this comment

Uh oh!

kertal left a comment

Choose a reason for hiding this comment

Uh oh!

majagrubic commented Feb 2, 2021

Uh oh!

majagrubic commented Feb 3, 2021

Uh oh!

kertal commented Feb 3, 2021

Uh oh!

kibanamachine commented Feb 3, 2021

💚 Build Succeeded

Metrics [docs]

Async chunks

Page load bundle

Saved Objects .kibana field count

History

Uh oh!

kertal left a comment

Choose a reason for hiding this comment

Uh oh!

kertal Feb 3, 2021

Choose a reason for hiding this comment

Uh oh!

majagrubic Feb 3, 2021

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

majagrubic commented Jan 23, 2021 •

edited

Loading