[Lens][TSVB] Ad-hoc dataViews for index pattern string mode in TSVB. by Kuznietsov · Pull Request #143500 · elastic/kibana

Kuznietsov · 2022-10-18T08:28:10Z

Summary

Completes part of #138236.
Added support of ad-hoc dataViews while converting TSVB visualizations, when index pattern string mode is turned on.

elasticmachine · 2022-10-18T12:07:32Z

Pinging @elastic/kibana-vis-editors @elastic/kibana-vis-editors-external (Team:VisEditors)

flash1293 · 2022-10-18T13:39:44Z

Doesn't work right for "overwrite data view":

Becomes:

dimaanj

Tested locally, changed logic when navigate from Discover works correctly.

dimaanj · 2022-10-19T10:16:02Z

Checked on main branch. Typing data view name in string mode doesn't give an error toast on each change.

tsvb.mp4

VladLasitsa

Found a problem: as in each change of model, we execute convert to lens logic, we will create a new ad hoc dataview on each typing action in index pattern field. It will lead to creation a lot of useless ad hoc dataviews in cache. Two possible way to resolve problem:

Move logic related to create ad hoc dataview to lens side
Create ad hoc dataview only when user click on "Edit in Lens" button

flash1293 · 2022-10-19T10:29:54Z

I think I like the first approach here, passing the spec to Lens instead of the id

Kuznietsov · 2022-10-26T09:41:49Z

@elastic/kibana-vis-editors, could you, please, review this PR? Thanks.

# Conflicts: # src/plugins/vis_types/timeseries/public/convert_to_lens/gauge/index.ts # src/plugins/vis_types/timeseries/public/convert_to_lens/index.test.ts # src/plugins/vis_types/timeseries/public/convert_to_lens/index.ts # src/plugins/vis_types/timeseries/public/convert_to_lens/metric/index.test.ts # src/plugins/vis_types/timeseries/public/convert_to_lens/metric/index.ts # src/plugins/vis_types/timeseries/public/convert_to_lens/timeseries/index.test.ts # src/plugins/vis_types/timeseries/public/convert_to_lens/timeseries/index.ts # src/plugins/vis_types/timeseries/public/convert_to_lens/top_n/index.test.ts # src/plugins/vis_types/timeseries/public/convert_to_lens/top_n/index.ts # src/plugins/vis_types/timeseries/public/convert_to_lens/types.ts # src/plugins/vis_types/timeseries/public/metrics_type.ts

# Conflicts: # src/plugins/vis_types/timeseries/public/convert_to_lens/metric/index.ts

flash1293 · 2022-10-31T11:11:47Z

In general it works fine, but there are too many data view lookups going on - every change to the configuration causes a bunch of them if index pattern string mode is used (which is a common thing).

This is what happens if multiple visualizations with index pattern string mode are used on a dashboard and the user hits refresh (after the chart loaded successfully previously):

For regular data views, there are no calls like this at all. Somehow we need to make sure these are cached.

flash1293

See above

Kuznietsov · 2022-10-31T11:18:25Z

In general it works fine, but there are too many data view lookups going on - every change to the configuration causes a bunch of them if index pattern string mode is used (which is a common thing)

@flash1293, that is how convertors are built and observation of "canNavigateToLens" on every change, and there is no way of fixing this. Can you explain, please, what you expect to see as a solution?

flash1293 · 2022-10-31T11:20:23Z

@Kunzetsov You already introduced an "adhoc data view cache" in the conversion logic, right? Could we just keep it around instead of resetting it?

Kuznietsov · 2022-10-31T11:29:30Z

@Kunzetsov You already introduced an "adhoc data view cache" in the conversion logic, right? Could we just keep it around instead of resetting it?

@flash1293, To be honest, we can't.

We should save ad-hoc dataviews only after click on "Edit in lens".
If the user is moving to the other part of the Kibana, all the created ad-hocs have to be cleaned up from the global ad-hoc dataviews cache. And after converting (checking if it is possible), all the dataViews, created by a call, are cleaned up.
If user is converting the vis to Lens, all the ad-hoc dataViews should be moved to Lens and kept in a global state.
Every change is calling "convertToLens", to detect if converting is possible and should be cleaned up after execution, because nobody guarantee, that user will convert the vis.
Even if we'll keep the cache between calls, we have to make an API call for checking if such index with the full name exists to fetch the time field (so, calls to the API will not reduce too much).

flash1293 · 2022-10-31T13:28:01Z

Fair considerations @Kunzetsov As it turned out the same issue occurs in other places as well, I will create an issue to solve it in the right place. However, we can't merge this before it's fixed as it will wreak havoc on more complex TSVB-heavy dashboards (performance wise)

Kuznietsov · 2022-10-31T13:32:12Z

Fair considerations @Kunzetsov As it turned out the same issue occurs in other places as well, I will create an issue to solve it in the right place. However, we can't merge this before it's fixed as it will wreak havoc on more complex TSVB-heavy dashboards (performance wise)

@flash1293, ok, will be waiting for the updates.

VladLasitsa · 2022-11-07T08:21:04Z

@elasticmachine merge upstream

kibanamachine · 2022-11-07T08:21:06Z

merge conflict between base and head

…vb-ad-hoc-dataview

…ld select

VladLasitsa · 2022-11-09T13:22:10Z

@elasticmachine merge upstream

kibana-ci · 2022-11-09T14:19:39Z

💚 Build Succeeded

Buildkite Build
Commit: b1c3aaf

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`visTypeTimeseries`	383	381	-2

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`lens`	1.3MB	1.3MB	+244.0B
`visTypeTimeseries`	509.2KB	509.7KB	+486.0B
total			+730.0B

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`visTypeTimeseries`	19.7KB	19.7KB	-36.0B

Unknown metric groups

ESLint disabled in files

id	before	after	diff
`osquery`	1	2	+1

ESLint disabled line counts

id	before	after	diff
`enterpriseSearch`	19	21	+2
`fleet`	59	65	+6
`osquery`	108	113	+5
`securitySolution`	440	446	+6
total			+19

Total ESLint disabled count

id	before	after	diff
`enterpriseSearch`	20	22	+2
`fleet`	67	73	+6
`osquery`	109	115	+6
`securitySolution`	517	523	+6
total			+20

History

💔 Build #86596 failed fe37115
💚 Build #86313 succeeded 2e4a088
💚 Build #85814 succeeded 5159470
💚 Build #83839 succeeded 3ad2a09
💔 Build #83808 failed 2888be6

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @VladLasitsa @Kunzetsov

flash1293

Tested around and everything works fine for me now, LGTM

Added addHocDataViews support for tsvb.

4b5687b

Kuznietsov changed the title ~~[Lens][TSVBAdded addHocDataViews support for tsvb.~~ [Lens][TSVB] Ad-hoc dataViews for index pattern string mode in TSVB. Oct 18, 2022

Kuznietsov requested a review from alexwizp October 18, 2022 08:29

Kuznietsov self-assigned this Oct 18, 2022

Kuznietsov added the WIP Work in progress label Oct 18, 2022

Kuznietsov mentioned this pull request Oct 18, 2022

[Lens] Convert legacy visualizations to Lens Phase I #138236

Closed

28 tasks

Kuznietsov added 2 commits October 18, 2022 11:36

Fixed collecting indexPatternIds from context.

11751a1

Fixed tests.

58c510e

Kuznietsov marked this pull request as ready for review October 18, 2022 12:07

Kuznietsov requested a review from a team as a code owner October 18, 2022 12:07

Kuznietsov requested a review from dimaanj October 18, 2022 12:44

Kuznietsov added 2 commits October 19, 2022 10:41

Merge branch 'main' into navigate-to-lens-tsvb-ad-hoc-dataview

df69ad5

Fixed overridden index pattern,

493eaf7

dimaanj reviewed Oct 19, 2022

View reviewed changes

dimaanj approved these changes Oct 19, 2022

View reviewed changes

VladLasitsa suggested changes Oct 19, 2022

View reviewed changes

Kuznietsov changed the title ~~[Lens][TSVB] Ad-hoc dataViews for index pattern string mode in TSVB.~~ [WIP][Lens][TSVB] Ad-hoc dataViews for index pattern string mode in TSVB. Oct 19, 2022

Kuznietsov added 7 commits October 26, 2022 19:52

Fixed table converter.

8ebf7b5

Merge branch 'main' into navigate-to-lens-tsvb-ad-hoc-dataview

316c394

# Conflicts: # src/plugins/vis_types/timeseries/public/convert_to_lens/metric/index.ts

Fixed merge conflict resolving mistake.

46e3e92

Merge branch 'main' into navigate-to-lens-tsvb-ad-hoc-dataview

0d5356c

Removed code, blocking the feature.

2888be6

Fixed tests.

3ad2a09

flash1293 requested changes Oct 31, 2022

View reviewed changes

Kuznietsov added blocked WIP Work in progress labels Oct 31, 2022

flash1293 mentioned this pull request Oct 31, 2022

Cache ad-hoc data views to avoid repeated field list calls #144253

Closed

VladLasitsa added 2 commits November 7, 2022 15:57

Merge remote-tracking branch 'upstream/main' into navigate-to-lens-ts…

5159470

…vb-ad-hoc-dataview

Remove adhoc dataview service, add update button for index patter fie…

2e4a088

…ld select

VladLasitsa removed the blocked label Nov 8, 2022

VladLasitsa self-assigned this Nov 8, 2022

Merge branch 'main' into navigate-to-lens-tsvb-ad-hoc-dataview

fe37115

Merge branch 'main' into navigate-to-lens-tsvb-ad-hoc-dataview

b1c3aaf

flash1293 approved these changes Nov 9, 2022

View reviewed changes

VladLasitsa merged commit 3d7b01e into elastic:main Nov 9, 2022

Conversation

Kuznietsov commented Oct 18, 2022 • edited by VladLasitsa Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Uh oh!

elasticmachine commented Oct 18, 2022

Uh oh!

flash1293 commented Oct 18, 2022

Uh oh!

dimaanj left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dimaanj commented Oct 19, 2022

Uh oh!

VladLasitsa left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

flash1293 commented Oct 19, 2022

Uh oh!

Kuznietsov commented Oct 26, 2022

Uh oh!

flash1293 commented Oct 31, 2022

Uh oh!

flash1293 left a comment

Choose a reason for hiding this comment

Uh oh!

Kuznietsov commented Oct 31, 2022

Uh oh!

flash1293 commented Oct 31, 2022

Uh oh!

Kuznietsov commented Oct 31, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

flash1293 commented Oct 31, 2022

Uh oh!

Kuznietsov commented Oct 31, 2022

Uh oh!

VladLasitsa commented Nov 7, 2022

Uh oh!

kibanamachine commented Nov 7, 2022

Uh oh!

VladLasitsa commented Nov 9, 2022

Uh oh!

kibana-ci commented Nov 9, 2022

💚 Build Succeeded

Metrics [docs]

Module Count

Async chunks

Page load bundle

ESLint disabled in files

ESLint disabled line counts

Total ESLint disabled count

History

Uh oh!

flash1293 left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Kuznietsov commented Oct 18, 2022 •

edited by VladLasitsa

Loading

dimaanj left a comment •

edited

Loading

VladLasitsa left a comment •

edited

Loading

Kuznietsov commented Oct 31, 2022 •

edited

Loading