[ES|QL] Enabling the timepicker for indices without the @timestamp field by stratoula · Pull Request #184361 · elastic/kibana

stratoula · 2024-05-28T14:51:05Z

Summary

Based on elastic/elasticsearch#108421

Closes #180805

It allows the users to add an ?earliest and ?latest variable in the ES|QL editor.

When we are detecting this variable, we are also sending the values to ES.

Earliest is the from value of the date picker
Latest is the to value of the date picker

Usage in bucket

This enables 2 very important features:

I can use the values of the time picker in the bucket function
I can use the time picker for indices without @timestamp field

For reviewers

Although it seems as a big PR, the majority of the changes happen due to the signature change of the getESQLAdHocDataview
The ML changes are mostly because the ML code has a lot of repetition. I think the code needs to be refactored to have a central point (preferably the getESQLResults from the esql-utils. I will create an issue for the ML team.
I am not proposing this in bucket autocomplete because it doesnt work great in general for the date histogram case. We are working on autocomplete improvements so I am expecting this to be part of a follow up PR.
I want to talk to the docs team to add it in the docs.

Follow ups

Change the histogram to use the bucket instead of the date_trunc (needs investigation first)
Speak with the docs team about adding an example on our official docs

Flaky test runner

https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/6521

…fix'

stratoula · 2024-06-18T10:47:17Z

src/plugins/unified_histogram/public/services/lens_vis_service.ts

-      dataView.isTimeBased() &&
-      query &&
-      isOfAggregateQueryType(query) &&
-      getAggregateQueryMode(query) === 'esql' &&


ℹ️ Irrelevant with this PR, I just removed it as is not needed anymore (we removed SQL)

stratoula · 2024-06-18T10:48:51Z

/ci

ryankeairns · 2024-06-18T20:05:18Z

Looks interesting! I'm wondering how else we might use it... 🤔

In this PR, am I understanding correctly that ?earliest and ?latest are parameters being defined by us, in the code?
Separately, can users/authors define named parameters 'on the fly'?

stratoula · 2024-06-19T06:04:35Z

@ryankeairns yes exactly, these are system parameters, defined by us. Eventually being part of our autocomplete too.

Separately, can users/authors define named parameters 'on the fly'?

They should and I think a nice idea to do so is to combine controls with named parameters. Not part of this PR though, we need to think about it and it needs design input my friend

ryankeairns · 2024-06-19T16:11:18Z

@ryankeairns yes exactly, these are system parameters, defined by us. Eventually being part of our autocomplete too.

Separately, can users/authors define named parameters 'on the fly'?

They should and I think a nice idea to do so is to combine controls with named parameters. Not part of this PR though, we need to think about it and it needs design input my friend

Great, thanks for clarifying.

Regarding system parameters, might we consider a unique prefix/naming convention? Otherwise, we could run into more complexity in validation, later, where we have to block/error on the use of system param names when users attempt to add them to their query, unknowingly.

Something like...
?system_earliest , ?esql_earliest or ?.earliest

Co-authored-by: Julia Rechkunova <julia.rechkunova@gmail.com>

…esql/_esql_view.ts Co-authored-by: Julia Rechkunova <julia.rechkunova@gmail.com>

…utils/get_esql_data_view.test.ts Co-authored-by: Julia Rechkunova <julia.rechkunova@gmail.com>

jughosta · 2024-07-16T13:22:09Z

@stratoula

Would be great to probably extend getInitialESQLQuery logic so it can append where with parameters and current data view index pattern automatically when navigating to ES|QL mode. Does not have to be in this PR though.

You mean to translate KQL / DSL filters to where?

No, I didn't mean this. Let's say that a sample ecommerce data has order_date as the current data view time field. When switching on Discover from data view mode to ES|QL mode, I would like it to pick this time field up automatically and use from ecommerce | where order_date >= ?earliest and order_date < ?latest | limit 10 as the initial query instead of from ecommerce | limit 10 so as a user I don't have to type in the parameters myself.

Also this way we would teach how to use these named params.

stratoula · 2024-07-16T13:30:28Z

@jughosta I see now 👍 This is not a bad idea, I will add it as a discussion point for our next ESQL sync to see how the other feel about it!

nickofthyme

Vis changes LGTM

nreese · 2024-07-17T14:38:43Z

I tried using the named variables in maps and am getting an error. Does something else need to be done to use ?earliest and ?latest?

Update

I noticed in the inspector that discover queries contain params while maps queries do not

"params": [
    {
      "earliest": "2024-07-17T14:11:35.452Z"
    },
    {
      "latest": "2024-07-17T14:26:35.452Z"
    }
  ],

nreese · 2024-07-17T14:46:33Z

x-pack/plugins/maps/public/classes/sources/esql_source/esql_source.tsx

      const timeFilter = getTime(undefined, timeRange, {
        fieldName: this._descriptor.dateField,
      });
+      const namedParams = getEarliestLatestParams(this._descriptor.esql, timeRange);


It looks like the problem I flagged is happening because getEarliestLatestParams is in if (requestMeta.applyGlobalTime) block. Instead, getEarliestLatestParams should be moved outside of this if statement so params are always added.

I would assume users would disable "Apply global time range" when using ?earliest and ?latest since you would not want both.

Great find Nathan! I had to move the timeRange computation outside too. I hope it makes sense 04a243f

nreese · 2024-07-17T17:07:06Z

You will also need to update getApplyGlobalQuery and isTimeAware implementations in x-pack/plugins/maps/public/classes/sources/esql_source/esql_source.tsx. These methods should return true when getEarliestLatestParams returns values.

Without these changes, then changes to the global time selector are not applied to the layer.

stratoula · 2024-07-17T21:38:01Z

You will also need to update getApplyGlobalQuery and isTimeAware implementations in x-pack/plugins/maps/public/classes/sources/esql_source/esql_source.tsx. These methods should return true when getEarliestLatestParams returns values.

Without these changes, then changes to the global time selector are not applied to the layer.

done here bd30542

stratoula · 2024-07-17T21:38:19Z

/ci

nreese

kibana-gis changes LGTM
code review, tested in chrome.

elasticmachine · 2024-07-17T22:29:26Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: bd30542
Kibana Serverless Image: docker.elastic.co/kibana-ci/kibana-serverless:pr-184361-bd305428d242
Observability Deployment

Failed CI Steps

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`@kbn/es-types`	26	27	+1
`@kbn/esql-utils`	51	59	+8
total			+9

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`aiops`	563.2KB	563.6KB	+447.0B
`apm`	3.5MB	3.5MB	+447.0B
`dashboard`	532.7KB	533.2KB	+495.0B
`dataVisualizer`	794.9KB	796.8KB	+1.9KB
`discover`	812.5KB	812.0KB	-572.0B
`esql`	181.5KB	181.5KB	+43.0B
`lens`	1.5MB	1.5MB	+1.1KB
`maps`	3.0MB	3.0MB	+1.1KB
`ml`	4.6MB	4.6MB	+447.0B
`observabilityAIAssistantApp`	149.6KB	150.0KB	+447.0B
`securitySolution`	16.3MB	16.3MB	+1.5KB
`slo`	874.7KB	875.1KB	+447.0B
`stackAlerts`	75.0KB	76.5KB	+1.4KB
`unifiedHistogram`	67.2KB	67.2KB	-46.0B
`unifiedSearch`	222.0KB	222.0KB	-10.0B
total			+9.2KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`dashboard`	43.2KB	43.3KB	+55.0B
`data`	417.2KB	417.7KB	+442.0B
`dataVisualizer`	24.1KB	24.4KB	+312.0B
`discover`	42.7KB	43.7KB	+1.1KB
`kbnUiSharedDeps-srcJs`	3.2MB	3.2MB	+321.0B
`maps`	54.6KB	54.7KB	+55.0B
`stackAlerts`	24.8KB	24.9KB	+60.0B
total			+2.3KB

Unknown metric groups

API count

id	before	after	diff
`@kbn/es-types`	26	27	+1
`@kbn/esql-utils`	53	63	+10
total			+11

async chunk count

id	before	after	diff
`stackAlerts`	3	4	+1

History

💛 Build #222150 was flaky 04a243f
💔 Build #222101 failed de171db
💔 Build #222100 failed de171db
💚 Build #221791 succeeded a4ae461

Enabling the timepicker with named params

a8bd2c9

stratoula mentioned this pull request May 29, 2024

[ES|QL] Support time series data indices without @timestamp field #180805

Closed

stratoula and others added 8 commits May 29, 2024 14:07

Merge branch 'main' into named-params-system

0a34007

[CI] Auto-commit changed files from 'node scripts/lint_ts_projects --…

0a25b1e

…fix'

Merge branch 'main' into named-params-system

1dc36e4

Merge branch 'main' into named-params-system

bae44e5

Merge branch 'main' into named-params-system

e8ce552

Merge branch 'main' into named-params-system

b4d28aa

Remove files

d63c101

Remove hardcoded implementation

725ebf0

stratoula changed the title ~~Enabling the timepicker with named params for indices without the @timestamp field~~ [ES|QL] Enabling the timepicker with named params for indices without the @timestamp field Jun 13, 2024

stratoula added 11 commits June 17, 2024 09:36

Merge branch 'main' into named-params-system

186f76a

Merge branch 'main' into named-params-system

6c258df

Merge branch 'main' into named-params-system

c968784

Ast implementation

bd61d48

Merge branch 'main' into named-params-system

adcc017

timefield assertion in one place

23f7817

Merge branch 'main' into named-params-system

cfdd424

Fixes

dbe7616

Cleanup

99277f7

Revert

088cb77

More cleanuo

9582af1

stratoula commented Jun 18, 2024

View reviewed changes

More cleanups

fd8e0d5

stratoula changed the title ~~[ES|QL] Enabling the timepicker with named params for indices without the @timestamp field~~ [ES|QL] Enabling the timepicker for indices without the @timestamp field Jun 18, 2024

Merge branch 'main' into named-params-system

35418ba

stratoula and others added 6 commits July 16, 2024 06:10

Update test/functional/apps/discover/esql/_esql_view.ts

1d87fc3

Co-authored-by: Julia Rechkunova <julia.rechkunova@gmail.com>

Update x-pack/test_serverless/functional/test_suites/common/discover/…

1a1e6cf

…esql/_esql_view.ts Co-authored-by: Julia Rechkunova <julia.rechkunova@gmail.com>

Update src/plugins/discover/public/application/main/state_management/…

6ced788

…utils/get_esql_data_view.test.ts Co-authored-by: Julia Rechkunova <julia.rechkunova@gmail.com>

Update src/plugins/discover/public/application/main/state_management/…

3418758

…utils/get_esql_data_view.test.ts Co-authored-by: Julia Rechkunova <julia.rechkunova@gmail.com>

Update src/plugins/discover/public/application/main/state_management/…

d06a1f6

…utils/get_esql_data_view.test.ts Co-authored-by: Julia Rechkunova <julia.rechkunova@gmail.com>

Update src/plugins/discover/public/application/main/state_management/…

90984d4

…utils/get_esql_data_view.test.ts Co-authored-by: Julia Rechkunova <julia.rechkunova@gmail.com>

Address PR comments

073d956

Merge branch 'main' into named-params-system

a4ae461

nickofthyme approved these changes Jul 16, 2024

View reviewed changes

dgieselaar approved these changes Jul 17, 2024

View reviewed changes

Merge with main and resolve conflicts

de171db

nreese reviewed Jul 17, 2024

View reviewed changes

adcoelho approved these changes Jul 17, 2024

View reviewed changes

stratoula added 2 commits July 17, 2024 08:57

Fix i18n check

81db001

Fix maps usage

04a243f

Apply PR comments

bd30542

nreese mentioned this pull request Jul 17, 2024

[maps][ESQL] disable "Apply global time range" checkbox when statement contains ?earliest or ?latest #188602

Open

nreese approved these changes Jul 17, 2024

View reviewed changes

stratoula enabled auto-merge (squash) July 17, 2024 21:43

stratoula disabled auto-merge July 17, 2024 23:23

stratoula merged commit fcf2702 into elastic:main Jul 17, 2024

stratoula mentioned this pull request Jul 24, 2024

[Unified search][ES|QL] For dataviews without the @timestamp field, transition to ES|QL with the named params #189032

Closed

stratoula mentioned this pull request Oct 31, 2024

Change the timefield for ES|QL visualizations #198563

Closed

Conversation

stratoula commented May 28, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

For reviewers

Follow ups

Flaky test runner

Uh oh!

stratoula Jun 18, 2024

Choose a reason for hiding this comment

Uh oh!

stratoula commented Jun 18, 2024

Uh oh!

ryankeairns commented Jun 18, 2024

Uh oh!

stratoula commented Jun 19, 2024

Uh oh!

ryankeairns commented Jun 19, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jughosta commented Jul 16, 2024

Uh oh!

stratoula commented Jul 16, 2024

Uh oh!

nickofthyme left a comment

Choose a reason for hiding this comment

Uh oh!

nreese commented Jul 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Update

Uh oh!

nreese Jul 17, 2024

Choose a reason for hiding this comment

Uh oh!

stratoula Jul 17, 2024

Choose a reason for hiding this comment

Uh oh!

nreese commented Jul 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

stratoula commented Jul 17, 2024

Uh oh!

stratoula commented Jul 17, 2024

Uh oh!

nreese left a comment

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Jul 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💛 Build succeeded, but was flaky

Failed CI Steps

Metrics [docs]

Public APIs missing comments

Async chunks

Page load bundle

API count

async chunk count

History

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

11 participants

stratoula commented May 28, 2024 •

edited

Loading

ryankeairns commented Jun 19, 2024 •

edited

Loading

nreese commented Jul 17, 2024 •

edited

Loading

nreese commented Jul 17, 2024 •

edited

Loading

elasticmachine commented Jul 17, 2024 •

edited

Loading