[Metrics UI] Remove TSVB dependency from Metrics Explorer APIs by simianhacker · Pull Request #74804 · elastic/kibana

simianhacker · 2020-08-11T20:24:34Z

Summary

This PR removes the TSVB dependency from Metrics Explorer and introduces a new library called "Metrics API". The goal of this library is to create a replacement backend for both the Metrics Explorer, Snapshot APIs, and alert executors. This new library supports the following features:

Multiple group bys
Multiple metrics defined using ES native aggregations
Optional alignment of the timeseries to the end of the time range. By default ES aligns the buckets to the beginning of the time range
Drop last bucket, similar to how TSVB works
Auto bucketing with a floor, this uses the same syntax >=1m as TSVB
Paginate through groupings
Everything is Typescript
Doesn't depend on Metric UI's source or the KIbana request context

Here is an example of how to use this new service.

import { query } from 'server/lib/metrics';
import { MetricsAPIRequest, MetricsAPIResponse } from 'common/http_api';
import { createSearchClient } from 'server/lib/create_search_client';

const options: MetricsAPIRequest = {
  timerange: {
    field: '@timestamp',
    from: new Date('2020-01-01T00:00:00Z').getTime(),
    to: new Date('2020-01-01T01:00:00Z').getTime(),
    interval: '>=1m',
  },
  limit: 9,
  dropLastBucket: true,
  indexPattern: 'metrics-*',
  metrics: [
    { id: 'metric_0', aggregations: { metric_0: { avg: { field: 'system.cpu.user.pct' } } } },
  ],
};

const client = createSearchClient(requestContext, framework);
const response: MetricsAPIResponse = await query(client, options);

One of the goals of this new service is to provide a consistent way to query time series metrics from anywhere with in the Metrics UI app. The search client function can be swapped out with a function (with the same signature, ESSearchClient) that makes calls using a search strategy.

I plan to turn the Metrics API into a Search Strategy in a separate PR which will make using this library even more ubiquitous. I'm also going to submit a follow up PR to use this API for the backend for the Snapshot API.

Testing this PR

The goal of this PR is to go unnoticed by the user. The only change to the integration test was the ID field when the user didn't specify any group by's. Before this change it returned "Everything" as the ID, now it returns * and we display the "Everything" title only in the UI when it matches this value.

Ingest data from Metricbeat running on your localhost
Open Metrics Explorer on Master and create a visualization
Check out this new PR, the visualization and data should be identical.

Checklist

Delete any items that are not applicable to this PR.

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Unit or functional tests were updated or added to match the most common scenarios

elasticmachine · 2020-08-12T19:03:46Z

Pinging @elastic/logs-metrics-ui (Team:logs-metrics-ui)

…with-metric-api-for-metrics-explorer

Zacqary

Couple style nits but looks good overall

x-pack/plugins/infra/public/pages/metrics/metrics_explorer/components/chart_title.tsx

x-pack/plugins/infra/server/lib/metrics/lib/calculate_bucket_size/calculate_auto.ts

x-pack/plugins/infra/server/lib/metrics/lib/calculate_bucket_size/unit_to_seconds.ts

x-pack/plugins/infra/server/lib/metrics/lib/convert_histogram_buckets_to_timeseries.ts

x-pack/plugins/infra/server/routes/metrics_explorer/lib/find_interval_for_metrics.ts

…ponents/chart_title.tsx Co-authored-by: Zacqary Adam Xeper <Zacqary@users.noreply.github.com>

…ize/calculate_auto.ts Co-authored-by: Zacqary Adam Xeper <Zacqary@users.noreply.github.com>

…buckets_to_timeseries.ts Co-authored-by: Zacqary Adam Xeper <Zacqary@users.noreply.github.com>

…nterval_for_metrics.ts Co-authored-by: Zacqary Adam Xeper <Zacqary@users.noreply.github.com>

…with-metric-api-for-metrics-explorer

Zacqary

Small typo in how many days are in a year

x-pack/plugins/infra/server/lib/metrics/lib/calculate_bucket_size/unit_to_seconds.ts

Co-authored-by: Zacqary Adam Xeper <Zacqary@users.noreply.github.com>

kibanamachine · 2020-08-13T17:14:25Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 92e9e16

Build metrics

@kbn/optimizer bundle module count

id	value	diff	baseline
infra	1057	+1	1056

async chunks size

id	value	diff	baseline
infra	3.5MB	+3.5KB	3.5MB

page load bundle size

id	value	diff	baseline
infra	275.5KB	+431.0B	275.1KB

History

💚 Build #67831 succeeded 1f1206c
💔 Build #67830 failed de23105
💔 Build #67826 failed db171df
💚 Build #67778 succeeded cca2b58
💔 Build #67767 failed f15f3257dcf7752c427b4ad259988a62f812544d

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

Zacqary

Checked out and tested this and it seems to be working! Thanks for doing this!

…ic#74804) * [Metrics UI] Remove TSVB dependency from Metrics Explorer APIs * Update x-pack/plugins/infra/public/pages/metrics/metrics_explorer/components/chart_title.tsx Co-authored-by: Zacqary Adam Xeper <Zacqary@users.noreply.github.com> * Update x-pack/plugins/infra/server/lib/metrics/lib/calculate_bucket_size/calculate_auto.ts Co-authored-by: Zacqary Adam Xeper <Zacqary@users.noreply.github.com> * Update x-pack/plugins/infra/server/lib/metrics/lib/calculate_bucket_size/calculate_auto.ts Co-authored-by: Zacqary Adam Xeper <Zacqary@users.noreply.github.com> * Update x-pack/plugins/infra/server/lib/metrics/lib/convert_histogram_buckets_to_timeseries.ts Co-authored-by: Zacqary Adam Xeper <Zacqary@users.noreply.github.com> * Update x-pack/plugins/infra/server/lib/metrics/lib/convert_histogram_buckets_to_timeseries.ts Co-authored-by: Zacqary Adam Xeper <Zacqary@users.noreply.github.com> * Update x-pack/plugins/infra/server/routes/metrics_explorer/lib/find_interval_for_metrics.ts Co-authored-by: Zacqary Adam Xeper <Zacqary@users.noreply.github.com> * Fixing some names, changing some units * Reverting TSVB calculate_auto; fixing names in infra * Fixing translation names * Fixing typo Co-authored-by: Zacqary Adam Xeper <Zacqary@users.noreply.github.com> Co-authored-by: Zacqary Adam Xeper <Zacqary@users.noreply.github.com>

… (#75017) * [Metrics UI] Remove TSVB dependency from Metrics Explorer APIs * Update x-pack/plugins/infra/public/pages/metrics/metrics_explorer/components/chart_title.tsx Co-authored-by: Zacqary Adam Xeper <Zacqary@users.noreply.github.com> * Update x-pack/plugins/infra/server/lib/metrics/lib/calculate_bucket_size/calculate_auto.ts Co-authored-by: Zacqary Adam Xeper <Zacqary@users.noreply.github.com> * Update x-pack/plugins/infra/server/lib/metrics/lib/calculate_bucket_size/calculate_auto.ts Co-authored-by: Zacqary Adam Xeper <Zacqary@users.noreply.github.com> * Update x-pack/plugins/infra/server/lib/metrics/lib/convert_histogram_buckets_to_timeseries.ts Co-authored-by: Zacqary Adam Xeper <Zacqary@users.noreply.github.com> * Update x-pack/plugins/infra/server/lib/metrics/lib/convert_histogram_buckets_to_timeseries.ts Co-authored-by: Zacqary Adam Xeper <Zacqary@users.noreply.github.com> * Update x-pack/plugins/infra/server/routes/metrics_explorer/lib/find_interval_for_metrics.ts Co-authored-by: Zacqary Adam Xeper <Zacqary@users.noreply.github.com> * Fixing some names, changing some units * Reverting TSVB calculate_auto; fixing names in infra * Fixing translation names * Fixing typo Co-authored-by: Zacqary Adam Xeper <Zacqary@users.noreply.github.com> Co-authored-by: Zacqary Adam Xeper <Zacqary@users.noreply.github.com> Co-authored-by: Zacqary Adam Xeper <Zacqary@users.noreply.github.com>

* master: (23 commits) Adding API test for custom link transaction example (elastic#74238) [Uptime] Singular alert (elastic#74659) Revert "attempt excluding a codeowners directory" (elastic#75023) [Metrics UI] Remove TSVB dependency from Metrics Explorer APIs (elastic#74804) Remove degraded state from ES status service (elastic#75007) [Reporting/Functional] unskip pagination test (elastic#74973) [Resolver] Stale query string values are removed when resolver's component instance ID changes. (elastic#74979) Add public url to Workplace Search plugin (elastic#74991) [Reporting] Update more Server Types for TaskManager (elastic#74915) [I18n] verify select icu-message options are in english (elastic#74963) Make data.search.aggs available on the server. (elastic#74472) [Security Solution][Resolver] Graph Control Tests and Update Simulator Selectors (elastic#74680) attempt excluding a codeowners directory [ML] DF Analytics: allow failed job to be stopped by force via the UI (elastic#74710) Add kibana-core-ui-designers team (elastic#74970) [Metrics UI] Fix inventory footer misalignment (elastic#74707) Remove legacy optimizer (elastic#73154) Update design-specific GH code-owners (elastic#74877) skip test Reporting paginates content elastic#74922 [Metrics UI] Add Jest tests for alert previews (elastic#74890) ...

simianhacker mentioned this pull request Aug 11, 2020

[Metrics UI] Consolidate Inventory and Metrics Explorer APIs #71575

Closed

17 tasks

simianhacker force-pushed the replace-tsvb-with-metric-api-for-metrics-explorer branch 7 times, most recently from 05ff5e0 to f15f325 Compare August 12, 2020 18:22

[Metrics UI] Remove TSVB dependency from Metrics Explorer APIs

ad02fdf

simianhacker force-pushed the replace-tsvb-with-metric-api-for-metrics-explorer branch from f15f325 to ad02fdf Compare August 12, 2020 18:54

simianhacker added Feature:Metrics UI Metrics UI feature Team:Infra Monitoring UI - DEPRECATED DEPRECATED - Label for the Infra Monitoring UI team. Use Team:obs-ux-infra_services v7.10.0 v8.0.0 release_note:skip Skip the PR/issue when compiling release notes labels Aug 12, 2020

simianhacker marked this pull request as ready for review August 12, 2020 19:03

simianhacker requested a review from a team as a code owner August 12, 2020 19:03

Merge branch 'master' of github.com:elastic/kibana into replace-tsvb-…

cca2b58

…with-metric-api-for-metrics-explorer

Zacqary requested changes Aug 12, 2020

View reviewed changes

simianhacker and others added 8 commits August 12, 2020 15:20

Update x-pack/plugins/infra/public/pages/metrics/metrics_explorer/com…

5717722

…ponents/chart_title.tsx Co-authored-by: Zacqary Adam Xeper <Zacqary@users.noreply.github.com>

Update x-pack/plugins/infra/server/lib/metrics/lib/calculate_bucket_s…

f99585e

…ize/calculate_auto.ts Co-authored-by: Zacqary Adam Xeper <Zacqary@users.noreply.github.com>

Update x-pack/plugins/infra/server/lib/metrics/lib/calculate_bucket_s…

68077f1

…ize/calculate_auto.ts Co-authored-by: Zacqary Adam Xeper <Zacqary@users.noreply.github.com>

Update x-pack/plugins/infra/server/lib/metrics/lib/convert_histogram_…

0fa99e4

…buckets_to_timeseries.ts Co-authored-by: Zacqary Adam Xeper <Zacqary@users.noreply.github.com>

Update x-pack/plugins/infra/server/lib/metrics/lib/convert_histogram_…

5ff126e

…buckets_to_timeseries.ts Co-authored-by: Zacqary Adam Xeper <Zacqary@users.noreply.github.com>

Update x-pack/plugins/infra/server/routes/metrics_explorer/lib/find_i…

db171df

…nterval_for_metrics.ts Co-authored-by: Zacqary Adam Xeper <Zacqary@users.noreply.github.com>

Fixing some names, changing some units

84a89b5

Merge branch 'master' of github.com:elastic/kibana into replace-tsvb-…

a6f36ae

…with-metric-api-for-metrics-explorer

simianhacker requested a review from a team August 12, 2020 22:47

Reverting TSVB calculate_auto; fixing names in infra

de23105

simianhacker requested review from Zacqary and removed request for a team August 12, 2020 22:55

Fixing translation names

1f1206c

Zacqary requested changes Aug 13, 2020

View reviewed changes

x-pack/plugins/infra/server/lib/metrics/lib/calculate_bucket_size/unit_to_seconds.ts Outdated Show resolved Hide resolved

Fixing typo

92e9e16

Co-authored-by: Zacqary Adam Xeper <Zacqary@users.noreply.github.com>

simianhacker requested a review from Zacqary August 13, 2020 16:23

Zacqary mentioned this pull request Aug 13, 2020

[Metrics UI] [Discuss] Combining Inventory and Metrics Explorer? #74972

Closed

Zacqary approved these changes Aug 13, 2020

View reviewed changes

simianhacker merged commit 1632391 into elastic:master Aug 14, 2020

simianhacker mentioned this pull request Aug 14, 2020

[7.x] [Metrics UI] Remove TSVB dependency from Metrics Explorer APIs (#74804) #75017

Merged

simianhacker deleted the replace-tsvb-with-metric-api-for-metrics-explorer branch April 17, 2024 15:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Metrics UI] Remove TSVB dependency from Metrics Explorer APIs#74804

[Metrics UI] Remove TSVB dependency from Metrics Explorer APIs#74804
simianhacker merged 13 commits intoelastic:masterfrom
simianhacker:replace-tsvb-with-metric-api-for-metrics-explorer

simianhacker commented Aug 11, 2020 •

edited

Loading

Uh oh!

elasticmachine commented Aug 12, 2020

Uh oh!

Zacqary left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Zacqary left a comment

Uh oh!

Uh oh!

kibanamachine commented Aug 13, 2020

Uh oh!

Zacqary left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

simianhacker commented Aug 11, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Testing this PR

Checklist

Uh oh!

elasticmachine commented Aug 12, 2020

Uh oh!

Zacqary left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Zacqary left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kibanamachine commented Aug 13, 2020

💚 Build Succeeded

Build metrics

@kbn/optimizer bundle module count

async chunks size

page load bundle size

History

Uh oh!

Zacqary left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

simianhacker commented Aug 11, 2020 •

edited

Loading