[Logs UI] Logs overview queries for the observability dashboard by afgomez · Pull Request #70413 · elastic/kibana

afgomez · 2020-07-01T09:52:21Z

Summary

This PR provides log overview data for the observability dashboard.

Checklist

Delete any items that are not applicable to this PR.

Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios

x-pack/plugins/infra/public/utils/logs_overview_fetchers.ts

jasonrhodes

Looks like I'm getting an error that I can't quite track down. The error seems to be a vague one related to the esSearcher observable:

VM2611:1 POST http://localhost:5601/internal/search/es 400 (Bad Request)

logs_overview_fetchers.ts:44 Error: Bad Request
    at Fetch.fetchResponse (fetch.ts:162)
    at async interceptResponse (intercept.ts:58)
    at async http:/localhost:5601/9007199254740991/bundles/core/core.entry.js:50572

Is the esSearcher API documented anywhere or are you just piecing it together reading code?

I trigger this with the following patch:

diff --git a/x-pack/plugins/observability/public/application/index.tsx b/x-pack/plugins/observability/public/application/index.tsx
index 21a9fabf445..7e9e3bc6358 100644
--- a/x-pack/plugins/observability/public/application/index.tsx
+++ b/x-pack/plugins/observability/public/application/index.tsx
@@ -9,10 +9,28 @@ import { EuiThemeProvider } from '../../../../legacy/common/eui_styled_component
 import { AppMountParameters, CoreStart } from '../../../../../src/core/public';
 import { Home } from '../pages/home';
 import { PluginContext } from '../context/plugin_context';
+import { getDataHandler } from '../data_handler';
 
 export const renderApp = (core: CoreStart, { element }: AppMountParameters) => {
   const i18nCore = core.i18n;
   const isDarkMode = core.uiSettings.get('theme:darkMode');
+
+  const getData = async () => {
+    const now = new Date();
+    const params = {
+      startTime: '2020-06-29T11:38:23.747Z',
+      endTime: now.toString(),
+      bucketSize: '30s',
+    };
+    const data = await getDataHandler('infra_logs')?.fetchData(params);
+    console.log('logs data', { data });
+  };
+
+  getData().catch((e) => {
+    console.log('an error occurred while getting obs data');
+    console.log(e);
+  });
+
   ReactDOM.render(
     <PluginContext.Provider value={{ core }}>
       <EuiThemeProvider darkMode={isDarkMode}>

and then navigating to /app/observability

x-pack/plugins/infra/public/utils/logs_overview_fetchers.ts

cauemarcondes · 2020-07-02T10:52:56Z

@afgomez as we discussed today can you return the logs rate per minute in the series, and the average in the stats, please?

I've done the same on APM: https://github.com/elastic/kibana/blob/master/x-pack/plugins/apm/server/lib/observability_dashboard/get_transaction_coordinates.ts#L50

afgomez · 2020-07-02T13:57:44Z

@jasonrhodes seems an issue with how the date is passed. The startTime is in ISO string, but the endTime is in epoch_millis. I'm not sure ES lets you mix and match (therefore the 400 error).

jasonrhodes · 2020-07-02T14:39:20Z

@jasonrhodes seems an issue with how the date is passed. The startTime is in ISO string, but the endTime is in epoch_millis. I'm not sure ES lets you mix and match (therefore the 400 error).

Oh I did .toString() so it's not millis but that's not the right string method, I need toISOString() 🤦

jasonrhodes

Looks great, nice work.

jasonrhodes · 2020-07-02T15:44:17Z

@afgomez if you look at the tests here that Chris added for metrics, could we duplicate those for some logs tests for our fetcher methods? #70529

We could merge this and do it immediately in another PR to unblock this for Caue if you want, but I'd like to get tests in ASAP.

afgomez · 2020-07-02T16:06:01Z

@jasonrhodes I was planning to :)

jasonrhodes · 2020-07-02T16:18:53Z

@jasonrhodes I was planning to :)

Yes but now I get credit for being the one who suggested it. #leadering

The graphs need to show the log rate per minute instead of the log count. This commit adds two helper functions to normalize those values.

x-pack/plugins/infra/public/utils/logs_overview_fetchers.ts

cauemarcondes

Observability registration LGTM.

kibanamachine · 2020-07-03T14:48:53Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: c2d130c

Build metrics

@kbn/optimizer bundle module count

id	value	diff	baseline
infra	606	+3	603

History

💔 Build #58648 failed 929afde
💚 Build #58017 succeeded 0dfddfd
💚 Build #57798 succeeded 92607df

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

afgomez · 2020-07-03T14:52:01Z

Merging now. There's a test missing that I will add in a separate PR.

* master: [APM-UI] e2e speed up build (elastic#70704) skip flaky suite (elastic#70764) skip flaky suite (elastic#70762) [Security Solution][Endpoint] Update to new manifest format (without compression) (elastic#70752) [functional tests] test url field formatter on dashboard and discover (elastic#70736) logout from transform_poweruser user in after method of transform tests (elastic#70644) [SECURITY] Bug fix for topN on draggables (elastic#70450) [Logs UI] Reorganise log rate anomaly table (elastic#69516) Update dependency @elastic/charts to v19.7.0 (elastic#69791) Add googlecloud metricbeat module to Kibana Home (elastic#70652) [Logs UI] Logs overview queries for the observability dashboard (elastic#70413) [Lens] Fitting functions (elastic#69820)

* master: [APM-UI] e2e speed up build (elastic#70704) skip flaky suite (elastic#70764) skip flaky suite (elastic#70762) [Security Solution][Endpoint] Update to new manifest format (without compression) (elastic#70752) [functional tests] test url field formatter on dashboard and discover (elastic#70736) logout from transform_poweruser user in after method of transform tests (elastic#70644) [SECURITY] Bug fix for topN on draggables (elastic#70450) [Logs UI] Reorganise log rate anomaly table (elastic#69516) Update dependency @elastic/charts to v19.7.0 (elastic#69791) Add googlecloud metricbeat module to Kibana Home (elastic#70652) [Logs UI] Logs overview queries for the observability dashboard (elastic#70413)

* actions/feature: fixed type errors [APM-UI] e2e speed up build (elastic#70704) skip flaky suite (elastic#70764) skip flaky suite (elastic#70762) [Security Solution][Endpoint] Update to new manifest format (without compression) (elastic#70752) [functional tests] test url field formatter on dashboard and discover (elastic#70736) logout from transform_poweruser user in after method of transform tests (elastic#70644) [SECURITY] Bug fix for topN on draggables (elastic#70450) [Logs UI] Reorganise log rate anomaly table (elastic#69516) Update dependency @elastic/charts to v19.7.0 (elastic#69791) Add googlecloud metricbeat module to Kibana Home (elastic#70652) [Logs UI] Logs overview queries for the observability dashboard (elastic#70413)

…#70413) (#70730) Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

Alejandro Fernández Gómez added 6 commits July 1, 2020 10:59

Tweak the typing of the fetcher functions

0685b41

Extract mocked stats and series

6209620

Pass arguments to fetchLogsOverview

bd39e10

Query data plugin and get the required aggregations

69abef1

Process aggregations into a consumable format

8261fa4

Extract query parts onto builder functions

62884e0

afgomez added Feature:Logs UI Logs UI feature release_note:skip Skip the PR/issue when compiling release notes Team:Infra Monitoring UI - DEPRECATED DEPRECATED - Label for the Infra Monitoring UI team. Use Team:obs-ux-infra_services v7.9.0 labels Jul 1, 2020

Perform hasData check

92607df

afgomez force-pushed the 68531-logs-overview-queries branch from d985d6d to 92607df Compare July 1, 2020 09:57

cauemarcondes reviewed Jul 1, 2020

View reviewed changes

x-pack/plugins/infra/public/utils/logs_overview_fetchers.ts Outdated Show resolved Hide resolved

Alejandro Fernández Gómez added 3 commits July 1, 2020 16:39

Fetch metadata from the source configuration

f6215c0

Use source status API to check for data presence

966a7e7

Extract default sourceId to a constant

2b10df8

afgomez marked this pull request as ready for review July 1, 2020 14:42

afgomez requested a review from a team as a code owner July 1, 2020 14:42

afgomez mentioned this pull request Jul 1, 2020

[Observability] Homepage experience (Milestone 1) #68176

Closed

jasonrhodes reviewed Jul 1, 2020

View reviewed changes

x-pack/plugins/infra/public/utils/logs_overview_fetchers.ts Outdated Show resolved Hide resolved

Extract DEFAULT_SOURCE_ID constant to a shared location

0dfddfd

jasonrhodes requested changes Jul 2, 2020

View reviewed changes

cauemarcondes reviewed Jul 2, 2020

View reviewed changes

x-pack/plugins/infra/public/utils/logs_overview_fetchers.ts Outdated Show resolved Hide resolved

jasonrhodes approved these changes Jul 2, 2020

View reviewed changes

Alejandro Fernández Gómez added 2 commits July 3, 2020 11:46

Add test for hasData

865942e

Normalize values

929afde

The graphs need to show the log rate per minute instead of the log count. This commit adds two helper functions to normalize those values.

cauemarcondes reviewed Jul 3, 2020

View reviewed changes

x-pack/plugins/infra/public/utils/logs_overview_fetchers.ts Outdated Show resolved Hide resolved

afgomez requested a review from jasonrhodes July 3, 2020 11:07

Alejandro Fernández Gómez added 3 commits July 3, 2020 13:47

Pass application path

2ec953f

Merge branch 'master' into 68531-logs-overview-queries

f07a384

Fix types after merge from master

c2d130c

afgomez requested a review from cauemarcondes July 3, 2020 13:28

cauemarcondes approved these changes Jul 3, 2020

View reviewed changes

afgomez merged commit f3573f3 into elastic:master Jul 3, 2020

afgomez deleted the 68531-logs-overview-queries branch July 3, 2020 14:59

This was referenced Jul 3, 2020

[Logs UI] Complete test suite for #70413 #70729

Closed

[7.x] [Logs UI] Logs overview queries for the observability dashboard (#70413) #70730

Merged

afgomez pushed a commit that referenced this pull request Jul 6, 2020

[7.x] [Logs UI] Logs overview queries for the observability dashboard (…

108818a

…#70413) (#70730) Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

Conversation

afgomez commented Jul 1, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Checklist

Uh oh!

Uh oh!

Uh oh!

jasonrhodes left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cauemarcondes commented Jul 2, 2020

Uh oh!

afgomez commented Jul 2, 2020

Uh oh!

jasonrhodes commented Jul 2, 2020

Uh oh!

jasonrhodes left a comment

Choose a reason for hiding this comment

Uh oh!

jasonrhodes commented Jul 2, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

afgomez commented Jul 2, 2020

Uh oh!

jasonrhodes commented Jul 2, 2020

Uh oh!

Uh oh!

cauemarcondes left a comment

Choose a reason for hiding this comment

Uh oh!

kibanamachine commented Jul 3, 2020

💚 Build Succeeded

Build metrics

@kbn/optimizer bundle module count

History

Uh oh!

afgomez commented Jul 3, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

afgomez commented Jul 1, 2020 •

edited

Loading

jasonrhodes left a comment •

edited

Loading

jasonrhodes commented Jul 2, 2020 •

edited

Loading