[Logs UI] Add category anomalies to anomalies page by Kerry350 · Pull Request #70982 · elastic/kibana

Kerry350 · 2020-07-07T16:33:39Z

Summary

Overview

Implements an API endpoint for retrieving anomalies.
Sorting and pagination has been moved from the client side to the anomalies API.
- The logic around next / previous page is encapsulated in the client side hook.
- The ES "hack" to reverse the sort order and use search_after is used to implement paging backwards.
Category anomalies are rendered in the table.
LogEntryRateExample resources have been renamed LogEntryExample as they're in no way specific to log rate.

Notes / caveats

I was unsure on the best route to take regarding the document tiebreaker for the sorting. Usually we would use the source configurations' tiebreaker property, but the anomalies are ML documents and not Filebeat documents. I'd considered the _id, but the docs advise strongly against this unless the _id is duplicated in another field that has doc value enabled, given we don't control that that wasn't possible. I've used _doc, which is our default. I know the doc properties have their own caveats, but wasn't sure what else could be used.
The auto refresh of the page really doesn't work very well with the pagination. Given the pagination and sorting is now determined server side, if the time range changes, then the pagination needs to be reset. If the time range shifts, the information we have is incorrect, and we need to re-evaluate. Two possible solutions:
- Turn auto refresh off by default, and have it as an opt in thing.
- Make the refresh rate a bit longer - it's currently every 30 seconds, maybe this could be every 5 minutes?

If somebody is really looking to stream their anomalies, rather than page through them and look at historical data, it's just a case of setting the sort (on start time) and staying on page 1.

Pagination options (page size) are technically all hooked up server and client side, but not exposed in the UI. I didn't know if we wanted this, or just to just stick with a sensible number (I've used 25).
On the design dataset and category are in bold in the rows. Given recommendations seem to be to use i18n.translate over FormattedMessage now I omitted the bold. If we think this is really important, it can be changed.

Next steps

We should add dataset filtering. This is technically in the design for the adding categories work, but wasn't part of the outlined work. I'd like to open up this PR now due to FF, and an already large changeset, but that would be the next logical step around this functionality. (PR open)
There is definitely some further cleanup that can be done, e.g. we can remove some of the alternate formattings of the log rate results data now.

Testing

It's useful to have varied data. E.g. both log rate and category anomalies, and have these from more than one dataset.

- Adds an anomalies API endpoint - Adds server side sorting / pagination to anomalies endpoint - Renders category examples in the anomalies table - Renames log entry rate examples to log entry examples

elasticmachine · 2020-07-07T16:33:42Z

Pinging @elastic/logs-metrics-ui (Team:logs-metrics-ui)

weltenwort

👀 I'll review this over the next days with the intention to minimize the conflicts between this and the modules list I'm working on (#64757).

weltenwort

I'm submitting this in several parts to shorten the feedback cycle. I hope it won't be too inconvenient that I'm jumping around between various aspects and comment on them as I notice them.

I'm seeing a weird effect that seems to be related to the pagination. This is how I can reproduce it:

load the page
disable the auto-refresh
paginate to the second page
click "refresh" besides the datepicker

This causes the "no data" screen to be displayed for me. Another click on "refresh" causes data to be shown again, but the pagination state is lost (i.e. I'm on page one again).

x-pack/plugins/infra/common/http_api/log_analysis/results/log_entry_anomalies.ts

x-pack/plugins/infra/public/pages/logs/log_entry_rate/sections/anomalies/table.tsx

Kerry350 · 2020-07-09T12:13:39Z

@weltenwort

This causes the "no data" screen to be displayed for me. Another click on "refresh" causes data to be shown again, but the pagination state is lost (i.e. I'm on page one again).

Hmm, I'm struggling to replicate this.

Following the same steps I see the following:

So I'm not thrown incorrectly into the "no data" state. I am returned to page 1, which would be expected as clicking "refresh" would change the time range we're looking at, and as soon as that changes pagination would need to be reset.

I've tried a few different date expressions etc 🤔

…entry_anomalies.ts Co-authored-by: Felix Stürmer <weltenwort@users.noreply.github.com>

weltenwort · 2020-07-09T12:29:30Z

Ok, it seems I didn't control for all the variables 🙈 I can reproduce it if I expand a row on the second page and then click "refresh".

Kerry350 · 2020-07-09T12:57:02Z

@weltenwort

Still no luck reproducing with the expanded row, interesting 🤔

weltenwort

While testing the page and reading the code I came a cross a few things that we might have to handle:

As mentioned below, the examples don't seem to be filtered by the category pattern.
The page doesn't work with only the categories job deployed. This is somewhat intertwined with the ml job list flyout, so it might make sense if I tackled this in #71132 or even a follow-up PR. What would the page look like without log rate data?
When deploying the log rate job anew, there was quite a long period where the anomalies API reported results but the page showed the empty state prompt:

This is probably also a relic of the previous focus of the page on the log entry rate results. The "hasResults" checks and related conditions might need refining.

x-pack/plugins/infra/public/pages/logs/log_entry_rate/sections/anomalies/expanded_row.tsx

x-pack/plugins/infra/server/lib/log_analysis/queries/common.ts

x-pack/plugins/infra/public/pages/logs/log_entry_rate/page_results_content.tsx

Kerry350 · 2020-07-09T13:37:12Z

As mentioned below, the examples don't seem to be filtered by the category pattern.
The example API used by the categories page does that. Should that API's functionality be merged into this one?

Hmm. Yeah, I missed this to be honest. Just applied the "examples around" like log rate anomalies. I'll get this working, probably with the outlook to merge what needs to be merged in a followup. I've merged certain things along the way, and left others separate.

The page doesn't work with only the categories job deployed. This is somewhat intertwined with the ml job list flyout, so it might make sense if I tackled this in #71132 or even a follow-up PR. What would the page look like without log rate data?

That would be great if you don't mind. I guess we'd want to show a callout in place of the chart without log rate data, telling the user to create the job.

When deploying the log rate job anew, there was quite a long period where the anomalies API reported results but the page showed the empty state prompt:

Ah, yes, need to tweak those has data checks (like you said).

…es.ts Co-authored-by: Felix Stürmer <weltenwort@users.noreply.github.com>

…b.com:Kerry350/kibana into 64755-expand-anomalies-page-to-add-categories

…es.ts Co-authored-by: Felix Stürmer <weltenwort@users.noreply.github.com>

…b.com:Kerry350/kibana into 64755-expand-anomalies-page-to-add-categories

…/anomalies/table.tsx Co-authored-by: Felix Stürmer <weltenwort@users.noreply.github.com>

…b.com:Kerry350/kibana into 64755-expand-anomalies-page-to-add-categories

…alies-page-to-add-categories

Kerry350 · 2020-07-13T12:43:19Z

@weltenwort I think everything is addressed and working now. Main things:

I've refactored the anomalies hooks state to use a reducer, this makes it (imo) a lot more readable, as well as eradicating the bugs.
Empty datasets should now work fine. I'm still trying to verify this 100% (having trouble with the dev cluster), but the fix is there.
For the jobId extraction at the expanded row level I've actually added the jobId to the anomaly result returned from the API. We might want to revisit this, I'm not sure. Originally, the log rate module context was being utilised at that level. I didn't want to pull in the category module context there as that would almost definitely start treading on the toes of your work, which seemed unnecessary as this works fine. On the other hand given how unassuming the anomlies API is (it doesn't expect or require particular jobs to exist, it'll just try to gather everything) it could be useful to have that property.
Fixed an additional bug with the "View anomaly in machine learning" link - the category ID needed passing through for category anomalies.

We should probably rethink the philosophy when and how we check for jobs, when we use the stored custom_settings instead of the source config and what parameters we pass at some point.

Definitely. I think there's a few things we can improve once this all comes together and we have a chance to breathe and reassess.

weltenwort

Well done regarding the reducer, feels pretty consistent and much easier to read now.

Great job overall, this was definitely a challenging enhancement to pivot the tab's focus like that 👏

weltenwort · 2020-07-13T13:52:02Z

x-pack/plugins/infra/public/pages/logs/log_entry_rate/sections/anomalies/expanded_row.tsx

-    sourceConfiguration: { sourceId },
-  } = useLogEntryRateModuleContext();
+}> = ({ anomaly, timeRange }) => {
+  const { sourceId } = useLogSourceContext();


Yes, this reduces the scope of the dependency 👍

weltenwort · 2020-07-13T14:31:30Z

x-pack/plugins/infra/public/pages/logs/log_entry_rate/sections/anomalies/chart.tsx

-          tooltip={tooltipProps}
-          baseTheme={isDarkMode ? DARK_THEME : LIGHT_THEME}
-          xDomain={{ min: timeRange.startTime, max: timeRange.endTime }}
+    <>


Oh, ok then... I would have expected something like

return !isLoading && !series.length ? ( <EuiEmptyPrompt ...> ) : ( <LoadingOverlayWrapper ...> );

to work as well.

weltenwort · 2020-07-13T14:38:22Z

Forgot to mention, the empty category example retrieval now works for me 👍

And I think putting the jobId into the anomaly response is totally reasonable - it's not a leak of abstraction since anomalies are an ML concept in the first place.

weltenwort · 2020-07-13T14:54:28Z

Another late thought, sorry 🙈 As the failing i18n check points out we lost a bit of accessibility of the loading spinner. Do we want to resurrect that LoadingOverlayContent?

Kerry350 · 2020-07-13T15:28:41Z

@weltenwort Thanks for the reviews 🙏

Pushed up those last little tweaks, hopefully this can go green in the background 🤞

kibanamachine · 2020-07-13T17:08:42Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 667f9c9

Build metrics

@kbn/optimizer bundle module count

id	value	diff	baseline
infra	607	+1	606

History

💔 Build #60868 failed 6cc04bb
💔 Build #60841 failed 298b66d
💔 Build #60811 failed ec0152c
💔 Build #60601 failed 4ff5799
💔 Build #60367 failed d84a7f9

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

* Add category anomalies to anomalies page Co-authored-by: Felix Stürmer <weltenwort@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

* Add category anomalies to anomalies page Co-authored-by: Felix Stürmer <weltenwort@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: Felix Stürmer <weltenwort@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

Add category anomalies to anomalies page

fbed94b

- Adds an anomalies API endpoint - Adds server side sorting / pagination to anomalies endpoint - Renders category examples in the anomalies table - Renames log entry rate examples to log entry examples

Kerry350 added release_note:enhancement v8.0.0 Feature:Logs UI Logs UI feature Team:Infra Monitoring UI - DEPRECATED DEPRECATED - Label for the Infra Monitoring UI team. Use Team:obs-ux-infra_services v7.9.0 labels Jul 7, 2020

Kerry350 added this to the Logs UI 7.9 milestone Jul 7, 2020

Kerry350 requested a review from a team July 7, 2020 16:33

Kerry350 self-assigned this Jul 7, 2020

weltenwort reviewed Jul 7, 2020

View reviewed changes

sgrodzicki linked an issue Jul 8, 2020 that may be closed by this pull request

[Logs UI] Show category anomalies in anomaly table #64755

Closed

node scripts/i18n_check --fix

f585973

Kerry350 mentioned this pull request Jul 8, 2020

[Logs UI] Anomalies page dataset filtering #71110

Merged

weltenwort reviewed Jul 9, 2020

View reviewed changes

Kerry350 and others added 7 commits July 9, 2020 13:17

Update x-pack/plugins/infra/common/http_api/log_analysis/results/log_…

9ab603f

…entry_anomalies.ts Co-authored-by: Felix Stürmer <weltenwort@users.noreply.github.com>

Update x-pack/plugins/infra/common/http_api/log_analysis/results/log_…

27b63a2

…entry_anomalies.ts Co-authored-by: Felix Stürmer <weltenwort@users.noreply.github.com>

Update x-pack/plugins/infra/common/http_api/log_analysis/results/log_…

ce78758

…entry_anomalies.ts Co-authored-by: Felix Stürmer <weltenwort@users.noreply.github.com>

Update x-pack/plugins/infra/common/http_api/log_analysis/results/log_…

96ef88b

…entry_anomalies.ts Co-authored-by: Felix Stürmer <weltenwort@users.noreply.github.com>

Update x-pack/plugins/infra/common/http_api/log_analysis/results/log_…

926feed

…entry_anomalies.ts Co-authored-by: Felix Stürmer <weltenwort@users.noreply.github.com>

Update x-pack/plugins/infra/common/http_api/log_analysis/results/log_…

9cc61f2

…entry_anomalies.ts Co-authored-by: Felix Stürmer <weltenwort@users.noreply.github.com>

Update x-pack/plugins/infra/common/http_api/log_analysis/results/log_…

c700591

…entry_anomalies.ts Co-authored-by: Felix Stürmer <weltenwort@users.noreply.github.com>

Add aria labels

0f919ee

weltenwort reviewed Jul 9, 2020

View reviewed changes

Kerry350 added 2 commits July 9, 2020 14:50

Amend pluralisation on variables and fix useEffect dependency

23346a3

Ensure category anomlies fetch examples relevant to the category

c540e33

Kerry350 and others added 15 commits July 13, 2020 11:21

Extract variable from common

f2e44e6

Update x-pack/plugins/infra/server/lib/log_analysis/log_entry_anomali…

1dd7c03

…es.ts Co-authored-by: Felix Stürmer <weltenwort@users.noreply.github.com>

Merge branch '64755-expand-anomalies-page-to-add-categories' of githu…

c4f8725

…b.com:Kerry350/kibana into 64755-expand-anomalies-page-to-add-categories

Update x-pack/plugins/infra/server/lib/log_analysis/log_entry_anomali…

fbf5c09

…es.ts Co-authored-by: Felix Stürmer <weltenwort@users.noreply.github.com>

Merge branch '64755-expand-anomalies-page-to-add-categories' of githu…

0e5441b

…b.com:Kerry350/kibana into 64755-expand-anomalies-page-to-add-categories

Update x-pack/plugins/infra/public/pages/logs/log_entry_rate/sections…

a2c1d2f

…/anomalies/table.tsx Co-authored-by: Felix Stürmer <weltenwort@users.noreply.github.com>

Merge branch '64755-expand-anomalies-page-to-add-categories' of githu…

9198baf

…b.com:Kerry350/kibana into 64755-expand-anomalies-page-to-add-categories

Amend message handling

51a1936

Wrap no data state with a loading overlay wrapper

ec0152c

Merge remote-tracking branch 'upstream/master' into 64755-expand-anom…

751ae96

…alies-page-to-add-categories

Ensure expanded rows extract their job id from the anomaly record

706239b

Update variable naming

c5b2243

Amend i18n syntax

f95947b

Ensure category is passed to category links

99a3e50

Commit

298b66d

Appease eslint

6cc04bb

weltenwort approved these changes Jul 13, 2020

View reviewed changes

Kerry350 added 3 commits July 13, 2020 16:03

Change conditional syntax

d3a9380

Add LoadingOverlayContent back for improved accessbility

47b17de

node scripts/i18n_check --fix

667f9c9

Kerry350 merged commit 4db5816 into elastic:master Jul 13, 2020

Kerry350 mentioned this pull request Jul 13, 2020

[7.x] [Logs UI] Add category anomalies to anomalies page (#70982) #71490

Merged

weltenwort mentioned this pull request Jul 13, 2020

[Logs UI] Show log analysis ML jobs in a list #71132

Merged

Conversation

Kerry350 commented Jul 7, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Overview

Notes / caveats

Next steps

Testing

Uh oh!

elasticmachine commented Jul 7, 2020

Uh oh!

weltenwort left a comment

Choose a reason for hiding this comment

Uh oh!

weltenwort left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Kerry350 commented Jul 9, 2020

Uh oh!

weltenwort commented Jul 9, 2020

Uh oh!

Kerry350 commented Jul 9, 2020

Uh oh!

weltenwort left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Kerry350 commented Jul 9, 2020

Uh oh!

Kerry350 commented Jul 13, 2020

Uh oh!

weltenwort left a comment

Choose a reason for hiding this comment

Uh oh!

weltenwort Jul 13, 2020

Choose a reason for hiding this comment

Uh oh!

weltenwort Jul 13, 2020

Choose a reason for hiding this comment

Uh oh!

weltenwort commented Jul 13, 2020

Uh oh!

weltenwort commented Jul 13, 2020

Uh oh!

Kerry350 commented Jul 13, 2020

Uh oh!

kibanamachine commented Jul 13, 2020

💚 Build Succeeded

Build metrics

@kbn/optimizer bundle module count

History

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Kerry350 commented Jul 7, 2020 •

edited

Loading