Skip to content

[Logs UI] Show log analysis ML jobs in a list#71132

Merged
weltenwort merged 17 commits intoelastic:masterfrom
weltenwort:logs-ui-add-ml-module-list
Jul 13, 2020
Merged

[Logs UI] Show log analysis ML jobs in a list#71132
weltenwort merged 17 commits intoelastic:masterfrom
weltenwort:logs-ui-add-ml-module-list

Conversation

@weltenwort
Copy link
Copy Markdown
Member

@weltenwort weltenwort commented Jul 8, 2020

Summary

This modifies the ML job setup flyout of the anomalies tab to offer a list of the two available modules. Via the list each of the modules' jobs can be created or re-created.

closes #64757

Previews

image

image

image

Review and test notes

  • The implementation can be seen as rather simplistic because it hard-codes the two available modules in the results page and flyout. The advantage is that the changeset is significantly smaller than of a fully generic implementation.

  • The possible setup state were reduced because the visibility state of the flyout and its pages is tracked in a separate flyout state (shared via the context).

  • The category page flyout only contains the category setup form and should function the same as before.

  • Warnings for outdated configurations or definitions can be triggered by modifying the jobs with queries like the following:

    POST _ml/anomaly_detectors/kibana-logs-ui-${spaceId}-default-log-entry-categories-count/_update
    {
      "custom_settings": {
        "created_by": "ml-module-logs-ui-categories",
        "job_revision": 0, // decrement this to cause definition warning
        "logs_source_config": {
          "indexPattern": "filebeat-*", // add something here to cause configuration warning
          "timestampField": "@timestamp",
          "bucketSpan": 900000,
          "datasetFilter": {
            "type": "includeAll"
          }
        }
      }
    }
    
    POST _ml/anomaly_detectors/kibana-logs-ui-${spaceId}-default-log-entry-rate/_update
    {
      "custom_settings": {
        "created_by": "ml-module-logs-ui-analysis",
        "job_revision": 2, // decrement this to cause definition warning
        "logs_source_config": {
          "indexPattern": "filebeat-*", // add something here to cause configuration warning
          "timestampField": "@timestamp",
          "bucketSpan": 900000
        }
      }
    }
    

Future work

  • Make module management and setup hooks multi-module-aware.
  • Combine the module state warning callouts into a combined callout (possible per module).
  • Show the module state warnings in the flyout and the setup forms.
  • Move the setup states to a provider so they survive closing the flyout.

@weltenwort weltenwort added release_note:enhancement v8.0.0 Feature:Logs UI Logs UI feature Team:Infra Monitoring UI - DEPRECATED DEPRECATED - Label for the Infra Monitoring UI team. Use Team:obs-ux-infra_services v7.9.0 Logs Epic: Anomaly Explorer labels Jul 8, 2020
@weltenwort weltenwort added this to the Logs UI 7.9 milestone Jul 8, 2020
@weltenwort weltenwort self-assigned this Jul 8, 2020
@weltenwort weltenwort changed the title [Logs UI] [skip ci] Show log analysis ML jobs in a list [Logs UI] [skip-ci] Show log analysis ML jobs in a list Jul 8, 2020
@weltenwort weltenwort force-pushed the logs-ui-add-ml-module-list branch 2 times, most recently from bbabb39 to e5a5dbd Compare July 8, 2020 18:00
@weltenwort weltenwort force-pushed the logs-ui-add-ml-module-list branch 2 times, most recently from 8c92106 to e4d4394 Compare July 10, 2020 16:17
@weltenwort weltenwort changed the title [Logs UI] [skip-ci] Show log analysis ML jobs in a list [Logs UI] Show log analysis ML jobs in a list Jul 10, 2020
@weltenwort weltenwort force-pushed the logs-ui-add-ml-module-list branch from 564acf7 to 0a55573 Compare July 13, 2020 13:35
Copy link
Copy Markdown
Contributor

@Kerry350 Kerry350 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome work 🎉

I've left a few comments, but nothing that should block approval. I also didn't want to comment on anything that was already covered under "Future work".

All the states worked well for me (including outdated configurations / definitions):

Screenshot 2020-07-13 16 36 34
Screenshot 2020-07-13 16 37 21
Screenshot 2020-07-13 16 37 34
Screenshot 2020-07-13 16 38 34
Screenshot 2020-07-13 16 39 00

Only thing to mention out of these screenshots is the "success" callout rendered for me with the log entry job, should it do the same for just the categories job?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it a problem that some user facing text uses "ML" and some uses "Machine Learning"?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good question, I'm trying to find that out...

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry that it took a while to find an answer. The preferred variant is to spell out "Machine Learning" (capitalized if it refers to the app) and only use the abbreviation if there's too little room.

@weltenwort
Copy link
Copy Markdown
Member Author

Thank you for the timely review. I mostly addressed your comments, but I'm still looking for writer input regarding the ML vs Machine Learning question.

Only thing to mention out of these screenshots is the "success" callout rendered for me with the log entry job, should it do the same for just the categories job?

That condition could indeed be better. I just changed it to "when any of the jobs have just been created and no log rate or anomalies are found". I imagine that would be the situation in which the user would be most unsure. Does that make sense?

@weltenwort weltenwort force-pushed the logs-ui-add-ml-module-list branch from 3bbf3e2 to e412dec Compare July 13, 2020 18:26
@weltenwort weltenwort marked this pull request as ready for review July 13, 2020 18:28
@weltenwort weltenwort requested a review from a team as a code owner July 13, 2020 18:28
@elasticmachine
Copy link
Copy Markdown
Contributor

Pinging @elastic/logs-metrics-ui (Team:logs-metrics-ui)

@weltenwort
Copy link
Copy Markdown
Member Author

ℹ️ I rebased this on master because #70982 has been merged.

@kibanamachine
Copy link
Copy Markdown
Contributor

💚 Build Succeeded

Build metrics

@kbn/optimizer bundle module count

id value diff baseline
infra 599 -8 607

History

  • 💚 Build #60986 succeeded 3bbf3e2d80fb1c8502a112e2a28cbff1c2d80a2c
  • 💚 Build #60976 succeeded 9eb33f1472d05d0cda6224dd82f0abe404d6bb2d
  • 💔 Build #60864 failed 0a5557370d80fa5f6d3159f6690e1071d2374128
  • 💚 Build #60796 succeeded 564acf74c9890ee265cfd6bc983b1cafd4fc49e6
  • 💔 Build #60624 failed 9f460539a174745fe032653e472537bb20fc447f

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

Copy link
Copy Markdown
Contributor

@Kerry350 Kerry350 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That condition could indeed be better. I just changed it to "when any of the jobs have just been created and no log rate or anomalies are found". I imagine that would be the situation in which the user would be most unsure. Does that make sense?

Makes sense, thanks for the explanation 👍

Reapproving now it's moved out of draft. Thanks for checking up on the "ML" vs "Machine Learning" copy (can always come in a followup if needed).

@weltenwort
Copy link
Copy Markdown
Member Author

I'm still waiting for new insights on the copy, but I'll merge while the Jenkins and GitHub stars are aligned.

@weltenwort weltenwort merged commit ff7b736 into elastic:master Jul 13, 2020
@weltenwort weltenwort deleted the logs-ui-add-ml-module-list branch July 13, 2020 21:29
weltenwort added a commit that referenced this pull request Jul 14, 2020
Backports the following commits to 7.x:
 - [Logs UI] Show log analysis ML jobs in a list (#71132)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Feature:Logs UI Logs UI feature release_note:enhancement Team:Infra Monitoring UI - DEPRECATED DEPRECATED - Label for the Infra Monitoring UI team. Use Team:obs-ux-infra_services v7.9.0 v8.0.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Logs UI] ML job list

4 participants