[ML] APM modules configs for RUM Javascript and NodeJS by darnautov · Pull Request #53792 · elastic/kibana

darnautov · 2019-12-24T12:54:36Z

Summary

Add APM modules configurations for RUM JS and NodeJS
Custom URLs support for APM

Checklist

This was checked for cross-browser compatibility, including a check against IE11
Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
~~Documentation was added for features that require explanation or tutorials~~
Unit or functional tests were updated or added to match the most common scenarios
~~This was checked for keyboard-only and screenreader accessibility~~

elasticmachine · 2019-12-24T12:54:38Z

Pinging @elastic/ml-ui (:ml)

kibanamachine · 2019-12-24T14:12:17Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request
Commit: 5ad8bab356055b199f701fa863e76468531747cd

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

blaklaybul

requesting a few minor changes

blaklaybul · 2019-12-24T15:30:24Z

x-pack/legacy/plugins/ml/server/models/data_recognizer/modules/apm_jsbase/manifest.json

Detect problematic spans and identify user agents that are potentially causing issues.

Fixed in 144c0ba

blaklaybul · 2019-12-24T15:36:05Z

x-pack/legacy/plugins/ml/server/models/data_recognizer/modules/apm_nodejs/manifest.json

Detect abnormal traces, anomalous spans, and identify periods of decreased throughput.

Fixed in 144c0ba

.../ml/server/models/data_recognizer/modules/apm_nodejs/ml/abnormal_trace_durations_nodejs.json

kibanamachine · 2019-12-24T16:53:14Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: c6039ecddfa888a594473d11b848b1533ebf3216

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

darnautov · 2020-01-01T17:22:43Z

@elasticmachine merge upstream

walterra

Code LGTM

alvarezmelissa87

LGTM ⚡️

...er/models/data_recognizer/modules/apm_jsbase/ml/datafeed_abnormal_span_durations_jsbase.json

...a_recognizer/modules/apm_jsbase/ml/datafeed_anomalous_error_rate_for_user_agents_jsbase.json

...r/models/data_recognizer/modules/apm_jsbase/ml/datafeed_high_count_by_user_agent_jsbase.json

x-pack/legacy/plugins/ml/server/models/data_recognizer/modules/apm_nodejs/manifest.json

...er/models/data_recognizer/modules/apm_nodejs/ml/datafeed_abnormal_span_durations_nodejs.json

...erver/models/data_recognizer/modules/apm_nodejs/ml/datafeed_decreased_throughput_nodejs.json

...erver/models/data_recognizer/modules/apm_jsbase/ml/datafeed_decreased_throughput_jsbase.json

...erver/models/data_recognizer/modules/apm_nodejs/ml/datafeed_decreased_throughput_nodejs.json

x-pack/legacy/plugins/ml/server/models/data_recognizer/modules/apm_jsbase/manifest.json

...erver/models/data_recognizer/modules/apm_jsbase/ml/datafeed_decreased_throughput_jsbase.json

...gins/ml/server/models/data_recognizer/modules/apm_jsbase/ml/decreased_throughput_jsbase.json

...s/ml/server/models/data_recognizer/modules/apm_jsbase/ml/abnormal_span_durations_jsbase.json

...gins/ml/server/models/data_recognizer/modules/apm_jsbase/ml/decreased_throughput_jsbase.json

blaklaybul

Looks good so far! Some comments:

remove scroll_size and chunking_config parameters.
update detector descriptions

blaklaybul · 2020-01-06T15:17:03Z

...s/ml/server/models/data_recognizer/modules/apm_jsbase/ml/abnormal_span_durations_jsbase.json

+    "bucket_span": "15m",
+    "detectors": [
+      {
+        "detector_description": "high_mean(\"span.duration.us\") partitionfield=\"span.type\"",


"description": "increased span duration"

blaklaybul · 2020-01-06T15:17:30Z

...odels/data_recognizer/modules/apm_jsbase/ml/anomalous_error_rate_for_user_agents_jsbase.json

+    "bucket_span": "15m",
+    "detectors": [
+      {
+        "detector_description": "high_non_zero_count partitionfield=\"user_agent.name\"",


"description": "high error rate for user agent"

blaklaybul · 2020-01-06T15:17:59Z

...gins/ml/server/models/data_recognizer/modules/apm_jsbase/ml/decreased_throughput_jsbase.json

+    "bucket_span": "15m",
+    "detectors": [
+      {
+        "detector_description": "low_count",


"description": "low throughput"

blaklaybul · 2020-01-06T15:18:23Z

.../ml/server/models/data_recognizer/modules/apm_jsbase/ml/high_count_by_user_agent_jsbase.json

+    "bucket_span": "15m",
+    "detectors": [
+      {
+        "detector_description": "high_non_zero_count partitionfield=\"user_agent.name\"",


"description": "high request rate for user agent"

blaklaybul · 2020-01-06T15:18:55Z

...s/ml/server/models/data_recognizer/modules/apm_nodejs/ml/abnormal_span_durations_nodejs.json

+    "bucket_span": "15m",
+    "detectors": [
+      {
+        "detector_description": "high_mean(\"span.duration.us\") partitionfield=\"span.type\"",


"description": "increased span duration"

blaklaybul · 2020-01-06T15:19:15Z

.../ml/server/models/data_recognizer/modules/apm_nodejs/ml/abnormal_trace_durations_nodejs.json

+    "bucket_span": "15m",
+    "detectors": [
+      {
+        "detector_description": "high_mean(\"transaction.duration.us\") by \"transaction.name\"",


"description": "increased trace duration"

blaklaybul · 2020-01-06T15:19:42Z

...gins/ml/server/models/data_recognizer/modules/apm_nodejs/ml/decreased_throughput_nodejs.json

+    "bucket_span": "15m",
+    "detectors": [
+      {
+        "detector_description": "low_count",


"description": "low throughput"

blaklaybul

Detector descriptions look good, but you should be able to remove the chunking_config parameter from the 2 decreased_throughput datafeeds as well

darnautov · 2020-01-07T09:39:24Z

@blaklaybul 374c4eb ?

blaklaybul

LGTM

blaklaybul

additional changes to custom URLs and datafeed filter.

blaklaybul · 2020-01-07T16:01:26Z

...r/models/data_recognizer/modules/apm_jsbase/ml/datafeed_high_count_by_user_agent_jsbase.json

+    "bool": {
+      "must": [
+        { "bool": { "filter": { "term": { "agent.name": "js-base" } } } },
+        { "bool": { "filter": [{ "exists": { "field": "user_agent.name" } }] } }


please add an additional filter for transaction to the must:

{"bool": {"filter": {"term": {"processor.event": "transaction"}}}}

...gins/ml/server/models/data_recognizer/modules/apm_nodejs/ml/decreased_throughput_nodejs.json

...gins/ml/server/models/data_recognizer/modules/apm_jsbase/ml/decreased_throughput_jsbase.json

darnautov · 2020-01-08T13:11:31Z

@elasticmachine merge upstream

peteharverson

Tested latest edits and LGTM

blaklaybul

Latests changes lgtm

darnautov · 2020-01-10T17:27:42Z

@elasticmachine merge upstream

darnautov · 2020-01-10T20:22:52Z

@elasticmachine merge upstream

kibanamachine · 2020-01-10T21:48:04Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: fe6058a

History

💔 Build #19354 failed ba69c89
💚 Build #18687 succeeded 7f465fe
💚 Build #18671 succeeded 2d1fe5c
💚 Build #18317 succeeded 374c4eb
💚 Build #18115 succeeded ebe6353

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

* [ML] apm modules * [ML] apm modules * [ML] update mocha test * [ML] fix config * [ML] single line JSON formatting for queries * [ML] remove an empty path component with a trailing slash * [ML] change detector descriptions, remove scroll size * [ML] remove chunking_config from datafeeds * [ML] fix configs Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

* master: (69 commits) [Graph] Fix various a11y issues (elastic#54097) Add ApplicationService app status management (elastic#50223) logs in one time (elastic#54447) Deprecate using `elasticsearch.ssl.certificate` without `elasticsearch.ssl.key` and vice versa (elastic#54392) [Optimizer] Fix a stack overflow with watch_cache when it attempts to delete very large folders. (elastic#54457) Security - Role Mappings UI (elastic#53620) [SIEM] [Detection engine] Permission II (elastic#54292) Allow User to Cleanup Repository from UI (elastic#53047) [Detection engine] Some UX for rule creation (elastic#54471) share specific instances of some ui packages (elastic#54079) [ML] APM modules configs for RUM Javascript and NodeJS (elastic#53792) [APM] Delay rendering invalid license notification (elastic#53924) [Graph] Improve error message on graph requests (elastic#54230) [ILM] Kibana should allow a min_age setting of 0ms in ILM policy phases (elastic#53719) Unit Tests for common/lib (elastic#53736) [Graph] Only show explorable fields (elastic#54101) remove linting rule exception for markdown (elastic#54232) [Monitoring] Fetch shard data more efficiently (elastic#54028) [Maps] Add hiddenLayers option to embeddable map input (elastic#54355) Pass termOrder and hasTermsAgg properties to serializeThresholdWatch function (elastic#54391) ...

* [ML] apm modules * [ML] apm modules * [ML] update mocha test * [ML] fix config * [ML] single line JSON formatting for queries * [ML] remove an empty path component with a trailing slash * [ML] change detector descriptions, remove scroll size * [ML] remove chunking_config from datafeeds * [ML] fix configs Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

darnautov added release_note:enhancement :ml Feature:Anomaly Detection ML anomaly detection v8.0.0 v7.6.0 labels Dec 24, 2019

darnautov requested review from alvarezmelissa87, blaklaybul and peteharverson December 24, 2019 12:54

darnautov requested a review from a team as a code owner December 24, 2019 12:54

darnautov self-assigned this Dec 24, 2019

blaklaybul suggested changes Dec 24, 2019

View reviewed changes

darnautov requested a review from walterra December 27, 2019 12:52

darnautov force-pushed the ML-new-apm-jobs branch from c6039ec to 144c0ba Compare December 30, 2019 12:10

darnautov requested a review from blaklaybul December 30, 2019 12:11

walterra approved these changes Jan 2, 2020

View reviewed changes

alvarezmelissa87 approved these changes Jan 2, 2020

View reviewed changes

peteharverson requested changes Jan 2, 2020

View reviewed changes

peteharverson reviewed Jan 2, 2020

View reviewed changes

...gins/ml/server/models/data_recognizer/modules/apm_jsbase/ml/decreased_throughput_jsbase.json Outdated Show resolved Hide resolved

peteharverson reviewed Jan 2, 2020

View reviewed changes

...s/ml/server/models/data_recognizer/modules/apm_jsbase/ml/abnormal_span_durations_jsbase.json Outdated Show resolved Hide resolved

peteharverson reviewed Jan 2, 2020

View reviewed changes

...gins/ml/server/models/data_recognizer/modules/apm_jsbase/ml/decreased_throughput_jsbase.json Outdated Show resolved Hide resolved

darnautov added 5 commits January 6, 2020 10:22

[ML] apm modules

a5f509d

[ML] apm modules

c8505f5

[ML] update mocha test

c8b1ef7

[ML] fix config

9a015c3

[ML] single line JSON formatting for queries

f40d0a2

darnautov force-pushed the ML-new-apm-jobs branch from cbf8ddf to f40d0a2 Compare January 6, 2020 09:50

[ML] remove an empty path component with a trailing slash

9f8347d

darnautov requested a review from peteharverson January 6, 2020 12:18

blaklaybul suggested changes Jan 6, 2020

View reviewed changes

[ML] change detector descriptions, remove scroll size

ebe6353

darnautov requested a review from blaklaybul January 6, 2020 16:21

blaklaybul suggested changes Jan 6, 2020

View reviewed changes

[ML] remove chunking_config from datafeeds

374c4eb

darnautov requested a review from blaklaybul January 7, 2020 09:39

blaklaybul approved these changes Jan 7, 2020

View reviewed changes

blaklaybul suggested changes Jan 7, 2020

View reviewed changes

[ML] fix configs

2d1fe5c

darnautov requested a review from blaklaybul January 8, 2020 12:06

Merge branch 'master' into ML-new-apm-jobs

7f465fe

peteharverson approved these changes Jan 9, 2020

View reviewed changes

blaklaybul approved these changes Jan 10, 2020

View reviewed changes

Merge branch 'master' into ML-new-apm-jobs

ba69c89

Merge branch 'master' into ML-new-apm-jobs

fe6058a

darnautov merged commit 51c1a8f into elastic:master Jan 10, 2020

darnautov mentioned this pull request Jan 13, 2020

[7.x] [ML] APM modules configs for RUM Javascript and NodeJS (#53792) #54553

Merged

Conversation

darnautov commented Dec 24, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Checklist

Uh oh!

elasticmachine commented Dec 24, 2019

Uh oh!

kibanamachine commented Dec 24, 2019

💔 Build Failed

Uh oh!

blaklaybul left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kibanamachine commented Dec 24, 2019

💚 Build Succeeded

Uh oh!

darnautov commented Jan 1, 2020

Uh oh!

walterra left a comment

Choose a reason for hiding this comment

Uh oh!

alvarezmelissa87 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

blaklaybul left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

blaklaybul left a comment

darnautov commented Dec 24, 2019 •

edited

Loading

blaklaybul left a comment •

edited

Loading