Skip to content

[7.7] [ML] DF Analytics results table: use index pattern field format if one exists (#61709)#61990

Merged
alvarezmelissa87 merged 1 commit intoelastic:7.7from
alvarezmelissa87:backport/7.7/pr-61709
Mar 31, 2020
Merged

[7.7] [ML] DF Analytics results table: use index pattern field format if one exists (#61709)#61990
alvarezmelissa87 merged 1 commit intoelastic:7.7from
alvarezmelissa87:backport/7.7/pr-61709

Conversation

@alvarezmelissa87
Copy link
Copy Markdown
Contributor

@alvarezmelissa87 alvarezmelissa87 commented Mar 31, 2020

Backports the following commits to 7.7:

@alvarezmelissa87
[ML] DF Analytics results table: use index pattern field format if on…
9f33bc5 
…e exists (elastic#61709)

* classification: use index field format in results table

* regression: use index field format in results table

* outlier: use index field format in results table

* update types

* add destIndex specific fields when using sourceIndex for fields
@alvarezmelissa87 alvarezmelissa87 added backport This PR is a backport of another PR :ml Feature:Data Frame Analytics ML data frame analytics features labels Mar 31, 2020
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Mar 31, 2020

Pinging @elastic/ml-ui (:ml)

@kibanamachine
Copy link
Copy Markdown
Contributor

kibanamachine commented Mar 31, 2020

💛 Build succeeded, but was flaky

Test Failures

Kibana Pipeline / kibana-xpack-agent / X-Pack Detection Engine API Integration Tests.x-pack/test/detection_engine_api_integration/security_and_spaces/tests/create_rules·ts.detection engine api security and spaces enabled create_rules creating rules should create a single Machine Learning rule

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has failed 2 times on tracked branches: https://github.com/elastic/kibana/issues/61995

[00:00:00]       │
[00:00:00]         └-: detection engine api security and spaces enabled
[00:00:00]           └-> "before all" hook
[00:00:09]           └-: create_rules
[00:00:09]             └-> "before all" hook
[00:00:10]             └-: creating rules
[00:00:10]               └-> "before all" hook
[00:00:10]               └-> should create a single rule with a rule_id
[00:00:10]                 └-> "before each" hook: global before each
[00:00:10]                 └-> "before each" hook
[00:00:10]                   │ info [o.e.x.i.a.TransportPutLifecycleAction] [kibana-ci-immutable-ubuntu-18-tests-xl-1585667196512269935] adding index lifecycle policy [.siem-signals-default]
[00:00:10]                   │ info [o.e.c.m.MetaDataIndexTemplateService] [kibana-ci-immutable-ubuntu-18-tests-xl-1585667196512269935] adding template [.siem-signals-default] for index patterns [.siem-signals-default-*]
[00:00:10]                   │ info [o.e.c.m.MetaDataCreateIndexService] [kibana-ci-immutable-ubuntu-18-tests-xl-1585667196512269935] [.siem-signals-default-000001] creating index, cause [api], templates [.siem-signals-default], shards [1]/[1], mappings [_doc]
[00:00:10]                   │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-ubuntu-18-tests-xl-1585667196512269935] moving index [.siem-signals-default-000001] from [null] to [{"phase":"new","action":"complete","name":"complete"}] in policy [.siem-signals-default]
[00:00:10]                   │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-ubuntu-18-tests-xl-1585667196512269935] moving index [.siem-signals-default-000001] from [{"phase":"new","action":"complete","name":"complete"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-indexing-complete"}] in policy [.siem-signals-default]
[00:00:10]                 │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-ubuntu-18-tests-xl-1585667196512269935] moving index [.siem-signals-default-000001] from [{"phase":"hot","action":"unfollow","name":"wait-for-indexing-complete"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-follow-shard-tasks"}] in policy [.siem-signals-default]
[00:00:14]                 │ info [o.e.x.s.a.AuthenticationService] [kibana-ci-immutable-ubuntu-18-tests-xl-1585667196512269935] Authentication using apikey failed - api key has been invalidated
[00:00:14]                 │ proc [kibana]   log   [16:15:45.762] [error][plugins][siem] An error occurred during rule execution:
[00:00:14]                 │ proc [kibana] message: "[security_exception] missing authentication credentials for REST request [/auditbeat-*/_search?allow_no_indices=true&size=100&ignore_unavailable=true], with { header={ WWW-Authenticate={ 0="Bearer realm=\"security\"" & 1="ApiKey" & 2="Basic realm=\"security\" charset=\"UTF-8\"" } } }"
[00:00:14]                 │ proc [kibana] name: "Simple Rule Query"
[00:00:14]                 │ proc [kibana] id: "7490392f-1ffa-4181-8a9f-c10a70271625"
[00:00:14]                 │ proc [kibana] rule id: "rule-1"
[00:00:14]                 │ proc [kibana] signals index: ".siem-signals-default"
[00:00:14]                 │ info [o.e.x.s.a.AuthenticationService] [kibana-ci-immutable-ubuntu-18-tests-xl-1585667196512269935] Authentication using apikey failed - api key has been invalidated
[00:00:14]                 │ proc [kibana]   log   [16:15:45.768] [error][alerting][alerting][plugins][plugins] Executing Alert "7490392f-1ffa-4181-8a9f-c10a70271625" has resulted in Error: [security_exception] missing authentication credentials for REST request [/_security/user/_has_privileges], with { header={ WWW-Authenticate={ 0="Bearer realm=\"security\"" & 1="ApiKey" & 2="Basic realm=\"security\" charset=\"UTF-8\"" } } }
[00:00:14]                 └- ✓ pass  (4.4s) "detection engine api security and spaces enabled create_rules creating rules should create a single rule with a rule_id"
[00:00:14]               └-> "after each" hook
[00:00:14]                 │ info [o.e.c.m.MetaDataDeleteIndexService] [kibana-ci-immutable-ubuntu-18-tests-xl-1585667196512269935] [.siem-signals-default-000001/OwPAXXeoT2WD_tjtoqn2WQ] deleting index
[00:00:14]                 │ info [o.e.c.m.MetaDataIndexTemplateService] [kibana-ci-immutable-ubuntu-18-tests-xl-1585667196512269935] removing template [.siem-signals-default]
[00:00:15]               └-> should create a single rule without an input index
[00:00:15]                 └-> "before each" hook: global before each
[00:00:15]                 └-> "before each" hook
[00:00:15]                   │ info [o.e.x.i.a.TransportPutLifecycleAction] [kibana-ci-immutable-ubuntu-18-tests-xl-1585667196512269935] adding index lifecycle policy [.siem-signals-default]
[00:00:15]                   │ info [o.e.c.m.MetaDataIndexTemplateService] [kibana-ci-immutable-ubuntu-18-tests-xl-1585667196512269935] adding template [.siem-signals-default] for index patterns [.siem-signals-default-*]
[00:00:15]                   │ info [o.e.c.m.MetaDataCreateIndexService] [kibana-ci-immutable-ubuntu-18-tests-xl-1585667196512269935] [.siem-signals-default-000001] creating index, cause [api], templates [.siem-signals-default], shards [1]/[1], mappings [_doc]
[00:00:15]                   │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-ubuntu-18-tests-xl-1585667196512269935] moving index [.siem-signals-default-000001] from [null] to [{"phase":"new","action":"complete","name":"complete"}] in policy [.siem-signals-default]
[00:00:15]                   │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-ubuntu-18-tests-xl-1585667196512269935] moving index [.siem-signals-default-000001] from [{"phase":"new","action":"complete","name":"complete"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-indexing-complete"}] in policy [.siem-signals-default]
[00:00:15]                 │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-ubuntu-18-tests-xl-1585667196512269935] moving index [.siem-signals-default-000001] from [{"phase":"hot","action":"unfollow","name":"wait-for-indexing-complete"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-follow-shard-tasks"}] in policy [.siem-signals-default]
[00:00:20]                 │ info [o.e.x.s.a.AuthenticationService] [kibana-ci-immutable-ubuntu-18-tests-xl-1585667196512269935] Authentication using apikey failed - api key has been invalidated
[00:00:20]                 │ proc [kibana]   log   [16:15:51.807] [error][plugins][siem] An error occurred during rule execution:
[00:00:20]                 │ proc [kibana] message: "[security_exception] missing authentication credentials for REST request [/apm-*-transaction*%2Cauditbeat-*%2Cendgame-*%2Cfilebeat-*%2Cpacketbeat-*%2Cwinlogbeat-*/_search?allow_no_indices=true&size=100&ignore_unavailable=true], with { header={ WWW-Authenticate={ 0="Bearer realm=\"security\"" & 1="ApiKey" & 2="Basic realm=\"security\" charset=\"UTF-8\"" } } }"
[00:00:20]                 │ proc [kibana] name: "Simple Rule Query"
[00:00:20]                 │ proc [kibana] id: "cd8f6b63-d5bb-4124-a6ab-bd15187e6f7f"
[00:00:20]                 │ proc [kibana] rule id: "rule-1"
[00:00:20]                 │ proc [kibana] signals index: ".siem-signals-default"
[00:00:20]                 │ info [o.e.x.s.a.AuthenticationService] [kibana-ci-immutable-ubuntu-18-tests-xl-1585667196512269935] Authentication using apikey failed - api key has been invalidated
[00:00:20]                 │ proc [kibana]   log   [16:15:51.813] [error][alerting][alerting][plugins][plugins] Executing Alert "cd8f6b63-d5bb-4124-a6ab-bd15187e6f7f" has resulted in Error: [security_exception] missing authentication credentials for REST request [/_security/user/_has_privileges], with { header={ WWW-Authenticate={ 0="Bearer realm=\"security\"" & 1="ApiKey" & 2="Basic realm=\"security\" charset=\"UTF-8\"" } } }
[00:00:20]                 └- ✓ pass  (5.5s) "detection engine api security and spaces enabled create_rules creating rules should create a single rule without an input index"
[00:00:20]               └-> "after each" hook
[00:00:20]                 │ info [o.e.c.m.MetaDataDeleteIndexService] [kibana-ci-immutable-ubuntu-18-tests-xl-1585667196512269935] [.siem-signals-default-000001/gpq5tZmAS0uJbZV-WUT7Vg] deleting index
[00:00:20]                 │ info [o.e.c.m.MetaDataIndexTemplateService] [kibana-ci-immutable-ubuntu-18-tests-xl-1585667196512269935] removing template [.siem-signals-default]
[00:00:21]               └-> should create a single rule without a rule_id
[00:00:21]                 └-> "before each" hook: global before each
[00:00:21]                 └-> "before each" hook
[00:00:21]                   │ info [o.e.x.i.a.TransportPutLifecycleAction] [kibana-ci-immutable-ubuntu-18-tests-xl-1585667196512269935] adding index lifecycle policy [.siem-signals-default]
[00:00:21]                   │ info [o.e.c.m.MetaDataIndexTemplateService] [kibana-ci-immutable-ubuntu-18-tests-xl-1585667196512269935] adding template [.siem-signals-default] for index patterns [.siem-signals-default-*]
[00:00:21]                   │ info [o.e.c.m.MetaDataCreateIndexService] [kibana-ci-immutable-ubuntu-18-tests-xl-1585667196512269935] [.siem-signals-default-000001] creating index, cause [api], templates [.siem-signals-default], shards [1]/[1], mappings [_doc]
[00:00:21]                   │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-ubuntu-18-tests-xl-1585667196512269935] moving index [.siem-signals-default-000001] from [null] to [{"phase":"new","action":"complete","name":"complete"}] in policy [.siem-signals-default]
[00:00:21]                   │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-ubuntu-18-tests-xl-1585667196512269935] moving index [.siem-signals-default-000001] from [{"phase":"new","action":"complete","name":"complete"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-indexing-complete"}] in policy [.siem-signals-default]
[00:00:21]                 │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-ubuntu-18-tests-xl-1585667196512269935] moving index [.siem-signals-default-000001] from [{"phase":"hot","action":"unfollow","name":"wait-for-indexing-complete"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-follow-shard-tasks"}] in policy [.siem-signals-default]
[00:00:27]                 └- ✓ pass  (6.5s) "detection engine api security and spaces enabled create_rules creating rules should create a single rule without a rule_id"
[00:00:27]               └-> "after each" hook
[00:00:27]                 │ info [o.e.c.m.MetaDataDeleteIndexService] [kibana-ci-immutable-ubuntu-18-tests-xl-1585667196512269935] [.siem-signals-default-000001/cLsyhEBATreFsx7vwUi4ZA] deleting index
[00:00:27]                 │ info [o.e.c.m.MetaDataIndexTemplateService] [kibana-ci-immutable-ubuntu-18-tests-xl-1585667196512269935] removing template [.siem-signals-default]
[00:00:28]               └-> should create a single Machine Learning rule
[00:00:28]                 └-> "before each" hook: global before each
[00:00:28]                 └-> "before each" hook
[00:00:28]                   │ info [o.e.x.i.a.TransportPutLifecycleAction] [kibana-ci-immutable-ubuntu-18-tests-xl-1585667196512269935] adding index lifecycle policy [.siem-signals-default]
[00:00:28]                   │ info [o.e.c.m.MetaDataIndexTemplateService] [kibana-ci-immutable-ubuntu-18-tests-xl-1585667196512269935] adding template [.siem-signals-default] for index patterns [.siem-signals-default-*]
[00:00:28]                   │ info [o.e.c.m.MetaDataCreateIndexService] [kibana-ci-immutable-ubuntu-18-tests-xl-1585667196512269935] [.siem-signals-default-000001] creating index, cause [api], templates [.siem-signals-default], shards [1]/[1], mappings [_doc]
[00:00:28]                   │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-ubuntu-18-tests-xl-1585667196512269935] moving index [.siem-signals-default-000001] from [null] to [{"phase":"new","action":"complete","name":"complete"}] in policy [.siem-signals-default]
[00:00:28]                   │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-ubuntu-18-tests-xl-1585667196512269935] moving index [.siem-signals-default-000001] from [{"phase":"new","action":"complete","name":"complete"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-indexing-complete"}] in policy [.siem-signals-default]
[00:00:28]                 │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-ubuntu-18-tests-xl-1585667196512269935] moving index [.siem-signals-default-000001] from [{"phase":"hot","action":"unfollow","name":"wait-for-indexing-complete"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-follow-shard-tasks"}] in policy [.siem-signals-default]
[00:00:31]                 │ proc [kibana]   log   [16:16:03.069] [warning][plugins][siem] Machine learning job is not started:
[00:00:31]                 │ proc [kibana] job id: "some_job_id"
[00:00:31]                 │ proc [kibana] job status: "undefined"
[00:00:31]                 │ proc [kibana] datafeed status: "undefined"
[00:00:31]                 │ proc [kibana] name: "Simple ML Rule"
[00:00:31]                 │ proc [kibana] id: "caf99ac7-382a-44da-92e0-6e984f6371a4"
[00:00:31]                 │ proc [kibana] rule id: "rule-1"
[00:00:31]                 │ proc [kibana] signals index: ".siem-signals-default"
[00:00:33]                 └- ✖ fail: "detection engine api security and spaces enabled create_rules creating rules should create a single Machine Learning rule"
[00:00:33]                 │

Stack Trace

{ Error: expected { created_by: 'elastic',
  description: 'Simple Machine Learning Rule',
  enabled: true,
  false_positives: [],
  from: 'now-6m',
  immutable: false,
  interval: '5m',
  rule_id: 'rule-1',
  output_index: '.siem-signals-default',
  max_signals: 100,
  risk_score: 1,
  name: 'Simple ML Rule',
  references: [],
  severity: 'high',
  updated_by: 'elastic',
  tags: [],
  to: 'now',
  type: 'machine_learning',
  threat: [],
  version: 1,
  lists: [],
  actions: [],
  throttle: 'no_actions',
  last_failure_at: '2020-03-31T16:16:03.087Z',
  last_failure_message: 'Machine learning job is not started:\njob id: "some_job_id"\njob status: "undefined"\ndatafeed status: "undefined"\nname: "Simple ML Rule"\nid: "caf99ac7-382a-44da-92e0-6e984f6371a4"\nrule id: "rule-1"\nsignals index: ".siem-signals-default"',
  anomaly_threshold: 44,
  machine_learning_job_id: 'some_job_id' } to sort of equal { actions: [],
  created_by: 'elastic',
  description: 'Simple Machine Learning Rule',
  enabled: true,
  false_positives: [],
  from: 'now-6m',
  immutable: false,
  interval: '5m',
  rule_id: 'rule-1',
  output_index: '.siem-signals-default',
  max_signals: 100,
  risk_score: 1,
  name: 'Simple ML Rule',
  references: [],
  severity: 'high',
  updated_by: 'elastic',
  tags: [],
  to: 'now',
  type: 'machine_learning',
  threat: [],
  throttle: 'no_actions',
  lists: [],
  version: 1,
  anomaly_threshold: 44,
  machine_learning_job_id: 'some_job_id' }
    at Assertion.assert (/dev/shm/workspace/kibana/packages/kbn-expect/expect.js:100:11)
    at Assertion.eql (/dev/shm/workspace/kibana/packages/kbn-expect/expect.js:244:8)
    at Context.it (test/detection_engine_api_integration/security_and_spaces/tests/create_rules.ts:101:34)
  actual:
   '{\n  "actions": []\n  "anomaly_threshold": 44\n  "created_by": "elastic"\n  "description": "Simple Machine Learning Rule"\n  "enabled": true\n  "false_positives": []\n  "from": "now-6m"\n  "immutable": false\n  "interval": "5m"\n  "last_failure_at": "2020-03-31T16:16:03.087Z"\n  "last_failure_message": "Machine learning job is not started:\\njob id: \\"some_job_id\\"\\njob status: \\"undefined\\"\\ndatafeed status: \\"undefined\\"\\nname: \\"Simple ML Rule\\"\\nid: \\"caf99ac7-382a-44da-92e0-6e984f6371a4\\"\\nrule id: \\"rule-1\\"\\nsignals index: \\".siem-signals-default\\""\n  "lists": []\n  "machine_learning_job_id": "some_job_id"\n  "max_signals": 100\n  "name": "Simple ML Rule"\n  "output_index": ".siem-signals-default"\n  "references": []\n  "risk_score": 1\n  "rule_id": "rule-1"\n  "severity": "high"\n  "tags": []\n  "threat": []\n  "throttle": "no_actions"\n  "to": "now"\n  "type": "machine_learning"\n  "updated_by": "elastic"\n  "version": 1\n}',
  expected:
   '{\n  "actions": []\n  "anomaly_threshold": 44\n  "created_by": "elastic"\n  "description": "Simple Machine Learning Rule"\n  "enabled": true\n  "false_positives": []\n  "from": "now-6m"\n  "immutable": false\n  "interval": "5m"\n  "lists": []\n  "machine_learning_job_id": "some_job_id"\n  "max_signals": 100\n  "name": "Simple ML Rule"\n  "output_index": ".siem-signals-default"\n  "references": []\n  "risk_score": 1\n  "rule_id": "rule-1"\n  "severity": "high"\n  "tags": []\n  "threat": []\n  "throttle": "no_actions"\n  "to": "now"\n  "type": "machine_learning"\n  "updated_by": "elastic"\n  "version": 1\n}',
  showDiff: true }

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@alvarezmelissa87 alvarezmelissa87 merged commit 2ee99e3 into elastic:7.7 Mar 31, 2020
@alvarezmelissa87 alvarezmelissa87 deleted the backport/7.7/pr-61709 branch March 31, 2020 18:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

backport This PR is a backport of another PR Feature:Data Frame Analytics ML data frame analytics features :ml

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@alvarezmelissa87 @elasticmachine @kibanamachine