Skip to content

[7.7] [SIEM] version 7.7 rule import (#61903)#62013

Merged
spong merged 1 commit intoelastic:7.7from
spong:backport/7.7/pr-61903
Mar 31, 2020
Merged

[7.7] [SIEM] version 7.7 rule import (#61903)#62013
spong merged 1 commit intoelastic:7.7from
spong:backport/7.7/pr-61903

Conversation

@spong
Copy link
Copy Markdown
Member

@spong spong commented Mar 31, 2020

Backports the following commits to 7.7:

* rule import

* Update x-pack/legacy/plugins/siem/server/lib/detection_engine/rules/prepackaged_rules/windows_credential_dumping_msbuild.json

Co-Authored-By: Garrett Spong <spong@users.noreply.github.com>

* Update add_prepackaged_rules_schema.ts

* Update rule.ts

* updates 'prebuilt_rules_loaded' data (elastic#61940)

Co-authored-by: Garrett Spong <spong@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: MadameSheema <snootchie.boochies@gmail.com>
@spong spong added the backport This PR is a backport of another PR label Mar 31, 2020
@kibanamachine
Copy link
Copy Markdown
Contributor

💛 Build succeeded, but was flaky


Test Failures

Kibana Pipeline / kibana-xpack-agent / X-Pack Detection Engine API Integration Tests.x-pack/test/detection_engine_api_integration/security_and_spaces/tests/create_rules·ts.detection engine api security and spaces enabled create_rules creating rules should create a single Machine Learning rule

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has failed 3 times on tracked branches: https://github.com/elastic/kibana/issues/61995

[00:00:00]       │
[00:00:00]         └-: detection engine api security and spaces enabled
[00:00:00]           └-> "before all" hook
[00:00:10]           └-: create_rules
[00:00:10]             └-> "before all" hook
[00:00:10]             └-: creating rules
[00:00:10]               └-> "before all" hook
[00:00:10]               └-> should create a single rule with a rule_id
[00:00:10]                 └-> "before each" hook: global before each
[00:00:10]                 └-> "before each" hook
[00:00:10]                   │ info [o.e.x.i.a.TransportPutLifecycleAction] [kibana-ci-immutable-centos-tests-xl-1585672100315530405] adding index lifecycle policy [.siem-signals-default]
[00:00:10]                   │ info [o.e.c.m.MetaDataIndexTemplateService] [kibana-ci-immutable-centos-tests-xl-1585672100315530405] adding template [.siem-signals-default] for index patterns [.siem-signals-default-*]
[00:00:10]                   │ info [o.e.c.m.MetaDataCreateIndexService] [kibana-ci-immutable-centos-tests-xl-1585672100315530405] [.siem-signals-default-000001] creating index, cause [api], templates [.siem-signals-default], shards [1]/[1], mappings [_doc]
[00:00:10]                   │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-centos-tests-xl-1585672100315530405] moving index [.siem-signals-default-000001] from [null] to [{"phase":"new","action":"complete","name":"complete"}] in policy [.siem-signals-default]
[00:00:10]                   │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-centos-tests-xl-1585672100315530405] moving index [.siem-signals-default-000001] from [{"phase":"new","action":"complete","name":"complete"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-indexing-complete"}] in policy [.siem-signals-default]
[00:00:10]                   │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-centos-tests-xl-1585672100315530405] moving index [.siem-signals-default-000001] from [{"phase":"hot","action":"unfollow","name":"wait-for-indexing-complete"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-follow-shard-tasks"}] in policy [.siem-signals-default]
[00:00:14]                 │ info [o.e.x.s.a.AuthenticationService] [kibana-ci-immutable-centos-tests-xl-1585672100315530405] Authentication using apikey failed - api key has been invalidated
[00:00:14]                 │ proc [kibana]   log   [17:42:41.997] [error][plugins][siem] An error occurred during rule execution:
[00:00:14]                 │ proc [kibana] message: "[security_exception] missing authentication credentials for REST request [/auditbeat-*/_search?allow_no_indices=true&size=100&ignore_unavailable=true], with { header={ WWW-Authenticate={ 0="Bearer realm=\"security\"" & 1="ApiKey" & 2="Basic realm=\"security\" charset=\"UTF-8\"" } } }"
[00:00:14]                 │ proc [kibana] name: "Simple Rule Query"
[00:00:14]                 │ proc [kibana] id: "36640fc1-8f19-4a08-a391-6c35ffb0ca48"
[00:00:14]                 │ proc [kibana] rule id: "rule-1"
[00:00:14]                 │ proc [kibana] signals index: ".siem-signals-default"
[00:00:14]                 │ info [o.e.x.s.a.AuthenticationService] [kibana-ci-immutable-centos-tests-xl-1585672100315530405] Authentication using apikey failed - api key has been invalidated
[00:00:14]                 │ proc [kibana]   log   [17:42:42.006] [error][alerting][alerting][plugins][plugins] Executing Alert "36640fc1-8f19-4a08-a391-6c35ffb0ca48" has resulted in Error: [security_exception] missing authentication credentials for REST request [/_security/user/_has_privileges], with { header={ WWW-Authenticate={ 0="Bearer realm=\"security\"" & 1="ApiKey" & 2="Basic realm=\"security\" charset=\"UTF-8\"" } } }
[00:00:15]                 └- ✓ pass  (5.0s) "detection engine api security and spaces enabled create_rules creating rules should create a single rule with a rule_id"
[00:00:15]               └-> "after each" hook
[00:00:15]                 │ info [o.e.c.m.MetaDataDeleteIndexService] [kibana-ci-immutable-centos-tests-xl-1585672100315530405] [.siem-signals-default-000001/DKYXyWQzSSSG6_3ZF7LghQ] deleting index
[00:00:15]                 │ info [o.e.c.m.MetaDataIndexTemplateService] [kibana-ci-immutable-centos-tests-xl-1585672100315530405] removing template [.siem-signals-default]
[00:00:15]               └-> should create a single rule without an input index
[00:00:15]                 └-> "before each" hook: global before each
[00:00:15]                 └-> "before each" hook
[00:00:15]                   │ info [o.e.x.i.a.TransportPutLifecycleAction] [kibana-ci-immutable-centos-tests-xl-1585672100315530405] adding index lifecycle policy [.siem-signals-default]
[00:00:15]                   │ info [o.e.c.m.MetaDataIndexTemplateService] [kibana-ci-immutable-centos-tests-xl-1585672100315530405] adding template [.siem-signals-default] for index patterns [.siem-signals-default-*]
[00:00:16]                   │ info [o.e.c.m.MetaDataCreateIndexService] [kibana-ci-immutable-centos-tests-xl-1585672100315530405] [.siem-signals-default-000001] creating index, cause [api], templates [.siem-signals-default], shards [1]/[1], mappings [_doc]
[00:00:16]                   │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-centos-tests-xl-1585672100315530405] moving index [.siem-signals-default-000001] from [null] to [{"phase":"new","action":"complete","name":"complete"}] in policy [.siem-signals-default]
[00:00:16]                   │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-centos-tests-xl-1585672100315530405] moving index [.siem-signals-default-000001] from [{"phase":"new","action":"complete","name":"complete"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-indexing-complete"}] in policy [.siem-signals-default]
[00:00:16]                 │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-centos-tests-xl-1585672100315530405] moving index [.siem-signals-default-000001] from [{"phase":"hot","action":"unfollow","name":"wait-for-indexing-complete"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-follow-shard-tasks"}] in policy [.siem-signals-default]
[00:00:20]                 └- ✓ pass  (4.5s) "detection engine api security and spaces enabled create_rules creating rules should create a single rule without an input index"
[00:00:20]               └-> "after each" hook
[00:00:20]                 │ info [o.e.c.m.MetaDataDeleteIndexService] [kibana-ci-immutable-centos-tests-xl-1585672100315530405] [.siem-signals-default-000001/ajTby-G6Q72kJGyg3MODHg] deleting index
[00:00:20]                 │ info [o.e.c.m.MetaDataIndexTemplateService] [kibana-ci-immutable-centos-tests-xl-1585672100315530405] removing template [.siem-signals-default]
[00:00:20]               └-> should create a single rule without a rule_id
[00:00:20]                 └-> "before each" hook: global before each
[00:00:20]                 └-> "before each" hook
[00:00:20]                   │ info [o.e.x.s.a.AuthenticationService] [kibana-ci-immutable-centos-tests-xl-1585672100315530405] Authentication using apikey failed - api key has been invalidated
[00:00:20]                   │ proc [kibana]   log   [17:42:48.029] [error][plugins][siem] An error occurred during rule execution:
[00:00:20]                   │ proc [kibana] message: "[security_exception] missing authentication credentials for REST request [/_security/user/_has_privileges], with { header={ WWW-Authenticate={ 0="Bearer realm=\"security\"" & 1="ApiKey" & 2="Basic realm=\"security\" charset=\"UTF-8\"" } } }"
[00:00:20]                   │ proc [kibana] name: "Simple Rule Query"
[00:00:20]                   │ proc [kibana] id: "6c64ead5-07cd-4c30-b259-5490c9b45ac3"
[00:00:20]                   │ proc [kibana] rule id: "rule-1"
[00:00:20]                   │ proc [kibana] signals index: ".siem-signals-default"
[00:00:20]                   │ info [o.e.x.s.a.AuthenticationService] [kibana-ci-immutable-centos-tests-xl-1585672100315530405] Authentication using apikey failed - api key has been invalidated
[00:00:20]                   │ proc [kibana]   log   [17:42:48.043] [error][alerting][alerting][plugins][plugins] Executing Alert "6c64ead5-07cd-4c30-b259-5490c9b45ac3" has resulted in Error: [security_exception] missing authentication credentials for REST request [/_security/user/_has_privileges], with { header={ WWW-Authenticate={ 0="Bearer realm=\"security\"" & 1="ApiKey" & 2="Basic realm=\"security\" charset=\"UTF-8\"" } } }
[00:00:20]                   │ info [o.e.x.i.a.TransportPutLifecycleAction] [kibana-ci-immutable-centos-tests-xl-1585672100315530405] adding index lifecycle policy [.siem-signals-default]
[00:00:21]                   │ info [o.e.c.m.MetaDataIndexTemplateService] [kibana-ci-immutable-centos-tests-xl-1585672100315530405] adding template [.siem-signals-default] for index patterns [.siem-signals-default-*]
[00:00:21]                   │ info [o.e.c.m.MetaDataCreateIndexService] [kibana-ci-immutable-centos-tests-xl-1585672100315530405] [.siem-signals-default-000001] creating index, cause [api], templates [.siem-signals-default], shards [1]/[1], mappings [_doc]
[00:00:21]                   │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-centos-tests-xl-1585672100315530405] moving index [.siem-signals-default-000001] from [null] to [{"phase":"new","action":"complete","name":"complete"}] in policy [.siem-signals-default]
[00:00:21]                   │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-centos-tests-xl-1585672100315530405] moving index [.siem-signals-default-000001] from [{"phase":"new","action":"complete","name":"complete"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-indexing-complete"}] in policy [.siem-signals-default]
[00:00:21]                   │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-centos-tests-xl-1585672100315530405] moving index [.siem-signals-default-000001] from [{"phase":"hot","action":"unfollow","name":"wait-for-indexing-complete"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-follow-shard-tasks"}] in policy [.siem-signals-default]
[00:00:25]                 └- ✓ pass  (4.5s) "detection engine api security and spaces enabled create_rules creating rules should create a single rule without a rule_id"
[00:00:25]               └-> "after each" hook
[00:00:25]                 │ info [o.e.c.m.MetaDataDeleteIndexService] [kibana-ci-immutable-centos-tests-xl-1585672100315530405] [.siem-signals-default-000001/5jhon87bSl-Kz8TdTqotQg] deleting index
[00:00:25]                 │ info [o.e.c.m.MetaDataIndexTemplateService] [kibana-ci-immutable-centos-tests-xl-1585672100315530405] removing template [.siem-signals-default]
[00:00:25]               └-> should create a single Machine Learning rule
[00:00:25]                 └-> "before each" hook: global before each
[00:00:25]                 └-> "before each" hook
[00:00:25]                   │ info [o.e.x.i.a.TransportPutLifecycleAction] [kibana-ci-immutable-centos-tests-xl-1585672100315530405] adding index lifecycle policy [.siem-signals-default]
[00:00:26]                   │ info [o.e.c.m.MetaDataIndexTemplateService] [kibana-ci-immutable-centos-tests-xl-1585672100315530405] adding template [.siem-signals-default] for index patterns [.siem-signals-default-*]
[00:00:26]                   │ info [o.e.c.m.MetaDataCreateIndexService] [kibana-ci-immutable-centos-tests-xl-1585672100315530405] [.siem-signals-default-000001] creating index, cause [api], templates [.siem-signals-default], shards [1]/[1], mappings [_doc]
[00:00:26]                   │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-centos-tests-xl-1585672100315530405] moving index [.siem-signals-default-000001] from [null] to [{"phase":"new","action":"complete","name":"complete"}] in policy [.siem-signals-default]
[00:00:26]                   │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-centos-tests-xl-1585672100315530405] moving index [.siem-signals-default-000001] from [{"phase":"new","action":"complete","name":"complete"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-indexing-complete"}] in policy [.siem-signals-default]
[00:00:26]                 │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-centos-tests-xl-1585672100315530405] moving index [.siem-signals-default-000001] from [{"phase":"hot","action":"unfollow","name":"wait-for-indexing-complete"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-follow-shard-tasks"}] in policy [.siem-signals-default]
[00:00:29]                 │ proc [kibana]   log   [17:42:56.257] [warning][plugins][siem] Machine learning job is not started:
[00:00:29]                 │ proc [kibana] job id: "some_job_id"
[00:00:29]                 │ proc [kibana] job status: "undefined"
[00:00:29]                 │ proc [kibana] datafeed status: "undefined"
[00:00:29]                 │ proc [kibana] name: "Simple ML Rule"
[00:00:29]                 │ proc [kibana] id: "9fb07162-0701-4796-8795-86e7aee40782"
[00:00:29]                 │ proc [kibana] rule id: "rule-1"
[00:00:29]                 │ proc [kibana] signals index: ".siem-signals-default"
[00:00:30]                 └- ✖ fail: "detection engine api security and spaces enabled create_rules creating rules should create a single Machine Learning rule"
[00:00:30]                 │

Stack Trace

{ Error: expected { created_by: 'elastic',
  description: 'Simple Machine Learning Rule',
  enabled: true,
  false_positives: [],
  from: 'now-6m',
  immutable: false,
  interval: '5m',
  rule_id: 'rule-1',
  output_index: '.siem-signals-default',
  max_signals: 100,
  risk_score: 1,
  name: 'Simple ML Rule',
  references: [],
  severity: 'high',
  updated_by: 'elastic',
  tags: [],
  to: 'now',
  type: 'machine_learning',
  threat: [],
  version: 1,
  lists: [],
  actions: [],
  throttle: 'no_actions',
  last_failure_at: '2020-03-31T17:42:56.271Z',
  last_failure_message: 'Machine learning job is not started:\njob id: "some_job_id"\njob status: "undefined"\ndatafeed status: "undefined"\nname: "Simple ML Rule"\nid: "9fb07162-0701-4796-8795-86e7aee40782"\nrule id: "rule-1"\nsignals index: ".siem-signals-default"',
  anomaly_threshold: 44,
  machine_learning_job_id: 'some_job_id' } to sort of equal { actions: [],
  created_by: 'elastic',
  description: 'Simple Machine Learning Rule',
  enabled: true,
  false_positives: [],
  from: 'now-6m',
  immutable: false,
  interval: '5m',
  rule_id: 'rule-1',
  output_index: '.siem-signals-default',
  max_signals: 100,
  risk_score: 1,
  name: 'Simple ML Rule',
  references: [],
  severity: 'high',
  updated_by: 'elastic',
  tags: [],
  to: 'now',
  type: 'machine_learning',
  threat: [],
  throttle: 'no_actions',
  lists: [],
  version: 1,
  anomaly_threshold: 44,
  machine_learning_job_id: 'some_job_id' }
    at Assertion.assert (/dev/shm/workspace/kibana/packages/kbn-expect/expect.js:100:11)
    at Assertion.eql (/dev/shm/workspace/kibana/packages/kbn-expect/expect.js:244:8)
    at Context.it (test/detection_engine_api_integration/security_and_spaces/tests/create_rules.ts:101:34)
  actual:
   '{\n  "actions": []\n  "anomaly_threshold": 44\n  "created_by": "elastic"\n  "description": "Simple Machine Learning Rule"\n  "enabled": true\n  "false_positives": []\n  "from": "now-6m"\n  "immutable": false\n  "interval": "5m"\n  "last_failure_at": "2020-03-31T17:42:56.271Z"\n  "last_failure_message": "Machine learning job is not started:\\njob id: \\"some_job_id\\"\\njob status: \\"undefined\\"\\ndatafeed status: \\"undefined\\"\\nname: \\"Simple ML Rule\\"\\nid: \\"9fb07162-0701-4796-8795-86e7aee40782\\"\\nrule id: \\"rule-1\\"\\nsignals index: \\".siem-signals-default\\""\n  "lists": []\n  "machine_learning_job_id": "some_job_id"\n  "max_signals": 100\n  "name": "Simple ML Rule"\n  "output_index": ".siem-signals-default"\n  "references": []\n  "risk_score": 1\n  "rule_id": "rule-1"\n  "severity": "high"\n  "tags": []\n  "threat": []\n  "throttle": "no_actions"\n  "to": "now"\n  "type": "machine_learning"\n  "updated_by": "elastic"\n  "version": 1\n}',
  expected:
   '{\n  "actions": []\n  "anomaly_threshold": 44\n  "created_by": "elastic"\n  "description": "Simple Machine Learning Rule"\n  "enabled": true\n  "false_positives": []\n  "from": "now-6m"\n  "immutable": false\n  "interval": "5m"\n  "lists": []\n  "machine_learning_job_id": "some_job_id"\n  "max_signals": 100\n  "name": "Simple ML Rule"\n  "output_index": ".siem-signals-default"\n  "references": []\n  "risk_score": 1\n  "rule_id": "rule-1"\n  "severity": "high"\n  "tags": []\n  "threat": []\n  "throttle": "no_actions"\n  "to": "now"\n  "type": "machine_learning"\n  "updated_by": "elastic"\n  "version": 1\n}',
  showDiff: true }

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@spong spong merged commit 6d3262d into elastic:7.7 Mar 31, 2020
@spong spong deleted the backport/7.7/pr-61903 branch March 31, 2020 18:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport This PR is a backport of another PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants