Skip to content

[7.x] [UA] Tight worker loop can cause high CPU usage (#60950)#62000

Merged
jloleysens merged 1 commit intoelastic:7.xfrom
jloleysens:backport/7.x/pr-60950
Apr 1, 2020
Merged

[7.x] [UA] Tight worker loop can cause high CPU usage (#60950)#62000
jloleysens merged 1 commit intoelastic:7.xfrom
jloleysens:backport/7.x/pr-60950

Conversation

@jloleysens
Copy link
Copy Markdown
Contributor

@jloleysens jloleysens commented Mar 31, 2020

Backports the following commits to 7.x:

@jloleysens @elasticmachine
[UA] Tight worker loop can cause high CPU usage (elastic#60950)
61baea9 
* Addded worker padding to save some CPU

* Updated comments

* Update worker scheduler and add a new util

The worker scheduler should only sleep when it cannot process any
in progress operations. Additionally, logic has been added
for handling of queue operations that have been in the queue for
a long time and may be viewed as still in small window of time
by wokers that do not have the credentials to process those
reindex operations.

* res 👉🏻resolve

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
@jloleysens jloleysens added the backport This PR is a backport of another PR label Mar 31, 2020
@kibanamachine
Copy link
Copy Markdown
Contributor

kibanamachine commented Mar 31, 2020

💛 Build succeeded, but was flaky

Test Failures

Kibana Pipeline / kibana-xpack-agent / X-Pack Detection Engine API Integration Tests.x-pack/test/detection_engine_api_integration/security_and_spaces/tests/create_rules·ts.detection engine api security and spaces enabled create_rules creating rules should create a single Machine Learning rule

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has failed 3 times on tracked branches: https://github.com/elastic/kibana/issues/61995

[00:00:00]       │
[00:00:00]         └-: detection engine api security and spaces enabled
[00:00:00]           └-> "before all" hook
[00:00:09]           └-: create_rules
[00:00:09]             └-> "before all" hook
[00:00:09]             └-: creating rules
[00:00:09]               └-> "before all" hook
[00:00:09]               └-> should create a single rule with a rule_id
[00:00:09]                 └-> "before each" hook: global before each
[00:00:09]                 └-> "before each" hook
[00:00:09]                   │ info [o.e.x.i.a.TransportPutLifecycleAction] [kibana-ci-immutable-centos-tests-xl-1585668584212196307] adding index lifecycle policy [.siem-signals-default]
[00:00:09]                   │ info [o.e.c.m.MetaDataIndexTemplateService] [kibana-ci-immutable-centos-tests-xl-1585668584212196307] adding template [.siem-signals-default] for index patterns [.siem-signals-default-*]
[00:00:09]                   │ info [o.e.c.m.MetaDataCreateIndexService] [kibana-ci-immutable-centos-tests-xl-1585668584212196307] [.siem-signals-default-000001] creating index, cause [api], templates [.siem-signals-default], shards [1]/[1], mappings [_doc]
[00:00:09]                   │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-centos-tests-xl-1585668584212196307] moving index [.siem-signals-default-000001] from [null] to [{"phase":"new","action":"complete","name":"complete"}] in policy [.siem-signals-default]
[00:00:09]                   │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-centos-tests-xl-1585668584212196307] moving index [.siem-signals-default-000001] from [{"phase":"new","action":"complete","name":"complete"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-indexing-complete"}] in policy [.siem-signals-default]
[00:00:10]                   │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-centos-tests-xl-1585668584212196307] moving index [.siem-signals-default-000001] from [{"phase":"hot","action":"unfollow","name":"wait-for-indexing-complete"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-follow-shard-tasks"}] in policy [.siem-signals-default]
[00:00:14]                 │ info [o.e.x.s.a.AuthenticationService] [kibana-ci-immutable-centos-tests-xl-1585668584212196307] Authentication using apikey failed - api key has been invalidated
[00:00:14]                 │ proc [kibana]   log   [16:41:00.001] [error][plugins][siem] An error occurred during rule execution:
[00:00:14]                 │ proc [kibana] message: "[security_exception] missing authentication credentials for REST request [/auditbeat-*/_search?allow_no_indices=true&size=100&ignore_unavailable=true], with { header={ WWW-Authenticate={ 0="Bearer realm=\"security\"" & 1="ApiKey" & 2="Basic realm=\"security\" charset=\"UTF-8\"" } } }"
[00:00:14]                 │ proc [kibana] name: "Simple Rule Query"
[00:00:14]                 │ proc [kibana] id: "88e0c40f-8f67-4f41-a16d-f0ac15e0a79f"
[00:00:14]                 │ proc [kibana] rule id: "rule-1"
[00:00:14]                 │ proc [kibana] signals index: ".siem-signals-default"
[00:00:14]                 │ info [o.e.x.s.a.AuthenticationService] [kibana-ci-immutable-centos-tests-xl-1585668584212196307] Authentication using apikey failed - api key has been invalidated
[00:00:14]                 │ proc [kibana]   log   [16:41:00.009] [error][alerting][alerting][plugins][plugins] Executing Alert "88e0c40f-8f67-4f41-a16d-f0ac15e0a79f" has resulted in Error: [security_exception] missing authentication credentials for REST request [/_security/user/_has_privileges], with { header={ WWW-Authenticate={ 0="Bearer realm=\"security\"" & 1="ApiKey" & 2="Basic realm=\"security\" charset=\"UTF-8\"" } } }
[00:00:14]                 └- ✓ pass  (4.8s) "detection engine api security and spaces enabled create_rules creating rules should create a single rule with a rule_id"
[00:00:14]               └-> "after each" hook
[00:00:14]                 │ info [o.e.c.m.MetaDataDeleteIndexService] [kibana-ci-immutable-centos-tests-xl-1585668584212196307] [.siem-signals-default-000001/-r5plPzrS86I3xcshVnMrw] deleting index
[00:00:15]                 │ info [o.e.c.m.MetaDataIndexTemplateService] [kibana-ci-immutable-centos-tests-xl-1585668584212196307] removing template [.siem-signals-default]
[00:00:15]               └-> should create a single rule without an input index
[00:00:15]                 └-> "before each" hook: global before each
[00:00:15]                 └-> "before each" hook
[00:00:15]                   │ info [o.e.x.i.a.TransportPutLifecycleAction] [kibana-ci-immutable-centos-tests-xl-1585668584212196307] adding index lifecycle policy [.siem-signals-default]
[00:00:15]                   │ info [o.e.c.m.MetaDataIndexTemplateService] [kibana-ci-immutable-centos-tests-xl-1585668584212196307] adding template [.siem-signals-default] for index patterns [.siem-signals-default-*]
[00:00:15]                   │ info [o.e.c.m.MetaDataCreateIndexService] [kibana-ci-immutable-centos-tests-xl-1585668584212196307] [.siem-signals-default-000001] creating index, cause [api], templates [.siem-signals-default], shards [1]/[1], mappings [_doc]
[00:00:15]                   │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-centos-tests-xl-1585668584212196307] moving index [.siem-signals-default-000001] from [null] to [{"phase":"new","action":"complete","name":"complete"}] in policy [.siem-signals-default]
[00:00:15]                   │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-centos-tests-xl-1585668584212196307] moving index [.siem-signals-default-000001] from [{"phase":"new","action":"complete","name":"complete"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-indexing-complete"}] in policy [.siem-signals-default]
[00:00:15]                 │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-centos-tests-xl-1585668584212196307] moving index [.siem-signals-default-000001] from [{"phase":"hot","action":"unfollow","name":"wait-for-indexing-complete"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-follow-shard-tasks"}] in policy [.siem-signals-default]
[00:00:19]                 │ info [o.e.x.s.a.AuthenticationService] [kibana-ci-immutable-centos-tests-xl-1585668584212196307] Authentication using apikey failed - api key has been invalidated
[00:00:19]                 │ proc [kibana]   log   [16:41:05.052] [error][plugins][siem] An error occurred during rule execution:
[00:00:19]                 │ proc [kibana] message: "[security_exception] missing authentication credentials for REST request [/apm-*-transaction*%2Cauditbeat-*%2Cendgame-*%2Cfilebeat-*%2Cpacketbeat-*%2Cwinlogbeat-*/_search?allow_no_indices=true&size=100&ignore_unavailable=true], with { header={ WWW-Authenticate={ 0="Bearer realm=\"security\"" & 1="ApiKey" & 2="Basic realm=\"security\" charset=\"UTF-8\"" } } }"
[00:00:19]                 │ proc [kibana] name: "Simple Rule Query"
[00:00:19]                 │ proc [kibana] id: "798d15af-b4f5-4de1-afb9-1ea438a1ec51"
[00:00:19]                 │ proc [kibana] rule id: "rule-1"
[00:00:19]                 │ proc [kibana] signals index: ".siem-signals-default"
[00:00:19]                 │ info [o.e.x.s.a.AuthenticationService] [kibana-ci-immutable-centos-tests-xl-1585668584212196307] Authentication using apikey failed - api key has been invalidated
[00:00:19]                 │ proc [kibana]   log   [16:41:05.058] [error][alerting][alerting][plugins][plugins] Executing Alert "798d15af-b4f5-4de1-afb9-1ea438a1ec51" has resulted in Error: [security_exception] missing authentication credentials for REST request [/_security/user/_has_privileges], with { header={ WWW-Authenticate={ 0="Bearer realm=\"security\"" & 1="ApiKey" & 2="Basic realm=\"security\" charset=\"UTF-8\"" } } }
[00:00:19]                 └- ✓ pass  (4.5s) "detection engine api security and spaces enabled create_rules creating rules should create a single rule without an input index"
[00:00:19]               └-> "after each" hook
[00:00:19]                 │ info [o.e.c.m.MetaDataDeleteIndexService] [kibana-ci-immutable-centos-tests-xl-1585668584212196307] [.siem-signals-default-000001/3tXp6pIVSgePuiyv11iKdg] deleting index
[00:00:20]                 │ info [o.e.c.m.MetaDataIndexTemplateService] [kibana-ci-immutable-centos-tests-xl-1585668584212196307] removing template [.siem-signals-default]
[00:00:20]               └-> should create a single rule without a rule_id
[00:00:20]                 └-> "before each" hook: global before each
[00:00:20]                 └-> "before each" hook
[00:00:20]                   │ info [o.e.x.i.a.TransportPutLifecycleAction] [kibana-ci-immutable-centos-tests-xl-1585668584212196307] adding index lifecycle policy [.siem-signals-default]
[00:00:20]                   │ info [o.e.c.m.MetaDataIndexTemplateService] [kibana-ci-immutable-centos-tests-xl-1585668584212196307] adding template [.siem-signals-default] for index patterns [.siem-signals-default-*]
[00:00:20]                   │ info [o.e.c.m.MetaDataCreateIndexService] [kibana-ci-immutable-centos-tests-xl-1585668584212196307] [.siem-signals-default-000001] creating index, cause [api], templates [.siem-signals-default], shards [1]/[1], mappings [_doc]
[00:00:20]                   │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-centos-tests-xl-1585668584212196307] moving index [.siem-signals-default-000001] from [null] to [{"phase":"new","action":"complete","name":"complete"}] in policy [.siem-signals-default]
[00:00:20]                   │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-centos-tests-xl-1585668584212196307] moving index [.siem-signals-default-000001] from [{"phase":"new","action":"complete","name":"complete"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-indexing-complete"}] in policy [.siem-signals-default]
[00:00:20]                 │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-centos-tests-xl-1585668584212196307] moving index [.siem-signals-default-000001] from [{"phase":"hot","action":"unfollow","name":"wait-for-indexing-complete"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-follow-shard-tasks"}] in policy [.siem-signals-default]
[00:00:24]                 └- ✓ pass  (4.6s) "detection engine api security and spaces enabled create_rules creating rules should create a single rule without a rule_id"
[00:00:24]               └-> "after each" hook
[00:00:25]                 │ info [o.e.c.m.MetaDataDeleteIndexService] [kibana-ci-immutable-centos-tests-xl-1585668584212196307] [.siem-signals-default-000001/xV9Bu566Sk-81wKC2OnKyg] deleting index
[00:00:25]                 │ info [o.e.c.m.MetaDataIndexTemplateService] [kibana-ci-immutable-centos-tests-xl-1585668584212196307] removing template [.siem-signals-default]
[00:00:25]               └-> should create a single Machine Learning rule
[00:00:25]                 └-> "before each" hook: global before each
[00:00:25]                 └-> "before each" hook
[00:00:25]                   │ info [o.e.x.i.a.TransportPutLifecycleAction] [kibana-ci-immutable-centos-tests-xl-1585668584212196307] adding index lifecycle policy [.siem-signals-default]
[00:00:25]                   │ info [o.e.c.m.MetaDataIndexTemplateService] [kibana-ci-immutable-centos-tests-xl-1585668584212196307] adding template [.siem-signals-default] for index patterns [.siem-signals-default-*]
[00:00:25]                   │ info [o.e.c.m.MetaDataCreateIndexService] [kibana-ci-immutable-centos-tests-xl-1585668584212196307] [.siem-signals-default-000001] creating index, cause [api], templates [.siem-signals-default], shards [1]/[1], mappings [_doc]
[00:00:25]                   │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-centos-tests-xl-1585668584212196307] moving index [.siem-signals-default-000001] from [null] to [{"phase":"new","action":"complete","name":"complete"}] in policy [.siem-signals-default]
[00:00:25]                   │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-centos-tests-xl-1585668584212196307] moving index [.siem-signals-default-000001] from [{"phase":"new","action":"complete","name":"complete"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-indexing-complete"}] in policy [.siem-signals-default]
[00:00:25]                 │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-centos-tests-xl-1585668584212196307] moving index [.siem-signals-default-000001] from [{"phase":"hot","action":"unfollow","name":"wait-for-indexing-complete"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-follow-shard-tasks"}] in policy [.siem-signals-default]
[00:00:28]                 │ proc [kibana]   log   [16:41:14.276] [warning][plugins][siem] Machine learning job is not started:
[00:00:28]                 │ proc [kibana] job id: "some_job_id"
[00:00:28]                 │ proc [kibana] job status: "undefined"
[00:00:28]                 │ proc [kibana] datafeed status: "undefined"
[00:00:28]                 │ proc [kibana] name: "Simple ML Rule"
[00:00:28]                 │ proc [kibana] id: "d5ccd323-07d1-4de2-ad66-c4f2dc09b011"
[00:00:28]                 │ proc [kibana] rule id: "rule-1"
[00:00:28]                 │ proc [kibana] signals index: ".siem-signals-default"
[00:00:30]                 └- ✖ fail: "detection engine api security and spaces enabled create_rules creating rules should create a single Machine Learning rule"
[00:00:30]                 │

Stack Trace

{ Error: expected { created_by: 'elastic',
  description: 'Simple Machine Learning Rule',
  enabled: true,
  false_positives: [],
  from: 'now-6m',
  immutable: false,
  interval: '5m',
  rule_id: 'rule-1',
  output_index: '.siem-signals-default',
  max_signals: 100,
  risk_score: 1,
  name: 'Simple ML Rule',
  references: [],
  severity: 'high',
  updated_by: 'elastic',
  tags: [],
  to: 'now',
  type: 'machine_learning',
  threat: [],
  version: 1,
  lists: [],
  actions: [],
  throttle: 'no_actions',
  last_failure_at: '2020-03-31T16:41:14.296Z',
  last_failure_message: 'Machine learning job is not started:\njob id: "some_job_id"\njob status: "undefined"\ndatafeed status: "undefined"\nname: "Simple ML Rule"\nid: "d5ccd323-07d1-4de2-ad66-c4f2dc09b011"\nrule id: "rule-1"\nsignals index: ".siem-signals-default"',
  anomaly_threshold: 44,
  machine_learning_job_id: 'some_job_id' } to sort of equal { actions: [],
  created_by: 'elastic',
  description: 'Simple Machine Learning Rule',
  enabled: true,
  false_positives: [],
  from: 'now-6m',
  immutable: false,
  interval: '5m',
  rule_id: 'rule-1',
  output_index: '.siem-signals-default',
  max_signals: 100,
  risk_score: 1,
  name: 'Simple ML Rule',
  references: [],
  severity: 'high',
  updated_by: 'elastic',
  tags: [],
  to: 'now',
  type: 'machine_learning',
  threat: [],
  throttle: 'no_actions',
  lists: [],
  version: 1,
  anomaly_threshold: 44,
  machine_learning_job_id: 'some_job_id' }
    at Assertion.assert (/dev/shm/workspace/kibana/packages/kbn-expect/expect.js:100:11)
    at Assertion.eql (/dev/shm/workspace/kibana/packages/kbn-expect/expect.js:244:8)
    at Context.it (test/detection_engine_api_integration/security_and_spaces/tests/create_rules.ts:101:34)
  actual:
   '{\n  "actions": []\n  "anomaly_threshold": 44\n  "created_by": "elastic"\n  "description": "Simple Machine Learning Rule"\n  "enabled": true\n  "false_positives": []\n  "from": "now-6m"\n  "immutable": false\n  "interval": "5m"\n  "last_failure_at": "2020-03-31T16:41:14.296Z"\n  "last_failure_message": "Machine learning job is not started:\\njob id: \\"some_job_id\\"\\njob status: \\"undefined\\"\\ndatafeed status: \\"undefined\\"\\nname: \\"Simple ML Rule\\"\\nid: \\"d5ccd323-07d1-4de2-ad66-c4f2dc09b011\\"\\nrule id: \\"rule-1\\"\\nsignals index: \\".siem-signals-default\\""\n  "lists": []\n  "machine_learning_job_id": "some_job_id"\n  "max_signals": 100\n  "name": "Simple ML Rule"\n  "output_index": ".siem-signals-default"\n  "references": []\n  "risk_score": 1\n  "rule_id": "rule-1"\n  "severity": "high"\n  "tags": []\n  "threat": []\n  "throttle": "no_actions"\n  "to": "now"\n  "type": "machine_learning"\n  "updated_by": "elastic"\n  "version": 1\n}',
  expected:
   '{\n  "actions": []\n  "anomaly_threshold": 44\n  "created_by": "elastic"\n  "description": "Simple Machine Learning Rule"\n  "enabled": true\n  "false_positives": []\n  "from": "now-6m"\n  "immutable": false\n  "interval": "5m"\n  "lists": []\n  "machine_learning_job_id": "some_job_id"\n  "max_signals": 100\n  "name": "Simple ML Rule"\n  "output_index": ".siem-signals-default"\n  "references": []\n  "risk_score": 1\n  "rule_id": "rule-1"\n  "severity": "high"\n  "tags": []\n  "threat": []\n  "throttle": "no_actions"\n  "to": "now"\n  "type": "machine_learning"\n  "updated_by": "elastic"\n  "version": 1\n}',
  showDiff: true }

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@jloleysens jloleysens merged commit 33261ff into elastic:7.x Apr 1, 2020
@jloleysens jloleysens deleted the backport/7.x/pr-60950 branch April 1, 2020 08:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

backport This PR is a backport of another PR

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@jloleysens @kibanamachine