Kibana version: 8.2.0-SNAPSHOT (details below show the problem seems to have started around March 1 2022)
Elasticsearch version:
same as Kibana
Original install method (e.g. download page, yum, from source, etc.):
- deploiyed via cloud, on cloud-proud CFT region. But problem was initially triaged and seen apart from cloud.
The issue was first found by the kbn-alert-load Rules Performance tester, by the Response Ops team. I can fwd you a slack message that relates, though I will try to capture the most relevant data here. I can also forward you more details, like the kbn-alert-load jenkins job output that can be re-run on a latest/later-new snapshot easily to re-run the test if we like.
Describe the bug:
The test starts up several cloud clusters, and individually assesses their memory usage and other metrics after creating a variable number of rules a variable % of which can create alerts from their execution.
Screenshots (if relevant):
Errors in browser console (if relevant):
n/a
Provide logs and/or server output (if relevant):
Here are the things from slack that we've already gathered:
Questions asked (and answered, mostly):
Is this repeatable or was this a s single run that could be off (I guess not)?
Yes, repeatable with the kbn-alert-load tool and locally
Is that also reproducible locally?
not applicable to the tool's use, but yes others see memory concerns locally
From Brandon K: Anecdotally, I'm seeing the memory usage increase from 8.1 to 8.2 even when there are no alerting rules running. The following are heap usages when starting up a new Kibana instance and logging in:
8.1: 403 MB
8.2: 574 MB
For an RSS increase, the first thing that would be interesting to understand whether it’s an increase in native memory or heap memory.
Heap dumps from Brandon K are here:
https://drive.google.com/drive/folders/1n0jjJ_H3oEbViMYjd8eKKqqcwM2mbcOX?usp=sharing
NOTE: we suspected possibly a node version? But it does not seem to be the Node version btw, 8.0 uses Node 16.13.2 (see https://github.com/elastic/kibana/blob/8.0/.node-version) as well as main (https://github.com/elastic/kibana/blob/main/.node-version).
Patrick M offered up a script he had used to create many running rules locally, if helpful:
https://gist.github.com/pmuellr/f30331660ae032a0a7ccf2767aea3900
Kibana version: 8.2.0-SNAPSHOT (details below show the problem seems to have started around March 1 2022)
Elasticsearch version:
same as Kibana
Original install method (e.g. download page, yum, from source, etc.):
The issue was first found by the kbn-alert-load Rules Performance tester, by the Response Ops team. I can fwd you a slack message that relates, though I will try to capture the most relevant data here. I can also forward you more details, like the kbn-alert-load jenkins job output that can be re-run on a latest/later-new snapshot easily to re-run the test if we like.
Describe the bug:
The test starts up several cloud clusters, and individually assesses their memory usage and other metrics after creating a variable number of rules a variable % of which can create alerts from their execution.
Screenshots (if relevant):
Errors in browser console (if relevant):
n/a
Provide logs and/or server output (if relevant):
Here are the things from slack that we've already gathered:
Questions asked (and answered, mostly):
Is this repeatable or was this a s single run that could be off (I guess not)?
Yes, repeatable with the kbn-alert-load tool and locally
Is that also reproducible locally?
not applicable to the tool's use, but yes others see memory concerns locally
From Brandon K: Anecdotally, I'm seeing the memory usage increase from 8.1 to 8.2 even when there are no alerting rules running. The following are heap usages when starting up a new Kibana instance and logging in:
8.1: 403 MB
8.2: 574 MB
For an RSS increase, the first thing that would be interesting to understand whether it’s an increase in native memory or heap memory.
Heap dumps from Brandon K are here:
https://drive.google.com/drive/folders/1n0jjJ_H3oEbViMYjd8eKKqqcwM2mbcOX?usp=sharing
NOTE: we suspected possibly a node version? But it does not seem to be the Node version btw, 8.0 uses Node 16.13.2 (see https://github.com/elastic/kibana/blob/8.0/.node-version) as well as main (https://github.com/elastic/kibana/blob/main/.node-version).
Patrick M offered up a script he had used to create many running rules locally, if helpful:
https://gist.github.com/pmuellr/f30331660ae032a0a7ccf2767aea3900