The problem
Kibana crashes with ECONNRESET error when connection to Elasticsearch cluster is lost (I assume this is the problem)
Expectation
Kibana should not crash when connection to Elasticsearch cluster is lost
Reproduction
This happens on my local developer machine when running yarn start and connecting to an external (cloud based) elasticsearch cluster. If I close the laptop lid and thus disable network, Kibana crashes after about 1 hour.
First I see the following lines:
[warning][kibanaUsageCollection][plugins] Average event loop delay threshold exceeded 350ms. Received 2654.3637022916055ms. See https://ela.st/kibana-scaling-considerations for more information about scaling Kibana.
[error][plugins][taskManager] [Task Poller Monitor]: Observable Monitor: Hung Observable restarted after 33000ms of inactivity
[warning][plugins][taskManager] Detected potential performance issue with Task Manager. Set 'xpack.task_manager.monitored_stats_health_verbose_log.enabled: true' in your Kibana.yml to enable debug logging
[info][status] Kibana is now unavailable (was available)
[info][status] Kibana is now available (was unavailable)
Then after a few more minutes I see:
[warning][kibanaUsageCollection][plugins] Average event loop delay threshold exceeded 350ms. Received 577.03415997404ms. See https://ela.st/kibana-scaling-considerations for more information about scaling Kibana.
[error][plugins][taskManager] [Task Poller Monitor]: Observable Monitor: Hung Observable restarted after 33000ms of inactivity
[warning][plugins][taskManager] Detected potential performance issue with Task Manager. Set 'xpack.task_manager.monitored_stats_health_verbose_log.enabled: true' in your Kibana.yml to enable debug logging
And then it crashes with:
Unhandled Promise rejection detected:
ConnectionError: read ECONNRESET
at ClientRequest.onError (/Users/sqren/elastic/kibana/node_modules/@elastic/elasticsearch/lib/Connection.js:116:16)
at ClientRequest.emit (events.js:400:28)
at TLSSocket.socketErrorListener (_http_client.js:475:9)
at TLSSocket.emit (events.js:400:28)
at emitErrorNT (internal/streams/destroy.js:106:8)
at emitErrorCloseNT (internal/streams/destroy.js:74:3)
at processTicksAndRejections (internal/process/task_queues.js:82:21) {
meta: {
body: null,
statusCode: null,
headers: null,
meta: {
context: null,
request: [Object],
name: 'elasticsearch-js',
connection: [Object],
attempts: 0,
aborted: false
}
},
isBoom: true,
isServer: true,
data: null,
output: {
statusCode: 503,
payload: {
statusCode: 503,
error: 'Service Unavailable',
message: 'read ECONNRESET'
},
headers: {}
},
[Symbol(SavedObjectsClientErrorCode)]: 'SavedObjectsClient/esUnavailable'
}
Terminating process...
server crashed with status code 1
The problem
Kibana crashes with
ECONNRESETerror when connection to Elasticsearch cluster is lost (I assume this is the problem)Expectation
Kibana should not crash when connection to Elasticsearch cluster is lost
Reproduction
This happens on my local developer machine when running
yarn startand connecting to an external (cloud based) elasticsearch cluster. If I close the laptop lid and thus disable network, Kibana crashes after about 1 hour.First I see the following lines:
Then after a few more minutes I see:
And then it crashes with: