I am testing support for multiple elasticsearch_nodes and found it is working when I stop one of the es node but Dev tools -> Console is not falling back to the alive es node.
This is how I tested it.
- start 2 es nodes
ES_PATH_CONF=./conf1 elasticsearch/bin/elasticsearch -Enode.name=node1 -Ecluster.name=elasticsearch -Epath.data=node1_data -d
ES_PATH_CONF=./conf2 elasticsearch/bin/elasticsearch -Enode.name=node2 -Ecluster.name=elasticsearch -Epath.data=node2_data -d
-
start kibana with the following set in kibana.yml
elasticsearch.hosts: ["http://localhost:9200","http://localhost:9201"]
-
make sure you can use the most feature of kibana such as Console and Discover.
For instance, I did GET / in http://localhost:5601/app/kibana#/dev_tools/console?_g=()
response
{
"name" : "node1",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "xk8_BJHFQU-4NXSA_TBcdQ",
"version" : {
"number" : "6.6.0",
"build_flavor" : "default",
"build_type" : "tar",
"build_hash" : "a9861f4",
"build_date" : "2019-01-24T11:27:09.439740Z",
"build_snapshot" : false,
"lucene_version" : "7.6.0",
"minimum_wire_compatibility_version" : "5.6.0",
"minimum_index_compatibility_version" : "5.0.0"
},
"tagline" : "You Know, for Search"
}
GET /_cat/nodes?v
ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
127.0.0.1 32 99 15 2.03 mdi - node2
127.0.0.1 36 99 15 2.03 mdi * node1
- check kibana connection.
[wave:tools]$ k_pid=$(lsof -i:5601 | grep node | awk -F ' ' '{print $2}')
[wave:tools]$ lsof -P -n -p $k_pid | grep TCP
node 96456 surfer 25u IPv4 0x716ccc6fe8bedf5f 0t0 TCP 127.0.0.1:5601 (LISTEN)
node 96456 surfer 26u IPv4 0x716ccc7000d04f5f 0t0 TCP 127.0.0.1:56327->127.0.0.1:9201 (ESTABLISHED)
node 96456 surfer 27u IPv4 0x716ccc6ff6c7e25f 0t0 TCP 127.0.0.1:56338->127.0.0.1:9200 (ESTABLISHED)
node 96456 surfer 28u IPv4 0x716ccc6fd42375df 0t0 TCP 127.0.0.1:56328->127.0.0.1:9200 (ESTABLISHED)
node 96456 surfer 29u IPv4 0x716ccc6ffa3d125f 0t0 TCP 127.0.0.1:56211->127.0.0.1:9200 (ESTABLISHED)
node 96456 surfer 30u IPv4 0x716ccc6ffb25a5df 0t0 TCP 127.0.0.1:56212->127.0.0.1:9201 (ESTABLISHED)
node 96456 surfer 31u IPv4 0x716ccc6ffab9ef5f 0t0 TCP 127.0.0.1:56644->127.0.0.1:9201 (ESTABLISHED)
node 96456 surfer 32u IPv4 0x716ccc6ff8affc5f 0t0 TCP 127.0.0.1:56683->127.0.0.1:9200 (ESTABLISHED)
node 96456 surfer 33u IPv4 0x716ccc7004072f5f 0t0 TCP 127.0.0.1:56645->127.0.0.1:9201 (ESTABLISHED)
node 96456 surfer 34u IPv4 0x716ccc70011635df 0t0 TCP 127.0.0.1:56215->127.0.0.1:9200 (ESTABLISHED)
node 96456 surfer 35u IPv4 0x716ccc6ff6c01f5f 0t0 TCP 127.0.0.1:56684->127.0.0.1:9200 (ESTABLISHED)
node 96456 surfer 36u IPv4 0x716ccc6ffb55425f 0t0 TCP 127.0.0.1:56216->127.0.0.1:9201 (ESTABLISHED)
node 96456 surfer 37u IPv4 0x716ccc70045a05df 0t0 TCP 127.0.0.1:56217->127.0.0.1:9200 (ESTABLISHED)
node 96456 surfer 38u IPv4 0x716ccc700459fc5f 0t0 TCP 127.0.0.1:56218->127.0.0.1:9201 (ESTABLISHED)
node 96456 surfer 39u IPv4 0x716ccc6fe80f125f 0t0 TCP 127.0.0.1:56219->127.0.0.1:9200 (ESTABLISHED)
node 96456 surfer 41u IPv4 0x716ccc6ff0c7d5df 0t0 TCP 127.0.0.1:56228->127.0.0.1:9200 (ESTABLISHED)
node 96456 surfer 42u IPv4 0x716ccc6feb2545df 0t0 TCP 127.0.0.1:56237->127.0.0.1:9201 (ESTABLISHED)
node 96456 surfer 43u IPv4 0x716ccc6ff71715df 0t0 TCP 127.0.0.1:56238->127.0.0.1:9201 (ESTABLISHED)
node 96456 surfer 44u IPv4 0x716ccc6fd10fef5f 0t0 TCP 127.0.0.1:56239->127.0.0.1:9201 (ESTABLISHED)
node 96456 surfer 47u IPv4 0x716ccc6fd423925f 0t0 TCP 127.0.0.1:56329->127.0.0.1:9201 (ESTABLISHED)
node 96456 surfer 48u IPv4 0x716ccc6fd4236c5f 0t0 TCP 127.0.0.1:56330->127.0.0.1:9200 (ESTABLISHED)
node 96456 surfer 49u IPv4 0x716ccc6ffd05e8df 0t0 TCP 127.0.0.1:56331->127.0.0.1:9201 (ESTABLISHED)
node 96456 surfer 52u IPv4 0x716ccc6ffb552f5f 0t0 TCP 127.0.0.1:56389->127.0.0.1:9200 (ESTABLISHED)
[wave:tools]$
- stop the es whose name is
node1 by locating the pid by jps -v | grep -i conf1
[wave:6.6.0]$ jps -v | grep -i conf1
96419 Elasticsearch -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.io.tmpdir=/tmp2 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=data -XX:ErrorFile=logs/hs_err_pid%p.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:logs/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=32 -XX:GCLogFileSize=64m -Des.path.home=/Users/surfer/elastic/labs/6.6.0/elasticsearch -Des.path.conf=/Users/surfer/elastic/labs/6.6.0/conf1 -Des.distribution.flavor=default -Des.distribution.type=tar
[wave:6.6.0]$ kill -9 96419
- check kibana connection again and see that kibana is now exclusively connected to port 9201 which is the second es node.
[wave:tools]$ lsof -P -n -p $k_pid | grep TCP
node 96456 surfer 25u IPv4 0x716ccc6fe8bedf5f 0t0 TCP 127.0.0.1:5601 (LISTEN)
node 96456 surfer 26u IPv4 0x716ccc7000d04f5f 0t0 TCP 127.0.0.1:56327->127.0.0.1:9201 (ESTABLISHED)
node 96456 surfer 27u IPv4 0x716ccc6ff6c015df 0t0 TCP 127.0.0.1:56803->127.0.0.1:9201 (ESTABLISHED)
node 96456 surfer 28u IPv4 0x716ccc6ffd05df5f 0t0 TCP 127.0.0.1:56813->127.0.0.1:9201 (ESTABLISHED)
node 96456 surfer 29u IPv4 0x716ccc6ffd05d5df 0t0 TCP 127.0.0.1:56814->127.0.0.1:9201 (ESTABLISHED)
node 96456 surfer 30u IPv4 0x716ccc6ffb25a5df 0t0 TCP 127.0.0.1:56212->127.0.0.1:9201 (ESTABLISHED)
node 96456 surfer 31u IPv4 0x716ccc6ffafbcc5f 0t0 TCP 127.0.0.1:56722->127.0.0.1:9201 (ESTABLISHED)
node 96456 surfer 32u IPv4 0x716ccc6fd4b8d25f 0t0 TCP 127.0.0.1:56815->127.0.0.1:9201 (ESTABLISHED)
node 96456 surfer 33u IPv4 0x716ccc6fe905025f 0t0 TCP 127.0.0.1:56723->127.0.0.1:9201 (ESTABLISHED)
node 96456 surfer 34u IPv4 0x716ccc6fe07a4f5f 0t0 TCP 127.0.0.1:56816->127.0.0.1:9201 (ESTABLISHED)
node 96456 surfer 36u IPv4 0x716ccc6ffb55425f 0t0 TCP 127.0.0.1:56216->127.0.0.1:9201 (ESTABLISHED)
node 96456 surfer 38u IPv4 0x716ccc700459fc5f 0t0 TCP 127.0.0.1:56218->127.0.0.1:9201 (ESTABLISHED)
node 96456 surfer 39u IPv4 0x716ccc6ffafddc5f 0t0 TCP 127.0.0.1:56842->127.0.0.1:9201 (ESTABLISHED)
node 96456 surfer 40u IPv4 0x716ccc6fdf8005df 0t0 TCP 127.0.0.1:56841->127.0.0.1:9201 (ESTABLISHED)
node 96456 surfer 42u IPv4 0x716ccc6feb2545df 0t0 TCP 127.0.0.1:56237->127.0.0.1:9201 (ESTABLISHED)
node 96456 surfer 43u IPv4 0x716ccc6ff71715df 0t0 TCP 127.0.0.1:56238->127.0.0.1:9201 (ESTABLISHED)
node 96456 surfer 44u IPv4 0x716ccc6fd10fef5f 0t0 TCP 127.0.0.1:56239->127.0.0.1:9201 (ESTABLISHED)
node 96456 surfer 47u IPv4 0x716ccc6fd423925f 0t0 TCP 127.0.0.1:56329->127.0.0.1:9201 (ESTABLISHED)
node 96456 surfer 49u IPv4 0x716ccc6ffd05e8df 0t0 TCP 127.0.0.1:56331->127.0.0.1:9201 (ESTABLISHED)
- Doing
GET / in http://localhost:5601/app/kibana#/dev_tools/console?_g=() again comes back with this resposne, but Kibana itself is working such as Discover is fully functional.
{
"message": "Client request error: connect ECONNREFUSED 127.0.0.1:9200",
"statusCode": 502,
"error": "Bad Gateway"
}
I am testing support for multiple elasticsearch_nodes and found it is working when I stop one of the es node but
Dev tools -> Consoleis not falling back to the alive es node.This is how I tested it.
start kibana with the following set in kibana.yml
elasticsearch.hosts: ["http://localhost:9200","http://localhost:9201"]make sure you can use the most feature of kibana such as Console and Discover.
For instance, I did
GET /inhttp://localhost:5601/app/kibana#/dev_tools/console?_g=()response
GET /_cat/nodes?vnode1by locating the pid byjps -v | grep -i conf1GET /inhttp://localhost:5601/app/kibana#/dev_tools/console?_g=()again comes back with this resposne, but Kibana itself is working such as Discover is fully functional.