There currently are many instances of build failures where the Mixed Cluster Tests are failing because the test is waiting on a connection to check the cluster health but this fails with:
12:48:36 [ant:get] Error opening connection java.io.IOException: Server returned HTTP response code: 503 for URL: http://[::1]:40959/_cluster/health?wait_for_nodes=>=4&wait_for_status=yellow
Example for failures: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.0+java9-periodic/1185/consoleFull
12:47:52 Task has not declared any outputs.
12:48:06 [ant:echo] ==> [Thu Nov 02 12:48:06 UTC 2017] checking health: http://[::1]:40959/_cluster/health?wait_for_nodes=>=4&wait_for_status=yellow
12:48:06 [echo] ==> [Thu Nov 02 12:48:06 UTC 2017] checking health: http://[::1]:40959/_cluster/health?wait_for_nodes=>=4&wait_for_status=yellow
12:48:06 [ant:get] Getting: http://[::1]:40959/_cluster/health?wait_for_nodes=>=4&wait_for_status=yellow
12:48:06 [get] Getting: http://[::1]:40959/_cluster/health?wait_for_nodes=>=4&wait_for_status=yellow
12:48:06 [ant:get] To: /var/lib/jenkins/workspace/elastic+elasticsearch+6.0+java9-periodic/qa/mixed-cluster/build/cluster/v5.6.4-SNAPSHOT#mixedClusterTestCluster node0/cwd/wait.success
12:48:06 [get] To: /var/lib/jenkins/workspace/elastic+elasticsearch+6.0+java9-periodic/qa/mixed-cluster/build/cluster/v5.6.4-SNAPSHOT#mixedClusterTestCluster node0/cwd/wait.success
12:48:36 [ant:get] Error opening connection java.io.IOException: Server returned HTTP response code: 503 for URL: http://[::1]:40959/_cluster/health?wait_for_nodes=>=4&wait_for_status=yellow
12:48:36 [get] Error opening connection java.io.IOException: Server returned HTTP response code: 503 for URL: http://[::1]:40959/_cluster/health?wait_for_nodes=>=4&wait_for_status=yellow
12:48:36 [ant:get] Error opening connection java.io.IOException: Server returned HTTP response code: 503 for URL: http://[::1]:40959/_cluster/health?wait_for_nodes=>=4&wait_for_status=yellow
12:48:36 [get] Error opening connection java.io.IOException: Server returned HTTP response code: 503 for URL: http://[::1]:40959/_cluster/health?wait_for_nodes=>=4&wait_for_status=yellow
12:48:36 [ant:get] Error opening connection java.io.IOException: Server returned HTTP response code: 503 for URL: http://[::1]:40959/_cluster/health?wait_for_nodes=>=4&wait_for_status=yellow
12:48:36 [get] Error opening connection java.io.IOException: Server returned HTTP response code: 503 for URL: http://[::1]:40959/_cluster/health?wait_for_nodes=>=4&wait_for_status=yellow
12:48:36 [ant:get] Error opening connection java.io.IOException: Server returned HTTP response code: 503 for URL: http://[::1]:40959/_cluster/health?wait_for_nodes=>=4&wait_for_status=yellow
12:48:36 [get] Error opening connection java.io.IOException: Server returned HTTP response code: 503 for URL: http://[::1]:40959/_cluster/health?wait_for_nodes=>=4&wait_for_status=yellow
12:48:36 [ant:get] Error opening connection java.io.IOException: Server returned HTTP response code: 503 for URL: http://[::1]:40959/_cluster/health?wait_for_nodes=>=4&wait_for_status=yellow
12:48:36 [get] Error opening connection java.io.IOException: Server returned HTTP response code: 503 for URL: http://[::1]:40959/_cluster/health?wait_for_nodes=>=4&wait_for_status=yellow
12:48:36 [ant:get] Error opening connection java.io.IOException: Server returned HTTP response code: 503 for URL: http://[::1]:40959/_cluster/health?wait_for_nodes=>=4&wait_for_status=yellow
12:48:36 [get] Error opening connection java.io.IOException: Server returned HTTP response code: 503 for URL: http://[::1]:40959/_cluster/health?wait_for_nodes=>=4&wait_for_status=yellow
12:48:36 Node 0 output:
12:48:36 [ant:get] Error opening connection java.io.IOException: Server returned HTTP response code: 503 for URL: http://[::1]:40959/_cluster/health?wait_for_nodes=>=4&wait_for_status=yellow
12:48:36 |-----------------------------------------
12:48:36 [get] Error opening connection java.io.IOException: Server returned HTTP response code: 503 for URL: http://[::1]:40959/_cluster/health?wait_for_nodes=>=4&wait_for_status=yellow
12:48:36 [ant:get] Error opening connection java.io.IOException: Server returned HTTP response code: 503 for URL: http://[::1]:40959/_cluster/health?wait_for_nodes=>=4&wait_for_status=yellow
12:48:36 [get] Error opening connection java.io.IOException: Server returned HTTP response code: 503 for URL: http://[::1]:40959/_cluster/health?wait_for_nodes=>=4&wait_for_status=yellow
12:48:36 [ant:get] Error opening connection java.io.IOException: Server returned HTTP response code: 503 for URL: http://[::1]:40959/_cluster/health?wait_for_nodes=>=4&wait_for_status=yellow
12:48:36 [get] Error opening connection java.io.IOException: Server returned HTTP response code: 503 for URL: http://[::1]:40959/_cluster/health?wait_for_nodes=>=4&wait_for_status=yellow
12:48:36 [ant:get] Error opening connection java.io.IOException: Server returned HTTP response code: 503 for URL: http://[::1]:40959/_cluster/health?wait_for_nodes=>=4&wait_for_status=yellow
12:48:36 | failure marker exists: false
12:48:36 | pid file exists: true
12:48:36 | http ports file exists: true
12:48:36 [get] Error opening connection java.io.IOException: Server returned HTTP response code: 503 for URL: http://[::1]:40959/_cluster/health?wait_for_nodes=>=4&wait_for_status=yellow
12:48:36 [ant:get] Can't get http://[::1]:40959/_cluster/health?wait_for_nodes=>=4&wait_for_status=yellow to /var/lib/jenkins/workspace/elastic+elasticsearch+6.0+java9-periodic/qa/mixed-cluster/build/cluster/v5.6.4-SNAPSHOT#mixedClusterTestCluster node0/cwd/wait.success
12:48:36 [get] Can't get http://[::1]:40959/_cluster/health?wait_for_nodes=>=4&wait_for_status=yellow to /var/lib/jenkins/workspace/elastic+elasticsearch+6.0+java9-periodic/qa/mixed-cluster/build/cluster/v5.6.4-SNAPSHOT#mixedClusterTestCluster node0/cwd/wait.success
The above failure also has:
12:48:36 | [2017-11-02T12:48:36,421][WARN ][r.suppressed ] path: /_cluster/health, params: {wait_for_status=yellow, wait_for_nodes=>=4}
12:48:36 | org.elasticsearch.discovery.MasterNotDiscoveredException: null
12:48:36 | at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$4.onTimeout(TransportMasterNodeAction.java:209) [elasticsearch-5.6.4-SNAPSHOT.jar:5.6.4-SNAPSHOT]
12:48:36 | at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:311) [elasticsearch-5.6.4-SNAPSHOT.jar:5.6.4-SNAPSHOT]
12:48:36 | at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:238) [elasticsearch-5.6.4-SNAPSHOT.jar:5.6.4-SNAPSHOT]
12:48:36 | at org.elasticsearch.cluster.service.ClusterService$NotifyTimeout.run(ClusterService.java:1056) [elasticsearch-5.6.4-SNAPSHOT.jar:5.6.4-SNAPSHOT]
12:48:36 | at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569) [elasticsearch-5.6.4-SNAPSHOT.jar:5.6.4-SNAPSHOT]
12:48:36 | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167) [?:?]
12:48:36 | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641) [?:?]
12:48:36 | at java.lang.Thread.run(Thread.java:844) [?:?]
In one of the node logs.
There currently are many instances of build failures where the Mixed Cluster Tests are failing because the test is waiting on a connection to check the cluster health but this fails with:
Example for failures: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.0+java9-periodic/1185/consoleFull
The above failure also has:
In one of the node logs.