Skip to content

kibana/x-pack on tribe node issue #20544

@obudiman

Description

@obudiman

Elasticsearch version: 5.0.0-alpha5

Plugins installed: x-pack@5.0.0-alpha5

JVM version: java-1.8.0-openjdk-1.8.0.101-3.b13.24.amzn1.x86_64

OS version: Linux 4.4.11-23.53.amzn1.x86_64

Description of the problem including expected versus actual behavior:
Kibana not usable when installed on tribe node with X-pack. We have followed the workaround as suggested on github #3114, but when trying to start the kibana, we found the following error messages on kibana log:

{"type":"log","@timestamp":"2016-09-19T06:57:33Z","tags":["status","plugin:xpack_main@5.0.0-alpha5-SNAPSHOT","error"],"pid":25694,"state":"red","message":"Status changed from yellow to red - Request Timeout after 30000ms","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2016-09-19T06:57:33Z","tags":["status","plugin:graph@5.0.0-alpha5-SNAPSHOT","error"],"pid":25694,"state":"red","message":"Status changed from yellow to red - Request Timeout after 30000ms","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2016-09-19T06:57:33Z","tags":["status","plugin:reporting@5.0.0-alpha5-SNAPSHOT","error"],"pid":25694,"state":"red","message":"Status changed from yellow to red - Request Timeout after 30000ms","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2016-09-19T06:57:33Z","tags":["status","plugin:security@5.0.0-alpha5-SNAPSHOT","error"],"pid":25694,"state":"red","message":"Status changed from yellow to red - Request Timeout after 30000ms","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2016-09-19T06:57:43Z","tags":["error","monitoring-ui"],"pid":25694,"level":"error","message":"Request Timeout after 30000ms","error":{"message":"Request Timeout after 30000ms","name":"Error","stack":"Error: Request Timeout after 30000ms\n    at /usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:336:15\n    at [object Object].<anonymous    (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:365:7)\n    at Timer.listOnTimeout (timers.js:92:15)"}}

Looking further on the ES log, I can see the following error messages which seems to indicate it tries to create those indices from the tribe node:

[2016-09-19 16:58:33,210][DEBUG][action.admin.indices.create] [dcmarhel6d] timed out while retrying [indices:admin/create] after failure (timeout [1m])
[2016-09-19 16:58:33,210][DEBUG][action.admin.indices.create] [dcmarhel6d] timed out while retrying [indices:admin/create] after failure (timeout [1m])
[2016-09-19 16:58:33,239][INFO ][xpack.security.audit.logfile] [transport] [access_granted] origin_type=[rest], origin_address=[127.0.0.1], principal=[dcma.appservice], action=[indices:data/write/bulk]
[2016-09-19 16:58:33,239][INFO ][xpack.security.audit.logfile] [transport] [access_granted] origin_type=[rest], origin_address=[127.0.0.1], principal=[dcma.appservice], action=[indices:admin/create], indices=[_data]
[2016-09-19 16:58:33,240][DEBUG][action.admin.indices.create] [dcmarhel6d] no known master node, scheduling a retry
[2016-09-19 16:58:33,240][INFO ][xpack.security.audit.logfile] [transport] [access_granted] origin_type=[rest], origin_address=[127.0.0.1], principal=[dcma.appservice], action=[indices:admin/create], indices=[_xpack]
[2016-09-19 16:58:33,240][DEBUG][action.admin.indices.create] [dcmarhel6d] no known master node, scheduling a retry

We've tried to set the tribe node to be a data node to see if it will create the indices, but it didn't work although the kibana would start just fine. After reverting the data node to be a tribe node, we still found the same error messages.

Steps to reproduce:

  1. Install kibana on tribe node
  2. Follow workaround as stated on github return term vectors and some statistics for a document #3114 (pointing kibana to elected master node and create dashboard,search,visualization before repointing it back to tribe node)
  3. Check kibana.stdout and elasticsearch.log

Metadata

Metadata

Assignees

No one assigned

    Labels

    :Distributed/DistributedA catch all label for anything in the Distributed Area. Please avoid if you can.

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions