Skip to content

IllegalStateException: cannot send more responses than requests #6775

Closed
Closed
IllegalStateException: cannot send more responses than requests#6775
@devoncrouse

Description

@devoncrouse
devoncrouse
opened on Apr 4, 2016

We have some existing infrastructure automation for Elasticsearch (that we like), and Kibana seems to break upon upgrading to ES 2.3/Kibana 4.5:

  • Cluster status is green (checked via curl from Kibana host, using same address provided to -e arg)
  • Head plugin loads and also shows green, with .kibana index created
  • Kibana loads into Oops! page with fatal error: Courier Fetch Error: unhandled courier request error: socket hang up

Kibana log output on each request of /:

{"type":"response","@timestamp":"2016-04-04T23:08:59+00:00","tags":[],"pid":5,"method":"get","statusCode":304,"req":{"url":"/bundles/node_modules/font-awesome/fonts/fontawesome-webfont.woff2","method":"get","headers":{"host":"10.100.1.213:22281","connection":"keep-alive","cache-control":"max-age=0","origin":"http://10.100.1.213:22281","if-modified-since":"Mon, 28 Mar 2016 23:29:56 GMT","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.110 Safari/537.36","accept":"*/*","referer":"http://10.100.1.213:22281/app/kibana","accept-encoding":"gzip, deflate, sdch","accept-language":"en-US,en;q=0.8"},"remoteAddress":"192.168.57.51","userAgent":"192.168.57.51","referer":"http://10.100.1.213:22281/app/kibana"},"res":{"statusCode":304,"responseTime":2,"contentLength":9},"message":"GET /bundles/node_modules/font-awesome/fonts/fontawesome-webfont.woff2 304 2ms - 9.0B"}
{"type":"response","@timestamp":"2016-04-04T23:09:01+00:00","tags":[],"pid":5,"method":"get","statusCode":200,"req":{"url":"/","method":"get","headers":{"host":"10.100.1.213:22281","connection":"keep-alive","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.110 Safari/537.36","accept-encoding":"gzip, deflate, sdch","accept-language":"en-US,en;q=0.8"},"remoteAddress":"192.168.57.51","userAgent":"192.168.57.51"},"res":{"statusCode":200,"responseTime":2,"contentLength":9},"message":"GET / 200 2ms - 9.0B"}
{"type":"response","@timestamp":"2016-04-04T23:09:01+00:00","tags":[],"pid":5,"method":"get","statusCode":200,"req":{"url":"/app/kibana","method":"get","headers":{"host":"10.100.1.213:22281","connection":"keep-alive","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.110 Safari/537.36","referer":"http://10.100.1.213:22281/","accept-encoding":"gzip, deflate, sdch","accept-language":"en-US,en;q=0.8"},"remoteAddress":"192.168.57.51","userAgent":"192.168.57.51","referer":"http://10.100.1.213:22281/"},"res":{"statusCode":200,"responseTime":4,"contentLength":9},"message":"GET /app/kibana 200 4ms - 9.0B"}
{"type":"response","@timestamp":"2016-04-04T23:09:02+00:00","tags":[],"pid":5,"method":"get","statusCode":304,"req":{"url":"/bundles/commons.style.css?v=9889","method":"get","headers":{"host":"10.100.1.213:22281","connection":"keep-alive","accept":"text/css,*/*;q=0.1","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.110 Safari/537.36","referer":"http://10.100.1.213:22281/app/kibana","accept-encoding":"gzip, deflate, sdch","accept-language":"en-US,en;q=0.8","if-none-match":"\"def64d76817ecadf60b9fd69e48a547847fb9d5e-gzip\"","if-modified-since":"Mon, 28 Mar 2016 23:29:56 GMT"},"remoteAddress":"192.168.57.51","userAgent":"192.168.57.51","referer":"http://10.100.1.213:22281/app/kibana"},"res":{"statusCode":304,"responseTime":4,"contentLength":9},"message":"GET /bundles/commons.style.css?v=9889 304 4ms - 9.0B"}
{"type":"response","@timestamp":"2016-04-04T23:09:02+00:00","tags":[],"pid":5,"method":"get","statusCode":304,"req":{"url":"/bundles/kibana.style.css?v=9889","method":"get","headers":{"host":"10.100.1.213:22281","connection":"keep-alive","accept":"text/css,*/*;q=0.1","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.110 Safari/537.36","referer":"http://10.100.1.213:22281/app/kibana","accept-encoding":"gzip, deflate, sdch","accept-language":"en-US,en;q=0.8","if-none-match":"\"94def40ee55a84529a383c1277923ea08a31e75c-gzip\"","if-modified-since":"Mon, 28 Mar 2016 23:29:56 GMT"},"remoteAddress":"192.168.57.51","userAgent":"192.168.57.51","referer":"http://10.100.1.213:22281/app/kibana"},"res":{"statusCode":304,"responseTime":5,"contentLength":9},"message":"GET /bundles/kibana.style.css?v=9889 304 5ms - 9.0B"}
{"type":"response","@timestamp":"2016-04-04T23:09:02+00:00","tags":[],"pid":5,"method":"get","statusCode":304,"req":{"url":"/bundles/commons.bundle.js?v=9889","method":"get","headers":{"host":"10.100.1.213:22281","connection":"keep-alive","accept":"*/*","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.110 Safari/537.36","referer":"http://10.100.1.213:22281/app/kibana","accept-encoding":"gzip, deflate, sdch","accept-language":"en-US,en;q=0.8","if-none-match":"\"e2a5dea356ee1a329b1a9a57379d5559b20f493e-gzip\"","if-modified-since":"Mon, 28 Mar 2016 23:29:56 GMT"},"remoteAddress":"192.168.57.51","userAgent":"192.168.57.51","referer":"http://10.100.1.213:22281/app/kibana"},"res":{"statusCode":304,"responseTime":7,"contentLength":9},"message":"GET /bundles/commons.bundle.js?v=9889 304 7ms - 9.0B"}
{"type":"response","@timestamp":"2016-04-04T23:09:02+00:00","tags":[],"pid":5,"method":"get","statusCode":304,"req":{"url":"/bundles/kibana.bundle.js?v=9889","method":"get","headers":{"host":"10.100.1.213:22281","connection":"keep-alive","accept":"*/*","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.110 Safari/537.36","referer":"http://10.100.1.213:22281/app/kibana","accept-encoding":"gzip, deflate, sdch","accept-language":"en-US,en;q=0.8","if-modified-since":"Mon, 28 Mar 2016 23:29:56 GMT"},"remoteAddress":"192.168.57.51","userAgent":"192.168.57.51","referer":"http://10.100.1.213:22281/app/kibana"},"res":{"statusCode":304,"responseTime":1,"contentLength":9},"message":"GET /bundles/kibana.bundle.js?v=9889 304 1ms - 9.0B"}
{"type":"response","@timestamp":"2016-04-04T23:09:03+00:00","tags":[],"pid":5,"method":"get","statusCode":304,"req":{"url":"/bundles/src/ui/public/images/kibana.svg","method":"get","headers":{"host":"10.100.1.213:22281","connection":"keep-alive","accept":"image/webp,image/*,*/*;q=0.8","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.110 Safari/537.36","referer":"http://10.100.1.213:22281/app/kibana","accept-encoding":"gzip, deflate, sdch","accept-language":"en-US,en;q=0.8","if-modified-since":"Mon, 28 Mar 2016 23:29:56 GMT"},"remoteAddress":"192.168.57.51","userAgent":"192.168.57.51","referer":"http://10.100.1.213:22281/app/kibana"},"res":{"statusCode":304,"responseTime":2,"contentLength":9},"message":"GET /bundles/src/ui/public/images/kibana.svg 304 2ms - 9.0B"}
{"type":"response","@timestamp":"2016-04-04T23:09:04+00:00","tags":[],"pid":5,"method":"post","statusCode":502,"req":{"url":"/elasticsearch/_mget?timeout=0&ignore_unavailable=true&preference=1459811342955","method":"post","headers":{"host":"10.100.1.213:22281","connection":"keep-alive","content-length":"62","accept":"application/json, text/plain, */*","origin":"http://10.100.1.213:22281","kbn-version":"4.5.0","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.110 Safari/537.36","content-type":"application/json;charset=UTF-8","referer":"http://10.100.1.213:22281/app/kibana","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.8"},"remoteAddress":"192.168.57.51","userAgent":"192.168.57.51","referer":"http://10.100.1.213:22281/app/kibana"},"res":{"statusCode":502,"responseTime":11,"contentLength":9},"message":"POST /elasticsearch/_mget?timeout=0&ignore_unavailable=true&preference=1459811342955 502 11ms - 9.0B"}

This automation has been pretty stable throughout the 2.x upgrades, so I thought I'd raise this in case something bigger happened here. All nodes are Dockerized using bridged networking and cloud-aws EC2 discovery.

Also, I spun up the official kibana:4.5 Docker image with "cmd": "kibana -e http://my-es:9200" with the same result, despite the green status from my-es:9200...

Does anything above stand out, by chance? Thanks in advance!

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions