I think this is happening fairly often in production, but I have finally come up with a simple reproduction where TSVB does not provide any error message.
The context for this reproduction step is that @gchaps is updating documentation for metrics:max_buckets, which is a TSVB-only setting, I was looking into how this setting actually takes effect. I was also thinking about better error handling in Lens, which is how I noticed this case.
Steps to reproduce:
- Go to TSVB and set the time range to Last 30 days
- Change the time interval to 1 minute/60 seconds, and you'll see this error and a blank chart:
-
Go to the Kibana advanced settings and increase the metrics:max_buckets limit from 2000 to 100000, which is higher than Elasticsearch allows
-
Go back to TSVB and choose "Last 30 days" with an interval of 30 seconds.
-
After issuing the query, TSVB will show a "No results" screen, but no error message. The error is completely silent.
I added some logging to the TSVB server, and Elasticsearch is providing the following response:
{
"statusCode": 503,
"errBody": {
"id": "FlNma2FBT3VuUTJXTWRQOFVtZ3otRWcbWFZCVVN4SkxTM2FaanMxamhYTHZ5dzo5ODU3",
"is_partial": true,
"is_running": false,
"start_time_in_millis": 1615319517469,
"expiration_time_in_millis": 1615319578803,
"response": {
"took": 962,
"timed_out": false,
"terminated_early": false,
"num_reduce_phases": 0,
"_shards": {
"total": 1,
"successful": 0,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 0,
"relation": "gte"
},
"max_score": null,
"hits": []
}
},
"error": {
"type": "status_exception",
"reason": "error while executing search",
"caused_by": {
"type": "search_phase_execution_exception",
"reason": "",
"phase": "fetch",
"grouped": true,
"failed_shards": [],
"caused_by": {
"type": "too_many_buckets_exception",
"reason": "Trying to create too many buckets. Must be less than or equal to: [65535] but was [86401]. This limit can be set by changing the [search.max_buckets] cluster level setting.",
"max_buckets": 65535
}
}
}
}
}
This is clearly an error response, and TSVB is silent about it.
cc @timroes @alexwizp
I think this is happening fairly often in production, but I have finally come up with a simple reproduction where TSVB does not provide any error message.
The context for this reproduction step is that @gchaps is updating documentation for
metrics:max_buckets, which is a TSVB-only setting, I was looking into how this setting actually takes effect. I was also thinking about better error handling in Lens, which is how I noticed this case.Steps to reproduce:
Go to the Kibana advanced settings and increase the
metrics:max_bucketslimit from2000to100000, which is higher than Elasticsearch allowsGo back to TSVB and choose "Last 30 days" with an interval of 30 seconds.
After issuing the query, TSVB will show a "No results" screen, but no error message. The error is completely silent.
I added some logging to the TSVB server, and Elasticsearch is providing the following response:
This is clearly an error response, and TSVB is silent about it.
cc @timroes @alexwizp