The healthcheck gets into an unrecoverable state when a saved object is written to the kibana index before it exists.
To reproduce this:
- Startup kibana and es
- Open discover and prepare to save a new saved search
- In a new terminal execute
sleep 10; curl -XDELETE http://localhost:9200/.kibana
- before the 10 second timer is done, get in position to save the search
- as soon as the delete goes through, save the search and observe the kibana server begin attempting to add the "visualization" type to the mapping every 2.5 second.
We've discussed this with the elasticsearch team as a part of elastic/elasticsearch#23685 and the way to fix it is using index templates. Rather than checking for the kibana index in the healthcheck we should probably:
- Check for the kibana index template
- create it if it doesn't exist
- patch it for missing types/properties (in case of an upgrade, or new plugin install)
- Check for the config doc
- create it if it doesn't exist
- will autocreate the kibana index if it also didn't exist
- Read the kibana index mapping
- ensure that defined properties match their expected settings
- add any missing properties (in case of an upgrade, or new plugin install)
The healthcheck gets into an unrecoverable state when a saved object is written to the kibana index before it exists.
To reproduce this:
sleep 10; curl -XDELETE http://localhost:9200/.kibanaWe've discussed this with the elasticsearch team as a part of elastic/elasticsearch#23685 and the way to fix it is using index templates. Rather than checking for the kibana index in the healthcheck we should probably: