Deprecation log datastream is missing settings after upgrade #80265
Description
The problem probably affects docker distribution only.
deprecation log data stream is created in 7.16 with following index template
deprecation-indexing-template.json
{
"index_patterns": [".logs-deprecation.*"],
"priority": 1000,
"data_stream": {
"hidden": true
},
"composed_of": [
".deprecation-indexing-mappings",
".deprecation-indexing-settings"
],
"allow_auto_create": true,
"_meta": {
"description": "default template for Stack deprecation logs index template installed by x-pack",
"managed": true
},
"version": ${xpack.deprecation.indexing.template.version}
}
deprecation-indexing-settings.json
{
"template": {
"settings": {
"index": {
"hidden" : true,
"auto_expand_replicas" : "0-1",
"lifecycle": {
"name": ".deprecation-indexing-ilm-policy"
},
"codec": "best_compression",
"query": {
"default_field": ["message"]
}
}
}
},
"_meta": {
"description": "default settings for Stack deprecation logs index template installed by x-pack",
"managed": true
},
"version": ${xpack.deprecation.indexing.template.version}
}
when a 7.16 instance is freshly started deprecation log datastream results with correct settings
/.logs-deprecation.elasticsearch-default/_settings
{
".ds-.logs-deprecation.elasticsearch-default-2021.11.03-000001": {
"settings": {
"index": {
"lifecycle": {
"name": ".deprecation-indexing-ilm-policy"
},
"codec": "best_compression",
"routing": {
"allocation": {
"include": {
"_tier_preference": "data_hot"
}
}
},
"hidden": "true",
"number_of_shards": "1",
"auto_expand_replicas": "0-1",
"provided_name": ".ds-.logs-deprecation.elasticsearch-default-2021.11.03-000001",
"query": {
"default_field": [
"message"
]
},
"creation_date": "1635937224704",
"number_of_replicas": "0",
"uuid": "PnjN0oe6Ro6gQYNGAFcVrg",
"version": {
"created": "7160099"
}
}
}
}
}
However if a 6.8 is created first, then the same data path is used to start 7.16 (upgrade process) then the deprecation log data stream (is is being created in 7.16) has incorrect settings (just default)
/.logs-deprecation.elasticsearch-default/_settings
{
".logs-deprecation.elasticsearch-default": {
"settings": {
"index": {
"routing": {
"allocation": {
"include": {
"_tier_preference": "data_content"
}
}
},
"number_of_shards": "1",
"provided_name": ".logs-deprecation.elasticsearch-default",
"creation_date": "1635936542141",
"number_of_replicas": "1",
"uuid": "plhubVeTRGuQanEOUbQTog",
"version": {
"created": "7160099"
}
}
}
}
}
STEPS TO REPRODUCE
docker pull docker.elastic.co/elasticsearch/elasticsearch:6.8.17
docker pull docker.elastic.co/elasticsearch/elasticsearch:7.16.0-SNAPSHOT
mkdir local_data docker run -v local_data:/usr/share/elasticsearch/data -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:6.8.17
docker run -v /Users/przemek/scratch/upgrade_6.8_7.16_docker/local_data:/usr/share/elasticsearch/data -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:7.16.0-SNAPSHOT
confirm settings with
curl --request GET \
--url http://localhost:9200/.logs-deprecation.elasticsearch-default/_settings
for tar distribution this works fine
start 6.8 from tar
copy data dir to 7.16
start 7.16 and observer deprecation log with correct settings
cc: @david-kow @pebrc