If an index is created and not all shards are allocated (i.e. go to state STARTED) before full restart of cluster, the restarted cluster tries to recover the index using Gateway but cannot find any metadata for it and shards remain unassigned.

The same happens for snapshot restore. If some shards don't finish recovery before full restart, they remain unassigned after restart. Even worse, if there are shards of a closed index that were there when we started the restore process, these are used to recover.

The same also happens when a shard that is initializing due to index creation fails (e.g. due to shard store being read-only).

To reproduce for index creation case:

• Set cluster.routing.allocation.enable to none as transient setting.
• create new index.
• restart cluster

The reason for this behavior is that the shard routing table, which contains creation context (created as new index or restore from snapshot), is not persisted with the index metadata.

Relates to #14739