Currently when executing an action (e.g., bulk, delete, or indexing operations) on all shards, if an exception occurs while executing the action on a replica shard we send a shard failure message to the master. However, we do not wait for the master to acknowledge this message and do not handle failures in sending this message to the master. This is problematic because it means that we will acknowledge the action and this can result in losing writes. For example, in a situation where a primary is isolated from the master and its replicas, the following sequence of events can occur:

1. we write to the local primary
2. we fail to write to the replicas
3. we fail in notifying the master to fail the replicas
4. the primary acknowledges the write to the client
5. the master notices the primary is gone and promotes one of the replicas to be primary

In this case, the replica will not have the write that was acknowledged to the client and this amounts to data loss.

Instead, if we waited on the master to acknowledge the shard failures we would never have acknowledged the write to the client in this case.

• Create listener mechanism for executing callbacks when exceptions occur sending a shard failure message to the master Add listener mechanism for failures to send shard failed #14295
• Add unit tests that show we wait until failure or success (do not have to handle the failures yet) Add timeout mechanism for sending shard failures #14707
• Add general support for cluster state batch updates Split cluster state update tasks into roles #14899
• Apply cluster state batch updates to shard failures Use general cluster state batching mechanism for shard failures #15016
• Handle when the node we thought was the master is no longer the master (e.g., master might have stepped down) -> find the actual master (e.g., wait for a new master to be elected) and retry the failed shard notice Wait for new master when failing shard #15748
• Fail shard failure requests from illegal sources Illegal shard failure requests #16275
• Master tells us we are no longer the primary -> fail the local shard, retry request on new primary Fail demoted primary shards and retry request #16415
• Handle failed shard has already been removed from the routing table -> okay Shard failure requests for non-existent shards #16089
• Handle master side of shard failures (do not respond to the node until the new cluster state is published, otherwise report failure or allow the node to timeout) Master should wait on cluster state publication when failing a shard #15468