Original comment by @jaymode:

When developing the users and roles API, a decision was made to use auto-expand replicas for the .security index so that users/roles were local to every data node ("auto_expand_replicas" : "0-all").

Security also caches users for 20 minutes since the last successful authentication on each node and roles are cached until they have been modified. I feel like we are duplicating efforts here by doing this and should instead use a sane limit for auto-expand replicas. Expanding to all nodes also doesn't really provide much if users are using dedicated client nodes as the user/role still needs to be retrieved from a data node if it is not cached.

So far we have not seen negative side effects from this, but if we decide to use this index for something with a much higher volume such as sessions, then we are adding a lot of load to the cluster since we have to replicate the operation to every data node.

@jasontedor @bleskes curious on your thoughts about this