Currently, leaderCache only updated when NotLeaderError. Why don't we update it when successfully receive resp from a range node? Because it make update too frequently? We can skip replicas.MoveToFront(i) to update it when after a short period(expired time).
And I think if we don't have expired time for leaderCache , it will be solidify a range leader for a gateway node. The random operation in SortByCommonAttributePrefix will be useless.
I also encounter a case like this: one of the three node disconnect from the network(named node C), other two node still work for a long time, then the "node C" recovered. "node C" has plenty of raft log entries to catch up, so "node C" is very busy at that time. But "node C" still in many gateway node's leaderCache(cached before node C disconnect from the network), they send lots of requests to "node C" at once when "node C" recovered. "node C" can't reply them in time because of the plenty of raft work. If we have expired time for leaderCache, the gateway node can give pressure to "node C" little by little when "node C" recovered.