Enhancement

In the cloud environment, the tidb clusters are commonly deployed across multiple "Access Zone"(AZ for short)s. Because the traffic cost across different AZ is much higher than within the same AZ. We can reduce the cost by dispatching read request to
the TiKV/TiFlash with a higher priority.

#28033 introduced a new tidb_replica_read option "closest-replica" that always dispatch all read request to the closest TiKV. But since the "follower read" feature need to propose an extra "readIndex" message through the raft state machine. This overhead may cause performance regression. For cop requests with small response, read from local AZ can not save the traffic because the extra "readIndex" messages are always cross AZ traffic. And the extra cpu usage to process these raft message can hurt the performance.

I propose to add a new option "closest-adaptive" for tidb_replica_read variable. This new option tries to dispatch request to the nearest store only when the response size is bigger than a certain threshold which is define by a new variable "tidb_adaptive_closest_read_threshold".

BTW, when the cop traffic from tidbs in different AZ are not evenly distributed, it may cause high load in some store while other are low loaded. This can cause high tail-lantency and low overall throughput. So when follow-read is enabled, the tikv client may need to track the load of each storage and force fallback to leader read when some stores' load are high.