My team is building a service. Each client of the service is reading from its own Redis stream. Each stream gets new entries appended after a random delay of 1sec-1min. We want clients to get the new data as soon as possible, and we want to serve about 100k-1M clients at the same time.

It's awesome that XREAD can read from multiple streams, this can greatly improve resource utilization: less empty reads, connections blocked for less time. Unfortunately this doesn't quite work in Redis Cluster due to the requirement that all the keys must hash to the same slot.

With our target numbers I'm not sure we can afford not to read multiple streams with single connection. We are currently considering the following hack, where we would call a lua script passing multiple stream keys (that we know are currently on the same node) as arguments:

local out = {}
for i = 2, #ARGV, 2 do
        local xs = redis.call('XRANGE', ARGV[i], ARGV[i + 1], '+', 'COUNT', ARGV[1])
        out[#out + 1] = {ARGV[i], xs}
end
return out

however, the script has quite a bit of overhead compared to XREAD...

I know Redis Cluster multikey issue has been discussed multiple times before (e.g. https://github.com/antirez/redis/issues/5061, https://github.com/antirez/redis/issues/5118):

this would be a bomb - you'd have code that works today because of an accident of how the slots are distributed between shards but that can break when an admin very legitimately migrates some of the slots to another shard.

but how about adding an option to disable this check for clients that understand the risk?

Maybe a new config option enable-local-cross-slot (similar to enable-cross-slot option in Redis Cluster Proxy project)?