Writable replicas: yes or no? #9760
Replies: 5 comments 11 replies
-
|
Writable replicas create some seemingly-useful caching possibilities, like pushing shared cache state to replicas through replication while allowing the replicas to develop location-specific caches through writes. Fly describes this pattern: https://fly.io/blog/last-mile-redis/ I'm not sure how widely used this approach is, but it's cool! |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for sharing the use case, cool indeed and serves as the perfect serve for pinging @mrkurt :) |
Beta Was this translation helpful? Give feedback.
-
|
this is exactly how I use redis for driving my api rate counters |
Beta Was this translation helpful? Give feedback.
-
|
Please don't remove replica writes. Or, if possible, create a way to have loosely coupled replicas.
We use them for globally coordinated caches (see above URL).
Yes, we write to the leader to propagate changes everywhere, purge keys, etc. Writes to replicas will overwrite those changes. We even do things like
No, not necessarily. Replicas can have long lived, replica only data. We sometimes set TTLs on the leader and then reset them on the replicas. It's a handy way to push a possible change to all the replicas and then let them vanish if they don't get used. |
Beta Was this translation helpful? Give feedback.
-
|
@mrkurt so you use EXPIRE on the master, and PERSIST on the replica? AFAIK that's not gonna have any effect, the master will still propagate DEL when the time comes. There's currently a mechanism in redis that allows EXPIRE to be used on replicas (and the replica will delete the keys locally when the time comes), but this mechanism isn't very consistent, it can get messed up in some situations, for instance when a replica is promoted I assume you're not interested in promoting replicas, and also have issues with a command arriving from the master, failing on the replica due to the different data set (maybe you don't use any commands / combinations that can cause that). p.s. which version of redis do you use? |
Beta Was this translation helpful? Give feedback.
-
|
Well now you made me wonder if we're actually using TTLs like I thought. We may not be. 😅 We definitely set TTLs on replica writes sometimes. These replicas will never be promoted. We have devs using replica writes across a bunch of versions. I think mostly less than 1 year old. |
Beta Was this translation helpful? Give feedback.
-
|
@mrkurt thanks for your report, seems you use writable-replica to build active-active, if I'm right I don't think it's a good use case, since the change on replica cannot propagate to master. And another thing: from 4.0 we use proxy replication mechanism, in replication stream replicas work as a proxy, replica propagate only write commands come from master to sub-replica, users' write commands on writable-replica would not propagate to sub-replica. That means the writable-replica would become an isolated data island. I don't know which version you are using, and do you notice that. |
Beta Was this translation helpful? Give feedback.
-
|
No we're not doing active-active. This is what we're doing: https://fly.io/blog/last-mile-redis/ I don't think Redis-as-a-cache fits into normal database cluster semantics, really. We use replication to implement a loosely coupled caching model. We usually want the replicas to be islands, because caches across geographies need different data. We also want to propagate some changes to all caches. Redis replication + writable replicas solves that nicely. If you are trying to build an elegant replication model you're probably building something we don't use Redis for. |
Beta Was this translation helpful? Give feedback.
-
|
Hi – I use writes on replicas! The big case that I don't see addressed here yet is when consistency is not important but availability and cache performance is. Consider a distributed application that needs to migrate from one Redis instance to another. Redis is used as a flat cache where write order isn't important and a (Redis) cache miss incurs a performance penalty but no errors: With replica writes enabled, we can update the application configuration(s) to point to the new Redis instance without any additional coordination. Cache hit ratio stays high during the transition because a meaningful percentage of writes from the old Redis flow through to the new Redis where both reads and writes are happening. Eventually the application stops writing to the old Redis altogether, the new Redis is promoted, and business continues as normal before the migration event occurred. If replica writes did not exist, I think we would either have to take (1) an availability hit to coordinate the application to switch the destination of all writes at once, or (2) a drop in cache hit ratio by disconnecting replication before pointing any writes at the new Redis. So to answer @itamarhaber's questions:
|
Beta Was this translation helpful? Give feedback.
-
|
Hi @bpo thanks for your report. The case you said is very common, many users in our platform migrate data from one instance to another everyday, but we use more elegant way to solve:
So I don't think redis needs handle this case internally, using a middle ware is more reasonable and flexible. BTW, redis is not only cache. |
Beta Was this translation helpful? Give feedback.
-
|
We also have a tool for that (not open source). The solution we implemented is a feature called "pseudo-master", and "pseudo-replica". The difference is that, if you don't solve this "pseudo-master" thing, you'll end up with data corruption even if you don't have writes on the target db. |
Beta Was this translation helpful? Give feedback.
-
|
Yes, the target master has data corruption issue due to expire and eviction, but in bop's case, it's not a problem because redis performs as a cache service. About the "pseudo-master" and "pseudo-replica" feature, it's a good idea, and we had intended to implement the similar feature, but we didn't, we discussed with some users in our platform and got some conclusions:
regardless of above, "pseudo-master" and "pseudo-replica" is a good idea, and using a tool (like redis-shake or your tool) is the right way in the migrate case. |
Beta Was this translation helpful? Give feedback.
-
|
i don't think i agree with the 3 bullets you wrote, but i don't think there's a point to argue about that (specifically if you agree that the in any case, i wanted to stress here that this is essentially a case of a writable replica, and even if we add the pseudo-master, it's still a case of writable replica (just a bit less buggy). Maybe we feel a little bit better about it because unlike |
Beta Was this translation helpful? Give feedback.
-
|
this dont work if their is any module added, like redis search |
Beta Was this translation helpful? Give feedback.
-
|
Is writable replica still supported? On performing any 'STORE' operation like 'ZRANGESTORE' or 'ZUNIONSTORE' on a replica node of a Redis Cluster, I a getting MOVED exception even when 'replica-read-only' is set to 'no'. Is there any other configuration required to make a replica writable? My use case for Writable replica - Redis version - |
Beta Was this translation helpful? Give feedback.
-
|
@srishtikeshari Did you find any solution? |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
TL;DR Since time immemorial, Redis has allowed stand-alone and cluster replicas to be configured so they can accept write requests. If you have an opinion about this feature, please add your voice to this discussion.
The primary use of Redis replicas is to provide a highly-available database service via failover and promotion. Replicas can also be used to offload reads from the master. However, a client can write to a replica under certain conditions (with the proper configuration and/or connection settings).
The motivations for making replicas writable, as far as we're aware, were mainly to support these use cases:
a. Allow creating (and possibly deleting) temporary keys, such as those that
ZUNIONSTOREproducesb. Allow using commands that have the potential to write to keys, even if they don't necessarily do so, such as with using
SORTin its read-only invocation form.However, with Redis v6.2 and the upcoming v7, a slew of new
_ROcommands and complementary non-STOREvariants have been added to address just that.If that's indeed the case, namely that writable replicas are no longer really needed, the next question is whether they are needed at all. Supporting write replicas not only adds to the overall complexity of the project but also introduces several edge cases that, while rare, have damage potential during runtime. While the effort can be invested to address some/all of the issues, we're looking to assess the need and requirements from the feature.
So the questions we are looking to answer are mainly:
Thanks in advance for your participation!
Beta Was this translation helpful? Give feedback.
All reactions