-
Notifications
You must be signed in to change notification settings - Fork 24.4k
expire command may cause data inconsistent. #5171
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
expire command may cause data inconsistent. #5171
Conversation
|
I added a test, it passed without your PR. |
|
Forgive me, my previous tcl test was wrong. |
|
Hi @trevor211 thanks~~ |
|
Hi @kingpeterpaule , i think you should also consider |
|
Hello and thanks for submitting this. I don't think we should fix this issue. I ping @oranagra and @soloestoy to see other POVs, but IMHO it is more important than two instances, a master and a slave, with a very different absolute clock value, are still able from the external to have more or less the same TTL visibility of a key, compared to fixing this "key reappearing" problem, that is btw not a consistency problem per se (because master and slave will have identical data here, there is no technical desync). To better focus how this patch impacts Redis is a much more serious way that it attempts to fix problems, see this: Master A clock is X. I send to the master I'm using Slave B to scale reads. However This is a far bigger problem than the fact that a key that expired in the master, in case of a replication link delay and a failover in the middle, may re-appear. Note that without failover, there is no actual problem, because in the replication stream there will be immediately a Taking this open for further discussion but IMHO there is nothing to fix... |
|
Hi, Maybe a more serious problem here, or another view of the same / similar problem, is what happens when an AOF is loaded. one way to look at it, may be to imagine that we respect the user's intent, so if he used EXPIRE and not EXPIREAT, the key should live for another 24 hours? p.s. i imagine that if we fix this, we may need to fix other commands (like RESTORE) |
|
@soloestoy i think i remember that change, but for some reason i can't find it in the code.. can you give me a hint? |
|
Hi @oranagra , it's function if (cmd->proc == expireCommand || cmd->proc == pexpireCommand ||
cmd->proc == expireatCommand) {
/* Translate EXPIRE/PEXPIRE/EXPIREAT into PEXPIREAT */
buf = catAppendOnlyExpireAtCommand(buf,cmd,argv[1],argv[2]); |
|
@oranagra I think it makes sense to rewrite EXPIRE as EXPIREAT when propagating to AOF but do not rewrite when propagating to slaves. When propagating to AOF, we are sure using the same clock value, but when propagating to slaves, we do not have that guarantee. |
|
@oranagra AOF rewrites EXPIRE as EXPIREAT since the very start, actually EXPIREAT exists because of that. However I think like @trevor211 that while it is sane to do that for AOF, because we want the time to logically pass when the server is restarted, on the contrary on the slave we want them to honor the same high level contract of expires that the master has, that is, we want the time to be counted logically starting from the moment the write was received. This usually provides the most coherent behavior from the point of view of the external users, with TTLs that are similar from the POV of the external observer. Of course the issue that @kingpeterpaule points in this issue is true, but it is a corner case, not the normal operations. And such corner case is not even particularly unexpected or surprising IMHO, because in the extreme case of a failover immediately after the stream was received what happens is not an inconsistency but just a jump back in time that is proportional to the replication delay that there is between the master and slave. Now in that case of a big delay in the replication link, jump back in time is always possible after a failover. So after thinking at the different arguments, I still think it's not the case to fix this issue. Thank you a lot for informing this discussion! |
|
Closing this due to Salvatore's post above (disadvantages outweigh the advantages) |
…#8357) This commit adds tests to make sure that relative and absolute expire commands are propagated as is to replicas and stop any future attempt to change that without a proper discussion. see #8327 and #5171 Additionally it slightly improve the AOF test that tests the opposite (always propagating absolute times), by covering more commands, and shaving 2 seconds from the test time.
…redis#8357) This commit adds tests to make sure that relative and absolute expire commands are propagated as is to replicas and stop any future attempt to change that without a proper discussion. see redis#8327 and redis#5171 Additionally it slightly improve the AOF test that tests the opposite (always propagating absolute times), by covering more commands, and shaving 2 seconds from the test time.
Consider following commands order:
Usually the master and slave have different expire time point and the differ should be very small. But If the expire command is not replicated to slave and some network problem happens or slave down.
When slave reboot, it send psync2 to master, and psync is accepted. In this scenario, the differ may be large and slave have newer expire time point. More serious, if failover happens, the slave becomes new master, a user may find that a already expired key which is in old master that could visit in new master.
Data inconsistent happens.