Recently there was an incident that involved mixing Replication, Sentinel, with masters having snapshots/AOF disabled. This setup was not supported by Redis, since slaves try to connect with the master in order to be their exact copy, and Redis Sentinel operated clusters require to have persistence turned on in order to be able to recover their state after restarts.

While publicly discussing the incident, Michel Martens and Jason Watkins argued that to have an explicit support for diskless Redis setups using replication could be useful.

In the past I received requests to better support pure in-memory setups with replication, however I was always extremely skeptical about this setup for a simple reason: replication used to require the ability to persist on disk for the first synchronization of slaves. If you are forced to persist, it is not really an in-memory setup, it only got the disadvantages of requiring disk I/O, and, at the same time the disadvantage of the dataset being volatile after restarts. So my argument was: let's enable persistence in replicated setups, or if you want really want just a cache, use masters with consistent hashing.

However this scenario changed because of the new feature we are introducing: diskless replication. At this point it will be possible to have a master-slaves setup where disk is not touched for synchronization with slaves. The feature was originally conceived more in order to cope with latency issues in slow disk environments like EC2, however it has more interesting consequences for Redis, since it allows to have truly diskless master-slaves setups if also persistence is disabled at all.

However for this diskless setups to be useful, the Redis server must support a new mode that allows the data set to survive to the reboot of the master. Especially in absence of network partitions, if we have N instances with one master and N-1 slaves, the dataset should survive the restart of N-1 instances.

This issue introduces the simplest feature that can provide this ability, and at the same time plays well with Sentinel. All this by providing backward compatibility, since most setups rely on the ability of slaves to automatically reconnect and resynchronize with the master each time the replication link gets disconnected.

Why replication and persistence turned off don’t play well today

While Redis is an in-memory database, replication totally rely on the fact that Redis is disk backed: first resynchronizations relied on RDB snapshots in the master side, slaves stored an RDB file on disk before loading it in order to survive restarts with the ability to load the data set, and so forth.

Basically a fundamental requirement was the ability to restore the previous state (or a state that represented a point in time state in the near past, when AOF is not used) after restarts. Masters with persistence turned off violates this requirements, however they accept slaves in order to support scaling reads with multiple nodes.

Once a master has replication turned off, a restart is semantically equivalent to a FLUSHALL operation. The server returns online with the previous configuration, but with a wiped data set.
Slaves trying to resynchronize will try to copy the master’s exact state, effectively wiping their own data set as well.

In order to support purely in-memory setups, the user should be able to rely on the fact that redundancy is obtained via multiple replicated nodes that can fail in an independent way, so that as long as N-1 nodes are restarted, the data set is able to survive.

Different point of views, different solutions

The obvious way to support this use case is to stop the automatism that allow slaves to reconnect to the master after a reboot. For example a possible approach would be for slaves to stop the replication process as soon as they discover the master was rebooted, that is, the master “runid” changed.
Let’s call this solution “conservative slaves”.

Another approach is the reverse one, to have “conservative masters”. Conservative masters don’t trust their status after a reboot, if configured to do so. After a reboot, the master remains offline, replying to requests with errors, until it is unblocked in order to return either an (empty) master, or a slave of some other master.

The two approaches have advantages and disadvantages, however conservative masters have the big advantage of playing well with Redis Sentinel.

In a master reboot event, if the Sentinel timeout is set so large that the reboot does not trigger the failure detector, the error reported by the conservative master will trigger a failure anyway. Moreover, as soon as a new master will be elected, the old master will be configured to replicate with the new one, which in turn unblock it into a normal state.

Basically a conservative master, if used together with Sentinel, guarantees that each master reboot triggers a Sentinel failover.

Feature description

The feature will be called protected restart, and can be enabled in the configuration file with the following configuration directive:

protected-restart yes

It only affects instances that are restarted configured as masters, regardless of the fact the instance is configured to persist on disk or not.

After a protected restart, the server will reply with a -PROTECTED error, waiting a SLAVEOF command in order to be properly configured as a master (using slaveof no one), or as a slave.

Slaves will not be able to replicate from a protected master, since it will reply with an error to synchronization attempts.

Masters in protected state will sill be able to reply to INFO commands in the proper way, however they will reply to PING with an error, triggering the Sentinel failure detector and most other monitoring systems checking for PONG replies to validate the instance state.

The first time a master-slave setup is created, the system administrator requires to manually unblock the master with the SLAVEOF NO ONE command.

Mixed mode

It sounds reasonalbe to also support an intermediate model where:

1. Masters have persistence disabled.
2. Slaves have persistence enabled.

Of course, when an instance is reconfigured, it switches persistence on/off automatically. In this way we can have more data safety in slaves, which would allow to use a diskless master in single datacenter setups without big concerns.

This could be obtained by extending the diskless-operations configuration option in order to support three different models:

1. diskless-operations no # Like today.
2. diskless-operations full # Masters and slaves with persistence turned off. Master safe restarts.
3. diskless-operations master # Only masters will not persist regardless of saving/AOF config. Master safe restarts.