What is Redis AOF and How to Use It

Redis Append Only File (AOF) is one of the persistence mechanisms in Redis used to log all write operations received by the Redis server. This log of operations is then used to rebuild the dataset when Redis restarts, preventing data loss.

How Redis AOF Works

By default, AOF is disabled in Redis. To enable it, the appendonly directive in the Redis configuration file needs to be set to yes:

appendonly yes

With AOF enabled, Redis will log every write command to a file in the Redis directory. The default name for this file is appendonly.aof.

When Redis restarts, it will replay the AOF file from beginning to end, executing all the write commands to rebuild the original dataset. This is very similar to a database transaction log.

Some key things to note about Redis AOF:

• AOF only logs write operations, not read operations. Gets, keys, etc. are not recorded.
• Commands are logged in a sequential order to prevent data inconsistency.
• AOF file can grow very large over time as more commands get logged. Redis provides rewrite/compaction mechanisms to deal with this.

AOF Persistence Models

There are three persistence models that control how Redis handles AOF file writes:

1. Always (default)

In this model, Redis logs every write command to the AOF file before returning success to the client. This provides the highest durability guarantees since every returned write is already on disk.

Downside is potential performance impact for latency-sensitive workloads, as every write has to wait for the disk I/O operation to complete.

Configured using:

appendfsync always

2. Everysec

Redis will log commands to AOF every second, rather than waiting for every command to log synchronously. Provides a nice balance between performance and durability guarantees.

Some writes could still be lost in event of crash in between writes. Risk is low over 1 second intervals.

Configured using:

appendfsync everysec

3. No

Redis will lazy write when OS flushes disk buffers. No waiting or syncing from Redis.

Potential for large data loss in event of crash/power loss. Persistence guarantees are weak.

Configured using:

appendfsync no

For most use cases, the default of always is recommended for durability. For caches the risks may be less so everysec can be considered.

AOF File Rewriting

Over time the AOF file keeps growing as more write commands get logged. To tackle this, Redis provides a file rewrite/compaction mechanism.

Redis will periodically spawn a child process that reads the contents of the current AOF file and creates a new compacted AOF file with redundancy removed.

Once done, the original AOF file is replaced with this compacted file. This allows Redis to maintain a small disk footprint for the AOF file.

Some ways to configure AOF rewriting:

# How many changes since last rewrite before auto trigger
auto-aof-rewrite-min-size 64mb

# Max size AOF file can grow to before triggering rewrite
auto-aof-rewrite-percentage 100 

# Sets append only file name 
appendfilename appendonly.aof

The rewrite thresholds can be tuned based on storage and workload requirements.

Manual rewrite of the AOF file can be triggered using:

BGREWRITEAOF

This can be useful before restarting Redis to reduce startup time.

Checking AOF File Consistency

The AOF file being logs of all write ops, it‘s important for it to remain consistent for Redis to function correctly on restarts.

The redis-check-aof tool can be used to scan AOF file proactively for errors:

$ redis-check-aof --fix appendonly.aof

This performs a test pass of loading the AOF file contents back into a Redis instance to check if dataset can load without errors. Any errors are logged.

The --fix attempts to correct certain errors if possible. Serious errors may require AOF file reset.

Resetting the AOF File

In event of non-fixable corruption or consistency issues with the Redis AOF file, resetting it may be required.

1. Stop Redis Server

Stop the Redis instance safely using:

$ redis-cli shutdown

2. Delete existing AOF file

Remove the old appendonly.aof file from the Redis data directory.

3. Start Redis server

Restart Redis server process.

This will trigger Redis to initialize a new empty AOF file, allowing for a clean start.

Existing data in memory will still be available after restart. But a full shutdown/restart will lead to complete flush.

Securing the AOF File

Since AOF file contains full command logs, it offers visibility into everything written to the datasets. Any sensitive data gets leaked.

It‘s important to take security precautions:

• Enable authentication so only authorized clients can access Redis
• Encrypt Redis network traffic if untrusted networks are involved
• Use filesystem encryption to protect the AOF file at rest
• Restrict filesystem permissions to appendonly.aof as much as feasible

Monitoring access logs around the AOF file location can also help detect foul play if any.

Advantages of Redis AOF

Here are some benefits provided by the AOF persistence model:

Total Command Log

• Complete sequential history of every write allowing for better audits and debugging of production issues. Changes can be deterministically rolled back or replayed if found to be problematic.

Faster Restarts

• Clean shutdowns don‘t require loading dataset back from disk on restart. Server is immediately ready to accept connections back.

Better Data Integrity

• Commands are serialized in the AOF leading to guaranteed consistency during restarts or crashes. Partial/failed writes cannot happen.

More Data Available After Crashes

• AOF allows partial recovery of data from up till the last successfully synced write commands.Better than losing complete dataset.

Easier Backup & Restore

• AOF log files better lend themselves to taking backups – standard tools can take continuous incremental backups.Restore just means replaying the AOF logs.

Append Only Model

• New commands get efficiently appended to end of file. Random file access/locking avoided. Scales better than updating in place.

Disadvantages of Redis AOF

The main drawbacks to balance when using an AOF based approach:

Slower Performance

• Syncing every operation leads to measurable impact on latency depending on persistence model chosen. Memory usage also increases with buffered writes.

Storage Overheads

• An infinitely growing AOF file constantly receiving logs of all write traffic implies large storage needs over time. requires periodic rewriting/compaction.

Security Concerns

• Having cleartext commanding logs exposes what kind of data is being written into Redis. Additional encryption care needs applied.

Restart Times Can Vary

• The recovery process replaying a large AOF file on restart can take significantly variable times. All clients wait on server availability.

So while AOF does provide strong persistence guarantees, the capacity planning implications of storage, memory and variable come back up times need accounted.

Conclusion

Redis AOF provides excellent protection against catastrophic data loss by being a continuous append-only log of all writes against Redis. It allows durable dataset persistence crucial for many databases and data systems.

With the different sync configurations, flexibility is offered between persistence guarantees versus performance. Tuned well, AOF can give necessary disaster recovery abilities without too many downsides for applications.