Whenever we introduce new data structures or new encodings, we need to adjust the RDB's version and encoding format. Internally in the server, this is not an issue. However, changes to RDB create a substantial adaption workload for external tools.
For example, some data migration tools like redis-shake, which needs to parse the RDB file during full synchronization, has to put effort into adapting to new versions whenever there are changes in the RDB (RDB version as well as many changes in storage structures). The tool needs to parse RDB to extract key-value pairs and transform them into restore commands before sending them to the target instance.
Also, there are many detailed issues to be addressed, such as the restore command's parameters not exceeding 500MB (due to proto-max-bulk-len limitation), meaning that when dealing with larger payloads, it's necessary to analyze the specific storage format for splitting, and then reverse-engineer it back into commands for replay (for example, a large hash would be converted into multiple HSET commands).
Furthermore, instances on old versions cannot parse RDB from new versions and cannot use the restore command for data migration (some special scenarios may require rolling back to a previous version by using data migration tools).
To solve the problems and to ensure that migration tools are not affected by changes in RDB format, we can add a new method for full synchronization: using the AOF file (where AOF specifically refers to a collection of commands, not using an RDB preamble). While full synchronization between master and replica still uses the RDB file, data migration tools could declare the file format they wish to use during full synchronization via the REPLCONF command. By doing so, data migration tools can simply forward commands without needing to parse RDB, allowing the full synchronization data to be directly passed through to the target instance, thus simplifying the adaptation work.
Whenever we introduce new data structures or new encodings, we need to adjust the RDB's version and encoding format. Internally in the server, this is not an issue. However, changes to RDB create a substantial adaption workload for external tools.
For example, some data migration tools like redis-shake, which needs to parse the RDB file during full synchronization, has to put effort into adapting to new versions whenever there are changes in the RDB (RDB version as well as many changes in storage structures). The tool needs to parse RDB to extract key-value pairs and transform them into restore commands before sending them to the target instance.
Also, there are many detailed issues to be addressed, such as the restore command's parameters not exceeding 500MB (due to
proto-max-bulk-lenlimitation), meaning that when dealing with larger payloads, it's necessary to analyze the specific storage format for splitting, and then reverse-engineer it back into commands for replay (for example, a large hash would be converted into multiple HSET commands).Furthermore, instances on old versions cannot parse RDB from new versions and cannot use the restore command for data migration (some special scenarios may require rolling back to a previous version by using data migration tools).
To solve the problems and to ensure that migration tools are not affected by changes in RDB format, we can add a new method for full synchronization: using the AOF file (where AOF specifically refers to a collection of commands, not using an RDB preamble). While full synchronization between master and replica still uses the RDB file, data migration tools could declare the file format they wish to use during full synchronization via the REPLCONF command. By doing so, data migration tools can simply forward commands without needing to parse RDB, allowing the full synchronization data to be directly passed through to the target instance, thus simplifying the adaptation work.