We have been evaluating the STREAMS branch of redis, and are very pleased with its behaviour. It fulfils a really important pattern for our workflow:
upstream data source emits events into a redis stream
production consumer uses XREAD to pull new data immediately into the live system. The live system's database atomically updates the last read messageID along with assimilating the new data, in a transaction.
non-production (e.g. development) environments can be spun up using a backup image of the production system. By using XREAD, whatever messages were queued since the backup was generated are replayed in the same order, quickly bringing the new environment into alignment with our production system, without any additional load on the upstream data source.
The only minor issue is that we would prefer to truncate the streams based on maximum message age, rather than by using MAXLEN. Some days may have many more messages than other days, and what really matters is that we are guaranteed the messages go back to prior to the last backup image.
Ideally for us, we could specify the oldest message we wish to keep. Could be by relative or absolute time, it doesn't really matter to me. Whatever is easiest or most logical to implement, so for example:
XADD MAXAGE ~ 8640000 mystream foo bar
or
XADD MINVAL ~ 1510704127488 mystream foo bar
should both remove the initial entries in the stream, up until the designated timepoint.
I imagine it would be important to support the use of MAXAGE and MAXLEN simultaneously. In my mind this should remove the maximum number of entries of the individual options.
We have been evaluating the STREAMS branch of redis, and are very pleased with its behaviour. It fulfils a really important pattern for our workflow:
The only minor issue is that we would prefer to truncate the streams based on maximum message age, rather than by using MAXLEN. Some days may have many more messages than other days, and what really matters is that we are guaranteed the messages go back to prior to the last backup image.
Ideally for us, we could specify the oldest message we wish to keep. Could be by relative or absolute time, it doesn't really matter to me. Whatever is easiest or most logical to implement, so for example:
XADD MAXAGE ~ 8640000 mystream foo baror
XADD MINVAL ~ 1510704127488 mystream foo barshould both remove the initial entries in the stream, up until the designated timepoint.
I imagine it would be important to support the use of MAXAGE and MAXLEN simultaneously. In my mind this should remove the maximum number of entries of the individual options.