-
Notifications
You must be signed in to change notification settings - Fork 24.5k
Description
First of all, thanks very much for all your passion and this awesome piece of software! Can't wait to see how it evolves even further (Redis-Cluster etc.).. keep up the good work :-)
Why a LIMIT option for BLPOP?
For Redis-backed queuing components (such as Resque or Sidekiq), BLPOP is an essential command for pulling new jobs. However, right now every BLPOP only delivers at most one single entry, which is a problem when jobs are small and very frequent, because pulling the jobs creates lots of overhead. One workaround is to use several connections pulling jobs, again at the cost of connection and bandwidth overhead.
My proposal thus is to have a command that allows retrieving multiple list items in a blocking way, i.e. a LIMIT option for BLPOP. This would minimize overhead and increase performance for very frequent pulls. Other queueing backends support this functionality already (e.g. Amazon SQS, where the LIMIT can be 10 at max).
An example using the resulting command specification would be:
BLPOP list1 list2 list3 0 LIMIT 10
Where up to 10 items are popped from list1, list2, or list3 respectively, waiting for the first item indefinitely in case there is none.
LIMIT option with delay parameter
One question that arises IMHO is how to deal with pulling items in batches that are added one by one. In this case, the first BLPOP would return one item, and only the next BLPOP would return subsequent items (that were added in the meantime).
This is probably not a big issue, but it could be even optimized by giving a second parameter for LIMIT that specifies a delay (in milliseconds) that is triggered when popping the first item after having waited for it. This would further maximize the throughput of very frequent pulls.
The same example including the second proposal:
BLPOP list1 list2 list3 0 LIMIT 10 5
Pops up to 10 items from the lists. If there is none, waiting for the first item plus another 5 milliseconds, popping up to 9 more items, then returning them.
Other blocking commands
The same functionality could naturally be applied to other blocking commands, such as BRPOP and BRPOPLPUSH.
Thanks very much for considering.