Skip to content

BLPOP with LIMIT option #936

@arnodirlam

Description

@arnodirlam

First of all, thanks very much for all your passion and this awesome piece of software! Can't wait to see how it evolves even further (Redis-Cluster etc.).. keep up the good work :-)

Why a LIMIT option for BLPOP?

For Redis-backed queuing components (such as Resque or Sidekiq), BLPOP is an essential command for pulling new jobs. However, right now every BLPOP only delivers at most one single entry, which is a problem when jobs are small and very frequent, because pulling the jobs creates lots of overhead. One workaround is to use several connections pulling jobs, again at the cost of connection and bandwidth overhead.

My proposal thus is to have a command that allows retrieving multiple list items in a blocking way, i.e. a LIMIT option for BLPOP. This would minimize overhead and increase performance for very frequent pulls. Other queueing backends support this functionality already (e.g. Amazon SQS, where the LIMIT can be 10 at max).

An example using the resulting command specification would be:
BLPOP list1 list2 list3 0 LIMIT 10

Where up to 10 items are popped from list1, list2, or list3 respectively, waiting for the first item indefinitely in case there is none.

LIMIT option with delay parameter

One question that arises IMHO is how to deal with pulling items in batches that are added one by one. In this case, the first BLPOP would return one item, and only the next BLPOP would return subsequent items (that were added in the meantime).

This is probably not a big issue, but it could be even optimized by giving a second parameter for LIMIT that specifies a delay (in milliseconds) that is triggered when popping the first item after having waited for it. This would further maximize the throughput of very frequent pulls.

The same example including the second proposal:
BLPOP list1 list2 list3 0 LIMIT 10 5

Pops up to 10 items from the lists. If there is none, waiting for the first item plus another 5 milliseconds, popping up to 9 more items, then returning them.

Other blocking commands

The same functionality could naturally be applied to other blocking commands, such as BRPOP and BRPOPLPUSH.

Thanks very much for considering.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions