Skip to content

async.queue performance suffers greatly with large queues #756

@bbrowning

Description

@bbrowning

While performance testing a library that uses async.queue, I noticed that the performance of async.queue degrades massively as the number of items in the queue increases. After a certain point, V8 spends so much time shuffling memory around that the performance of everything else slows to a crawl.

The culprit appears to be how the current implementation calls shift on the tasks array every time an item is processed. I prototyped an alternate implementation that maintains an index into the array and performance is vastly improved with larger queue sizes. For the purposes of this testing, the entire async.queue implementation was copied and modified to form a new async.indexedQueue function so side-by-side comparison could be easily done.

https://gist.github.com/bbrowning/b7355b731eb025608228

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions