Truthfully I'm not sure if I should be filing an issue here or over at split-array-stream, but I figured I would start here and see what everyone else thinks.

I recently started work on a BigQuery feature (googleapis/nodejs-bigquery#484) that was supposed to be a big performance boost for dealing with large result sets. However what I ended up finding was that it was significantly slower, which threw me off quite a bit. Debugging lead me to the paginator, it would appear in my BQ refactor one of the side effects was that the individual page size of a request went from about 18k results to 60k results. Interestingly enough, it only took about 40~50ms for it to process 18k results, however it takes a staggering 3-5 seconds to process 60k results.

My guess is that using setImmediate slows down the process pretty drastically. I tested a local patch where instead of using the split stream, I used a PassThrough that I would write to using a for loop and in the event that write() returned false, I would wait for the drain event before I continued the loop. This allowed me to process 60k results in about 40ms. This leads me to think that we should not be using setImmediate.