This pull-request teaches the lfs/transfer_queue how to retry an object more than once per transfer.

Previously

The transfer queue used to follow these steps when executing a transfer:

1. Receive a bunch of objects from calls to Add()
2. Start either the batchApiRoutine or individualApiRoutine based on its capabilities.
3. Attempt to transfer the object(s), and commit to a retry if it failed with a retriable error.
   1. Collect the transfers to be retried in the retryCollector
   2. Close the transfer channel to kick off all of the objects to be retried.
   3. Re-add all of the objects to be retried and re-process them once.
4. Exit.

This works fine, but limits the transfer queue to:

• Responding to one set of retries per transfer.
• Responding to one retry per unique OID in the queue.

Now

Now, the transfer queue knows how to immediately retry an object. It works like this:

1. Steps 1-4 from above.
2. If an object failed, and can be retried, enqueue a retry.
3. Collect the retry, and...
   1. Make sure that we have the budget to retry the object.
   2. Add() it to the next batch, or API channel (in legacy mode)
   3. If in batch mode, immediately flush the batch, forcing the batchApiRoutine function to receive a new batch.
4. Wait until all objects have been either transferred, or abandoned.

(The ability to flush a batch was introduced in #1528, and enables the ability to immediately retry an object in a batch.)

In order to make sure that all items are processed before exiting, the way we treat the internal waitgroup q.wait changed slightly. Previously, q.wait was incremented every time we try to preform a transfer on an item. To prevent a situation where the WaitGroup would reach zero while waiting in between failing an object and retrying it, the WaitGroup is now only incremented the first time an object begins a transfer.

The transferqueue also now keeps track of the number of retries made per OID in order to prevent infinitely retrying that transfer. Currently, the maximum number of retries per object is set to 1, in order to keep the same behavior as from before this PR which is why there are no new tests.

I've also left some line comments throughout the code to clarify some further things.

/cc @technoweenie @rubyist @sinbad for 👀 and 💭