Goals

Support pausing of in progress requests, which we can resume without needing rerequest and duplicate data.

Implementation

• First, add the Traverser class in ipldutil, which allows us to step through a selector traversal block by block, rather than simply get a callback whenever we load a block
• Second, build a runner for the Traverser that exits either when the traverse completes or when it is told to pause
• Add a hook for each outgoing block which gives you the options to pause, based on the link, size, and size on wire of the last block sent
• store the traverser with each request (along with the loader we're using and whether the request is paused) and maintain request data for the case that the request is paused rather than completed.
• upon unpause, put the request data back in the peer request but with a higher priority
• for resumed requests, rather than process hooks and prepare request data, just resume the traversal by calling the runner again
• Having added block hooks, as well as request hooks and persistence options to the responsemanager, this class is getting large and overwhelmed with hook processing, so factor each of these three concepts to its own class

For discussion

I believe the traverser concept can simplify the requesting side as well but have not dug into it yet.
Also, there are enough architectural shifts here we may want to update the archtecture doc.