If Close is called externally before a request is attempted, then we
will accidentally attempt to send to a closed channel, causing a panic.

To avoid this, we can check to see if Close has been called, using a
done channel. If this channel is ever done, we drop any incoming errors,
requests or pipes - we don't need them, since we're done.

Signed-off-by: Justin Chadwell <me@jedevc.com>

If we get io.ErrClosedPipe in pushWriter.Write, there are three possible
scenarios:

- The request has failed, we need to attempt a reset, so we can expect a
  new pipe incoming on pipeC.
- The request has failed, we don't need to attempt a reset, so we can
  expect an incoming error on errC.
- Something else externally has called Close, so we can expect the done
  channel to be closed.

This patch ensures that we block for as long as possible (while still
handling each of the above cases, so we avoid hanging), to make sure
that we properly return an appropriate error message each time.

Signed-off-by: Justin Chadwell <me@jedevc.com>

Signed-off-by: Justin Chadwell <me@jedevc.com>

If a writer continually asks to be reset then it should always succeed -
it should be the responsibility of the underlying content.Writer to
stop producing ErrReset after some amount of time and to instead return
the underlying issue - which pushWriter already does today, using the
doWithRetries function.

doWithRetries already has a separate cap for retries of 6 requests (5
retries after the original failure), and it seems like this would be
previously overridden by content.Copy's max number of 5 attempts, hiding
the original error.

Signed-off-by: Justin Chadwell <me@jedevc.com>

If sending two messages from goroutine X:

	a <- 1
	b <- 2

And receiving them in goroutine Y:

	select {
	case <- a:
	case <- b:
	}

Either branch of the select can trigger first - so when we call
.setError and .Close next to each other, we don't know whether the done
channel will close first or the error channel will receive first - so
sometimes, we get an incorrect error message.

We resolve this by not sending both signals - instead, we can have
.setError *imply* .Close, by having the pushWriter call .Close on
itself, after receiving an error.

Signed-off-by: Justin Chadwell <me@jedevc.com>

We also need an additional check to avoid setting both the error and
response which can create a race where they can arrive in the receiving
thread in either order.

If we hit an error, we don't need to send the response.

> There is a condition where the registry (unexpectedly, not to spec)
> returns 201 or 204 on the put before the body is fully written. I would
> expect that the http library would issue close and could fall into a
> deadlock here. We could just read respC and call setResponse. In that
> case ErrClosedPipe would get returned and Commit shouldn't be called
> anyway.

Signed-off-by: Justin Chadwell <me@jedevc.com>