/wait

Hey Snow, I'm throwing this your way as Matt's out.
I am hoping to land this PR this week because it has some test infra I want to use elsewhere, but given IMO the lifetime change is pretty high risk and we're not sure if we're cutting the release in under 2 weeks, I'm inclined to get an LGTM (making sure all changes pass on all CI with the flag on) then flip the default false right before landing with a TODO to flip true after the release.

Ah, so when TCP does writes, it puts data in the connection buffer, waits for an event, and then writes on the next epoll loop. for UDP we just write directly to the socket. If the write fails, we close the connection under the stack of the write, and here we delete the upstream request while doing work in the upstream request, resulting in a use-after-free. This adds a (really synthetic) test of writing fail due to an injected error, and fixes up the code so we don't have a use-after-free.
Some day I'd like to fix TCP to do inline writes because from a perf perspective it's pretty helpful (cc @wbpcode) but before we think of doing that we need to actually handle inline writes correctly which this PR does (and regression tests)