Problem

In dynamic routing environments, upstreams may move IP addresses and intermediate gateways may not gracefully terminate those TCP connections. This is particularly problematic in environments were upstreams may exist on shared infrastructure such as AWS Application Load Balancers.

Solution

We need to provide a reset deadline for these TCP connections to rotate them, even if the TCP connections are still active. Unfortunately, to the best of my knowledge, the Go standard library does not provide such dials to rotate active TCP connections after some deadline period.

We implement this logic by lazily rotating the entire HTTP Transport object after deadline intervals expire.

This will create some extra garbage but this should be nominal in most environments.