Title: Proxy protocol header intermittently uses wrong addresses

Description:
Proxy protocol injection (envoy.transport_sockets.upstream_proxy_protocol) works as expected to start out, but once you start connecting from different IPs, the injected proxy protocol header ends up randomly having addresses from previous connections.
The log shows the right IPs but proxy protocol header intermittently sends the IPs not from the current connection, but from one of the previous ones.
I thought maybe setting connection_pool_per_downstream_connection: true would help but it did not.

Repro steps:
Have envoy listen on 127.0.0.1:9000 and tcp proxy to a cluster that listens for connections, reads the proxy protocol header, and then promptly closes the connection.

# run this about 20 times:
curl --interface 127.0.0.2 127.0.0.1:9000
# then run this about 20 times
curl --interface 127.0.0.3 127.0.0.1:9000
# then run this about 20 times
curl --interface 127.0.0.4 127.0.0.1:9000

The expectation here and what the logs show is 20 connections each from the 3 separate addresses above. What the proxy protocol sends, ends up being a random mix.

Suspected cause:
I'm thinking this has something to do with connection pooling as the following comment suggests that it's not exactly a 1 to 1 setup currently: #4684 (comment)

Potentially related: #2800 #11694