-
Notifications
You must be signed in to change notification settings - Fork 5.3k
Discussion: envoy internal connection #11725
Description
I was exploring dispatching h2 CONNECT to listeners of the same envoy process.
The mental topology is something like below.
curl -> H2 CONNECT -> ListenerA_HCM -> TCP -> ListenerB_TcpProxy.
Without any change listenerA_HCM -> TCP -> ListenerB_TcpProxy will be carried by a TCP connection.
It is expensive in terms of
- Resource usage: tcp 4 tuple socket id and consider the TIME_WAIT, ephemeral ports
- Latency: it take extra epoll cycle to relay the data
- CPU usage: annoying memory copy from and to kernel.
From my point of view, where envoy's role is the sidecar, the tunnel HCM is pluggable. The ListenerB_TcpPoxy carries the business logic. This TcpProxy should kept the minimum changes when ListenerA_HCM switching on and off. I believe the deployment is significantly different from envoy as central proxy.
To reach the minimum change at TcpProxy listener, the reasonable approach is to create a non-socket based connection while providing the connection interface: read/write so that
- The cluster used by tunnel HCM could create such a virtual connection,
- Dispatcher create both
VirtualClientConnectionandVirtualServerConnection - TcpListener find the filter chain and create filter chain for
VirtualServerConnection, optionally bypass listener filter.
I think the major challenges includes simulating the delay close, half close, watermarks, supporting other transport socket, etc. But not all are necessary at the very beginning.
This approach could be an optimization for any envoy cluster connecting to envoy itself.
Pros:
- Build on top of the existing abstraction: listener, cluster, host, connection.
- The new functionaries are extensions. The lifetime of above core components are not impacted.
- Almost zero overhead for anyone who doesn't use it
- Least control plane change.
Cons:
- The pipe-ish connection doesn't have the highest performance among the alternative solutions
- The new simulated connection is not tcp feature-complete (I doubt if we need the whole).
Alternative solutions
Cluster network filter, or http upstream filter: The dispatch of connections is achieved by X clusters cluster/route instead of X listener. Migration from traditional listener filter chain match to cluster match is non-trivial. It's a huge burden when we switching the tunnel.