Skip to content

Discussion: envoy internal connection #11725

@lambdai

Description

@lambdai

I was exploring dispatching h2 CONNECT to listeners of the same envoy process.
The mental topology is something like below.

curl -> H2 CONNECT -> ListenerA_HCM -> TCP -> ListenerB_TcpProxy.

Without any change listenerA_HCM -> TCP -> ListenerB_TcpProxy will be carried by a TCP connection.
It is expensive in terms of

  1. Resource usage: tcp 4 tuple socket id and consider the TIME_WAIT, ephemeral ports
  2. Latency: it take extra epoll cycle to relay the data
  3. CPU usage: annoying memory copy from and to kernel.

From my point of view, where envoy's role is the sidecar, the tunnel HCM is pluggable. The ListenerB_TcpPoxy carries the business logic. This TcpProxy should kept the minimum changes when ListenerA_HCM switching on and off. I believe the deployment is significantly different from envoy as central proxy.

To reach the minimum change at TcpProxy listener, the reasonable approach is to create a non-socket based connection while providing the connection interface: read/write so that

  1. The cluster used by tunnel HCM could create such a virtual connection,
  2. Dispatcher create both VirtualClientConnection and VirtualServerConnection
  3. TcpListener find the filter chain and create filter chain for VirtualServerConnection, optionally bypass listener filter.

I think the major challenges includes simulating the delay close, half close, watermarks, supporting other transport socket, etc. But not all are necessary at the very beginning.

This approach could be an optimization for any envoy cluster connecting to envoy itself.

Pros:

  1. Build on top of the existing abstraction: listener, cluster, host, connection.
  2. The new functionaries are extensions. The lifetime of above core components are not impacted.
  3. Almost zero overhead for anyone who doesn't use it
  4. Least control plane change.

Cons:

  1. The pipe-ish connection doesn't have the highest performance among the alternative solutions
  2. The new simulated connection is not tcp feature-complete (I doubt if we need the whole).

Alternative solutions
Cluster network filter, or http upstream filter: The dispatch of connections is achieved by X clusters cluster/route instead of X listener. Migration from traditional listener filter chain match to cluster match is non-trivial. It's a huge burden when we switching the tunnel.

CC @alyssawilk @PiotrSikora

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/httpstalestalebot believes this issue/PR has not been touched recently

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions