Recent changes to the socket related API no longer allow overriding methods such as bind or connect. This is both a bug report and proposal for how to bring this functionality back.

The "socket API" here and below refers to a collection of methods for creating communication endpoints in use by Envoy which mirrors Posix sockets API.

Background:
Envoy provide ability to extend name resolution mechanism by registering a custom address "resolver" factory identified by a string ID. Custom resolver by specifying its ID in the envoy.config.core.v3.SocketAddress configuration proto. A resolver returns Envoy::Network::Address::Instance interface to an IP address object.

Previously socket API was part of the Envoy::Network::Address::Instance which while admittedly unorthodox allowed vendors to override default behavior of the API based on the type of resolver that created given IP address object.

Recent redesign spread socket API between several interfaces:

• Envoy::Network::SocketInterface for creating descriptor objects. This interface is implemented by an object which is a singleton in Envoy process and can be customized through a factory.
• Envoy::Network::IoHandle for return values from SocketInterface methods.
• Envoy::Network::Socket. Objects of this interface are created from the Envoy::Network::IoHandle` to provide specializations for posix socket API.

The objects that implement Envoy::Network::Socket are created internally by Envoy and their types can not be modified by vendors. Since this interface defines methods like bind, listen and connect the behavior of these methods can be overridden no longer.

However some of our use cases require completely custom implementation of binding or connecting to some IP addresses (i.e. Cloud IPs). The customization is not just limited to providing additional socket options, but requires making calls to proprietary subsystems.

This proposal outlines new class hierarchy that would allow complete control over implementation of the socket API.
Stream socket descriptors can be placed into one of the 3 categories based on the use.

1. Listening socket used by a server to accept new connections from clients.
2. Connecting socket used by a client to establish new connection to a server.
3. Data socket that is used to send a receive data between peers.

As such class hierarchy can be designed as follows:

1. Socket interface with various methods common to all socket types (i.e. get/setsockopt)
2. DataSocket : Socket for sending and receiving data.
3. ListeningSocket : Socket with methods specific to listening sockets. A callback that accepts new connections provides DataSocket to communicate with the peer.
4. ConnectingSocket : Socket with methods specific to connecting sockets. A callback is invoked with the DataSocket when new connection is established.
5. SocketInterface has methods added for creating ListeningSocket and ConnectingSocket.

NB: We could make ConnectingSocket derive from DataSocket to reduce the number of changes in the code.

The SocketInterface implementation may need to distinguish between different types of IP addresses that are provided to it at the time of socket creation so it can instantiate different implementations of the socket interfaces. Use of RTTI is not desirable as it may require pulling in type definitions for IP address classes. However providing the string ID of the resolver that created the address is enough to cover existing use cases. As such the Envoy::Network::Address::Instance is proposed to be extended with the resolverId() method returning the unique ID of the resolver.

Datagram sockets have the superset of the DataSocket API with the additional bind and connect methods (if we use connect on UDP sockets).To support datagram sockets the following changes are proposed:

1. DatagramSocket : DataSocket with added bind method.
2. SocketInterface extended with the addition of the method for creating datagram sockets.

This class hierarchy would allow vendors to provide custom implementations for all classes involved in establishing a connection:

1. Custom name resolvers through named factories.
2. Custom SocketInterface that can instantiate different implementations of the Socket interface family that would allow all its methods to be overridden.