# Managing Network Communications in C

> As explained in the Network Communications page, eXtremeDB release 7.1.1795 and later allow the IPv6 address protocol for specifying node addresses. The following section describes issues that can be important for developers porting applications that used network communications in prior versions of eXtremeDB. The OpenSSL section describes how to enable SSL security for network communications.

> ## IPv4 vs. IPv6 issues

> In places where network addresses are represented in the form "<host>:<port>", the IPv6 addresses must be enclosed in the square brackets, for example: "[::1]:5000" or "[fe80::21e:4fff:fe9c:5398%1]:5000". This is to avoid conflicts with ':' in the IPv6 address. In places where IP addresses are represented as `uint4`, the type was changed from `uint4` to `struct mco_inetaddr_t`. This structure is used to represent IPv4 or IPv6 addresses (e.g. in instances such as `SqlServer::session_info_t::peer_addr`).

> Doing network name resolution, the eXtremeDB Network Layer relies on the function `getaddrinfo()` and its settings on the target system. (The only exception to this is when turning on the dual stack on the Windows.)

> ### API Changes from Previous Implementation

> A full list of the C API changes from previous versions to eXtremeDB release 7.1.1795 is as follows:

> - The function `uint4 mco_HA_get_ip_addr(mco_channel_h chan)` is declared deprecated (as it returns IP as `uint4`). As a replacement, the function`void mco_HA_get_inet_addr(mco_channel_h chan, mco_inetaddr_t *addr)` was added
> - The types of `SqlServer::session_info_t::peer_addr` and `SqlServer::query_info_t::peer_addr` where changed from `uint4` to `mco_inetaddr_t`
> - A new constructor for class SqlServer was added. It takes as a parameter the structure `SqlServer::OpenParameters`, which combines all existing parameters and adds new parameter `const char *netInterface` , which is the address of the network interface to listen on. For example:

> >         "0.0.0.0" (IPv4 ADDR_ANY)

> > or

> >         "[::1]" (IPv6 localhost)

> > ### Internal API changes (in mconet.h):

> > New constants `MCO_SOCK_INETV4_DOMAIN` and `MCO_SOCK_INETV6_DOMAIN` were added to the enum `MCO_SOCK_DOMAIN`. These constants allow explicitly defining a socket as only IPv4 or only IPv6. The old value `MCO_SOCK_INET_DOMAIN` (which was the default) created a socket where the domain is determined by the network address (at the moment of `connect()` or `bind()`).

> > The following structure was added to represent IP addresses (IPv4 and IPv6) :

> >         typedef struct mco_inetaddr_t_
> >         {
> >             union
> >             {
> >                 uint1 u1[16];
> >                 uint2 u2[8];
> >                 uint4 u4[4];
> >             };
> >             uint4 scope_id;
> >         } mco_inetaddr_t;

> > The following new function was added:

> >         MCO_RET mco_net_parse_address(const char *address, char *hostname,
> >                             int hostname_len,
> >                             int *port,
> >                             const char * *endptr);

> > This function allows parsing an address string in the form "<host>:<port>" (it is aware of square brackets around IPv6)

> > The function parameters are as follows:

> > > `const char *address` - the address string

> > > `char *hostname` - a buffer for the <host> part of the address

> > > `int hostname_len` - the size of the hostname buffer

> > > `int *port` - the parsed port number

> > > `const char * *endptr` - If `endptr` is not NULL, the function stores the address of the first character after the IP address in `*endptr`. (This can be used to continue parsing after loading the IP address)

> > The following new function `int mco_net_ntoa(const mco_inetaddr_t *addr, char *str)` was added to convert an `mco_inetaddr_t` structure to a string representation. The length of the buffer pointed to by `str` must be at least `MCO_NET_INETADDR_STRING_LEN.`

> > The signature of the `mco_socket_t::connect()` was changed from

> >         int (*connect)(struct mco_socket_t *s,
> >                     char const* host,
> >                     int port,
> >                     int max_attempts,
> >                     timer_unit delay);

> > to

> >         int (*connect)(struct mco_socket_t *s,
> >                     char const* host,
> >                     int port,
> >                     timer_unit timeout);

> > The signature of `mco_socket_t::get_peer_info()` was changed from

> >         int (*get_peer_info)(struct mco_socket_t* s,
> >                     uint4 *ip,
> >                     int *port);

> > to

> >         int (*get_peer_info)(struct mco_socket_t* s,
> >                     mco_peer_info_t *pinfo );

> > where structure mco_peer_info_t is defined as

> >     	typedef struct mco_peer_info_t_
> >     	{
> >     		mco_inetaddr_t ip;
> >     		int            port;
> >     		mco_size_t     rx_bytes;
> >     		mco_size_t     tx_bytes;
> >     	} mco_peer_info_t;

> > In addition to `ip` and `port` it contains two counters `rx_bytes` and `tx_bytes` that allow determining the number of bytes sent or received to the wire. (For compressed sockets these are the sizes of compressed data).

> > Compression can be combined with SSL encryption - in this case, the data is first compressed and then encrypted.

> > The "compression_level" parameter was also added to corresponding structures and API functions of all eXtremeDB modules that use `mco_socket_t` (i.e. HA, Cluster, IoT, RSQL). Note that all communicating parts must use the same value for the "compression_level" parameter.

> > > Note that if compression is not used (`compression_level` is set to 0), the Zlib library is not necessary

> > The new constant `MCO_NET_CAP_IPV6` was added. The function `mco_net_get_capabilities()` returns this flag if IPv6 is supported. The new macro `MCO_CFG_NET_SUPPORT_IPV6` in file `mcocfg.h` enables IPv6 support (enabled by default).

> ## SSL support

> The `OpenSSL` binary libraries are loaded dynamically by the eXtremeDB runtime (by the `mconet` library). To build the `mconet` library with `OpenSSL` support, the environment variable `MCO_OPENSSL_INCLUDE_PATH` should be set to `OpenSSL` include directory, for example:

>         make x64=on DEBUG=on MCO_OPENSSL_INCLUDE_PATH=/home/user/OpenSSL/include host target

> On `Windows` systems it is necessary to first install the `OpenSSL` version 1.0.2 libraries which can be downloaded from https://slproweb.com/products/Win32OpenSSL.html. Then define the environment variable with a command like the following:

>         set MCO_OPENSSL_INCLUDE_PATH=c:\OpenSSL-Win64\include

> > ### A note on mcossl for Mac OS X

> > At the time of writing, the latest stable Mac OS X release ships with OpenSSL 0.9.8, while `libmcossl` requires OpenSSL 1.1.1. Thus, OpenSSL 1.1.1 must be retrieved and built manually by the user. The `MCO_OPENSSL_INCLUDE_PATH` environment variable must be set when building the eXtremeDB libraries as described above. Moreover, the `DYLD_LIBRARY_PATH` environment variable must point to the directory containing the OpenSSL 1.1.1 libraries. Otherwise xSQL (and mcossl-enabled applications) will fail to launch.

> ### Usage

> In order to use `OpenSSL` networking in an application, the user system is required to have a local `OpenSSL` installation. The `OpenSSL` headers and the libraries must be available to the compiler.

> The application must initialize the `OpenSSL` subsystem by calling function `mco_ssl_init()` and perform cleanup calling function `mco_ssl_cleanup()`. Two other functions are available, `mco_ssl_params_init()` and `mco_ssl_load_verify_locations()`. The former initializes a `mco_ssl_params_t` structure and sets its fields to default values, while the latter allows the application to load CA (Certificate Authority) certificates from a file (using the `ca_file` parameter), or a directory (using the `ca_path` parameter). (Please refer to the OpenSSL Parameters page for details on setting the values of these parameters.)

> In order to enable OpenSSL communications, the application must pass SSL settings to the eXtremeDB subsystems. The eXtremeDB High Availability structure `mco_HA_master_params_t`, the eXtremeDB Cluster structure `mco_clnw_tcp_params_t` (and the constructor of the RemoteSqlEngine, DistributedSqlEngine and SqlServer C++ classes) have void pointer parameter `ssl_params` that must point to a properly configured `mco_ssl_params_t` structure. When this pointer is set to NULL, ordinary TCP communications are used.

> Care should be taken to avoid freeing the `mco_ssl_params_t` structure, since the eXtremeDB runtime stores a pointer to it.

> C API functions

> The eXtremeDB OpenSSL API functions are the following:

>         MCO_RET mco_ssl_init()

> Initializes the global OpenSSL context and sets up the synchronization primitives required for multi-threaded applications.

>         void mco_ssl_cleanup();

> Frees the global OpenSSL context and synchronization primitives.

>         void mco_ssl_params_init(mco_ssl_params_t *params);

> Sets the `mco_ssl_params_t` structure’s default values.

>         MCO_RET mco_ssl_load_verify_locations(const char *ca_file, const char *ca_path);

> Loads the CA verification locations (see `SSL_CTX_load_verify_locations()`). Please note that the contents of the directory specified in `ca_path` should be set up using the `c_rehash` utility (see the corresponding OpenSSL manual page for details).

> ### Quick start

> In order to enable OpenSSL support in your application, please follow these steps:

> - Link your application with library `libmconet`(`mconet.lib` on Windows);
> - Call `mco_ssl_init()`;
> - Call `mco_ssl_load_verify_locations()` to load CA certificates, peer verification may fail otherwise. Please see the OpenSSL manual page for details;
> - Allocate and initialize the `mco_ssl_params_t` structure. Please be careful not to deallocate the structure after passing it to the eXtremeDB runtime. Set at least the following parameters:
>   - `tmp_dh`: many ciphers will not work without it, see the OpenSSL manual page for details and instructions on configuring it (OpenSSL includes the `dhparam` utility that can be used);
>   - `certificate`, or `certificate_file`;
>   - `private_key,` or `private_key_file`.
>   - For eXtremeDB High Availability applications, pass a pointer to the `mco_ssl_params_t` structure in parameter `ssl_params` of `mco_HA_master_params_t`; for eXtremeDB Cluster applications, pass a pointer to the `mco_ssl_params_t` structure in parameter `ssl_params` of `mco_clnw_tcp_params_t`; and for RemoteSQL applications, pass a pointer to the `mco_ssl_params_t` structure in parameter `ssl_params` of the constructor for the C++ class RemoteSqlEngine, DistributedSqlEngine or SqlServer.

> It is extremely important to note that although the default `mco_ssl_params_t` values are chosen to provide optimal security, a careful review and deep understanding of all settings is crucial for safe and reliable communications.

> There are tutorials available on self-signed certificates generation using OpenSSL. A complicated one: https://jamielinux.com/docs/openssl-certificate-authority/create-the-root-pair.html. A simple one: http://datacenteroverlords.com/2012/03/01/creating-your-own-ssl-certificate-authority/.

> ## Debugging OpenSSL

> Debugging OpenSSL applications can be tricky. A few approaches may be used.

> - to debug the client functionality, use the OpenSSL `s_server` utility to start a server and connect to it from your OpenSSL application. Pass the `-debug`, `-msg` and `-state` parameters to this utility to get a more detailed output, and look for error messages;
> - alternatively, use the OpenSSL `s_client` utility to establish a connection to your OpenSSL application. Pass the `-debug`, `-msg` and `-state` parameters for more detailed output;
> - as a temporary measure, one may disable peer verification by setting the `verify_mode` in `mco_ssl_params_t` to `MCO_SSL_VERIFY_NONE`. If the application starts working, the certificate and/or private key are probably set incorrectly, or CA certificates are not set. Don’t forget to re-enable peer verification when done.

> ## Implementation details

> In order to avoid massive changes to the existing eXtremeDB codebase and to allow it to be built without OpenSSL, the SSL settings (`mco_ssl_params_t`) are passed to the eXtremeDB High Availability subsystems using a void pointer.

> Currently, `UDP SSL` encryption is not supported, as well as local sockets communications. Any applications data that is sent via UDP sockets (including multicast sockets) is not encrypted.

> The `mconet` library creates `TCP` sockets and establishes TCP connections. Once a TCP connection is established, an `SSL/TLS` handshake is attempted, and upon its successful completion, an SSL connection is established, which is used for further communications.

> Although it is not obvious from the manual, OpenSSL’s `SSL_read()` and `SSL_write()` functions must not be called simultaneously from different threads for the same socket. Thus, all read and write operations are guarded by a latch. Also see http://stackoverflow.com/questions/14467630/clarifying-openssl-0-9-8l-concurrency-support-can-ssl-instances-be-used-by-mul

> In the existing eXtremeDB High Availability and eXtremeDB Cluster codebase, some processes use the `select()` call to wait for a non-blocking TCP socket’s connection establishment. Obviously, this will not work with SSL connections: while a TCP connection can be established asynchronously, SSL handshake on a non-blocking socket requires a series of `SSL_read()` and `SSL_write()` calls. So, when a TCP connection is established (asynchronously), the SSL handshake is done in a blocking manner.

> OpenSSL requires a number of synchronization primitives to be set up in order for multi-threaded applications to work correctly. This is done in the `mco_ssl_init()` function, which must be called before any SSL operations are done. Then `mco_ssl_cleanup()` deallocates these primitives.