We discussed this a long time ago: ZeroMQ sockets are not thread-safe, so a lot of threads end up creating their own client objects that create connections to other nodes if communication is required. This causes a lot of unnecessary TCP connections to get created, leading to running over default socket limits, it can lead to worse slow start behaviour on those sockets and other similar scaling issues. NNG is a library made by the original authors of 0mq after learning from its mistakes and nanomsg's mistakes.

Crucially, nng sockets are thread-safe and can be used from multiple threads. Even patterns such as req-rep can be done on the same socket from multiple threads by using nng_context objects (instead of the req-router-dealer-rep 0mq pattern).

One big difference between nng and 0mq sockets is that send operations will block until a receiving endpoint is connected, rather than going blindly into a queue that might never connect to an endpoint. This broke a few tests which were connecting to non-existent endpoints, so they needed fixing. Some tests also created the sending endpoints and sent messages before creating the corresponding receivers which led to deadlocks. Passing the NNG_FLAG_NONBLOCK flag wouldn't solve this issue as in that case the send function returns an error EAGAIN, so some parts of the code (mostly tests) need to be changed to make sure receivers become available at some point during send calls.

To-do:

• Replace 0MQ with NNG in CMake
• Switch the message/endpoint classes to wrap NNG sockets&contexts
• Find instances of thread-local sockets and replace them with global sockets
• Add a thread-safe concurrent map wrapper to replace manual handling of mutexes+maps, because quite a few of the thread-local maps and also some global maps use the same patterns over and over making it prone to mistakes with forgotten locks.
• Replace map+mutex usage with the concurrent map type where appropriate

Closes #266