.NET HttpClient extensibility: Support for UnixDomain Sockets & Named Pipes

.NET HttpClient is main interface for sending HTTP requests and receiving HTTP responses from a resource identified by a URI. Generally this API supported TCP stack. In .NET 5 , library is extended to support other transports such as Unix Domain Sockets & NamedPipes because many web server on Unix supported HTTP server implementations using Unix Domain Sockets . Windows also added Unix domain socket support in Windows 10 ( for more details see here)

ASP.NET Core web server (Kestrel) also added support for UNIX domain sockets and named pipes( from .NET 8 on-wards).

Alternative transport options is introduced in HttpClient SocketsHttpHandler. With this it is possible to connect to server with Unix Domain Socket or Named Pipes.

HttpClient with Unix Domain Socket

SocketsHttpHandler socketsHttpHandler = new SocketsHttpHandler();
// Custom connection callback that connects to Unixdomain Socket
socketsHttpHandler.ConnectCallback = async (sockHttpConnContext, ctxToken) =>
{
    Uri dockerEngineUri = new Uri("unix:///var/run/docker.sock");
    var socket = new Socket(AddressFamily.Unix, SocketType.Stream, ProtocolType.IP);

    var endpoint = new UnixDomainSocketEndPoint(dockerEngineUri.AbsolutePath);
    await socket.ConnectAsync(endpoint, ctxToken);
    return new NetworkStream(socket);
};

// create HttpClient with SocketsHttpHandler
var httpClient  = new HttpClient(socketsHttpHandler);
// make Http Request .

HttpClient with Named Pipe

SocketsHttpHandler socketsHttpHandler = new SocketsHttpHandler();
// Custom connection callback that connects to NamedPiper server
socketsHttpHandler.ConnectCallback = async (sockHttpConnContext, ctxToken) =>
{
    Uri dockerEngineUri = new Uri("npipe://./pipe/docker_engine");
    NamedPipeClientStream pipeClientStream = new NamedPipeClientStream(dockerEngineUri.Host,
                                            dockerEngineUri.Segments[2],
                                            PipeDirection.InOut, PipeOptions.Asynchronous);
    await pipeClientStream.ConnectAsync(ctxToken);
    return pipeClientStream;
};

// create HttpClient with SocketsHttpHandler
var httpClient  = new HttpClient(socketsHttpHandler);
// make Http Request .
        

Complete demonstration sample that uses HttpClient connecting to Docker daemon serving REST API via NamedPipe on Windows & Unix Domain socket on Linux.


using System.IO.Pipes;
using System.Net.Http;
using System.Net.Http.Json;
using System.Net.Sockets;
using System.Runtime.InteropServices;

HttpClient client = CreateHttpClientConnectionToDockerEngine();
String dockerUrl = "http://localhost/v1.41/containers/json";
var containers = client.GetFromJsonAsync<List<Container>>(dockerUrl).GetAwaiter().GetResult();
Console.WriteLine("Container List:...");
foreach (var item in containers)
{
    Console.WriteLine(item);
}


// Create HttpClient to Docker Engine using NamedPipe & UnixDomain
HttpClient CreateHttpClientConnectionToDockerEngine()
{
    SocketsHttpHandler socketsHttpHandler =
        RuntimeInformation.IsOSPlatform(OSPlatform.Windows) switch
        {
            true => GetSocketHandlerForNamedPipe(),
            false => GetSocketHandlerForUnixSocket(),
        };
    return new HttpClient(socketsHttpHandler);

    // Local function to create Handler using NamedPipe
    static SocketsHttpHandler GetSocketHandlerForNamedPipe()
    {
        Console.WriteLine("Connecting to Docker Engine using Named Pipe:");
        SocketsHttpHandler socketsHttpHandler = new SocketsHttpHandler();
        // Custom connection callback that connects to NamedPiper server
        socketsHttpHandler.ConnectCallback = async (sockHttpConnContext, ctxToken) =>
        {
            Uri dockerEngineUri = new Uri("npipe://./pipe/docker_engine");
            NamedPipeClientStream pipeClientStream = new NamedPipeClientStream(dockerEngineUri.Host,
                                                    dockerEngineUri.Segments[2],
                                                    PipeDirection.InOut, PipeOptions.Asynchronous);
            await pipeClientStream.ConnectAsync(ctxToken);
            return pipeClientStream;
        };
        return socketsHttpHandler;
    }
    // Local function to create Handler using Unix Socket
    static SocketsHttpHandler GetSocketHandlerForUnixSocket()
    {
        Console.WriteLine("Connecting to Docker Engine using Unix Domain Socket:");
        SocketsHttpHandler socketsHttpHandler = new SocketsHttpHandler();
        // Custom connection callback that connects to Unixdomain Socket
        socketsHttpHandler.ConnectCallback = async (sockHttpConnContext, ctxToken) =>
        {
            Uri dockerEngineUri = new Uri("unix:///var/run/docker.sock");
            var socket = new Socket(AddressFamily.Unix, SocketType.Stream, ProtocolType.IP);

            var endpoint = new UnixDomainSocketEndPoint(dockerEngineUri.AbsolutePath);
            await socket.ConnectAsync(endpoint, ctxToken);
            return new NetworkStream(socket);
        };
        return socketsHttpHandler;
    }
}

/// <summary>
/// Record class to hold the container information
/// </summary>
/// <param name="Names"></param>
/// <param name="Image"></param>
/// <param name="ImageID"></param>
/// <param name="Command"></param>
/// <param name="State"></param>
/// <param name="Status"></param>
/// <param name="Created"></param>
public record Container(List<String> Names, String Image, String ImageID,
                        String Command, String State, String Status, 
                        int Created);

Conclusion:

.NET HttpClient extensibility allows connecting to HTTP server implemented using non TCP transport such as Unix Domain Socket & Named Pipes.

Reference:

SocketsHttpHandler Extension Points

Docker Engine REST API Reference

Beautiful visualization of SSH port forward

SSH port forwarding syntax is confusing . Here are some of the best blog posts that provides break down of SSH port forward command line syntax with beautiful visualization.

Unix domain socket support in Windows

UNIX domain sockets enable efficient inter process communication between the processes running on the same machine.,UNIX domain socket uses file pathname to identify the server instead of  an IP address and port. Support for UNIX domain sockets has existed on many flavors UNIX & LINUX for the longest time but this facility was no available on Windows. On Windows  if you want to do  Local IPC  then you had to use named pipes . Named pipe had different API’s then sockets., This difference made porting UNIX domain sockets based application to Windows difficult.

Windows 10 Version 1803  brings native support of UNIX domain sockets to Windows. You can program in “C”  with Windows SDK version 1803 (10.0.17134 or greater ) and program in C# using .NET core 2.1 ,

Here is how to check support for UNIX domain sockets on Windows.

sc query afunix” from a Windows admin command prompt

image

Here is the C#  demo sample


using System;
using System.IO;
using System.Net.Sockets;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
namespace UnixSocketsDemo
{
class UnixSocketsOnWindows
{
public static void Demo()
{
string path = Path.Combine(Path.GetTempPath(), Path.GetRandomFileName());
var task = Task.Run(() =>
{
StartServer(path);
});
// wait for server to start
Thread.Sleep(2000);
StartClient(path);
Console.ReadLine();
}
private static void StartClient(String path)
{
var endPoint = new UnixDomainSocketEndPoint(path);
try
{
using (var client = new Socket(AddressFamily.Unix, SocketType.Stream, ProtocolType.Unspecified))
{
client.Connect(endPoint);
Console.WriteLine($"[Client] Connected to … ..{path}");
String str = String.Empty;
var bytes = new byte[100];
while (!str.Equals("exit", StringComparison.InvariantCultureIgnoreCase))
{
Console.WriteLine("[Client]Enter something: ");
var line = Console.ReadLine();
client.Send(Encoding.UTF8.GetBytes(line));
Console.Write("[Client]From Server: ");
int byteRecv = client.Receive(bytes);
str = Encoding.UTF8.GetString(bytes, 0, byteRecv);
Console.WriteLine(str);
}
}
}
finally
{
try { File.Delete(path); }
catch { }
}
}
private static void StartServer(String path)
{
var endPoint = new UnixDomainSocketEndPoint(path);
try
{
using (var server = new Socket(AddressFamily.Unix, SocketType.Stream, ProtocolType.Unspecified))
{
server.Bind(endPoint);
Console.WriteLine($"[Server] Listening … ..{path}");
server.Listen(1);
using (Socket accepted = server.Accept())
{
Console.WriteLine("[Server]Connection Accepted …" + accepted.RemoteEndPoint.ToString());
var bytes = new byte[100];
while (true)
{
int byteRecv = accepted.Receive(bytes);
String str = Encoding.UTF8.GetString(bytes, 0, byteRecv);
Console.WriteLine("[Server]Received " + str);
accepted.Send(Encoding.UTF8.GetBytes(str.ToUpper()));
}
}
}
}
finally
{
try { File.Delete(path); }
catch { }
}
}
}
class Program
{
static void Main(string[] args)
{
Console.WriteLine("Hello World!");
}
}
}

 

Resource

Compile and Run .NET program with just .NET framework on your system.

Compile and Run .NET program with just .NET framework on your system.

Did you know that every .NET framework installation also installs VB & C# compiler.

Using these compilers you can compile and run C# or VB.NET program on machine with just .NET framework installed. This can be useful on system where installing full visual studio is not an option.

Here is how to do it

If .NET framework installed on you system, C:\Windows\Microsoft.NET\Framework directory  should list all the versions. Here is what is on my machine.

image

Once you find required .NET version, directory content should list csc.exe and vbs.exe compiler executable. As shown below.

image

Now use C# or VB compiler to compile the program as follows. For example use .NET 2.0 compiler to generate .exe c:\Windows\Microsoft.NET\Framework\v2.0.50727\csc.exe Program.cs

image

Identifying process using TCP port in .NET

Currently I am working on a application  which uses third-party component for remote diagnostics.  This third party component internally uses 8081 tcp port for communication between the process and port was not configurable , which forced our application installer to make sure that no other process in the system is using port 8081 during install. We had to warn user during application install if any process is using required tcp port.

In .Net you can use System.Net.NetworkInformation.IPGlobalProperties to find out whether tcp port in use. But this API does not tell you which process using the port. We wanted to show user not only tcp port is in use , also provide information  about process , such as name , id and path .  We also had to make sure that it works from Windows XP to Windows 7 .

From MSDN documentation, I found that only to get the process id of the process using the tcp port is to use Win32 API . This means I had to use platform invoke(pinvoke) and carefully declare lot of Win32 structure in C#.

Other alternative is to use the netstat command , that provides the same details with appropriate command line switches. I used this technique various times ,when I was working on Linux systems. I decided to do the same here.

I wrote simple helper class which uses netstat and WMI to get the details about the process using the tcp port.
Here is how you use the sample

int port = 8081;
TcpHelperUtil tcpHelper = new TcpHelperUtil();
var details = tcpHelper.GetPortDetails(port);
if (details.Item1)
{
    Console.WriteLine("Port {0} in Use",port);
    Console.WriteLine(details.Item2.ToString());
}else
{
    Console.WriteLine("Port {0} is free ",port);
}

Here is the source code.

using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Linq;
using System.Management;

namespace ConsoleApp2
{

    internal class PortDetail
    {
        public int Port { get; set; }
        public int ProcessID { get; set; }
        public string ProcessName { get; set; }
        public string Path { get; set; }
        public override string ToString()
        {
            return string.Format(@" Process Name: {0} ,Process ID: {1} ,
                                    Port: {2} ,\nPath : {3}", ProcessName,
                                    ProcessID, Port, Path);
        }

    }
    /// <summary>
    /// Usage:
    /// int port = 8081
    /// TcpHelperUtil tcpHelper = new TcpHelperUtil();
    /// var details = tcpHelper.GetPortDetails(port);
    /// if (details.Item1)
    /// {
    ///     Console.WriteLine("Port {0} in Use",port);
    ///     Console.WriteLine(details.Item2.ToString());
    /// }else
    /// {
    ///     Console.WriteLine("Port {0} is free ",port);
    /// }
    ///
    /// </summary>
    class TcpHelperUtil
    {
        private const short MINIMUM_TOKEN_IN_A_LINE = 5;
        private const string COMMAND_EXE = "cmd";

        public TcpHelperUtil()
        {

        }

        public Tuple<bool, PortDetail> GetPortDetails(int port)
        {
            PortDetail PortDetail = new PortDetail();
            Tuple<bool, PortDetail> result = Tuple.Create(false, PortDetail);

            // execute netstat command for the given port
            string commandArgument = string.Format("/c netstat -an -o -p tcp|findstr \":{0}.*LISTENING\"", port);

            string commandOut = ExecuteCommandAndCaptureOutput(COMMAND_EXE, commandArgument);
            if (string.IsNullOrEmpty(commandOut))
            {
                // port is not in use
                return result;
            }

            var stringTokens = commandOut.Split(default(Char[]), StringSplitOptions.RemoveEmptyEntries);
            if (stringTokens.Length < MINIMUM_TOKEN_IN_A_LINE)
            {
                return result;
            }

            // split host:port
            var hostPortTokens = stringTokens[1].Split(new char[] { ':' });
            if (hostPortTokens.Length < 2)
            {
                return result;
            }

            int portFromHostPortToken = 0;
            if (!int.TryParse(hostPortTokens[1], out portFromHostPortToken))
            {
                return result;
            }
            if (portFromHostPortToken != port)
            {
                return result;
            }

            PortDetail.Port = port;
            PortDetail.ProcessID = int.Parse(stringTokens[4].Trim());
            Tuple<string, string> processNameAndPath = null;
            try
            {
                processNameAndPath = GetProcessNameAndCommandLineArgs(PortDetail.ProcessID);
                PortDetail.ProcessName = processNameAndPath.Item1;
                PortDetail.Path = processNameAndPath.Item2;
                result = Tuple.Create(true, PortDetail);
            }
            catch (Exception exp)
            {
                Console.WriteLine(exp.ToString());

            }

            return result;

        }
        /// <summary>
        /// Using WMI API to get process name and path instead of
        /// Process.GetProcessById, because if calling process ins
        /// 32 bit and given process id is 64 bit, caller will not be able to
        /// get the process name
        /// </summary>
        /// <param name="processID"></param>
        /// <returns></returns>
        private Tuple<string, string> GetProcessNameAndCommandLineArgs(int processID)
        {
            Tuple<string, string> result = Tuple.Create(string.Empty, string.Empty);
            string query = string.Format("Select Name,ExecutablePath from Win32_Process WHERE ProcessId='{0}'", processID);
            try
            {
                ObjectQuery wql = new ObjectQuery(query);
                ManagementObjectSearcher searcher = new ManagementObjectSearcher(wql);
                ManagementObjectCollection results = searcher.Get();

                // interested in first result.
                foreach (ManagementObject item in results)
                {
                    result = Tuple.Create<string, string>(Convert.ToString(item["Name"]),
                    Convert.ToString(item["ExecutablePath"]));
                    break;

                }
            }
            catch (Exception)
            {

                throw;
            }

            return result;

        }

        /// <summary>
        /// Execute the given command and captures the output
        /// </summary>
        /// <param name="commandName"></param>
        /// <param name="arguments"></param>
        /// <returns></returns>
        private string ExecuteCommandAndCaptureOutput(string commandName, string arguments)
        {
            string commandOut = string.Empty;
            Process process = new Process();
            process.StartInfo.FileName = commandName;
            process.StartInfo.Arguments = arguments;
            process.StartInfo.UseShellExecute = false;
            process.StartInfo.CreateNoWindow = true;
            process.StartInfo.RedirectStandardOutput = true;
            process.StartInfo.RedirectStandardError = true;
            process.Start();

            commandOut = process.StandardOutput.ReadToEnd();
            string errors = process.StandardError.ReadToEnd();
            try
            {
                process.WaitForExit(TimeSpan.FromSeconds(2).Milliseconds);
            }
            catch (Exception exp)
            {

                Console.WriteLine(exp.ToString());
            }
            return commandOut;
        }
    }
}

Simple way to share files from between Linux to windows

I often use both Windows and Linux environment at work. Sometimes I need to share files between Linux & windows workstation.

There are couples of ways to share files between Linux & Windows.

  1. Setup Samba server (CIFS share) on your Linux box, which can be accessed from any windows machine. Because In many of the Linux distros Samba server is not installed by default. It requires installing package and setting up configuration file to share folder.
  2. Run Ftp server on you Linux box and use ftp client from windows box to access it. Again this solution may require starting the ftp server on the Linux machine and configuring it to allow access to shares.
  3. Run SSH server on Linux machine and on windows box you use secure copy(scp) or similar SSH client to do file transfer. You can download scp client for windows putty download page.This also requires installing SSH server on Linux box ( for Ubuntu by default SSH server is not installed)

Even though many of the option listed are simple , it requires some effort to setup. You need to repeat the same when you re-install Linux.

Recently I discovered simple and elegant way to achive the same using python built in SimpleHTTPServer module.Advantage of this approach is python is by default installed on all Linux distributions. No need to install any additional package.

Here is how to do it

    1. From the command shell, go to the directory you want to share. For example if you want to share “/usr/home/john/files” .                                                                                                                                                                       $ cd /usr/home/john/files
    2. Run following command to share files. This command starts the webserver on port 8080 with directory browsing enabled .                                                                                                                                                                             $ python -m SimpleHTTPServer 8080
    3. On your windows box point your any Webrowser to http://<linux_machine_ip&gt;:8080 . This will list all the files present on the Linux box . Now select any files download to windows.

Resource