websocket in curl

WebSocket has been supported by curl as a non-experimental feature since version 8.11.0 (November 6 2024). With the upcoming release of version 8.16.0, we are taking it a step further.

With that release, you can use WebSocket in a pipe from the command line. Like this:

> curl --no-progress-meter -T . -N wss://echo.websocket.org/
Request served by 4d896d95b55478dsadsa
Hello!
Hello!
you just echo what I send?
you just echo what I send?
...

This command line contacts the “echo” server run by websocket.org which just sends back what it receives from the client. The command line options are:

• -T .: tells curl to upload the data from stdin, in non-blocking fashion.
• -N: to avoid buffering of output data, so that even a single character received from the server is on stdout right away.
• --no-progress-meter: to avoid curl cluttering stderr with information about how much was sent/received.

This is convenient if you want to test a WebSocket endpoint or do scripting utilities.

Usage in libcurl

For applications that use the libcurl API, this release means they can use the CURLOPT_READFUNCTION with WebSocket transfers. For an easy handle, it basically does the following:

CURL *easy;
...
curl_easy_setopt(easy, CURLOPT_UPLOAD, 1L);
curl_easy_setopt(easy, CURLOPT_READFUNCTION, my_read_cb);
curl_easy_setopt(easy, CURLOPT_READDATA, my_read_ctx);
curl_easy_setopt(easy, CURLOPT_WRITEFUNCTION, my_write_cb);
curl_easy_setopt(easy, CURLOPT_WRITEDATA, my_write_ctx);
...
result = curl_easy_perform(easy);

This will read WebSocket frames payload from my_read_cb and sends the frames to the server and, on receiving frames, writes the payload to my_write_cb. For the write callback, the frames meta data is available via curl_ws_meta(easy), so it knows the type of the frame, its overall length, etc. If it cares about that.

The write support has been there in previous versions. The read callback is new.

WebSocket Read Callback

A callback installed in CURLOPT_READFUNCTION could look like this:

size_t my_read_cb(char *buf, size_t nitems, size_t buflen, void *userdata)
{
    size_t blen = nitems * buflen;
    
    if(have_more_to_send) {
        size_t nwritten = 0;
        // write bytes to buf
        return nwritten;
    }
    else if(nothing_to_send_now) {
        return CURL_READFUNC_PAUSE;
    }       
    else { /* done sending */
        return 0;
    }
}

The nwritten bytes placed into buf are sent to the WebSocket server in a BINARY frame of exactly that length (for other frame types, see below).

Returning CURL_READFUNC_PAUSE unsurprisingly pauses the sending. Frames received from the server are still written out, but the my_read_cb is only called again when the transfer is unpaused via curl_easy_pause().

Returning 0 ends the sending of data. The receiving direction still remains open, however.

Sending Other Frames

If the read callback wants to send other frame types or larger frames, it can call curl_ws_start_frame(easy, frame_flags, frame_length). frame_length cannot be shorter than its nwritten return value, but it may be longer.

In that case, the read callback is expected to return the remaining payload bytes in future calls, so it needs to track where it is at. Only when the previous frame is complete can it call curl_ws_start_frame() again. Calling it before returns an error code.

Empty Frames

One last special thing: WebSocket frames may have no payload, e.g. a payload length of 0. This requires some special tweaks:

• When the read callback starts such a frame via curl_ws_start_frame(easy, frame_flags, 0), libcurl will treat a return value of 0 as the end of that frame and not as the end of sending.

• Similar, the write callback for a WebSocket transfer may get called with a length of 0 when receiving a frame without payload. For other protocols, it does not do that as it has no meaning. A websocket write callback needs to handle that gracefully.

Other Options?

The libcurl API offers two functions to send and receive WebSocket data: curl_ws_send() and curl_ws_recv(). While these work, they place a burden on applications which has led to some confusions in the past.

Calling curl_ws_send() may return CURLE_AGAIN, meaning the data could not be sent as the network buffers are full. The application was then expected to try again “later”, ideally when the connection’s socket had become writable again.

This is not so easy to achieve, especially in a portable manner. libcurl already implements this internally for all platforms. It seems therefore much more convenient for applications to use the callbacks. For applications doing several transfers in parallel, the callbacks are the much better option anyway.

Summary

With the full support of CURLOPT_READFUNCTION in WebSocket transfers, this is now the recommended way how applications of libcurl should use the protocol. It is non-blocking, it allows parallel transfers and it let’s the application send WebSocket frames in whichever way it needs.

The bidirectional support on the curl command line should allow easy testing and simple applications. If you need more in this area, let us know or brainstorm your ideas on a curl discussion.