DEV Community: JackMacWindows

Printing coredumps automatically with systemd and ReceiptIO

JackMacWindows — Fri, 25 Oct 2024 06:40:59 +0000

I had a fun idea one day after seeing some people online playing with receipt printers. I frequently run into app crashes on my Linux system, all of which generate core dumps for review later, and those are logged in the system log. Wouldn't it be funny if they were logged not only in software, but also in physical form? It took me a while to finally go through with it, but I now have a receipt printer set up to automatically print out a crash log for every crash on my system. Here's how I did it.

systemd-coredump

On Linux systems using systemd (which is the vast majority of them), a component called systemd-coredump is installed to automatically take in coredump files that the kernel generates, and processes the files before outputting them on disk. Coredump files are stored in /var/lib/systemd/coredump by default, and the coredumpctl command can be used to output various information about the coredumps. (This will be useful later.)

My first task was to get the system to run a command whenever a coredump occurred. I spent some time digging through man pages on systemd-coredump and coredumpctl, but I unfortunately came up with nothing - it doesn't seem to have any functionality to e.g. run a service on crash. I tried hacking with the inner functionality of the dump service, trying to hook into its trigger, but this didn't result in anything useful. I also thought about replacing systemd-coredump entirely, but then I'd be lacking the digest that it creates.

Then I came up with a solution...

Watching directories with systemd

I realized that I could simply have a command be triggered whenever the coredump directory gets written to, which only happens when a program crashes. systemd supports special unit files with the extension .path to watch a specific file or directory. When the path is updated, systemd triggers a .service unit with the same name, which can be a shell script with the commands I need. Perfect!

I put together a simple path unit /etc/systemd/system/print-coredump.path:

[Path]
PathChanged=/var/lib/systemd/coredump

[Install]
WantedBy=paths.target

And I made an accompanying service unit to run a bash script:

[Unit]
Description=Automatically print core dump

[Service]
Type=oneshot
ExecStart=/usr/local/bin/print-coredump

To enable watching, I simply had to run sudo systemctl enable --now print-coredump.path.

ReceiptIO

Next I needed a way to send text to the printer. I went googling around for a bit, and eventually stumbled upon ReceiptIO. ReceiptIO is a NodeJS library and program that automatically formats and prints out text (and images) using its own formatting language inspired by Markdown. It also has a web viewer to preview printouts, which was useful before I had the printer in my hands, and it also helped me understand what changes I'd need to make it pretty.

I installed it globally with npm: sudo npm install -g receiptio

To start with, I made a very simple bash script to print out the most recent coredump generated to an SVG (the default output if no printer is attached):

#!/bin/bash
sleep 1 # wait for the file to be fully written
coredumpctl info -1 | receiptio -i -o /home/jack/printtest.svg

As I expected, the output it generated was nowhere near formatted in a way that was readable. So I then spent some time writing some commands to process the coredump digest into a nicer format.

Formatting the output

I started by plugging in the output of coredumpctl info straight into the web viewer. My first challenge was that all of the text was centered horizontally - this looked really ugly. At first, I simply made all of the text align left, but the header info at the top didn't quite look right.

One thing that coredumpctl does in its output is that it formats the metadata block at the top in two columns using spaces. However, this doesn't translate well to the receipt printer, so my output was overflowing the lines, and it just didn't look good. To fix that, I used the column formatting commands that ReceiptIO provides. I used sed to replace every a: b with a:|b - | tells ReceiptIO to separate the sides into columns, and having no spaces on either side of it means that the left side is right aligned, and the right side is left aligned (i.e. the text is effectively centered). One caveat: I did decide to undo this change on the final line, which I wanted to keep centered - this involved another sed expression to change it back if the left side was architecture.

After that, I needed to do some formatting on the stack trace entries. I wanted to keep the "introduction" to each thread centered, but each frame would be left-aligned. This simply involved adding a | at the start of each frame line, which conveniently always starts with #. A simple sed expression fixed that. I also made it replace the empty lines with horizontal rules (- alone on a line) for further prettification.

My last gripe was that the underscores in function names in the traces were being interpreted as control characters - like in Markdown, _a_ and __a__ apply special formatting. This required one last sed expression to escape every _ in the output.

Building the command

Next I combined the changes into a single command line. Watch out, this is a pretty big command!

coredumpctl info -1 | sed -e 's/^ *//g' -e 's/: /:|/g' -e 's/^#/|#/g' -e 's/_/\\_/g' -e 's/architecture:|/architecture: /' -e 's/^$/-/g' | receiptio -d /dev/usb/lp0 -c 42

Let's break that really long sed line down into each part:

If you aren't aware of how sed works: basically, it's a simple text processing language that transforms each line of text passed into it via a pipe. The most basic command, and the only one used here, is s for substitution, which takes a regular expression pattern and replaces it with other text. It takes the syntax s/find/replace/flags, where flags may be empty to replace only once, or it can be g to replace all.
Each -e option is a single replacement command. This makes it so I don't have to run sed a bunch of different times - it can be done all in one command.
The first -e command gets rid of any spaces at the start of a line, which basically removes the column formatting that coredumpctl applies.
The second -e command replaces all :s with :|, placing metadata lines into two columns as described above.
The third -e command left-aligns all lines starting with # (stack frame lines).
The fourth -e command escapes all _s with \_ (double backslash so sed doesn't see it as an escape itself).
The fifth -e command removes the column formatting from the final line (with architecture: in it).
The last -e command finds all empty lines and replaces them with a single -, which indicates a horizontal rule.

After that behemoth, the output is sent to receiptio, which prints to the printer over USB, with 42 columns of characters (the spec of my printer).

And sure enough it worked!

Further tweaking

One of the main things that bothered me about this print was that there was so much whitespace on the left side that wasn't being used, which could've been used for more text on the right. After looking through the docs, I found that I could use a special command, {w:}, to set the widths of each column to a specific value.

I played around a bit in the web viewer, and settled on the command {w:15 *}, which makes the left side 15 characters wide, and makes the right side the rest of the width (27 in my case). But I had one problem - the stack traces were now limited to being 15 characters wide, so I needed to reset the columns with {w:*} after the header. This looked like a tough problem at first, since sed only sees the file as independent lines. However, I was able to fix it by making the command piggyback off of the horizontal rule replacement - in other words, now it would replace empty lines with both {w:*} and -.

Prepending the {w:15 *} command appeared tough at first, but with some internet digging, I managed to come up with this command:

{ echo "{w:15 *}" ; coredumpctl info -1 | sed -e 's/^ *//g' -e 's/: /:|/g' -e 's/^#/|#/g' -e 's/_/\\_/g' -e 's/architecture:|/architecture: /' -e 's/^$/{w:*}\n-/g' ; } | receiptio -d /dev/usb/lp0 -c 42

This uses piping on two combined commands: first an echo, and then the coredump command, now with the {w:*} commands filled in. This worked fine, but I soon had to swap it out for a better solution.

Next I added a little logo to the top - it was kinda boring with just text. I took the monochrome Linux logo from the kernel, scaled it up, and added a ! after it to indicate a crash. Then I saved it as a monochrome PNG. ReceiptIO can display an image using the {i:} command, which takes base64 of a PNG file as an argument. I didn't want to shove all this into the same command line, so I split it off into a separate file /usr/local/share/coredump-header.txt:

{i:iVBORw0KGgoAAAANSUhEUgAAAWgAAADwCAAAAAAIzKGTAAAACXBIWXMAAAsTAAALEwEAmpwYAAAEWElEQVR42u3d23KjMBCE4WjL7//Ksxdbu8XGhgh0QBJf38VxsOtPq5lBEqT4oh76BQHQQBPQQANNQANNp/Xq+WFp5/XgaAJ6MqU+wzZlvCc4moAWHSdzY/kA4WigRUe73Fg4PTgaaNFRPTfi7c3B0QT00HoN8j3S24/B0QS06Kjd1HA0Ab1qdCR0ORpo0XFHKRIcTes4eryiPji6phrlFdAyGmgCGmjlXbfzeDo8v6fVexaOBlp0XKv/43DlRqx+WZWjgRYdZ5X+hkN8Wr/xNUtpERwtOqhfdKSMX6WdYOFoAvrJVcc2HziagJ6qXr4/Or5VIE/bhs/RQKs6zibewQWNs2HC0QT0Q6Jj26ekzURt/J8nJmcJaNGx13TEfidicpaA7qvC0/JAO2fNsBDQqo7MpmN7GWTtnoWjgRYdVdJjpsVgHA30k1uTWxuWnA0s24ZlveaFo4FWdZwNvIOmY6/2SJv0WKNn4WigRcfZMuNjemhYCGjRkVl7/Nh+xf8bW74VIRxNQNcck+NFRyr4prHoVAtHA63q6BdsHE1APyY62s2MTF14cDTQ8zYnHaKjyhPcDmZm500PjgZa1dG/3lggPTgaaNHRp8CyJIyAfkJ0pJ1X3rfVR8afx3KFB0d36sqB7qQ6T7rPnI1NebmRWXg0TI/E0aoOuqdhiTOD/fiGPxxNQC9ZdVQ/RecHSKvCQ9UxS3sC9EJVR1Q6gsukBPTC0dFugL8HyMcnQAVHE9DzVx3VB/Iys7QcDbTo6FBy7H3cvIUHR3c69wA9YdVhsxtHAy06Ris5OJqAXrXqaFfDpvl7Fo7uVKkCzdFAE9DLVR1n10ylx18J4WiggSaggQYagj4dOdAcDTQBDfTjNcEMy/sTW2bc5MLRHD175Twj6Ni5MDtReogOoNfSlU33p27/VVhs3JiqHC06qGLV8bF9aPRw2JxbAM3SvHA00E+Ojr0HxVZPj8zjBEcT0MNHx/EDpvs/mD44moAeODryn8q0TY+6QztO3jKdo0UH3Rgdhdc/U+306Ky6l3/NGf58Zko1UAP9E+ZKqF91B2x890K/IV/xs9KnlwoP7mSYxfmr+JI30LWJ3pXRfdLjT2RVaZR2OZfdPJ6jO1kd6C7B0eiO6LdMlf6bpR1zly5Hd3I70J30ajSK71poEYXp0ex7c7SMFh2DV1hjLlDnaKBFR/74vbzKnaMJ6IqN/FxVx4X0SOv+FzhadKg6WqRHxdAoO1SkNvnN0aJjtDNeWUFy87WO1OZoRUwicfSdnraApgvpKO1kXh2+dKplqozcqNPZvaVH+WEtcvzZHpbtDttvLwq60Q1DVB0aFrocHeM/BKH6VjuOFh1UGB3b1d0DDs8pig2OBnpR5W7MuLznPZ38q8xAmKXY4GigHx4dl2Pk8sKwWCg3OBroRzUsB43MdhQfDOQW05wz5gZHA63qyKlDrrU2+ZkTHE1ALxEdxNFAA01AA01AAw00NdBvS6PYP/o0f6cAAAAASUVORK5CYII=}
{w:15 *}

Then I updated the command to concatenate that file and stdin from the coredumpctl/sed chain:

#!/bin/bash
sleep 1
coredumpctl info -1 | sed -e 's/^ *//g' -e 's/: /:|/g' -e 's/^#/|#/g' -e 's/_/\\_/g' -e 's/architecture:|/architecture: /' -e 's/^$/{w:*}\n-/g' | cat /usr/local/share/coredump-header.txt - | receiptio -d /dev/usb/lp0 -c 42

This produced my final output:

Oops...

While writing this blog, I had a little accident. plasmashell, the KDE desktop manager, crashed, and successfully demonstrated that the script worked! But...

Yeah, uh, plasmashell has a LOT of threads. 220" (555cm) of threads, in fact. It was hilarious to watch it keep printing for a solid 20 seconds straight (this printer is fast!), but I don't need a useless piece of paper that's longer than the room I live in, especially when I get these sorts of crashes every day. To safeguard, I made my final edit: stopping the output after 100 lines (should be reasonable).

#!/bin/bash
sleep 1
coredumpctl info -1 | sed -e 's/^ *//g' -e 's/: /:|/g' -e 's/^#/|#/g' -e 's/_/\\_/g' -e 's/architecture:|/architecture: /' -e 's/^$/{w:*}\n-/g' | cat /usr/local/share/coredump-header.txt - | head -n 100 | receiptio -d /dev/usb/lp0 -c 42

Conclusion

This was probably a complete waste of my time and nearly $100 in acquiring the receipt printer and paper. But it was a really fun time to put it together, especially when I cried laughing at the sheer length of that plasmashell crash. It also taught me some things about receipt printers and coredumps. Maybe someone will decide it's worth it to set this up, which is why I'm putting this post out there. Anyway, remember folks: if you're gonna accidentally print over 6 yards of useless receipts, it had better not be coming out of your pocket.

Implementing coroutines in Swift using Swift Concurrency

JackMacWindows — Thu, 13 Jun 2024 04:00:00 +0000

One of my favorite features of Lua is its first-class support for coroutines. Recently, I started writing a new project using Swift, and I wanted to be able to use coroutines natively in my Swift code. In most Lua VMs, coroutines are a complex feature that require a lot of environmental support to be able to save and restore function calls. However, Swift includes the async and await keywords for pausing functions built into the language. Because of this, I decided to take a crack at using them to implement coroutines natively.

What are coroutines?

A coroutine is an object that represents a function which can be paused and resumed. This function may pause (or yield) itself at any time, which will return execution to the code that last resumed the coroutine. The coroutine can then be resumed later, and the code will pick up right where it left off.

A coroutine also represents a call stack, or a thread of execution. The main function can call other functions that can yield, at which point the entire coroutine pauses without affecting any of the calling functions. For example, assume a coroutine was initialized with a function foo. If foo calls bar, and at some point bar yields, foo's status will be completely unaffected, and in fact, foo will have no idea that bar even yielded.

In Lua's coroutines, yielding and resuming can also pass values between each coroutine. When yielding, a function can pass values back to its parent as return values from the corresponding resume call, and resuming can pass values to the corresponding yield call as well. This can be used to implement various design patterns, such as iterators, using a single function.

An example of an iterator using coroutines.

local function iterator(array)
    -- Loop over the array normally.
    for i = 1, #array do
        -- Send the index and value back to the function.
        coroutine.yield(i, array[i])
    end
    -- Send back nil to end the iteration.
    return nil
end

local array = {"foo", "bar", "baz"}
-- Loop using a coroutine as an iterator function.
-- The function returned by `coroutine.wrap` is called with `array` for each iteration.
for index, value in coroutine.wrap(iterator), array do
    print(index, value)
end

Coroutines can be used to implement a rudimentary form of cooperative multitasking. A set of tasks can each be placed in their own coroutines, and a master "coroutine manager" can resume each of those coroutines in order until they finish. When one coroutine yields, it lets another one continue its work. This can give the impression that the functions are running in parallel - they can update each other's states between yields, and may process information out-of-order from each other. Using this basic structure, more complex forms of multitasking can be implemented, like OS processes and threads.

A diagram of how execution flows between two parallel coroutines.

Swift Concurrency

Swift Concurrency is a feature of the Swift language added in Swift 5.5. Its main feature is the async and await keywords, which allow functions to delegate long-running tasks to separate threads while the main code continues. This allows cooperative multitasking in a structured manner. These keywords may be familiar to JavaScript or Python developers, who have likely used this construct in those languages before.

Functions which are designated async may use the await keyword to call another async function with set arguments. This pauses execution of the current function, and allows other asynchronous tasks to complete. Once the awaited function returns, the original function is resumed with the return value (if used) sent back as the result of await.

To call an async function without waiting for a result, a Task object is used. The Task constructor takes a single async function, which may then call other async functions. A Task wraps around a call stack of async functions, representing a single thread of execution which can be paused and resumed. Tasks are created in an execution pool, which schedules when each task will be run. Using await or calling Task.yield() will pause the current task and allows other Tasks to resume.

A model of how async functions can pause and resume in a single thread of execution.

Some of these things may sound like parts of coroutines as discussed above. Both coroutines and async functions are able to be paused and resumed at certain points. Both coroutines and await can pass values to and from a subtask. Both coroutines and Tasks represent a single thread of resumable execution.

However, there is one very important distinction between the two: coroutines have a parent-child relationship, where a coroutine resumes a child and yields to its parent; while Tasks and their main async functions are run in a pool, and thus have no parents. To mitigate this, I decided to implement my own Coroutine class to hold the parent-child relationship.

Initial implementation

To start, I created a Coroutine class with an initializer, a resume method, and a static yield function. The resume and yield methods take and return arrays of any value, which allows passing and returning multiple values. The initializer takes an async function for the body, which takes and returns [Any] arrays as well. To keep track of parent coroutines, I added a static running property, which holds the currently running coroutine.

public class Coroutine {
    public static var running: Coroutine? = nil
    public static func yield(with args: [Any]) async -> [Any] {}

    public init(_ body: ([Any]) async -> [Any]) {}
    public func resume(with args: [Any]) async -> [Any] {}
}

Each coroutine holds a single Task variable, which is what runs the functions and holds the call stack. To keep track of whether the coroutine is paused, running, normal (running but waiting on another coroutine), or dead, it also has a state property, which is an enum type.

    public enum State {
        case suspended
        case running
        case normal
        case dead
    }
    public var state: State = .suspended
    private var task: Task<Void, Never>! = nil // no return value, never throws

To implement the resuming and yielding functionality, I used the state variable to determine whether a task should continue. Resuming a coroutine involved setting the coroutine's status to running, and then waiting until it was no longer running. Likewise, yielding a coroutine would set the coroutine's status to suspended, and then waited until it was no longer suspended. This would ensure that only one coroutine was running at a time. An additional private member held the return values on each end.

    private var results: [Any] = [Any]()

    public func resume(with args: [Any] = [Any]()) async -> [Any] {
        // Set the currently running coroutine, and make the previous coroutine have normal status.
        let old = Coroutine.running
        old?.state = .normal
        Coroutine.running = self

        // Set the coroutine to running, pass the return values, and wait for its task to yield.
        results = args
        state = .running
        while state == .running {
            await Task.yield()
        }

        // Reset the running coroutine to its previous value, and return with the yield's return values.
        Coroutine.running = old
        old?.state = .running
        return results
    }

    public static func yield(with args: [Any] = [Any]()) async -> [Any] {
        if let coro = Coroutine.running {
            coro.results = args
            coro.state = .suspended
            while coro.state != .running {
                await Task.yield()
            }
            return coro.results
        }
    }

The initializer function simply created a new Task using a small wrapper to handle the first resume and last return.

    public init(_ body: ([Any]) async -> [Any]) {
        task = Task {
            // Wait for the task to be resumed for the first time.
            while self.state != .running {
                await Task.yield()
            }
            // Call the body function.
            let res = await body(self.results)
            // Set the coroutine as dead and set return values.
            self.state = .dead
            self.results = res
        }
    }

This approach worked, and my small test suite passed properly. However, astute readers will notice a huge hole in this approach. Task.yield() does not wait for anything - it simply lets other tasks step forward, and then resumes itself, which is why the while loop is required. This means that every coroutine is consuming 100% CPU until they get resumed, and because tasks can be delegated to multiple CPU cores, this can quickly overload the system.

Obviously, this isn't a suitable approach for a complete application. But luckily, there's a mechanism included in the concurrency features that helps fix this issue.

Continuations

In JavaScript, many older asynchronous functions use a callback parameter to specify what code to run once the async task completes. The function itself would return immediately, but the callback function would be called (often with result parameters) after the asynchronous task was finished, which would continue the program's execution. But this often led to callback hell, a situation where a program gets extremely deeply nested because it used multiple asynchronous functions in series:

fooAsyncCallback(a, b => {
    barAsyncCallback(b, c => {
        bazAsyncCallback(c, d => {
            d.processCallback(res => {
                console.log(res)
            })
        })
    })
})

To fix this, JavaScript introduced the Promise type, which allowed async functions to be called in a serial manner using chains of .then calls:

fooAsyncPromise(a)
    .then(b => barAsyncPromise(b))
    .then(c => bazAsyncPromise(c))
    .then(d => d.processPromise())
    .then(res => console.log(res))

Later on, async/await wrapped around this functionality by automatically breaking an async function into Promise callbacks during compilation, allowing true structured programming:

let b = await fooAsyncPromise(a)
let c = await barAsyncPromise(b)
let d = await bazAsyncPromise(c)
let res = await d.processPromise()
console.log(res)

But this requires functions to implement a function that returns Promises. If you're stuck with an old callback-based function, you normally have to break the chain and start a new one inside the callback. This is where the Promise constructor comes in. It takes another callback as an argument - but this callback is used to call the async function. The callback receives an argument called the resolver, which is used as the callback for the async function. This allows using callback-based functions with Promises and async.

let b = await fooAsyncPromise(a)
let c = await barAsyncPromise(b)
let d = await bazAsyncPromise(c)
let res = await new Promise(resolve => {
    d.processCallback(resolve) // call the function using the resolver function
    // calling resolve() will cause the await statement to continue
})
console.log(res)

Like JavaScript, Swift also has a procedure for using callback-style functions with async/await. A continuation represents the same thing as a JavaScript Promise, and works in a similar way.

To create a continuation, you use one of the with*Continuation global functions. There are four different functions, depending on whether you want a checked or unsafe continuation (more on that later), and a throwing or non-throwing callback function. These functions take a single block/closure, which takes a continuation object, which is then resumed inside the async function's callback.

Here's a translation of the above JavaScript code into Swift using closures:

let b = await fooAsync(a)
let c = await barAsync(b)
let d = await bazAsync(c)
let res = await withCheckedContinuation { continuation in
    d.process { result in
        continuation.resume(returning: result)
    }
}
print(res)

One useful feature of the continuation functions is that the currently running task gets paused until the continuation is resumed. This is super handy when we're looking for a way to pause a task unconditionally. But a drawback of continuations is that they need to be resumed exactly once - the task can't just wait on the same continuation multiple times for multiple yields, and it also can't just leave the task hanging if the continuation's no longer needed, or the task will leak resources.

(This is where the checked/unsafe variants come into play - checked continuations have built-in checks to make sure they are resumed exactly once, while unsafe continuations don't. Checked continuations are usually used while debugging, and can be migrated to unsafe continuations to optimize for speed.)

Throwing continuations can also have errors passed as resume values using the resume(throwing: Error) method, which propagates the error back to the with*ThrowingContinuation function. The coroutine can use this to send errors back to the parent coroutine.

The coroutine will store a property that holds a continuation for later use. To pause a task, the coroutine will create a new continuation, and store it in the continuation property. After that, it'll resume the old continuation with the results to send. Finally, the with*Continuation function's block returns, which pauses the task, and waits for the continuation to be resumed.

Putting it together

First, we'll update the resume function with continuations. We'll use withCheckedThrowingContinuation to create a checked, throwable continuation, which'll allow us to propagate errors back to the resume call.

    public enum CoroutineError: Error {
        case notSuspended
        case noCoroutine
        case cancel
    }

    private var continuation: CheckedContinuation<Void, Error>!
    public func resume(with args: [Any] = [Any]()) async throws -> [Any] {
        // Error if the coroutine isn't currently suspended.
        if state != .suspended {
            throw CoroutineError.notSuspended
        }

        // Set the currently running coroutine, and make the previous coroutine have normal status.
        let old = Coroutine.running
        old?.state = .normal
        Coroutine.running = self

        // NEW: Create a continuation, resume the coroutine, and wait for the coroutine to finish.
        self.state = .running
        let res = try await withCheckedThrowingContinuation {nextContinuation in
            let c = continuation!
            continuation = nextContinuation
            c.resume(returning: args)
        }

        // Reset the running coroutine to its previous value, and return with the yield's return values.
        Coroutine.running = old
        old?.state = .running
        return res
    }

The yield function will work in a similar way. We'll take advantage of throwing errors later on.

    public static func yield(with args: [Any] = [Any]()) async throws -> [Any] {
        // Yielding does not work if there is no currently running coroutine.
        if Coroutine.running == nil {
            throw CoroutineError.noCoroutine
        }
        // Set the currently running coroutine as suspended.
        let coro = Coroutine.running
        coro!.state = .suspended
        // Create a new continuation, and wait for its response.
        return try await withCheckedThrowingContinuation {continuation in
            let c = coro!.continuation!
            coro!.continuation = continuation
            c.resume(returning: args)
        }
    }

The initializer function will also be modified to use a continuation, instead of busy waiting for the first resume. But we need to wait a little bit for the task to store the coroutine - otherwise, the resume method could be called before the continuation is set.

    public init(for body: @escaping ([Any]) async throws -> [Any]) async {
        // Create the task.
        task = Task {
            // Create the continuation for the first resume.
            let args = try await withCheckedThrowingContinuation {continuation in
                self.continuation = continuation
            }
            do {
                // Call the body function.
                let res = try await body(args)

                // Set the coroutine as dead, and send the result back as the final yield.
                self.state = .dead
                self.continuation.resume(returning: res)
            } catch {
                // Catch any thrown errors, and throw them back to the parent resume.
                self.state = .dead
                self.continuation.resume(throwing: error)
            }
        }
        // Wait for the continuation to be created in the other task.
        while continuation == nil {
            await Task.yield()
        }
    }

This code will work fine for running coroutines normally. However, if a coroutine is deleted before its body returns or errors, the task will be left hanging because the continuation was never resumed. This will also print a warning message to the console, since we're using checked continuations.

To resolve this, we'll add a deinitializer to resume the continuation. We'll resume it with an error to make the function exit as quickly as possible. This means that body functions will need to make sure to propagate the .cancel error up to the main function, which is a bit annoying, but I haven't figured out to get around this yet.

    deinit {
        if _state == .suspended {
            continuation.resume(throwing: CoroutineError.cancel)
        }
    }

Finally, we need to tweak the initializer and yield to use a weak reference to the coroutine, as these will make the task enter a retain cycle until the task completes.

    public init(for body: @escaping ([Any]) async throws -> [Any]) async {
        // Create the task.
        // NEW: Use a weak self to avoid retaining the coroutine inside itself.
        task = Task { [weak self] in
            // Create the continuation for the first resume.
            let args = try await withCheckedThrowingContinuation {continuation in
                self!.continuation = continuation
            }
            do {
                // Call the body function.
                let res = try await body(args)

                // Set the coroutine as dead, and send the result back as the final yield.
                self?.state = .dead
                self?.continuation.resume(returning: res)
            } catch {
                // Catch any thrown errors, and throw them back to the parent resume.
                self?.state = .dead
                self?.continuation.resume(throwing: error)
            }
        }
        // Wait for the continuation to be created in the other task.
        while continuation == nil {
            await Task.yield()
        }
    }

    public static func yield(with args: [Any] = [Any]()) async throws -> [Any] {
        // Yielding does not work if there is no currently running coroutine.
        if Coroutine.running == nil {
            throw CoroutineError.noCoroutine
        }
        // Set the currently running coroutine as suspended.
        // NEW: Use an unowned reference to avoid retaining the coroutine after yielding.
        unowned let coro = Coroutine.running
        coro!.state = .suspended
        // Create a new continuation, and wait for its response.
        return try await withCheckedThrowingContinuation {continuation in
            let c = coro!.continuation!
            coro!.continuation = continuation
            c.resume(returning: args)
        }
    }

Wrapping up

Coroutines are a useful primitive for various tasks both synchronous and asynchronous. Using Swift's comprehensive concurrency model, we can implement a coroutine object in less than 100 lines of code. This approach can also be used in other languages that have similar constructs, including JavaScript.

The complete library source is listed in this Gist. This version includes a couple of additions, like the ability to call the coroutine directly to resume it. It's donated to the public domain, so feel free to use it in any project, but I'd appreciate a link back to this article as a reference.

On Writing a Sane API

JackMacWindows — Wed, 12 Jun 2024 04:00:00 +0000

Over my years on the ComputerCraft Discord server, I've had the opportunity to witness the creation of numerous APIs/libraries of all sorts. I've gotten to examine these APIs in depth, as well as answer questions involving the APIs that the creators or users have. As an API designer myself, I compare the designs of other APIs with my designs, and I've noticed a number of patterns that make or break an API design. I've seen plenty of designs that make me go "WTF???", and lots that I just can't understand, even at my advanced level of programming (not to toot my own horn).

This article outlines some rules for making a sane API, which is easy to use, understandable, and doesn't make the user spin in circles to make things with it. Note that when I use the term "API", I'm primarily referring to code libraries and their public interfaces, but a number of points can be applied to web APIs as well. Since I have the most experience in Lua APIs, I'll be focusing on Lua APIs, but this can also be applied to any other language.

Keep It Consistent

Consistency is key in any sane API design. By keeping things the same across the entire API, you reduce the number of surprises the developer will find when they try to use it. This applies for every part of the API, including, but not limited to:

Name formatting (including case, general length, and word choice)
Types of arguments and return values (don't use numbers for some functions, and string-numbers for others)
Order of arguments (if you put X/Y as the 2nd/3rd argument in one function, do that in all functions)
Property names (don't have one type use width/height and another use w/h)
Error message format (use similar word choice and placement for all errors)
Division of submodules (don't place some similar functions in one submodule, but others in the root)

Before writing the API, decide on some standards that you'll follow throughout the API. Then follow this standard in every public interface you write, and don't deviate from the standard unless absolutely required (e.g. you also implement functions for a different API than yours). Having the standard written down somewhere can be useful not only for yourself, but also for the developer. Even better, develop standards that you'll use in all your APIs. For example, I usually use names that are as few words as possible while still getting the idea across, with camelCase to separate words. If you have a personal standard, people who know how to use one API of yours won't have much trouble using another.

Make It Concise

There's no need to make your function/variable names overly descriptive. The documentation is where you can be highly descriptive about the function, but the names should be just long enough to describe what they do. A function or property should ideally be no more than four words long; a type name should ideally have no more than six words. Instead of calling your function drawBoxOnScreenWithSizeAndColor(x, y, w, h, c), use drawBox(x, y, width, height, color) - most IDEs will display the parameter names when typing or hovering over the function, making it unnecessary to describe the parameters in the type name. (Note that this does not necessarily apply to Objective-C, which typically uses descriptive message and parameter label names.) Using shorter names will make your code easier to read and look at, and also makes it easier to type, especially if the developer's working in an editor without autocomplete.

In addition, keep the number of arguments to functions as low as possible. Don't give your functions 13 optional arguments just to make it possible to construct an object in one call, or to specify different options in the process. If you're making a constructor, only use arguments that are absolutely required to create the object, and have the developer set properties or call setter methods for any additional options. For single-call functions that have a lot of options, consider having the developer pass a table/dictionary/object of options instead of positional arguments. This has the benefit of explicitly defining which arguments correspond to which option, and allows complete omission of any default arguments, as well as not requiring the developer to remember the order of arguments. Python has the ability to use named arguments, so prefer using named arguments when there are a large number of options. Lua even has shorthand for passing a single table argument, allowing named calls with a similar syntax to Python:

local function lotsOfArguments(options)
    if options.test then
        print(options.myArgument)
    end
    -- etc.
end
lotsOfArguments{test = true, myArgument = "Hello World!"}

Finally, instead of having large metafunctions that do everything, consider providing individual functions that just do one thing. This allows developers to combine your functions as they need without having to follow the same mindset you may have while writing the API. A function should have one concise purpose, and no more - if more is needed, the developer can combine functions to make it.

Maintain Modularity

In many languages, importable libraries are called modules. This is because the libraries are made to be modular, and able to be loaded and unloaded cleanly as one unit. When writing a library (especially in module-based languages), you should maintain this idea of modularity. The biggest rule to follow is to use as little global state as possible, especially environment-level globals. This means that you should not use any global or module-level variables to store state about the API. Instead, opt to use handles or state objects to store all state. Do not store things in the global environment of the caller (Lua _ENV, JS window, etc.). This can lead to very bad things happening when the "module" (it is no longer modular) is reloaded, plus possible conflicts with other "module"s. By not using global variables, you reduce the number of surprises that may happen due to side effects of function calls. This is especially important if you intend to allow the developer to use multiple copies of some object (e.g. Lua 5 allowing multiple parallel states (VM instances) in the same program), or you want to make your library multithreading-capable (global state leads to race conditions).

Another good way to modularize your code (as well as just general code cleanliness & organization) is to split similar functions into a submodule. This can be a separate file that you have to load manually (good for large/complex submodules), or just a simple namespace or table inside the main module (good for single-file modules). One example of this is in my AUKit library: I have the root aukit module store the loader functions + miscellaneous functions; an aukit.effects submodule (in the same file) holds functions that modify loaded audio in-place; and an aukit.stream submodule provides a number of streaming functions that all return a loader callback for use in aukit.play(). This goes along with the natural human tendency to categorize similar things, and will help the API make more sense, as well as keeping your code naturally organized.

Be Fool-Resistant

When writing libraries, I find that a lot of people tend to make their APIs understandable for them. This is okay if you're just making it for yourself, but it can become a problem when you want other people to start using the library. Just because you know what something means doesn't mean that everyone else who uses your API will know what it means too. A sane API should be clear to anyone with a base-level knowledge of the purpose of the library; e.g. an audio manipulation library should be clear to someone with a base level of audio editing. In addition, assume that they will do everything wrong - this helps avoid creating unintelligible errors and catastrophic failures when someone does do something wrong.

When you write functions that are public-facing, act like the user is a monkey with a typewriter, who only knows how to press keys to make things work. In reality, a lot of newer programmers are like this in a way, copying code they find online to make things work. This is good for learning, but once the programmer starts adjusting things to fit their needs better, they often make mistakes that even a high-level programmer wouldn't expect. Again, this is fine for the learner, but it is not good for your code. At the very least, check all inputs your code takes, especially in a dynamically typed language like Lua or JavaScript. Never assume the inputs are correct - in fact, assume the inputs are wrong in every single way possible before you start operating on it. This also gives the developer an opportunity to get an error message that actually describes what's wrong - getting an error of program.lua:15: bad argument #1 to 'loadFile' (expected string, got number) is much more understandable to the developer than api.lua:893: attempt to index a number value. You should always prefer to bail cleanly if something goes wrong - be ready for things to fail, and handle them as gracefully as possible with as descriptive of errors as possible (even if this means triggering a panic - just make sure there are no side-effects).

In addition, no matter how good your documentation is, you should assume that the user did not read it. Many times, people will jump into writing code just using autocompletion, rather than looking at the actual documentation of the functions (myself included). This doesn't make it okay to write crappy docs (more on this below), but you should make your API at least decently navigable without reading the docs. This not only helps the lazy be lazy, but it also contributes to keeping the code clean, as it forces you to write more understandable names as described above. The functions should be able to describe the docs nearly as well as the docs describe the function - this means that there should be no surprises when reading the docs for a function by name.

Write Good Documentation

The key to getting people to use your code is to document everything well. If it is not documented, or the docs are hard to read, people will avoid your code even if it is the best choice available. It's hard to use code if you don't know how to use it, and this gets worse as the complexity of the library (and thus the likelihood that a library is better than implementing your own) increases. A sane API extends beyond just having clean code - they also have clean documentation to help the developer as much as possible.

The first part in writing good docs is to describe everything properly. Proper function docs should contain:

A brief, one-line overview of the function
Further description of the function, including any special notes (if necessary)
Descriptions of each argument's purpose with types
An explanation of the return value
Examples of how to use the function (if desired, recommended if the function's complex)
Links to any related functions

This procedure should be followed for every public function in your API. Failure to document a function will result in decreased usage of that function. All documentation should be written in proper grammatically-correct English (or whatever language you choose - English is pretty standard, as is Chinese) as much as possible - use tools like Grammarly to check your docs for proper grammar (you may ignore warnings about using programming words). Native English speakers may find poor grammar in documentation to indicate poor code quality as well, even though a large number of developers speak English as a second language, so ensuring your docs are written well will help native speakers understand it better. Also, be specific about what you're writing about - try not to use pronouns (that, it, etc.) unless it negatively affects readability.

It's also a good idea to include module-level documentation to describe the API as a whole. This is where you write a multi-paragraph description of what it is, how it works, potential use cases, and full-formed examples of how to use the API. These help describe the API as a whole, and gives some more insight into usage beyond single functions. You can also declare the license statement here, as well as author and version.

To assist in writing documentation, I recommend you use a documentation generator such as Javadoc, Doxygen or LDoc. These tools allow you to type your docs directly in the code using comments. This means you don't have to jump to a different file or workspace to document your work, and you can keep everything in one file. This also allows people to read the docs directly in their code editor - in fact, many IDEs include automatic parsing of select doc syntaxes, showing the descriptions in the editor while writing code. Finally, the tool will automatically handle all styling of the resulting web page for you, so you don't need to manage styling or links at all (unless you want to make it look better - standard styles are typically very basic, but usable). To generate a webpage version, all you need to do is (in some tools) create a default config file and pass in the source file(s) you want to generate for, and it'll spit out HTML and CSS with the documentation extracted and formatted, suitable for uploading to places like GitHub Pages.

Writing documentation won't be helpful unless there's a place where developers can easily access it. If you have comment docs, this is a great first step, but it's also helpful to be able to bring up just the docs separately from the code (especially if the source is long). The easiest way to do this is to generate the docs as stated above, upload the docs in a /docs folder in a GitHub repo (preferably the same one with the code), and enable GitHub Pages on the repo (in Settings => Pages, then choose the source branch and path). This will make the docs automatically available at https://your-username.github.io/your-repository/. Once you do this, add a link to the docs to README.md (you have one, right?) and the repo URL field so it's easy to find them. (Note that making docs available as a webpage isn't strictly required - especially if you use Gists/single-file distribution methods - but it's a very helpful resource when deep in using the API.)

An Example

My favorite example of what I consider a sane API is my AUKit library (which I've referenced before). I used all of the concepts I've described when designing it:

I kept the functions consistent: all names are single words, all Audio methods return new values while all aukit.effects functions modify in-place
I kept it as terse as possible while still making sense: the single-word function names describe what they do, functions (mostly) do exactly one thing (and I should probably adjust a few functions to take named arguments as well)
I made it modular: functions with similar purposes are stored in separate subtables, no globals are used besides a defaultInterpolation value in the module table (which can be overridden locally), functions do not produce side effects
I made it fool-resistant: all arguments are checked before use, malformed files trigger a readable error
I wrote good documentation: all functions, methods and fields are fully documented both in inline comments as well as online through GitHub Pages

If you want to see how to follow this guide, you may examine AUKit's code and documentation page - I recommend it as a template of how good APIs look. Of course, this is my own project, so I can't give a perspective from someone else's view, but I'm personally quite proud of it.

Conclusion

Sane APIs are pretty much a necessity if you want to make a successful library. Consistency and conciseness is key when writing a public interface - not following these will cause confusion, making people turn away. Keeping the module modular means your library is easy to simply snap into a program without worrying about conflicts, as well as keeping your own code structure clean. Protecting your code from foolish inputs will make sure that you don't inadvertently cause catastrophe, and that the developer can get helpful feedback on their errors. Finally, writing good documentation for your code is one of the most important things when releasing the API for use by other developers. By following these guidelines, you'll be able to make a public interface for your code that people can look at and think, "That is a very sane API."

On Writing a ComputerCraft OS

JackMacWindows — Tue, 11 Jun 2024 04:00:00 +0000

One of the most common projects I've seen for ComputerCraft is to write an operating system. People look at the limited command-line interface that CraftOS provides, and think, "I want this to work like my normal computer does!" Time and time again, a new post pops up on the ComputerCraft forums or Discord either announcing an OS, or asking for help with an OS, or releasing an OS. Usually, there are some very obvious flaws in these "OS"es, ranging from poor design choices, to overstating what they are and underdelivering. There are many common misunderstandings and undersights that newbie developers run into when writing an operating system, and these end up creating mediocre products at best.

A Critical Distinction

The term "OS" is thrown around a lot, and in my opinion it's very overused. According to Wikipedia: "An operating system (OS) is system software that manages computer hardware, software resources, and provides common services for computer programs." However, much of this is not really possible to accomplish in ComputerCraft. Hardware (peripherals) is automatically handled inside the mod through the peripheral API, and hardware is exposed through a simple object-oriented-like interface. Contrast this to a real system, in which hardware has to be memory mapped or controlled through arbitrary I/O ports; components can have various interfaces that are not compatible with other versions of components of the same type; and access has to be abstracted for effective use. ComputerCraft's simplistic peripheral interfacing API removes most of the need to manage hardware manually. CraftOS does add in some extra code to allow accessing networked peripherals through the base peripheral API, but other than that, what the system already provides is enough for pretty much all use cases.

In addition, since CC uses Lua, a dynamically-allocated, garbage collected language, memory allocation is handled by the internals of the language, and the user does not have to, and cannot, manually manage these resources (outside of collectgarbage, which is not exposed in standard CC). Input and output operations are all handled through the event system, and there is no such thing as probing of input devices or setting up interrupts - every event that could be triggered externally is handled through the coroutine layer, and inputs are automatically translated into a usable format for the computer.

Software resources are slightly more representable in ComputerCraft. The filesystem provided by CC is pretty basic, only giving access to the raw filesystem representation on disk, and not supporting more advanced features such as file permissions and mounts (besides disk drives, which are automatically handled anyway). The terminal interface is also simplistic, and does not natively handle things such as newlines or control codes. CraftOS supplements this by providing write and print functions that handle newlines and the ability to redirect the terminal output to another output "device"/object. Process control is notably missing from CraftOS, since Lua is a natively single-tasking language (though advanced computers do have multishell, which lets you run multiple programs in separate tabs of the shell). However, the existence of coroutines makes it possible to have cooperative multitasking, and through debug hooks it's even possible to set up preemptive multitasking. Even if these services are provided by ComputerCraft, an operating system could override these services and add extra functionality that is missing from the base APIs.

The main part of an operating system that makes sense on ComputerCraft is the presence of "common services for computer programs" - or, as ComputerCraft refers to them, APIs. CraftOS comes with a number of APIs built-in that are loaded on boot, providing services such as value serialization and unserialization, simple painting functions, and a universal settings database. Other OSes may provide extra functionality, including cryptography, UI libraries, automatic event handling, etc. These sorts of APIs can be very useful for developers, and I'd argue this is the most valuable part of a ComputerCraft OS.

However, in practice, many programs billed as "operating systems" provide none of these services. The main goal of developers who make these is to create a desktop environment. They rely on the services provided by CraftOS, and do not extend from it further, besides possibly adding their own globals to access the UI components. These are not operating systems. These are called shells, and should be distinct from actual OSes. A shell is a program that just provides the services of the base OS (in this case, CraftOS) in a human-usable form. The term "shell" is usually used to refer to command-line applications, such as, well, shell. But GUI applications can be considered shells too! Usually, a GUI shell is split into multiple components: a window manager, a desktop manager, a file browser, etc. However, it's still completely valid for a shell to be one application that provides a UI.

There's also a large number of "OS"es that actually are shells - no nitpicking required. These are just replacements for the default CraftOS shell that have a dollar sign instead of a bracket, and maybe some extra programs to make it feel more "Unix-like". At least for GUI "OS"es, a different user interface is implemented - CLI "OS"es are pretty much always shells. (There are some that are not, but we'll get to that later.)

If you decide to make an "OS", consider whether it's actually an operating system. Does it change how the computer is interfaced with? Does it provide any extra APIs/libraries for programs to use? Does it do anything besides change how the user interacts with the computer? If not, your program is a shell, not an OS, and it should not be marketed as such.

You Are Not Original

If you think you've got some neat idea for an OS that's never been done before: throw that thinking in the garbage. If you can think of it, it's (more likely than not) been done before by someone else. Don't start an OS project thinking that it will be special in some way. Uniqueness in an OS is determined by a) its themeing of user interface, and b) the set of features it provides. There's tens to hundreds of "OS"es that provide a user login system - if your selling point is that it has multiuser support, reconsider how you're advertising the OS.

Linux Clones Are Overdone

Adding on to the previous part, one particular type of OS that has been done way too much are Linux "clones". These attempt to make a shell that looks like Bash, and add programs that are named like GNU programs. However, in 99% of these "OS"es, they do not actually clone anything else about Linux, and in most cases they're just reskins of CraftOS. Having two-letter programs, following the FHS, and putting a dollar sign in the shell does not make your "OS" (read: shell) anywhere near Linux.

For an OS to be an actual Linux clone, it would need to have things such as preemptive multitasking, kernel-userspace separation, a syscall layer, virtual filesystem mounts, a full permission system (possibly including ACLs), multiple users, etc. And this is just kernel features - for a proper GNU/Linux userspace you'd need to implement many other utilities that would be expected in /bin, /sbin, etc., as well as a proper Bourne shell (or any other POSIX shell) that supports scripting. This is a big task, and the type of developer that would naively write a "Linux clone" would likely not have the skills to properly accomplish this. They'd probably end up with a mediocre shell that pretends to look like Linux - just like the rest of the Linux clones on the CC forums.

Do note that it is possible to pull off a proper *nix-like OS. VorbaniOS is one that I've seen that appears to do it right. This has many of the features I described above, but it's quite a large project, and would likely take a developer months to get into a properly working state, let alone complete.

Writing An OS Is Hard

Taking on an operating system project requires a lot of time to complete. Since 2014, I have started multiple OS projects thinking that I would be able to get them finished. Perhaps this is a personal issue of motivation, but I believe this is more because I think of a cool idea, but once I get started on it I realize how much time it'll actually take.

To make an OS (and not a shell as previously described), a lot of components need to be assembled. The kernel is the largest part, and you'll likely spend a lot of time working on this, depending on what kinds of features the kernel has. (If the kernel just provides process management, then it won't take long - but you won't have a very good base to build on.) The kernel is also the first thing to work on, since everything builds upon it. Next is adding user libraries, which will consist of UI libraries (if present) and other utilities you'd want to provide to program. The other big portion (at least for OSes with a GUI) will be writing a proper windowing system, which will need to be able to support connections from multiple processes. Finally, you'll need to write programs that can run on your OS, as having an empty desktop or shell won't really be interesting to users.

All of this will require writing tens of thousands of lines of code across many files. In addition to writing code, a significant portion of time will also be spent planning the OS - figuring out how the kernel works, what kinds of services it will provide, what interfaces will be provided to programs, how to manage windows, etc. It might actually take longer to plan the OS than to actually write it. While writing code, you'll likely also realize that some of your earlier decisions were bad, and you want to change how something works. This will take time to rewrite whatever you want to change, as well as rearchitecting anything else that may be affected by the change.

I see very little of this actually done by developers of "OS"es - they just jump in and start writing code, but then they either drop or pause the project, or kind of throw whatever they have together so it looks like it works, but a lot of things are broken or missing. Before you start writing an OS, consider the amount of time it will take. Do you have the motivation to keep working on it, and will you continue to be motivated until it's done? Do you have the time to work on it? How committed are you to having it done in a reasonable amount of time? I've had a very loose idea of an OS for over two years that I've scrapped and rethought multiple times. I know I don't have the attention span to keep working on it continuously, and I don't have any sort of expectation on when I'll have anything done. Consider your ambition for completion before starting.

Do You Really Need A GUI? (aka Why?)

One of the most common features of "OS"es is a GUI, and many users of ComputerCraft have complained about the lack of a windowing system. Therefore, the next logical step for those users is "Well, I'll just make my own!" As stated above, they often underestimate the requirements of such a system, which requires multiprocessing, window management, layering, mouse hit detection, user interface libraries, and much more. But the real question to ask is, "Why do I want a GUI?"

I've seen a lot of users say they want their ComputerCraft computer to work like their real computer. But those users are missing the point of the mod itself. CC's goal is to provide a fantasy terminal that is similar to 80s microcomputers without the complexity of programming that they often had (though BASIC was pretty, well, basic). Even with this goal, CC is quite overpowered for this goal, and it already supports much more than any of those old computers could, even without any sort of graphics mode. Trying to get something like Windows 10 working in CC kind of defeats the point of the limitations. Certainly, it's cool to see stuff like this actually working in such an environment, but arguably it would be easier to use the computer in a basic CLI shell. Perhaps users that have never touched the command line would have an easier time, but in the end it may be better for them to just bite the bullet and learn. (At least CC's design is much easier to get used to than a plain Linux shell.)

Due to the limitations of CC (notably, the lack of proper graphics), it's not very easy to make a proper GUI. At the very least, you end up with a small 51x19 display area to try to show graphical content. If you use the drawing characters from \x80-\x9F, at least you get more resolution and square pixels - but with the caveat that you can only have two colors in any 2x3 area. Navigating this requires some smart techniques and algorithms that are beyond the typical type of user that would want to write a desktop environment.

Desktop-like GUIs aren't the only imitation that users desire - as previously stated, a lot of users try to make Linux "clones" because that's the only CLI they know. Just because a system has a CLI doesn't mean it has to work exactly like popular systems. Sticking to the design of some other system often causes there to be limitations in the design of what you're writing. This is especially true when comparing Linux, which is written in C and compiled to machine code, with CC, which uses Lua and has a lot more reflection and metaprogramming capabilities than compiled languages like C. Instead of trying to make a clone of something, try making something completely new. Use the benefits of the language you're writing with to your advantage! This will make your system much more robust, as programs will be able to use the system to the fullest.

Consider how important it is that you have a GUI. Is it worth the months it will likely take? Critically, do you know enough about how a desktop works that you can reasonably implement one yourself? There's many GUIs out there already too that will probably be better than yours - my favorite one recently being LevelOS, which does the desktop thing right (however, it's got a messy codebase, and it could be better designed - but it works!).

Just Do Something Else

As fun as it is to have an OS on your computer, there are many other things that could be more useful for ComputerCraft. And the types of programs you might include with an OS can, more often than not, be as useful or more useful than if they were only available in the OS. If you're new to programming in CC (or in general), you will pretty much always not be experienced enough to pursue an OS project. If you do have the experience, there are other types of programs that will be more useful. Try making something that works in CraftOS first, as every user of CC will be able to use it, and not just those who have your OS.

For new developers, turtle programs are likely the best first step for getting used to Lua and ComputerCraft. They're pretty simple to navigate, and tasks like moving a distance will introduce you to loops and other constructs. Make a farming script that automatically harvests a plot, collects the crops, replants it, and puts the crops in a chest. Or set up an automatic miner that can automatically collect only ores and returns to a chest when full. If you have a modpack, make an automatic reactor controller script, possibly with a cool UI for touch control on monitors. Try to make something that improves your or someone else's survival gameplay instead of something that's just going to be eye candy at best.

If you're more experienced with CC, and maybe not as interested in the Minecraft side of it (like me), try using the resources available in CC and push them to the limit. Make a music player that automatically manages speakers and can run outside of the locked tick speed of CC - maybe even with stereo support. Create a package manager (of which there are also many) that can handle deep dependency resolution and conflicts, with expandable repository support. My best recommendation is to make a cool game, of which I have not seen very many of. ~~I'm still waiting for someone to create a game using CraftOS-PC's graphics mode, so if you're interested, try giving it a shot - I'll promote the first good one on the front page.~~ (UPDATE: This promotion has now been taken, but I still encourage you to make a cool graphics mode game!)

There's plenty of other projects you could try making that do not have the complexity of an operating system, and are much more useful. Before diving head-first into an OS, make something that's actually useful for other people. It'll help the community much more than yet another sloppily-built UI system.

If You Say So...

To be frank, there are some valid reasons you might want to build an operating system. If you aren't concerned about quality, don't care about releasing it, and just want to give it a shot, then try it out! If it's an epic failure, so be it, and if you end up with a great product, then congratulations. In addition, if you know how to structure an operating system, have the skills required, and can stick to a goal, then making an OS is a fine long-term project. Most of my worries are about people who just think it up and want to jump right in without any preparations. Honestly, a lot of those people are just kids, who like Minecraft, like computers, and have big dreams about the kinds of things they want to make. I don't necessarily want to rain on their parade, but they are trying to compete in a market with those who are much older and wiser than they are, and if they continue without having a reality check, they will get a reality check in the form of comments on a forum post. I'd hope that those comments aren't hateful, but you can never really be sure about it.

If, after reading all of my demotivational ramblings, you still want to try making an OS, there are a few basic rules you should follow to make sure it doesn't suck like most of the other "OS"es out there.

Give It A Structure

For an OS project to be successful, it has to have a structure to it. Having multiple different modules for each part of the system is a must for a proper OS. Keep code that is not related separate, so do not have your scroll bar code inside the kernel (unlike Windows). If using source control like Git (which I strongly suggest), place each module in a different repository so you can keep track of each thing individually. Having organization in your code keeps things easy to maintain so you can find code when you need to access it.

Have A Plan

Just jumping into an OS without actually knowing what you're doing is a 100% guaranteed path to failure. Given the size of an operating system, there's a lot of details that you need to be sure about before you even start writing code. The interfaces provided by each module (and not just user interfaces, but programming interfaces) need to be set in stone before you can move on to dependent modules; otherwise, you end up going back and forth between modules trying to figure out how things work and what things are supposed to do. This is a big time suck, and will prolong the development process greatly. Think of how you want to complete each part before actually starting, as this will remove much hassle from writing the code.

Don't Build The Roof First

A common mistake I see from "OS" developers is that they start writing the first thing they want to see from their "OS" - the user interface. They write the windowing system first without even thinking about multiprocessing, libraries, etc., taking a do-it-as-you-go approach. This is not an effective programming strategy. Doing this leaves you with spaghetti code that does... something...?, and lots of dead code that you wrote for some purpose before, but don't actually need anymore and now it's just kind of sitting there. This strategy also leads to breaking best practices and structure, as you need to go back and forth to build the floor below you - eventually leading to the roof collapsing under the weak walls that were built to support the things you needed from the system.

Don't Pretend To Be Cool

So many new "OS" developers want to make their "OS" look cool, but without doing any of the work. They accomplish this by adding fake loading screens and pauses to make it look like the "OS" is doing something. Don't do this. Fake loading screens and the like may look cool to you, but they cripple usability and make you look like a fool when the user realizes their time is being wasted for the purpose of "realism". ComputerCraft OSes should pretty much never have any sort of loading time, since you have the full speed of the server's CPU, memory, and disk, but you're limited to 1 MB of disk space. (Some tasks do take a lot of processing, though; and HTTP requests can take a long time too.) In short, if your code contains even a single sleep call, it is a waste of time.

Think About The User (And Developer)

An effective OS should be easy to use, not only for the end user, but also for developers who want to write programs for your OS. Having a beautiful UI will draw people in, but poor UX will push them away. Make sure everything in your OS is understandable for users so they don't pull their hair out trying to figure out how things work. Label things well, use proper English (for OSes that target English-speaking audiences), and test every part of your OS to make sure it makes sense to you. If possible, have other people try it out before releasing, and get feedback on what strengths and weaknesses your interface has. The US government has a website at usability.gov that has tips on how to improve user experience. (Even though this targets websites, it can still be a good resource for other types of interfaces.)

Ensuring developers can use your OS's services is important if you want to have an ecosystem of programs. A poor structure will hurt developer usability, but having hard-to-understand code or a lack of documentation will surely cripple anyone who might want to make applications. Providing documentation for every API exposed to the developer is a crucial requirement for developer usability. Otherwise, the developer will have to pick through your code to figure out what things mean. Document not only what APIs do, but also what file formats they use, how they work, and give tips and example code on using the APIs. Think of each API exposed in the OS as if it were independent, and the developer did not know about the rest of the OS. If different APIs interact with each other/the kernel, provide links to documentation on those parts. The developer's experience is as important as the user's experience as long as you want to allow growth on your platform.

Consider Ditching CraftOS

One common failure point I see in various "OS"es and libraries/expansions is that they try to add a bunch of new and useful features while also sticking to the CraftOS APIs. Unfortunately, due to CraftOS's (poor) API design, a lot of things that may seem simple to add are actually not that simple. Take, for example, setting load paths for require. In normal Lua, this is as simple as setting package.path; however, since the shell creates a new copy of package.path and require that does not inherit old settings, and that any changes you make are discarded on program exit, there is no viable way to add a folder to the path without some really dirty hacks. Another thing I see is an attempt to add file permissions to fs, which is something I've done myself, and it often ends up with more dirty code that also breaks API compatibility.

If you really want to make an operating system that is well-designed, you should not try to keep CraftOS compatibility. Creating a new API based around the features you want to support in your OS will make the design of the system so much more clean and stable (as long as you design it well enough). Even though this will limit the number of pre-existing programs you can use, you can always write a shim library to convert CraftOS API calls to your OS's calls - or even use something like YellowBox to create a virtual machine with actual CraftOS. I'm following this design strategy in my upcoming operating system, which at the time of writing is TBA.

Follow Best Practices

Just like there's ettiquete in social interactions, there's a set of best practices that all programmers should follow when writing programs, including when making an operating system. In particular, ComputerCraft coding has certain rules that should be followed to avoid falling into specific pitfalls in CraftOS's design.

Use local variables as much as possible, including local functions. You should avoid global variables as much as possible, as they are slower, aren't automatically garbage collected, and can cause funky errors when other code uses the same variable name. A good OS design will not use any global variables except for a single root namespace where all kernel services are provided - or in some cases, no globals, instead using the coroutine boundary (syscalls) to transfer data.
Indent your code! The rule of thumb is that every time you write then, do, function, repeat, or {, increase the indentation level by 1, and when you write end, until, or }, decrease the level by 1. Your individual code style may dictate some exceptions, but this general rule should always be followed. Not indenting code makes it hard to read, like adding random line breaks or indentation in the middle of a paragraph. Being hard to read not only affects other people looking at your code, but yourself, as you traverse a large file and have to figure out where a block begins and ends.
Give functions and variables descriptive names. For example, button.position.x is much more understandable than b.px. This not helps other people understand how things work, but also helps you understand what things do a month from now when you forget how everything works. Some things, like iterator variables, may be alright to shorten, but remember that less ambiguity will make it easier to understand what the meaning of the variable actually is.
Avoid os.loadAPI - use require instead. os.loadAPI is deprecated, creates unnecessary globals, and does not work with relative paths. It also does funky things with the environment of the API, and can cause a lot of things to break. You should replace all usages of os.loadAPI with require, which is much better and is compatible with normal Lua. To summarize how to do this, simply return a table of functions at the end of your API to make it compatible with require (and also convert any global functions to locals), and use local myAPI = require "myAPI" to load it.

Conclusion

Making a ComputerCraft operating system is one of the most popular projects in CC, but most of these "OS"es are bad for many reasons; some of those reasons being poor design, a lack of structure, or just plain not being an OS, but a shell. New users think of a cool idea that they want to try (an OS), not realizing the scope of the project and the kind of dedication it actually requires. Some just want to be able to have a system like their normal computer, but they never really question why they want it, and they don't find the proper OSes out there that accomplish this goal without them having to write a single line. And for those who just want to go ahead anyway, a list of guidelines that developers should follow when making an OS will keep them from falling into the same traps that other "OS" developers fall into. If you want to make an OS in ComputerCraft, think about how much you actually want to dedicate into writing it, and what your exact goals are. More often than not, it may be better for novice users to use an already existing pre-built system like LevelOS or Opus OS, or - hear me out - just plain old CraftOS.

Running Lua C modules in a pure Lua environment (1)

JackMacWindows — Sun, 02 Jun 2024 00:07:40 +0000

Most desktop scripting languages are incomplete without a way to add extra functionality from native code. Modules written in C (or other compiled languages) are used to add extra functionality, expose system APIs, import high-performance libraries, and more. Lua stands out as a language that was built with C extensibility in mind - it has a simple C API which works both to extend the Lua environment, as well as to create the environment itself. As such, there's a wide ecosystem of Lua modules written in C, often wrapping system libraries in Lua APIs.

However, being targeted at embedded applications, developers often remove the ability to load C modules in the Lua environment, for security and/or compatibility reasons. This can wipe out a whole bunch of useful modules that aren't available in pure Lua form, such as complex compressors. My favorite Lua environment, ComputerCraft, is one of those - since the VM, Cobalt, is written in Java, it can't load C modules without a huge amount of work across the JNI boundary; and loading arbitrary code on a shared server is super insecure anyway.

For no particular reason, I wanted to be able to load these C modules somehow - I already managed to run C DOOM on CC in a RISC-V VM, so there had to be a way to load the modules. I had a few main goals in mind:

The loader has to be able to load the modules unmodified - this meant following the standard Lua loading process of calling a shared library's luaopen_* function.
The module code has to be compiled unmodified, using whatever compile steps the author provides (besides configuring it to target the right compiler). This means implementing the Lua API, plus any system APIs required (such as libc).
The module should integrate seamlessly with the rest of the outside Lua environment - all Lua values should be stored in the same environment that's loading the module, and not inside the VM (i.e. using another Lua VM on the C side, then copying the values over).

My devious solution: implement a RISC-V VM that emulates a Linux system; implement the Lua API in Lua and expose it to the VM; dynamically link and load the shared module file into the VM; and execute the functions required as one-shot calls on the VM.

RISC-V

If you're not familiar, RISC-V is a new-ish CPU architecture that focuses on being simple to implement, but highly scalable up to desktop architectures. It has a grand total of 40 instructions in the base 32-bit integer instruction set, and uses a load/store, register-register RISC ISA.

Back when working on DOOM-CC, I chose it as my target because of its low instruction count. I only needed to implement 48 instructions (integer + multiplication extension) to have a fully working system. It also has strong compiler support in GCC, so I didn't need to fiddle with finding a compiler and runtime library. Because I already had the emulator core written, I figured I'd just scoop it out and bring it over to this new project.

Since I didn't write about the DOOM project, I'll quickly go over what I remember of the development process for the RISC-V VM. I first spent some time studying the instruction formats. RISC-V has four core instruction formats (R, I, S, U) and two modified formats (B, J), for which I wrote a decoder function for each. All formats have a 7-bit opcode field, for which each opcode maps to a certain format. To handle this, I made a table mapping each opcode to the decoder function, and the clock function takes the opcode, finds the decoder function, and passes the decoded instruction to the execution function.

After that, I wrote a function for each instruction. The instruction table is indexed by the opcode first, but some instructions are further selected by the funct7 field in the instruction, so those instructions are further indexed by funct7 to get the executor function. Each function takes the CPU instance (self) and the decoded instruction, and executes the instruction requested before returning to the tick function.

For expediency, I decided to use LuaJIT and its FFI library to store the CPU's memory, which let me have views for 8-/16-/32-bit values over the same memory block, and is much faster than normal tables. This did mean that the code only worked under LuaJIT environments, but I could easily replace it with tables later once I wanted it to work with normal Lua.

Once the instructions were initially written, I needed a way to load binaries into the CPU for testing. I wrote up a little ELF loader that would simply read the required file header fields (like entrypoint), and copy all of the program headers into the memory space. (More on ELF later.) After that, I started running the CPU tests provided by the RISC-V authors. A lot of tweaking was required, but eventually I got it to pass all the tests.

Finally, I had to implement some way to access the CC environment from inside the VM. I used the RISC-V ECALL instruction to do this, which is also used by Linux for system calls (which will become relevant later). The function number is passed in register a7/x17, and arguments in registers a0-a6/x10-x16. I made a table of functions for this as well, with each of them implementing some function like drawing the graphics to the screen, or accessing files on the filesystem. After a bit of tweaking, I got it working... albeit, at such a low framerate that it's completely unplayable (even with LuaJIT!).

Lua in Lua

The first part I worked on was implementing the Lua C API. The essence of it is that all of the state is stored in Lua outside of the VM as native objects, and a shim C library calls out to the Lua side to do whatever operations are necessary. API calls are made through a single ECALL/syscall with ID 0x1b52, and the function to call is the first argument.

On the Lua side, each lua_State object is stored in a table, with the coroutine it applies to, the CPU object for the system, and the "stack" which stores all of the values that the API uses. There is one lua_State object at the start, which represents the main/current calling thread; and the pointer passed to functions is actually a number indexing into the table. Because some value types that are accessible to the C API aren't representable in plain Lua (like C functions and userdata pointers), those types are wrapped in tables (as well as tables themselves, so that it can differentiate between wrapped types and table values).

On the C side, most of the functions are simple wrappers around syscall, casting types as necessary. However, some functions return memory blocks or strings, which Lua can't natively create - instead of expecting Lua to allocate the memory region, the C function allocates the necessary memory, then passes the pointer to Lua to fill up. There's also a few functions that only use static pointers, so those are entirely implemented in C.

Here's a brief list of issues I ran into while testing, and how I fixed them:

Lua uses negative numbers to indicate stack positions relative to the top. However, I forgot that the RISC-V VM only deals with unsigned numbers. I had to add an extra function call to convert unsigned numbers to signed numbers before using the index.
There are also certain negative indices that point to special values - LUA_REGISTRYINDEX (-1001000) points to a secret global table, and indices below that point to upvalues in a C function. I eventually had to implement those to be able to use most modules.
Lua uses double floating point numbers for its native number type, but because I didn't implement the floating-point extension for RISC-V, doubles are passed in two integer registers, as according to the ABI. I had to make a quick set of functions to convert the two integers into a native Lua number, which used string.[un]pack to reinterpret the two integers as a double.
Moving values between tables and the stack involves wrapping/unwrapping the stack values. I had to make a few functions to do that sanely, and it also required rewriting some code I had previously to use them instead.

Setting up a cross-compiler

Now that I had some C code ready, I needed a compiler to turn it into RISC-V code. While I had a compiler set up for compiling 32-bit RISC-V already, it used the embedded Newlib C library and didn't target Linux systems. Since I wanted to have off-the-shelf modules run unmodified, I wanted to use the canonical C library for Linux, glibc, on top of Linux syscalls. This required building all of those components from scratch in a proper cross-compile toolchain.

I first looked through the Arch Linux AUR to see if anyone had anything for it, but only 64-bit toolchains were available. I wanted to avoid using super heavy toolchains anyway, since I only implemented the IM instruction set, while most Linux toolchains use the full GC/IMAFDCZicsr_Zfencei instruction set. I then proceeded to start building a cross-compiler by hand. I cloned the latest versions of GCC, glibc, and dependencies, but I soon realized that I was in over my head a bit - compiling it manually was gonna be pretty tough.

I then found that RISC-V has a build script for GCC+glibc already available, so I went and cloned that and built it. I had some issues with cloning one of the submodules (it was pointing to an invalid commit), but I was able to get around it by cloning each submodule individually, besides the one that was having problems (I didn't need it). Then I built the toolchain with:

./configure --prefix=/usr/riscv32-linux-gnu --with-arch=rv32im --with-abi=ilp32
make linux

The build went pretty quickly on my R9 5950X CPU. However, shortly into the glibc build process, I was greeted with an error - glibc requires the A (atomic) extension on RISC-V. No problem, I could implement that once required. After switching the arch to rv32ima, it finished successfully, and I was ready to build the project.

I first went to build "liblua" into a static library, which was just a few gcc -c commands to build the object files, and then an ar to pack them into liblua.a. I then tried to build a test module file against this library. My original plan was to build everything statically, so I wouldn't need any dynamic linking. I wanted to simply load the code from the module, find the symbol address in the symbol table, and then execute it. But no matter what I tried, I couldn't statically link glibc because the static library wasn't built to work with shared library files:

/usr/local/lib/gcc/riscv32-unknown-linux-gnu/13.2.0/../../../../riscv32-unknown-linux-gnu/bin/ld: /usr/local/sysroot/usr/lib/libc.a(malloc.o): relocation R_RISCV_TPREL_HI20 against `a local symbol' can not be used when making a shared object; recompile with -fPIC
/usr/local/lib/gcc/riscv32-unknown-linux-gnu/13.2.0/../../../../riscv32-unknown-linux-gnu/bin/ld: /usr/local/sysroot/usr/lib/libc.a(sysdep.o): relocation R_RISCV_TPREL_HI20 against `__libc_errno' can not be used when making a shared object; recompile with -fPIC
/usr/local/lib/gcc/riscv32-unknown-linux-gnu/13.2.0/../../../../riscv32-unknown-linux-gnu/bin/ld: BFD (GNU Binutils) 2.42 assertion fail /home/jack/Downloads/riscv32-gcc/riscv-gnu-toolchain/binutils/bfd/elfnn-riscv.c:2565
collect2: fatal error: ld terminated with signal 11 [Segmentation fault], core dumped
compilation terminated.

I was able to successfully build a shared library file for the module with the dynamically linked glibc file - but this now meant I had to write a dynamic linker.

The next post in the series will describe how dynamic linking works, and what I had to do to implement a dynamic linker for the virtual machine in Lua.

libcert: Implementing S/MIME PKI in Lua (2)

JackMacWindows — Wed, 17 Apr 2024 07:22:42 +0000

This is a continuation of my post from yesterday about libcert. Read it if you haven't - this jumps right in after the end of that post.

PKCS#7/CMS: Container for signatures (and other data types)

After getting the initial signature generation and validation working, I wanted to expand it to use a standard container format, instead of making my own blob thing. A bit of searching brought me the PKCS#7 standard; however, I soon found out that the original PKCS#7 was obsoleted by the Cryptographic Message Standard (CMS), which extends PKCS#7 for... something - I haven't dug deep enough to see exactly what's different. CMS is described in RFC 5652, which describes the format, and has the ASN.1 reference I needed, plus all the object IDs that are relevant for implementing the format.

The PKCS#7/CMS container doesn't just store signatures, however - it's also meant for storing hashes (or digests), encrypted data, enveloped data (which is encrypted data, plus the key encrypted one or more times, so multiple receivers can decrypt the same data using their own keys), authenticated data (e.g. HMAC, Poly1305), and authenticated enveloped data (which I would implement later on). To allow for future expansion, I decided to implement all of the data types, which meant copying the seven-page reference into Lua. And while I was working on copying the structures, I realized that I wanted to add type annotations as well - which meant copying all of the types again, to create virtual class definitions next to each structure. The library soon blew up to 700+ lines, but I eventually got it all written out.

Once I had the definitions written, and after some fiddling with the ASN.1 library, I had functions that could load and save PKCS#7 containers. I replaced the old signature object in the test script with a PKCS#7 SignedInfo structure, and tried signing and verifying, and it appeared to work. However, I wanted to try verifying my signature with OpenSSL, to make sure my file was in the right format. I ran openssl cms -verify on the file I was testing with, plus the signature extracted to a new file and the certificate I used in Lua; but I got an obscure error instead (not surprising coming from OpenSSL).

Some searching on the matter brought me to a pull request which implements Ed25519 signatures for openssl cms - apparently the signature format I need to use isn't quite supported yet in the main release. At least there was a fork with support, so I cloned the fork and built it. Once it was built, and after discovering that it would only load the new libraries with LD_LIBRARY_PATH, I got a different error - it was reporting that it had an unsupported algorithm, despite having the Ed25519 fork.

My first diagnostic was to generate a signature with OpenSSL itself, and then see what the differences were. Comparing the files in asn1js showed a different signature scheme - whereas I was generating the signature directly from the file contents with no signed attributes, OpenSSL stored a hash of the message as a signed attribute, and then signed the signed attributes object. I didn't want to have to do this method because it seemed like more work, but after perusing the OpenSSL source a bit, I found that it has two different code paths for these types of signature generation, and the Ed25519 PR only implemented the one with the signed attributes (for technical reasons related to OpenSSL internals). I then changed up my code to use signed attributes, which made OpenSSL accept the file, but it still failed verification. Not only that, but my test signature from OpenSSL was also failing. This had me really confused, so I spent some time debugging OpenSSL itself to figure out exactly where it was failing.

Through a lot of debugging, I eventually found that the encoded attributes OpenSSL was verifying were different from the one I signed. I then read in the X.690 standard that in DER encoding, the ASN.1 SET OF type, which is used to store the list of attributes, has an explicit encoding order - it sorts each entry by byte ordering for the encoding of each entry. Because I didn't do this sorting, but OpenSSL did, the DER encoding that I was signing didn't match the encoding that OpenSSL was verifying. I implemented some sorting in the ASN.1 library, and finally the file verified. (I also managed to fix verifying the test file after realizing that I was signing in text mode, but verifying in binary mode.)

Quick-and-dirty certificate trust chain

Now that I had a standardized signature format working, the first thing I needed to do to make the signatures actually secure was to implement a chain of trust. Just checking the signature with the provided certificate isn't enough - an adversary could easily swap out the signature and certificate with their own certificate. The signature of the certificate itself needs to be verified with its issuer, up until a trusted root is found.

My original plan was to have a daemon process that handled the certificate database, including the list of root certificates. A program would hand over the signature blob through IPC, and the daemon would check the signature using its root certificate list as the root of trust. However, it's really easy to make a program that spoofs system calls (including IPC) in Phoenix. After discussing it with others, I came to the conclusion that I should just leave it in a library, as OpenSSL does.

After some reorganization of my library, I started working on the chain verifier. It turned out to be a lot simpler than I expected. The verification process is a simple recursive algorithm that does the following steps:

Look for the issuing certificate in the list of all known certificates, which includes root certificates, plus other certificates from a container that may be needed to verify (including intermediate CA certs).
If the issuer couldn't be found, fail.
Encode the inner toBeSigned certificate object to DER, and check its signature using the public key of the issuer. If the signature is invalid, fail.
If the issuer certificate is in the list of root certificates, verification succeeded.
Otherwise, if the issuer is the same as the current certificate (i.e. the cert is self-signed), fail to avoid an infinite loop. (I later replaced this with checking a list of visited certificates, to avoid looping in cross-signature scenarios.)
Repeat from step 1 using the issuer certificate.

I then generated a couple more certificates and keys to test certification paths, and placed the original self-signed root into /etc/certs, which stores the root certificates. I also adjusted my PKCS#7 signature function to allow adding extra certificates, which is needed to carry required intermediate CA certificates with the leaf certificate. Surprisingly, my code worked first-try, with it successfully validating signatures signed by the root, an intermediate CA (issued by the root), and a leaf certificate (issued by the CA).

Implementing symmetric encryption

My next task was to tackle implementing encrypted PKCS#8 private keys. I originally wasn't planning on implementing encryption in the library, but after the reorganization, I decided to add in not only PKCS#8 decryption, but also PKCS#7 authenticated enveloped data.

My original intention was to only use ChaCha20+Poly1305 for encryption, which is fast, and is also the most prominent symmetric cipher available in Lua. However, I soon found out that there's no standard provisions for using it in anything except PKCS#7 as an authenticated encryption cipher - neither PKCS#8 key encryption, nor the key encryption used in PKCS#7 enveloped data, support it; they only accept AES, or the incredibly discouraged DES. Luckily, a ComputerCraft coder had recently written an AES library in Lua, which I was able to slightly modify to work in normal Lua. To be frank, I'm not sure how secure this library is (it did use a lot of global variables...), but it's the best I had, and I didn't want to spend another two days writing an AES library myself.

Unfortunately, PKCS#7 and #8 don't describe how the algorithms should actually be implemented. I first found RFC 3565, which describes how AES can be used as an encryption algorithm, including its OIDs. I then checked a password-encrypted PKCS#8 key generated with OpenSSL in an ASN.1 viewer, which said that it used an algorithm called PBES2. A search brought me to PKCS#5, standardized in RFC 8018, which describes the Password-Based Encryption Standard 2 (PBES2), plus the Password-Based Key Derivation Function 2 (PBKDF2), which I was already familiar with. The idea is that the encryption key is generated by running a pseudorandom function over the input password and a random salt thousands of times, and then encrypting with some cipher using that key. The PRF is usually HMAC from a SHA family hash function, and the cipher is usually AES in CBC mode. (Both are explicitly documented in PKCS#5.)

I then went to work implementing PBES2/PBKDF2. PBKDF2 was easy to implement - it's pretty common in ComputerCraft, and I was able to extract the implementation from a popular SHA-256 library. I needed to do a bit of finagling to get the PBKDF2 function, the SHA library I was using, and the AES library to all play nice together - some functions used lists of bytes, while other ones used strings. I sketched up a quick PKCS#8 key generation function, but didn't end up testing it out until later.

PKCS#7 2: Encrypted Boogaloo

Since I was already working on encryption, I went to work on implementing PKCS#7 auth-enveloped data after PKCS#8. RFC 5083 describes the data format, since it's an extension of the original CMS format, but it's basically a mix of the AuthenticatedData and EnvelopedData formats, so I mostly read the CMS standard. Enveloped data uses a unique scheme where, instead of relying on the receiver to have the encryption key for the data, it generates a unique key for the data, and then stores that key encrypted for the receiver. This allows the same encrypted data to be sent to multiple receivers, who all have different keys to decrypt the data and its key. This did mean that I had to use AES again, since again, ChaCha20 isn't supported for encrypting keys.

PKCS#7 defines four key wrapping methods:

key transit, which uses normal public-private key asymmetric encryption;
key agreement, which uses a key exchange algorithm between two key-certificate pairs to create a shared symmetric key;
key-encryption key, which uses a pre-shared key to symmetrically encrypt the key; and
password-based key, which uses a password to symmetrically encrypt the key.

I wanted to implement all of these, in case I or someone else eventually needs them. However, because I am using elliptic-curve asymmetric ciphers, which don't support encryption directly, I couldn't implement key transit; but I still implemented the other three. To support all of these different methods in the same encryption function, I had a function for each method, which returned an object to encrypt and decrypt keys into an abstract RecipientInfo ASN.1 object. The main encryption functions can transparently call the object to convert each RecipientInfo in the envelope to and from the key.

The first one I implemented was key agreement. This uses the X25519 key exchange algorithm, which takes the private key of one side and the public key of the other side, and returns a key which is guaranteed to be equivalent on each side. The constructor for the key encryptor takes a private key for the current side, plus the certificate for both sides. When encrypting a key for sending, it exchanges the private key with the receiver's certificate's public key, encrypts the key with the exchange key, and returns a new RecipientInfo object that stores references to the sender and receiver's certificates with the encrypted key. Decryption checks that the sender and receiver in the RecipientInfo match the given certificates, then exchanges the private key with the sender's certificate's public key to create the same exchange key, and finally decrypts the key with that exchange key.

Next was the key-exchange key method, which is fairly simple: it just runs AES on the encryption key using the given key. It also adds in an identifier for the key, which allows the receiver to identify the proper KEK info to check. The last one was the password method, which functions a lot like PKCS#8 PBES2. The encryptor generates a salt, and then runs PBKDF2 on the password and salt to generate a key, which is then used to encrypt the encryption key. The decryptor runs PBKDF2 on the password with the salt given in the RecipientInfo object, and then decrypts the encryption key using the PBKDF2 key.

Then I wrote the main encryption routines. The encryption function takes data with a list of key encryptors, and returns a PKCS#7 authenticated enveloped data structure. It works by generating a random key and nonce value, and then encrypts the data using ChaCha20. It also generates an authentication tag using Poly1305, which is important to verify the origin of the data. Afterward, it uses each encryptor object to generate a RecipientInfo for each recipient/key type, which are packed into the PKCS#7 structure, and returned to the user.

The decryptor takes the PKCS#7 structure and the same key encryptor list, and returns the decrypted data. This function first does some sanity checks on the structure, and then runs each RecipientInfo in the structure through each given encryptor to try to decrypt the encrypted key. Once a valid key is decrypted, the data is decrypted using the nonce stored in the structure, the authentication tag is checked to verify authenticity, and the data is returned.

Once again, I've been writing this for way too long, it's super late and I don't feel like writing more today. I'll pick it up again later with my struggles while testing, as well as PKCS#10 CSRs.

libcert: Implementing S/MIME PKI in Lua (1)

JackMacWindows — Tue, 16 Apr 2024 06:11:32 +0000

So, this is my first status update post to dev.to. This one's gonna be a bit longer than I'm aiming for for these types of posts, since I have to explain my entire project so far for any of my current progress to be understandable. But I've got plenty to talk about, so let's jump in.

Abstract

This section's a bit of an overview of my goal for this project. As part of my Phoenix operating system, I want to have some sort of code signing for applications, a lot like how macOS has code signing. Now I don't necessarily see it as being as strict - I'm not implementing a web service that checks validity, or enforcing code signing from my root on every single executable - but I'd like to at least toy around with the idea of a secure system, just to see whether/how it works.

One of my underlying principles in Phoenix's design is to reuse common protocols and formats as much as possible. That's why I'm using POSIX standards for the command line utilities, and dpkg/.deb for packaging. For cryptographic routines, I wanted to use a public key infrastructure design that was already commonplace. I could have used GPG/PGP, which is pretty common in the Linux world, but it's not as flexible - it lacks certificates and signing paths, which is pretty important for having distributed code signing in the way I imagined. (I also don't think it supports elliptic curve cryptography, which is the only type of encryption supported in CC at the moment - I really don't want to have to toil over making my own RSA implementation.) I liked the idea of X.509 certificates as used on the internet, and OpenSSL is commonly used to handle that, so I decided to pick that group of formats (which I soon learned is officially called S/MIME).

Initial research

I already knew some basics about X.509 certificates - they store a public key along with some metadata in an ASN.1 DER container, which is then signed by a certificate authority as assurance that the certificate is trusted. The certificate authority is then signed by another certificate, up to a root of trust that is stored in the operating system or security library like OpenSSL.

I had some experience with ASN.1 encoding and decoding before this - I wrote a script that extracts encryption keys from iOS update packages a long time ago, which involved parsing a DER-encoded keybag - but I hadn't actually worked with ASN.1 definitions before. At first, I started writing my own decoder for DER structures, referencing the Wikipedia article on DER, supplemented by the very thick X.680 and X.690 standards. I made it through a decent number of the tags, but eventually I got tired of working on it, so I searched to see if someone else had already done the work of making one. Fortunately, I found one. Unfortunately, it was very barebones, and lacked most of the tags I needed. To remedy this, I ported some of my code I'd written to this new one, filling in tags like object identifiers, which are extremely important in S/MIME. I also added stuff that really should have been there, like sets, optional values, and implicit tags.

One thing that was really nice about this new library is that it's structured - instead of just blasting the decoded data into a list of unnamed entries, it follows a structure that you give it, and not only names each entry as it reads them, but also can do type checking to make sure the structure it's reading in is correctly written. This ended up becoming extremely important, because I would have had quite a tough time with optional values and just general usage of these types if I had stuck with my own plan. It also meant that I could just slap a type definition on the resulting object without having to fiddle with figuring out what each entry in an array means manually.

Another benefit of this format came when I started reading the X.509 standard. The 236-page document would be ridiculous for me to read all of for this one little project, but one little section, Annex A, has a breakdown of every part of the X.509 certificate format, written in ASN.1 structure format. These were also included in a very useful blog post here, which let me know what each field meant. All I had to do was rewrite those definitions into Lua code, and the ASN.1 library would convert those definitions into an encoder and decoder for that type. I then spent many hours carefully transcribing those definitions into Lua, which, because of how the library was built, was surprisingly straightforward and elegant.

(An example of a definition ported to Lua:)

local TBSCertificate = asn1.sequence {
    {"version", asn1.default(asn1.explicit(0, asn1.integer), 1)},
    {"serialNumber", asn1.integer},
    {"signature", AlgorithmIdentifier},
    {"issuer", Name},
    {"validity", Validity},
    {"subject", Name},
    {"subjectPublicKeyInfo", SubjectPublicKeyInfo},
    {"issuerUniqueID", asn1.optional(asn1.implicit(1, asn1.bit_string))},
    {"subjectUniqueID", asn1.optional(asn1.implicit(2, asn1.bit_string))},
    {"extensions", asn1.optional(asn1.explicit(3, asn1.sequence_of(Extension)))}
}

However, one roadblock I ran into was figuring out what those darn object identifiers were. The X.509 standard makes no mention of any IDs for any purpose, including signature algorithms, public key algorithms, extensions, attributes, etc. (Rightfully so, it defines the certificate format, not what you put in them.) This meant I now had to run around finding these obscure identifiers for each purpose. I hit gold when I found some documentation for a Python crypto library, which had most of the IDs I needed. But it didn't have all of them - most notably, the OIDs for Ed25519 signature keys, which is the only signature algorithm I had access to in Lua. So, I went hunting for these OIDs online, and after an accidental detour through the unrelated RFC 5480 (which describes general-purpose EC curves, and not Ed25519), I found what I needed in RFC 8410. Once I put those all together, and after putting together a very quick and dirty PEM decoder (which is just Base64), I was able to load and print X.509 certificates, with all the keys named and everything.

I'd like to shout out asn1js, a super useful tool for reading out ASN.1 structures in the browser. I was able to use it to figure out why certain structures were getting encoded or decoded wrong, and its breakdown of structures into both a hierarchy and the hex view were very handy. I also appreciated its ability to annotate the field names using built-in structure definitions.

PKCS#8: Private Keys (and initial signatures)

Next was being able to use the private keys that go along with the certificates. You can't sign anything without a private key. From the Wikipedia page on X.509, I learned that the appropriate standard for private keys is PKCS#8 (or PKCS#1 for RSA, but I'm not using RSA, so it's irrelevant). Clicking through references brought me to the standard, RFC 5208. Luckily, it was just two structures (PKCS#8 is really simple), but I ran into a little roadblock: I had no idea how to encrypt the key. I'd eventually get back to this later, but for the time being I just left the encrypted key part as a stub.

Once I had loading certificates and keys working, I decided to take a crack at signature generation. I wrote up a quick script to read in a certificate and key file (generated by OpenSSL), and then write a signature blob (encoded in ASN.1/DER) with the Ed25519 signature over some file. Then it would read in the signed file, split off the signature blob, and then verify the signature using the certificate in the blob. (No chain validation yet.) After a good bit of debugging, I got it working:

Anyway, this is getting way longer than I was thinking at first - I've been writing this for over an hour. I'm gonna continue this more tomorrow, with my journey through PKCS#7, plus chain validation of certificates, and maybe PKCS#10 if I have time.

Hello, dev.to!

JackMacWindows — Tue, 16 Apr 2024 04:55:43 +0000

Hi, I'm JackMacWindows, a C++ and Lua programmer who focuses on embedded and low-level system programming. I've been coding for over 10 years, starting with C++ as an elementary schooler. (That was quite a choice for young me!) Since then, I've learned a lot about all sorts of programming and computer topics, from compression algorithms and networking layers, to music synthesis and video transcoding.

My main passion right now is working with the ComputerCraft mod for Minecraft, which adds a flexible Lua-based fantasy terminal to the game. I'm most known for CraftOS-PC, a cross-platform desktop and mobile C++ app which recreates the ComputerCraft environment outside Minecraft. I'm currently working on writing an operating system for it called Phoenix, which aims to implement a full-fledged POSIX/Linux-like environment all in Lua.

Outside of ComputerCraft coding, I also like to make various little utilities, apps and devices. One of those was DashBot, a genetic machine learning bot that plays Geometry Dash. iRCON, an iOS app for remotely administering Minecraft servers, has been on my mind recently, and I'm contemplating putting it on the App Store. I also gained some notoriety in the game music ripping scene for my UnkrawerterGBA project, as well as my work on ripping music from the Club Penguin DS games. My favorite non-ComputerCraft project recently has been my MIDI synthesizer/sound generator, which taught me a lot about developing for microcontrollers.

I'm planning on using this page as a blog/journal of things that I'm working on. I've wanted to have a place to be able to dump my mind after a good day of coding, and dev.to seems like a nice place to o that. Expect my posts to usually be more of a ramble about things, but every so often I might post a longer-form "essay" of sorts about a certain topic.