Motivation

Current NixOS integration tests rely heavily on QEMU, which can be slow and resource-intensive. This PR introduces systemd-nspawn as a lightweight container backend, significantly reducing test latency and enabling hardware passthrough scenarios that are difficult to achieve in VMs.

Key advantages of container tests

• Faster boot times and lower overhead
  ~25% improvement in test execution speed on Intel(R) Core(TM) Ultra 9 285HX

  benchmark of 24 machines running GNU hello

  
  
  nix-build test-backends-benchmark.nix -A hello-nspawn  27.76s user 1.50s system 155% cpu 18.803 total
  
  nix-build test-backends-benchmark.nix -A hello-qemu  36.80s user 1.92s system 140% cpu 27.548 total

• Container tests can be run in cheap VMs instead of bare-metal machines.
• Containers allow direct bind-mounting of host device nodes. This enables integration testing for CUDA code within the NixOS test framework.
• The implementation (see below) lays the groundwork for other machine backends (other container infrastructures, bare-metal etc.)

Try it out (2 simple steps!)

1. Configure the nix daemon to allow running systemd-nspawn:

 nix.settings.auto-allocate-uids = true;
 nix.settings.experimental-features = ["auto-allocate-uids" "cgroups"];
 nix.settings.extra-system-features = ["uid-range"];
 nix.settings.sandbox-paths = [ "/dev/net" ]; # to make nspawn↔qemu networking work

2. Run a container test, either

• nix-build -A nixosTests.nixos-test-driver.containers: Basic startup and inter-VLAN isolation (VMs and containers in parallel).
• nix-build -A nixosTests.test-containers-bittorrent: Complex networking (NAT/UPnP) involving multiple containers.

Implementation details

• Refactor the Python test-driver to move QEMU-specific logic into a QemuMachine class (inheriting from an abstract BaseMachine class).
• Introduce an NspawnMachine class replicates as much functionality of QemuMachine as possible using systemd-nspawn containers.
• Use the existing QEMU networking options (virtualisation.vlans) for containers.
• Bridge QEMU's VLANs to the containers, enabling VM↔container networking.

Debugging

To debug a failing container test, introduce enableDebugHook and sshBackdoor like so:

# nixos/tests/test-containers-backdoor.nix
{
  name = "containers-backdoor";

  containers.machine = { };

  sshBackdoor.enable = true;
  enableDebugHook = true;

  testScript = ''
    start_all()
    machine.succeed("false") # this will fail
  '';
}

The test will then print an ssh command on startup:

machine:  ssh -o User=root -o ProxyCommand="socat - UNIX-CLIENT:/run/systemd/nspawn/unix-export/machine/ssh" bash

Upon failure, the test will print a command to attach to it, e.g.

!!! Breakpoint reached, run 'sudo /nix/store/hb1v3cz5bd6qk8arhxy6bii0wcilg8wh-attach/bin/attach 6793584

Run this to get a shell inside the sandbox, where you can run the ssh command above to enter the container.

Note: Due to the nature of systemd-nspawn, interactive execution of the tests requires root privileges: sudo $(nix -L build --print-out-paths .#nixosTests.test-containers.driverInteractive)/bin/nixos-test-driver --interactive

Limitations of containers

• You cannot test kernel-specific changes (e. g. kernel modules) without also having them active on the host.
• Containers running in the Nix sandbox cannot run setuid wrappers (like sudo—though you can use runuser instead).
• Container tests do not support graphical applications (and taking screenshots of them).
• Containers running in the Nix sandbox don't allow many of the systemd hardening options (ProtectSystem= etc.) used by NixOS modules such as services.transmission among many others.
• Containers running in the sandbox have limited access to /dev, making it necessary to pass in needed paths, e. g. --option sandbox-paths /dev/net for VPN tests that create /dev/net/tun.

Credits and history

Based on the testing infrastructure from @clan-lol.
The heavy lifting for integrating this into nixpkgs was done by @jfly.

Things done

• Built on platform:
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• Tested, as applicable:
  • NixOS tests in nixos/tests.
  • Package tests at passthru.tests.
  • Tests in lib/tests or pkgs/test for functions and "core" functionality.
• Ran nixpkgs-review on this PR. See nixpkgs-review usage.
• Tested basic functionality of all binary files, usually in ./result/bin/.
• Nixpkgs Release Notes
  • Package update: when the change is major or breaking.
• NixOS Release Notes
  • Module addition: when adding a new NixOS module.
  • Module update: when the change is significant.
• Fits CONTRIBUTING.md, pkgs/README.md, maintainers/README.md and other READMEs.

Add a 👍 reaction to pull requests you find important.