Thanks to #71684 being merged recently, we do now have a test driver implementation in Python that allows for writing the imperative part of NixOS integration tests in Python instead of Perl.

Thanks @flokli @edolstra @Lassulus @adisbladis @domenkozar @garbas @jonringer for the support and @blitz @jtraue @marijanp for helping me porting the remaining Perl tests to Python.

In order to finish this project, we are going to perform the following tasks over the next weeks/months:

• Perl and Python Test driver feature-parity
  • implement dumpTTYContents Port remaining test-driver functions to python #72943
  • implement forwardPort Port remaining test-driver functions to python #72943
  • implement getWindowNames Nixos integration test ports #72835
  • implement requireActiveUnit Nixos integration test ports #72835
  • implement waitForWindow
  • implement waitUntilTTYMatches
  • fix emptyDiskImages behaviour nixos test driver doesn't work with multiple emptyDiskImages #73559
  • Collect kernel coverage data after test finished (as per nixos/qemu-vm: Option to use squashfs Nix store closure instead of virtfs access to host's store #72354 (comment)), removal PR nixos/testing: remove remaining coverage-data logic #97231
• Document all Python functions with docstrings
• Remove all global state/singletons from Python test driver (e.g. global log variable)
• Port all remaining Perl tests
  • cockroachdb nixosTests.cockroachdb: port to python #73934 nixosTests.cockroachdb: Port to python #80047
  • docker-containers broken: port nixosTests.docker-containers: Port to python #80049 (was renamed to oci-containers in the meantime and fixed)
  • docker-preloader broken: dockerPreloader fails due to "read-only file system" error #74143, port nixosTests.docker-preloader: Port to Python #80051
  • docker-tools broken: docker-tools test fails to run nix image #75081, port nixosTests.docker-tools: Port to Python #80048
  • ec2 nixosTests.ec2: Port tests that depend on common/ec2.nix #79696
  • hardened nixosTests.hardened: port to python #76708
  • krb5/deprecated-config nixosTests.krb5: Port to python #80130
  • krb5/example-config nixosTests.krb5: Port to python #80130
  • kubernetes/*
  • openstack-image nixosTests.ec2: Port tests that depend on common/ec2.nix #79696
  • partition (test deleted in nixos/tests/partition.nix: remove #87084)
  • redmine (fixed: nixosTests.redmine is broken #80061) nixosTests.redmine: Port to python #85747
  • rsyslogd nixosTests.rsyslogd: Port to python #80096
  • run-in-machine nixosTests.run-in-machine: Port to Python #80099
  • systemd-confinement nixosTests.systemd-confinement: Port to Python #80103
  • taskserver nixosTests.taskserver: Port to python #93413
  • virtualbox nixosTests.virtualbox: Port to python #94858
  • os-prober nixos/tests: Leftover tasks towards unified Python integration tests #72828 (comment)
  • acme
  • ammonite
  • atd Port tests to Python #72833
  • automysqlbackup
  • avahi Port tests to Python #72833
  • babeld Port tests to Python #72833 nixos/tests: migrate my tests to python #72834
  • bcachefs Port tests to Python #72833
  • beanstalkd Port tests to Python #72833
  • bees nixosTests.bees: port to python #74001
  • bind Port tests to Python #72833
  • bittorrent
  • blivet (broken, see nixos/tests/blivet: times out #33496)
  • boot-stage1 Port tests to Python #72833
  • borgbackup Port tests to Python #72833
  • buildbot nixosTests: Port more tests #78556
  • caddy Port tests to Python #72935
  • cadvisor Port tests to Python #72935
  • cassandranixos/cassandra: Port test to python #72897
  • ceph-multi-node WIP: nixos/ceph: port tests to python #73190
  • ceph-single-node WIP: nixos/ceph: port tests to python #73190
  • certmgr Port tests to Python #72935
  • cfssl Port tests to Python #72935
  • chromium nixosTests.chromium: Port to Python #79352
  • cjdns Port tests to Python #72935
  • clickhouse Port tests to python #73131
  • cloud-init Port tests to Python #72935
  • codimd nixos/codimd: port test to python test-driver #74036
  • colord Port tests to Python #72935
  • containers-bridge Nixos test containers python port #74218
  • containers-ephemeral nixosTests.containers*: port to python #74196
  • containers-extra_vethnixosTests.containers*: port rest to python #74761
  • containers-hosts nixosTests.containers*: port to python #74196
  • containers-imperative Nixos test containers python port #74218
  • containers-ipv4 Nixos test containers python port #74218
  • containers-ipv6 Nixos test containers python port #74218
  • containers-macvlans nixosTests.containers*: port rest to python #74761
  • containers-physical_interfaces nixosTests.containers*: port rest to python #74761
  • containers-portforward nixosTests.containers*: port rest to python #74761
  • containers-reloadable nixosTests.containers*: port to python #74196
  • containers-restart_networking nixosTests.containers*: port rest to python #74761
  • containers-tmpfs nixosTests.containers*: port to python #74196
  • couchdb Port tests to Python #72935
  • deluge Port tests to python #73131
  • dhparams Nixos tests ports #75084
  • dnscrypt-proxy Port tests to Python #72935
  • docker-edge
  • docker-registry nixos/docker-registry: port test to python test-driver #74033
  • docker-tools-overlay Nixos tests ports #75084
  • docker Port tests to Python #72935
  • documize Port tests to Python #72935
  • dovecot nixosTests.dovecot: port test to python #74004
  • ecryptfs Nixos tests ports #75084
  • emacs-daemon
  • env Nixos tests ports #75084
  • etcd-cluster nixosTests.etcd*: port to python #74005
  • etcd nixosTests.etcd*: port to python #74005
  • fancontrol Port tests to python #73131
  • ferm nixos/ferm: port test to python #74513
  • firefox Nixos integration test ports #72835
  • firewall Port tests to python #73131
  • fish Port tests to python #73131
  • flannel (broken nixos/tests/flannel fails #74941, PR nixosTests.flannel: port to python, unbreak #85252)
  • flatpak-builder
  • flatpak
  • fluentd Port tests to python #73131
  • fontconfig-default-fonts
  • fsck
  • fwupd
  • gdk-pixbuf
  • gitlab nixosTests.gitlab: port to python #73939
  • gitolite nixosTests.gitolite: port to python #74063
  • gjs
  • glib-networking
  • glusterfs nixosTests.glusterfs: port to python #74003
  • gnome-photos
  • gnome3-xorg nixosTests.gnome3-xorg: port to python/rewrite #73938
  • gnome3 nixos/tests/gnome3: port to python #74943
  • gocd-agent nixosTest.gocd-{agent,server}: port to python #74082
  • gocd-server nixosTest.gocd-{agent,server}: port to python #74082
  • google-oslogin/default nixosTests.google-oslogin: port to python #74002
  • gotify-server nixos/gotify-server: port test to python test-driver #74034
  • grafana nixos/tests: port some tests to python (loki, grafana, pgjwt, initrd-ssh, exporters) #72904
  • graphene
  • graphite nixosTests.graphite: port to python and mark as broken #76562 (python-twisted dependencies fail already in Perl)
  • graylog nixos/graylog: port test to python test-driver #74040
  • hadoop/hdfs
  • hadoop/yarn
  • haka nixosTests.haka: port to python #76707
  • handbrake Port tests to python #73131
  • haproxy nixosTests.haproxy: port test to python #75459 Port NixOs tests to python #75695
  • hibernate Port tests to python #73131
  • hitch/default
  • home-assistant nixosTests.home-assistant: port to python #74126
  • hound Port tests to python #73131
  • hydra/default
  • i3wm nixos/tests/i3wm: port to python #74938
  • icingaweb2 Port tests to python #73131
  • iftop
  • ihatemoney nixosTests: Port more tests #78556
  • incron Port tests to python #73131
  • influxdb nixosTests.influxdb: port to python #74069
  • initrd-network-ssh/default nixos/tests: port some tests to python (loki, grafana, pgjwt, initrd-ssh, exporters) #72904
  • initrd-network Port NixOs tests to python #75695
  • installernixosTests.installer: Port installer and ZFS test to python #78670
  • ipv6 nixosTests.ipv6: port to python #74758
  • jackett Port tests to python #73131
  • jellyfin
  • jenkins nixosTests.jenkins: port to python #74074
  • kerberos/heimdal nixos/kerberos: port tests to python #73961
  • kerberos/mit nixos/kerberos: port tests to python #73961
  • kernel-latest
  • kernel-lts
  • kernel-testing
  • kexec nixosTests.kexec: port to python and mark as broken #76560 (hangs forever already in Perl)
  • keymap nixosTests: Port more tests #78556
  • knot nixos/tests: migrate my tests to python #72834
  • ldap nixosTests.[open]ldap: port to python #74851
  • leaps Port NixOs tests to python #75695
  • libgdata
  • libxmlb
  • lidarr Port NixOs tests to python #75695
  • lightdm nixosTests.lightdm: port to python #73135
  • limesurvey (attempted in Port NixOs tests to python #75695, broke with python error on decoding incomplete utf-8 string), reattempt in nixosTests.limesurvey: port to python and drop badly utf encoded characters  #78555
  • login
  • loki nixos/tests: port some tests to python (loki, grafana, pgjwt, initrd-ssh, exporters) #72904
  • magnetico
  • mailcatcher Port NixOs tests to python #75695
  • matrix-synapse Nixos integration test ports #72835
  • mediawiki
  • memcached nixosTests.memcached: port to python #74312
  • metabase Port tests to Python #72857
  • minidlna Port tests to python #73131
  • miniflux Port tests to python #73131
  • minio nixosTests.minio: port to python #74070
  • misc nixosTests.misc: port to python #79064
  • mongodb nixosTests.mongodb: port to python #73933
  • moodle Port tests to Python #72887
  • morty Port tests to Python #72887
  • mosquitto Mosquitto test python #73827
  • mpd nixosTests.mpd: port to python #73935
  • mumble nixosTests.mumble: port to python #76410
  • munin nixos/tests/munin: port to python #74935
  • mutable-users Port NixOS tests to python #75703
  • mxisd Port NixOS tests to python #75703
  • mysql-backup nixosTests.mysql*: port to python #73992
  • mysql-replication nixosTests.mysql*: port to python #73992
  • mysql nixosTests.mysql*: port to python #73992
  • nat nixosTests.nat: port to python #74754
  • ndppd a9a2717
  • neo4j nixosTests.neo4j: port test to python #73991
  • nesting Port NixOS tests to python #75703
  • netdata nixosTests.netdata: port to python #74078
  • networkingProxynixosTests.networkingProxy: port to Python #78239
  • networking nixosTests.networking: Port tests to python #75721
  • nextcloud/basic
  • nextcloud/with-mysql-and-memcached
  • nextcloud/with-postgresql-and-redis
  • nexus
  • nfs nixos/nfs: Allow Kerberized NFS #73989
  • nghttpx Port NixOS tests to python #75703
  • nginx-sso nixosTests.nginx*: port to python #74072
  • nginx nixosTests.nginx*: port to python #74072
  • nix-ssh-serve nixosTests.nix-ssh-serve: port to python #73994
  • nixos-generate-config
  • novacomd Port NixOS tests to python #75703
  • nsd nixosTests.nsd: Port to python #80065
  • nzbget Port NixOS tests to python #75703
  • openarena
  • openldap nixosTests.[open]ldap: port to python #74851
  • opensmtpd opensmtpd: 6.4.2p1 -> 6.6.1p1 #72390
  • openssh
  • orangefs Port NixOS tests to python #75703
  • osquery nixosTests.osquery: port to python #74073, broken: osquery: remove #74081
  • osrm-backend Port NixOS tests to python #75703
  • ostree
  • overlayfs Port NixOS tests to python #75703
  • packagekit
  • pam-oath-login nixos/test: Use retry() in all looping functions that need timeouts #74898
  • pam-u2f nixosTests.pam-u2f: port test to python #75180
  • pantheon nixosTests.pantheon: port to python #73140
  • paperless Port NixOS tests to python #75701
  • pdns-recursor Port NixOS tests to python #75701
  • peerflix Port NixOS tests to python #75701
  • pgjwt nixos/tests: port some tests to python (loki, grafana, pgjwt, initrd-ssh, exporters) #72904
  • pgmanage Port NixOS tests to python #75701
  • php-pcre Port NixOS tests to python #75701
  • plasma5 nixosTests.plasma5: port to python #73134
  • plotinus nixosTests.plotinus: Port to Python #80067
  • postgis Port NixOS tests to python #75701
  • postgresql-wal-receiver nixosTests.postgresql-wal-receiver is broken #80060 nixosTests.postgresql-wal-receiver needs to be ported to python #96347
  • powerdns nixos/tinydns: port test to python #73059
  • pppd nixos/pppd: port test to python #73056
  • printing nixosTests.printing: Port to Python #79327
  • prometheus-exporters nixos/tests: port some tests to python (loki, grafana, pgjwt, initrd-ssh, exporters) #72904
  • prometheus nixos/prometheus: Port prometheus test to python #74055
  • proxy nixosTests.proxy: port to python #75462 nixosTests.limesurvey: port to python and drop badly utf encoded characters  #78555
  • quagga Port NixOS tests to python #75701
  • quake3
  • rabbitmq nixosTests.rabbitmq: port to python #74123
  • radarr Port tests to Python #72887
  • radicale nixosTests.radicale: port to python #74764
  • redis Port tests to Python #72887
  • riak nixosTests.riak: port to python #74124
  • roundcube Port tests to Python #72887 Port tests to python #73131
  • rspamd nixosTests.rspamd: port to python #75464
  • rss2email Port tests to Python #72887
  • rxe nixosTests.rxe: port to python #73568
  • samba nixos/samba: use tmpfiles, port test to python #73080
  • shiori
  • signal-desktop Port tests to Python #72887
  • simple
  • slurm nixos/slurm: fix test and X11 options #73179
  • smokeping nixos/smokeping: port test to python #73057
  • snapper Port tests to Python #72887
  • solr nixosTests.solr: Port to python #80063
  • sonarr Port NixOS tests to python #75701
  • strongswan-swanctl Port tests to Python #72887
  • sudo nixosTests.sudo: port to python #74076
  • switch-test Port NixOS tests to python #75701
  • syncthing-init nixosTests.syncthing*: port to python #74114
  • syncthing-relay nixosTests.syncthing*: port to python #74114
  • systemd-networkd-wireguard tests/systemd-networkd-wireguard: migrate to python test runner #73699
  • systemd-nspawn nixosTests.systemd-nspawn: convert test to python #73990
  • systemd-timesyncd Port NixOS tests to python #75701
  • systemd nixosTests.systemd: port to Python #78241
  • telegraf Port tests to Python #72887
  • tiddlywiki nixos/tests/tiddlywiki: port to python #74866
  • timezone nixos/tests/timezone: port to python #74858
  • tinydns nixos/tinydns: port test to python #73059
  • tor
  • trac Port tests to Python #72857
  • transmission
  • trezord Port tests to Python #72857
  • trickster Port tests to Python #72887
  • udisks2 Port gnome3 like tests to python and udisks2 #72860
  • upnp Port tests to Python #72887
  • uwsgi nixosTests.uwsgi: port to python #74061
  • vault
  • wireguard/default
  • wireguard/generated
  • wireguard/namespaces Port NixOs tests to python #75695
  • wordpress nixosTests.wordpress: port to python #73993
  • xautolock Port tests to Python #72887
  • xdg-desktop-portal
  • xfce4-14 Port xfce test to python #73138
  • xfce Port xfce test to python #73138
  • xmonad nixosTests.xmonad: port to python #73137
  • xmpp/ejabberd nixos/ejabberd: Port test to python #74254
  • xmpp/prosody-mysql nixosTests.prosody*: port to python #74031
  • xmpp/prosody nixosTests.prosody*: port to python #74031
  • xrdp nixos/tests/xrdp: port to python #74854
  • xss-lock
  • yabar Port tests to Python #72887
  • yggdrasil nixosTests.yggdrasil: port to python #73411
  • zfs nixosTests.installer: Port installer and ZFS test to python #78670
  • zookeeper Port tests to Python #72887
  • beegfs (broken & removed, see beegfs is broken #73998)
• broken, fix/discuss what to do:
  • hocker-fetchdocker/default haskellPackages.hocker marked as broken #95517, PR: nixosTests.hocker-fetchdocker #96076
  • ipfs (broken, see ipfs tests are broken #74000)
  • mathics nixosTests.mathics is broken #80055, PR: mathics: remove package, module and test #95505
  • mesos Mesos is broken #78557, PR: mesos: remove package, module and test (and chronos/marathon which depends on it) #95507
• Deprecate perl test driver
• Drop Perl test driver PR: nixos/lib/test*: remove perl test driver #96396

Additional work that would be awesome:

• Add type annotations
• Perform mypy checks
• Drop XML output, use Python-native logging (but it would be worthwhile to keep the nesting approach in the log) nixos/test-driver: use pythons logging module #96254
• Make output more readable by adding colors PR: nixos/test: colour machine names #96152