Possible performance improvements in the integration test suite #23723
Description
As a response to @yuwata's systemd/systemd-centos-ci#475 I introduce several time-saving features in the CentOS CI, which cut down the runtime of the CentOS CI jobs by ~25 - 35%. Apart from that, I noticed that there are several areas for further improvements, which would be beneficial for CIs in general, not only CentOS CI.
1) Check the sleep()s in the systemd-networkd test suite
There's quite a lot of sleep()s in the systemd-networkd test suite and some of them might be reduced or possibly dropped/replaced by other means:
test/test-network/systemd-networkd-tests.py:440: time.sleep(1)
test/test-network/systemd-networkd-tests.py:472: time.sleep(1)
test/test-network/systemd-networkd-tests.py:539: time.sleep(.2)
test/test-network/systemd-networkd-tests.py:604: time.sleep(sleep_sec)
test/test-network/systemd-networkd-tests.py:626: time.sleep(1)
test/test-network/systemd-networkd-tests.py:657: time.sleep(1)
test/test-network/systemd-networkd-tests.py:714: time.sleep(1)
test/test-network/systemd-networkd-tests.py:724: time.sleep(1)
test/test-network/systemd-networkd-tests.py:1323: time.sleep(1)
test/test-network/systemd-networkd-tests.py:1327: time.sleep(1)
test/test-network/systemd-networkd-tests.py:2363: time.sleep(1)
test/test-network/systemd-networkd-tests.py:2552: time.sleep(1)
test/test-network/systemd-networkd-tests.py:2582: time.sleep(1)
test/test-network/systemd-networkd-tests.py:2774: time.sleep(1)
test/test-network/systemd-networkd-tests.py:2810: time.sleep(1)
test/test-network/systemd-networkd-tests.py:2846: time.sleep(2)
test/test-network/systemd-networkd-tests.py:3089: time.sleep(1)
test/test-network/systemd-networkd-tests.py:3099: time.sleep(1)
test/test-network/systemd-networkd-tests.py:3172: time.sleep(2)
test/test-network/systemd-networkd-tests.py:3186: time.sleep(2)
test/test-network/systemd-networkd-tests.py:3194: time.sleep(2)
test/test-network/systemd-networkd-tests.py:3202: time.sleep(2)
test/test-network/systemd-networkd-tests.py:3211: time.sleep(2)
test/test-network/systemd-networkd-tests.py:3255: time.sleep(1)
test/test-network/systemd-networkd-tests.py:3419: time.sleep(1)
test/test-network/systemd-networkd-tests.py:3436: time.sleep(1)
test/test-network/systemd-networkd-tests.py:3442: time.sleep(2)
test/test-network/systemd-networkd-tests.py:4025: time.sleep(1)
test/test-network/systemd-networkd-tests.py:4112: time.sleep(1)
test/test-network/systemd-networkd-tests.py:4116: time.sleep(3)
test/test-network/systemd-networkd-tests.py:4297: time.sleep(1)
test/test-network/systemd-networkd-tests.py:4592: time.sleep(125)
test/test-network/systemd-networkd-tests.py:4829: time.sleep(125)
test/test-network/systemd-networkd-tests.py:4862: time.sleep(125)
test/test-network/systemd-networkd-tests.py:4891: time.sleep(125)
test/test-network/systemd-networkd-tests.py:5055: time.sleep(2)
test/test-network/systemd-networkd-tests.py:5115: time.sleep(130)
test/test-network/systemd-networkd-tests.py:5183: time.sleep(130)
test/test-network/systemd-networkd-tests.py:5215: time.sleep(3)
test/test-network/systemd-networkd-tests.py:5236: time.sleep(3)
test/test-network/systemd-networkd-tests.py:5254: time.sleep(3)
test/test-network/systemd-networkd-tests.py:5272: time.sleep(3)
test/test-network/systemd-networkd-tests.py:5290: time.sleep(3)
test/test-network/systemd-networkd-tests.py:5778: time.sleep(30)
test/test-network/systemd-networkd-tests.py:5780: time.sleep(30)
test/test-network/systemd-networkd-tests.py:5782: time.sleep(30)
test/test-network/systemd-networkd-tests.py:5784: time.sleep(30)2) Merge some of the integration tests
This is something that @bluca (and others) already mentioned, and this might save quite a noticeable chunk of time (especially after mergin the various TEST-XX-ISSUE-XXX tests, due to the VM setup overhead.
To give a better idea of the run times of various tests, here are timings taken from the CentOS CI:
C8S job (bare metal):
$ wget -q -O archive.zip https://jenkins-systemd.apps.ocp.ci.centos.org/job/upstream-centos8/8687/artifact/*zip*/archive.zip && zipgrep "wall clock" archive.zip | sort -brVk 9 | awk -F / '{ print $NF; }' | awk '{ printf "%-50s %s\n", $1, $NF }'
systemd-networkd-tests.py_PASS.log: 41:49.07
TEST-64-UDEV-STORAGE_PASS.log: 6:54.50
TEST-60-MOUNT-RATELIMIT_1_PASS.log: 4:33.66
TEST-06-SELINUX_PASS.log: 2:17.92
TEST-02-UNITTESTS_PASS.log: 1:56.64
TEST-35-LOGIN_PASS.log: 1:40.56
TEST-34-DYNAMICUSERMIGRATE_PASS.log: 1:39.89
TEST-04-JOURNAL_PASS.log: 1:22.89
TEST-50-DISSECT_PASS.log: 1:16.63
TEST-16-EXTEND-TIMEOUT_1_PASS.log: 1:12.47
TEST-24-CRYPTSETUP_PASS.log: 1:04.98
TEST-42-EXECSTOPPOST_PASS.log: 1:04.63
TEST-29-PORTABLE_PASS.log: 0:59.61
TEST-08-ISSUE-2730_PASS.log: 0:57.13
TEST-36-NUMAPOLICY_PASS.log: 0:52.73
TEST-73-LOCALE_PASS.log: 0:52.35
TEST-10-ISSUE-2467_PASS.log: 0:49.80
TEST-70-TPM2_PASS.log: 0:48.75
TEST-48-START-STOP-NO-RELOAD_PASS.log: 0:48.09
TEST-57-ONSUCCESS-UPHOLD_PASS.log: 0:45.97
TEST-49-RUNTIME-BIND-PATHS_PASS.log: 0:44.69
TEST-17-UDEV_PASS.log: 0:43.09
test-exec-deserialization.py_PASS.log: 0:43.08
TEST-13-NSPAWN-SMOKE_PASS.log: 0:42.80
TEST-41-ONESHOT-RESTART_PASS.log: 0:40.21
TEST-47-ISSUE-14566_PASS.log: 0:39.36
TEST-18-FAILUREACTION_PASS.log: 0:37.93
TEST-43-PRIVATEUSER-UNPRIV_PASS.log: 0:37.66
TEST-39-EXECRELOAD_PASS.log: 0:35.28
TEST-20-MAINPIDGAMES_PASS.log: 0:35.17
TEST-38-FREEZER_PASS.log: 0:34.34
TEST-66-DEVICE-ISOLATION_PASS.log: 0:34.26
TEST-51-ISSUE-16115_PASS.log: 0:33.42
TEST-55-OOMD_PASS.log: 0:32.80
TEST-40-EXEC-COMMAND-EX_PASS.log: 0:32.08
TEST-63-ISSUE-17433_PASS.log: 0:31.59
TEST-45-TIMEDATE_PASS.log: 0:31.04
TEST-67-INTEGRITY_PASS.log: 0:30.36
TEST-69-SHUTDOWN_PASS.log: 0:29.89
TEST-23-TYPE-EXEC_PASS.log: 0:29.41
TEST-14-MACHINE-ID_PASS.log: 0:28.48
TEST-72-SYSUPDATE_PASS.log: 0:28.14
TEST-56-EXIT-TYPE_PASS.log: 0:28.03
TEST-25-IMPORT_PASS.log: 0:27.82
TEST-33-CLEAN-UNIT_PASS.log: 0:27.75
TEST-30-ONCLOCKCHANGE_PASS.log: 0:27.58
TEST-27-STDOUTFILE_PASS.log: 0:25.61
TEST-54-CREDS_PASS.log: 0:25.47
TEST-28-PERCENTJ-WANTEDBY_PASS.log: 0:25.37
TEST-65-ANALYZE_PASS.log: 0:24.82
TEST-58-REPART_PASS.log: 0:24.81
TEST-01-BASIC_PASS.log: 0:24.52
TEST-05-RLIMITS_PASS.log: 0:24.13
TEST-26-SETENV_PASS.log: 0:24.04
TEST-71-HOSTNAME_PASS.log: 0:23.99
TEST-44-LOG-NAMESPACE_PASS.log: 0:23.89
TEST-15-DROPIN_PASS.log: 0:23.53
TEST-37-RUNTIMEDIRECTORYPRESERVE_PASS.log: 0:23.48
TEST-32-OOMPOLICY_PASS.log: 0:23.30
TEST-11-ISSUE-3166_PASS.log: 0:22.99
TEST-19-DELEGATE_PASS.log: 0:22.70
TEST-62-RESTRICT-IFACES_PASS.log: 0:22.52
TEST-31-DEVICE-ENUMERATION_PASS.log: 0:22.38
TEST-53-ISSUE-16347_PASS.log: 0:22.33
TEST-09-ISSUE-2691_PASS.log: 0:22.30
TEST-03-JOBS_PASS.log: 0:19.82
TEST-59-RELOADING-RESTART_PASS.log: 0:17.65
TEST-12-ISSUE-3171_PASS.log: 0:12.43
TEST-68-PROPAGATE-EXIT-STATUS_PASS.log: 0:11.57
TEST-22-TMPFILES_PASS.log: 0:11.20
TEST-46-HOMED_PASS.log: 0:10.56
TEST-07-ISSUE-1981_PASS.log: 0:09.38
TEST-52-HONORFIRSTSHUTDOWN_PASS.log: 0:08.01
Arch Linux (kvm):
$ wget -q -O archive.zip https://jenkins-systemd.apps.ocp.ci.centos.org/job/upstream-vagrant-archlinux/12350/artifact/*zip*/archive.zip && zipgrep "wall clock" archive.zip | sort -brVk 9 | awk -F / '{ print $NF; }' | awk '{ printf "%-50s %s\n", $1, $NF }'
systemd-networkd-tests.py_PASS.log: 39:18.83
TEST-64-UDEV-STORAGE_PASS.log: 9:00.08
TEST-55-OOMD_PASS.log: 7:05.63
TEST-34-DYNAMICUSERMIGRATE_PASS.log: 5:13.26
TEST-60-MOUNT-RATELIMIT_1_PASS.log: 4:27.06
TEST-46-HOMED_1_PASS.log: 3:02.79
TEST-02-UNITTESTS_PASS.log: 2:17.28
TEST-35-LOGIN_PASS.log: 1:40.27
TEST-04-JOURNAL_PASS.log: 1:21.81
TEST-39-EXECRELOAD_PASS.log: 1:14.67
TEST-24-CRYPTSETUP_PASS.log: 1:13.72
TEST-16-EXTEND-TIMEOUT_1_PASS.log: 1:11.60
TEST-36-NUMAPOLICY_PASS.log: 1:07.54
TEST-50-DISSECT_PASS.log: 1:05.11
TEST-42-EXECSTOPPOST_PASS.log: 0:59.00
TEST-08-ISSUE-2730_PASS.log: 0:56.00
TEST-38-FREEZER_PASS.log: 0:55.69
TEST-70-TPM2_PASS.log: 0:55.19
TEST-29-PORTABLE_PASS.log: 0:55.12
TEST-40-EXEC-COMMAND-EX_PASS.log: 0:53.93
TEST-66-DEVICE-ISOLATION_PASS.log: 0:53.86
TEST-41-ONESHOT-RESTART_PASS.log: 0:48.08
test-exec-deserialization.py_PASS.log: 0:42.67
TEST-57-ONSUCCESS-UPHOLD_PASS.log: 0:39.96
TEST-48-START-STOP-NO-RELOAD_PASS.log: 0:39.00
TEST-13-NSPAWN-SMOKE_PASS.log: 0:38.82
TEST-10-ISSUE-2467_1_PASS.log: 0:38.43
TEST-17-UDEV_PASS.log: 0:37.58
TEST-20-MAINPIDGAMES_PASS.log: 0:37.38
TEST-49-RUNTIME-BIND-PATHS_PASS.log: 0:37.19
TEST-23-TYPE-EXEC_PASS.log: 0:36.99
TEST-30-ONCLOCKCHANGE_PASS.log: 0:34.97
TEST-47-ISSUE-14566_PASS.log: 0:33.43
TEST-62-RESTRICT-IFACES_PASS.log: 0:31.35
TEST-51-ISSUE-16115_PASS.log: 0:28.17
TEST-67-INTEGRITY_PASS.log: 0:26.29
TEST-18-FAILUREACTION_PASS.log: 0:26.29
TEST-63-ISSUE-17433_PASS.log: 0:25.93
TEST-45-TIMEDATE_PASS.log: 0:25.40
TEST-27-STDOUTFILE_PASS.log: 0:25.16
TEST-15-DROPIN_PASS.log: 0:23.68
TEST-43-PRIVATEUSER-UNPRIV_PASS.log: 0:22.83
TEST-56-EXIT-TYPE_PASS.log: 0:22.39
TEST-69-SHUTDOWN_PASS.log: 0:22.33
TEST-25-IMPORT_1_PASS.log: 0:22.28
TEST-72-SYSUPDATE_PASS.log: 0:22.24
TEST-33-CLEAN-UNIT_PASS.log: 0:21.26
TEST-54-CREDS_PASS.log: 0:19.25
TEST-58-REPART_PASS.log: 0:19.15
TEST-44-LOG-NAMESPACE_PASS.log: 0:18.84
TEST-28-PERCENTJ-WANTEDBY_PASS.log: 0:18.59
TEST-03-JOBS_PASS.log: 0:18.40
TEST-26-SETENV_PASS.log: 0:17.84
TEST-01-BASIC_PASS.log: 0:17.48
TEST-65-ANALYZE_PASS.log: 0:17.35
TEST-37-RUNTIMEDIRECTORYPRESERVE_PASS.log: 0:17.27
TEST-19-DELEGATE_PASS.log: 0:16.81
TEST-05-RLIMITS_PASS.log: 0:16.58
TEST-59-RELOADING-RESTART_PASS.log: 0:16.57
TEST-32-OOMPOLICY_PASS.log: 0:16.12
TEST-53-ISSUE-16347_PASS.log: 0:15.87
TEST-11-ISSUE-3166_PASS.log: 0:15.41
TEST-14-MACHINE-ID_PASS.log: 0:14.89
TEST-31-DEVICE-ENUMERATION_PASS.log: 0:14.79
TEST-09-ISSUE-2691_PASS.log: 0:14.28
TEST-68-PROPAGATE-EXIT-STATUS_PASS.log: 0:12.57
TEST-22-TMPFILES_PASS.log: 0:11.13
TEST-12-ISSUE-3171_PASS.log: 0:10.30
TEST-07-ISSUE-1981_PASS.log: 0:07.86
TEST-52-HONORFIRSTSHUTDOWN_PASS.log: 0:06.97
TEST-06-SELINUX_PASS.log: 0:00.02