Skip to content

Commit 987f5e2

Browse files
debarshiraydebarshiray
committed
.zuul, playbooks, test/system: Optimize the CI on Fedora nodes
The test suite has expanded to 415 system tests. These tests can be very I/O intensive, because many of them copy OCI images from the test suite's image cache directory to its local container/storage store, create containers, and then delete everything to run the next test with a clean slate. This makes the system tests slow. Unfortunately, Zuul's max-job-timeout setting defaults to an upper limit of 3 hours or 10800 seconds for jobs [1], and this is what Software Factory uses [2]. So, there comes a point beyond which the CI can't be prevented from timing out by increasing the timeout. One way of scaling past this maximum time limit is to run the tests in parallel across multiple nodes. This has been implemented by splitting the system tests into different groups, which are run separately by different nodes. First, the tests were grouped into those that test commands and options accepted by the toolbox(1) binary, and those that test the runtime environment within the Toolbx containers. The first group has more tests, but runs faster, because many of them test error handling and don't do much I/O. The runtime environment tests take especially long on Fedora Rawhide nodes, which are often slower than the stable Fedora nodes. Possibly because Rawhide uses Linux kernels that are built with debugging enabled, which makes it slower. Therefore, this group of tests were further split for Rawhide nodes by the Toolbx images they use. Apart from reducing the number of tests in each group, this should also reduce the amount of time spent in downloading the images. The split has been implemented with Bats' tagging system that is available from Bats 1.8.0 [3]. Fortunately, commit 87eaeea already added a dependency on Bats >= 1.10.0. So, there's nothing to worry about. At the moment, Bats doesn't expose the tags being used to run the test suite to setup_suite() and teardown_suite() [4]. Therefore, the TOOLBX_TEST_SYSTEM_TAGS environment variable was used to optimize the contents of setup_suite(). [1] https://zuul-ci.org/docs/zuul/latest/tenants.html [2] Commit 83f28c5 83f28c52e47c2d44 #1548 [3] https://bats-core.readthedocs.io/en/stable/writing-tests.html [4] bats-core/bats-core#1006 #1551
1 parent e435704 commit 987f5e2

27 files changed

+455
-53
lines changed

‎.zuul.yaml‎

Lines changed: 98 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -49,71 +49,141 @@
4949
run: playbooks/unit-test.yaml
5050

5151
- job:
52-
name: system-test-fedora-rawhide
53-
description: Run Toolbx's system tests in Fedora Rawhide
54-
timeout: 10800
52+
name: system-test-fedora-rawhide-commands-options
53+
description: Run Toolbx's commands-options system tests in Fedora Rawhide
54+
timeout: 7200
5555
nodeset:
5656
nodes:
5757
- name: fedora-rawhide
5858
label: cloud-fedora-rawhide
5959
pre-run: playbooks/setup-env.yaml
60-
run: playbooks/system-test.yaml
60+
run: playbooks/system-test-commands-options.yaml
6161

6262
- job:
63-
name: system-test-fedora-41
64-
description: Run Toolbx's system tests in Fedora 41
65-
timeout: 9000
63+
name: system-test-fedora-rawhide-runtime-environment-arch-fedora
64+
description: Run Toolbx's (arch-fedora,runtime-environment) system tests in Fedora Rawhide
65+
timeout: 7200
66+
nodeset:
67+
nodes:
68+
- name: fedora-rawhide
69+
label: cloud-fedora-rawhide
70+
pre-run: playbooks/setup-env.yaml
71+
run: playbooks/system-test-runtime-environment-arch-fedora.yaml
72+
73+
- job:
74+
name: system-test-fedora-rawhide-runtime-environment-ubuntu
75+
description: Run Toolbx's (runtime-environment,ubuntu) system tests in Fedora Rawhide
76+
timeout: 7200
77+
nodeset:
78+
nodes:
79+
- name: fedora-rawhide
80+
label: cloud-fedora-rawhide
81+
pre-run: playbooks/setup-env.yaml
82+
run: playbooks/system-test-runtime-environment-ubuntu.yaml
83+
84+
- job:
85+
name: system-test-fedora-41-commands-options
86+
description: Run Toolbx's commands-options system tests in Fedora 41
87+
timeout: 6300
6688
nodeset:
6789
nodes:
6890
- name: fedora-41
6991
label: cloud-fedora-41
7092
pre-run: playbooks/setup-env.yaml
71-
run: playbooks/system-test.yaml
93+
run: playbooks/system-test-commands-options.yaml
7294

7395
- job:
74-
name: system-test-fedora-40
75-
description: Run Toolbx's system tests in Fedora 40
76-
timeout: 9000
96+
name: system-test-fedora-41-runtime-environment
97+
description: Run Toolbx's runtime-environment system tests in Fedora 41
98+
timeout: 6300
99+
nodeset:
100+
nodes:
101+
- name: fedora-41
102+
label: cloud-fedora-41
103+
pre-run: playbooks/setup-env.yaml
104+
run: playbooks/system-test-runtime-environment.yaml
105+
106+
- job:
107+
name: system-test-fedora-40-commands-options
108+
description: Run Toolbx's commands-options system tests in Fedora 40
109+
timeout: 6300
77110
nodeset:
78111
nodes:
79112
- name: fedora-40
80113
label: cloud-fedora-40
81114
pre-run: playbooks/setup-env.yaml
82-
run: playbooks/system-test.yaml
115+
run: playbooks/system-test-commands-options.yaml
116+
117+
- job:
118+
name: system-test-fedora-40-runtime-environment
119+
description: Run Toolbx's runtime-environment system tests in Fedora 40
120+
timeout: 6300
121+
nodeset:
122+
nodes:
123+
- name: fedora-40
124+
label: cloud-fedora-40
125+
pre-run: playbooks/setup-env.yaml
126+
run: playbooks/system-test-runtime-environment.yaml
127+
128+
- job:
129+
name: system-test-fedora-39-commands-options
130+
description: Run Toolbx's commands-options system tests in Fedora 39
131+
timeout: 6300
132+
nodeset:
133+
nodes:
134+
- name: fedora-39
135+
label: cloud-fedora-39
136+
pre-run: playbooks/setup-env.yaml
137+
run: playbooks/system-test-commands-options.yaml
83138

84139
- job:
85-
name: system-test-fedora-39
86-
description: Run Toolbx's system tests in Fedora 39
87-
timeout: 9000
140+
name: system-test-fedora-39-runtime-environment
141+
description: Run Toolbx's runtime-environment system tests in Fedora 39
142+
timeout: 6300
88143
nodeset:
89144
nodes:
90145
- name: fedora-39
91146
label: cloud-fedora-39
92147
pre-run: playbooks/setup-env.yaml
93-
run: playbooks/system-test.yaml
148+
run: playbooks/system-test-runtime-environment.yaml
94149

95150
- project:
96151
periodic:
97152
jobs:
98-
- system-test-fedora-rawhide
99-
- system-test-fedora-41
100-
- system-test-fedora-40
101-
- system-test-fedora-39
153+
- system-test-fedora-rawhide-commands-options
154+
- system-test-fedora-rawhide-runtime-environment-arch-fedora
155+
- system-test-fedora-rawhide-runtime-environment-ubuntu
156+
- system-test-fedora-41-commands-options
157+
- system-test-fedora-41-runtime-environment
158+
- system-test-fedora-40-commands-options
159+
- system-test-fedora-40-runtime-environment
160+
- system-test-fedora-39-commands-options
161+
- system-test-fedora-39-runtime-environment
102162
check:
103163
jobs:
104164
- unit-test
105165
- unit-test-migration-path-for-coreos-toolbox
106166
- unit-test-restricted
107-
- system-test-fedora-rawhide
108-
- system-test-fedora-41
109-
- system-test-fedora-40
110-
- system-test-fedora-39
167+
- system-test-fedora-rawhide-commands-options
168+
- system-test-fedora-rawhide-runtime-environment-arch-fedora
169+
- system-test-fedora-rawhide-runtime-environment-ubuntu
170+
- system-test-fedora-41-commands-options
171+
- system-test-fedora-41-runtime-environment
172+
- system-test-fedora-40-commands-options
173+
- system-test-fedora-40-runtime-environment
174+
- system-test-fedora-39-commands-options
175+
- system-test-fedora-39-runtime-environment
111176
gate:
112177
jobs:
113178
- unit-test
114179
- unit-test-migration-path-for-coreos-toolbox
115180
- unit-test-restricted
116-
- system-test-fedora-rawhide
117-
- system-test-fedora-41
118-
- system-test-fedora-40
119-
- system-test-fedora-39
181+
- system-test-fedora-rawhide-commands-options
182+
- system-test-fedora-rawhide-runtime-environment-arch-fedora
183+
- system-test-fedora-rawhide-runtime-environment-ubuntu
184+
- system-test-fedora-41-commands-options
185+
- system-test-fedora-41-runtime-environment
186+
- system-test-fedora-40-commands-options
187+
- system-test-fedora-40-runtime-environment
188+
- system-test-fedora-39-commands-options
189+
- system-test-fedora-39-runtime-environment

‎playbooks/system-test-commands-options.yaml‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
#
2+
# Copyright © 2021 – 2024 Red Hat, Inc.
3+
#
4+
# Licensed under the Apache License, Version 2.0 (the "License");
5+
# you may not use this file except in compliance with the License.
6+
# You may obtain a copy of the License at
7+
#
8+
# http://www.apache.org/licenses/LICENSE-2.0
9+
#
10+
# Unless required by applicable law or agreed to in writing, software
11+
# distributed under the License is distributed on an "AS IS" BASIS,
12+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
# See the License for the specific language governing permissions and
14+
# limitations under the License.
15+
#
16+
17+
---
18+
- hosts: all
19+
tasks:
20+
- include_tasks: build.yaml
21+
22+
- name: Run the commands-options system tests
23+
command: bats --filter-tags commands-options ./test/system
24+
environment:
25+
PODMAN: '/usr/bin/podman'
26+
TMPDIR: '/var/tmp'
27+
TOOLBX: '/usr/local/bin/toolbox'
28+
TOOLBX_TEST_SYSTEM_TAGS: 'arch-fedora,commands-options,custom-image,ubuntu'
29+
args:
30+
chdir: '{{ zuul.project.src_dir }}'

‎playbooks/system-test-runtime-environment-arch-fedora.yaml‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
#
2+
# Copyright © 2021 – 2024 Red Hat, Inc.
3+
#
4+
# Licensed under the Apache License, Version 2.0 (the "License");
5+
# you may not use this file except in compliance with the License.
6+
# You may obtain a copy of the License at
7+
#
8+
# http://www.apache.org/licenses/LICENSE-2.0
9+
#
10+
# Unless required by applicable law or agreed to in writing, software
11+
# distributed under the License is distributed on an "AS IS" BASIS,
12+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
# See the License for the specific language governing permissions and
14+
# limitations under the License.
15+
#
16+
17+
---
18+
- hosts: all
19+
tasks:
20+
- include_tasks: build.yaml
21+
22+
- name: Run the (arch-fedora,runtime-environment) system tests
23+
command: bats --filter-tags arch-fedora,runtime-environment ./test/system
24+
environment:
25+
PODMAN: '/usr/bin/podman'
26+
TMPDIR: '/var/tmp'
27+
TOOLBX: '/usr/local/bin/toolbox'
28+
TOOLBX_TEST_SYSTEM_TAGS: 'arch-fedora,runtime-environment'
29+
args:
30+
chdir: '{{ zuul.project.src_dir }}'

‎playbooks/system-test-runtime-environment-ubuntu.yaml‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
#
2+
# Copyright © 2021 – 2024 Red Hat, Inc.
3+
#
4+
# Licensed under the Apache License, Version 2.0 (the "License");
5+
# you may not use this file except in compliance with the License.
6+
# You may obtain a copy of the License at
7+
#
8+
# http://www.apache.org/licenses/LICENSE-2.0
9+
#
10+
# Unless required by applicable law or agreed to in writing, software
11+
# distributed under the License is distributed on an "AS IS" BASIS,
12+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
# See the License for the specific language governing permissions and
14+
# limitations under the License.
15+
#
16+
17+
---
18+
- hosts: all
19+
tasks:
20+
- include_tasks: build.yaml
21+
22+
- name: Run the (runtime-environment,ubuntu) system tests
23+
command: bats --filter-tags runtime-environment,ubuntu ./test/system
24+
environment:
25+
PODMAN: '/usr/bin/podman'
26+
TMPDIR: '/var/tmp'
27+
TOOLBX: '/usr/local/bin/toolbox'
28+
TOOLBX_TEST_SYSTEM_TAGS: 'runtime-environment,ubuntu'
29+
args:
30+
chdir: '{{ zuul.project.src_dir }}'

playbooks/system-test.yaml renamed to playbooks/system-test-runtime-environment.yaml

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -19,11 +19,12 @@
1919
tasks:
2020
- include_tasks: build.yaml
2121

22-
- name: Run system tests
23-
command: bats ./test/system
22+
- name: Run the runtime-environment system tests
23+
command: bats --filter-tags runtime-environment ./test/system
2424
environment:
2525
PODMAN: '/usr/bin/podman'
2626
TMPDIR: '/var/tmp'
2727
TOOLBX: '/usr/local/bin/toolbox'
28+
TOOLBX_TEST_SYSTEM_TAGS: 'arch-fedora,runtime-environment,ubuntu'
2829
args:
2930
chdir: '{{ zuul.project.src_dir }}'

‎test/system/001-version.bats‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,12 +15,14 @@
1515
# limitations under the License.
1616
#
1717

18+
# bats file_tags=commands-options
19+
1820
load 'libs/bats-support/load'
1921
load 'libs/bats-assert/load'
2022
load 'libs/helpers.bash'
2123

2224
setup() {
23-
bats_require_minimum_version 1.7.0
25+
bats_require_minimum_version 1.8.0
2426
_setup_environment
2527
}
2628

‎test/system/002-help.bats‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,8 @@
1515
# limitations under the License.
1616
#
1717

18+
# bats file_tags=commands-options
19+
1820
load 'libs/bats-support/load'
1921
load 'libs/bats-assert/load'
2022
load 'libs/helpers.bash'

‎test/system/101-create.bats‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,8 @@
1515
# limitations under the License.
1616
#
1717

18+
# bats file_tags=commands-options
19+
1820
load 'libs/bats-support/load'
1921
load 'libs/bats-assert/load'
2022
load 'libs/helpers'

‎test/system/102-list.bats‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,8 @@
1515
# limitations under the License.
1616
#
1717

18+
# bats file_tags=commands-options
19+
1820
load 'libs/bats-support/load'
1921
load 'libs/bats-assert/load'
2022
load 'libs/helpers'

‎test/system/103-container.bats‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,11 +15,14 @@
1515
# limitations under the License.
1616
#
1717

18+
# bats file_tags=commands-options
19+
1820
load 'libs/bats-support/load'
1921
load 'libs/bats-assert/load'
2022
load 'libs/helpers'
2123

2224
setup() {
25+
bats_require_minimum_version 1.8.0
2326
_setup_environment
2427
cleanup_all
2528
}

0 commit comments

Comments
 (0)