Add zuul-ci by martymichal · Pull Request #250 · containers/toolbox

martymichal · 2019-09-03T14:08:12Z

This adds configuration for zuul-ci provided by softwarefactory, the first batch of system tests that I scrambled together and a fix for bind mounting toolbox.sh inside of containers. This should be ready to be merged. If it will be, also close this #68.

softwarefactory-project-zuul · 2019-09-03T14:10:40Z

Build failed.

toolbox-test : FAILURE in 1m 40s

softwarefactory-project-zuul · 2019-09-03T14:23:37Z

Build failed.

toolbox-test : FAILURE in 1m 46s

TristanCacqueray · 2019-09-03T14:24:31Z

About the /run/media issue, you can try adding to the First comment of this change:

Depends-On: https://github.com/debarshiray/toolbox/pull/251

softwarefactory-project-zuul · 2019-09-03T14:28:31Z

Build succeeded.

toolbox-test : SUCCESS in 1m 56s

softwarefactory-project-zuul · 2019-09-04T09:52:53Z

Build failed.

toolbox-test-rpm-29 : NODE_FAILURE in 0s
toolbox-test-rpm-30 : SUCCESS in 1m 47s
toolbox-test-git : RETRY_LIMIT in 1m 11s

softwarefactory-project-zuul · 2019-09-04T10:24:38Z

Build failed.

toolbox-test-rpm-29 : NODE_FAILURE in 0s
toolbox-test-rpm-30 : SUCCESS in 2m 00s
toolbox-test-git : RETRY_LIMIT in 3m 58s

softwarefactory-project-zuul · 2019-09-04T11:03:20Z

Build failed.

toolbox-test-rpm-30 : SUCCESS in 2m 08s
toolbox-test-git : RETRY_LIMIT in 1m 33s

softwarefactory-project-zuul · 2019-09-04T11:19:37Z

Build failed.

toolbox-test-rpm-30 : SUCCESS in 1m 34s
toolbox-test-git : RETRY_LIMIT in 1m 05s

softwarefactory-project-zuul · 2019-09-04T11:57:34Z

Build failed.

toolbox-test-rpm-30 : SUCCESS in 2m 07s
toolbox-test-git : FAILURE in 1m 05s

softwarefactory-project-zuul · 2019-09-04T12:01:05Z

Build failed.

toolbox-test-rpm-30 : SUCCESS in 1m 47s
toolbox-test-git : FAILURE in 1m 48s

softwarefactory-project-zuul · 2019-09-04T12:10:58Z

Build failed.

toolbox-test-rpm-30 : SUCCESS in 2m 01s
toolbox-test-git : FAILURE in 1m 21s

softwarefactory-project-zuul · 2019-09-04T12:17:51Z

Build succeeded.

toolbox-test-rpm-30 : SUCCESS in 2m 07s
toolbox-test-git : SUCCESS in 1m 12s

softwarefactory-project-zuul · 2019-09-04T12:21:16Z

Build succeeded.

toolbox-test-rpm-30 : SUCCESS in 1m 55s
toolbox-test-git : SUCCESS in 1m 33s

softwarefactory-project-zuul · 2019-09-05T07:36:09Z

Build failed.

toolbox-test-rpm-30 : SUCCESS in 1m 39s
toolbox-test-git : FAILURE in 1m 10s

softwarefactory-project-zuul · 2019-09-05T08:19:12Z

Build failed.

toolbox-test-rpm-30 : SUCCESS in 1m 36s
toolbox-test-git : RETRY_LIMIT in 54s

softwarefactory-project-zuul · 2019-12-03T17:05:38Z

Build failed.

shellcheck : SUCCESS in 1m 28s
test-with-podman-stable-f30 : SUCCESS in 11m 04s
test-with-podman-stable-f31 : FAILURE in 14m 29s
test-with-podman-rawhide : SUCCESS in 10m 16s

softwarefactory-project-zuul · 2019-12-04T11:49:11Z

Build failed.

shellcheck : SUCCESS in 1m 57s
test-with-podman-stable-f30 : SUCCESS in 12m 27s
test-with-podman-stable-f31 : TIMED_OUT in 15m 44s
test-with-podman-rawhide : SUCCESS in 12m 31s

softwarefactory-project-zuul · 2019-12-04T12:50:27Z

Build succeeded.

shellcheck : SUCCESS in 1m 20s
test-with-podman-stable-f30 : SUCCESS in 9m 27s
test-with-podman-stable-f31 : SUCCESS in 8m 01s
test-with-podman-rawhide : SUCCESS in 10m 42s

debarshiray

This looks good.

Some quick comments:

Could you squash the "update" and "restructure" commits into some of the original commits? There doesn't seem to be any reason to keep them separate. Or did I miss something?

test/system/000-test.bats is empty. Should it be removed?

One thing that I couldn't help thinking is that the test suite involves a lot of shell scripting, which is odd since we are trying to move /usr/bin/toolbox away from being a shell script. It's not surprising that the Bash Automated Testing System involves Bash, but still. :) Anyway, let's not delay this anymore. The tests aren't that complicated, so we can move them to a different framework in future, if needed.

martymichal · 2020-01-16T09:33:08Z

I updated the PR. Let me know if there is anything else you want me to adjust.

I agree that it is a bit of a contradiction that I introduced BATS even though we want to move from shell/bash. At the time I started the PR it looked like the optimal solution because it's quite easy to get started. If in the future we find a better solution then I'm not against replacing.

softwarefactory-project-zuul · 2020-01-16T09:45:30Z

Build succeeded.

shellcheck : SUCCESS in 1m 15s
test-with-podman-stable-f30 : SUCCESS in 8m 38s
test-with-podman-stable-f31 : SUCCESS in 8m 29s
test-with-podman-rawhide : SUCCESS in 16m 24s

These tests are written using BATS (Bash Automated Testing System). I used a very helpful helpers.bash script from the libpod project (Thank you!) that I tweaked slightly. containers#68

containers#68

This adds several .yaml files that specify jobs (those in folder playbooks) and one that serves as the main config (.zuul.yaml). Tests and builds are currently executed on every change in PRs (ie., check and gating) and periodically (according to the documentation this pipeline should be run at least once a day). There are 4 tests in total: 1. 'ninja test' - does the same thing that Travis did 2. Fedora 30 - runs the system tests with current Podman and Toolbox in Fedora 30 3. Fedora 31 - the same but for Fedora 31 4. Fedora Rawhide - the same but for Fedora Rawhide containers#68

Travis was running 'ninja test' and that's now covered by Zuul. containers#68

debarshiray

Thanks for cleaning up the commits, @HarryMichal

debarshiray · 2020-01-22T15:16:16Z

playbooks/fedora-30/test-with-podman-stable.yaml

+        chdir: '{{ zuul.project.src_dir }}'
+
+    - name: Run system tests
+      command: bats ./test/system


Nitpick: we should swap the order of commits so that ./test/system is in place before we start referring to it.

playbooks/shellcheck.yaml

debarshiray · 2020-01-22T15:18:04Z

playbooks/fedora-rawhide/setup-rpm-env.yaml

+          - ShellCheck
+          - bash-completion
+          - udisks2
+          - wget


Typo? Or do we really need wget?

Very possibly a typo.

softwarefactory-project-zuul · 2020-01-22T15:33:08Z

Build succeeded.

shellcheck : SUCCESS in 1m 56s
test-with-podman-stable-f30 : SUCCESS in 11m 00s
test-with-podman-stable-f31 : SUCCESS in 7m 29s
test-with-podman-rawhide : SUCCESS in 12m 24s

The tests introduced by containers#250 have proven to be rather unstable due to mistakes in their design. The tests very quite chaotically structured. Because of that images were deleted and pulled too often which caused several false positives (containers#374, containers#372). This changes the strucutre of the tests in a major way. The tests (resp. commands) are now ran in a manner to kinda simulate the way Toolbox is used. From clean state, through creating containers, using them and in the end deleting them. This should reduce the strain on the bandwidth and possibly even speed up the tests themselves. More information in the README.md in the directory with the tests.

martymichal force-pushed the zuul-ci branch from d25508b to 7786fc9 Compare September 3, 2019 14:20

martymichal force-pushed the zuul-ci branch from 7786fc9 to 4d18abf Compare September 3, 2019 14:25

martymichal force-pushed the zuul-ci branch from fb764d1 to bd3bbe5 Compare September 4, 2019 10:08

martymichal force-pushed the zuul-ci branch from bd3bbe5 to 544296c Compare September 4, 2019 10:53

martymichal force-pushed the zuul-ci branch 2 times, most recently from 73c5c50 to ca6b9d8 Compare September 4, 2019 11:12

martymichal force-pushed the zuul-ci branch 3 times, most recently from 52ab49b to fb34595 Compare September 4, 2019 11:53

martymichal force-pushed the zuul-ci branch from fb34595 to 86484ed Compare September 4, 2019 11:57

martymichal force-pushed the zuul-ci branch from 86484ed to 87e9a44 Compare September 4, 2019 12:07

martymichal force-pushed the zuul-ci branch from 87e9a44 to 7b63676 Compare September 4, 2019 12:14

martymichal force-pushed the zuul-ci branch from 3fd26df to 307da2c Compare September 5, 2019 07:33

martymichal force-pushed the zuul-ci branch 2 times, most recently from 31fc35d to 7616924 Compare September 5, 2019 08:14

martymichal force-pushed the zuul-ci branch from 5113fb1 to e8818e5 Compare December 4, 2019 12:38

martymichal added the 3. Enhancement Improvement to an existing feature label Dec 5, 2019

debarshiray mentioned this pull request Jan 14, 2020

Check /usr/share/profile.d when bind mounting toolbox.sh #362

Merged

debarshiray requested changes Jan 14, 2020

View reviewed changes

martymichal force-pushed the zuul-ci branch from e8818e5 to b7529c9 Compare January 16, 2020 09:26

martymichal and others added 4 commits January 22, 2020 15:54

Add system test scripts

b5cdc57

These tests are written using BATS (Bash Automated Testing System). I used a very helpful helpers.bash script from the libpod project (Thank you!) that I tweaked slightly. containers#68

Add a README file for system tests

da4fea2

containers#68

Remove Travis

c86a715

Travis was running 'ninja test' and that's now covered by Zuul. containers#68

debarshiray force-pushed the zuul-ci branch from b7529c9 to c86a715 Compare January 22, 2020 15:18

debarshiray reviewed Jan 22, 2020

View reviewed changes

debarshiray merged commit c86a715 into containers:master Jan 22, 2020

martymichal mentioned this pull request Feb 10, 2020

Rework the system tests #375

Merged

Conversation

martymichal commented Sep 3, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

softwarefactory-project-zuul bot commented Sep 3, 2019

Uh oh!

softwarefactory-project-zuul bot commented Sep 3, 2019

Uh oh!

TristanCacqueray commented Sep 3, 2019

Uh oh!

softwarefactory-project-zuul bot commented Sep 3, 2019

Uh oh!

softwarefactory-project-zuul bot commented Sep 4, 2019

Uh oh!

softwarefactory-project-zuul bot commented Sep 4, 2019

Uh oh!

softwarefactory-project-zuul bot commented Sep 4, 2019

Uh oh!

softwarefactory-project-zuul bot commented Sep 4, 2019

Uh oh!

softwarefactory-project-zuul bot commented Sep 4, 2019

Uh oh!

softwarefactory-project-zuul bot commented Sep 4, 2019

Uh oh!

softwarefactory-project-zuul bot commented Sep 4, 2019

Uh oh!

softwarefactory-project-zuul bot commented Sep 4, 2019

Uh oh!

softwarefactory-project-zuul bot commented Sep 4, 2019

Uh oh!

softwarefactory-project-zuul bot commented Sep 5, 2019

Uh oh!

softwarefactory-project-zuul bot commented Sep 5, 2019

Uh oh!

softwarefactory-project-zuul bot commented Dec 3, 2019

Uh oh!

softwarefactory-project-zuul bot commented Dec 4, 2019

Uh oh!

softwarefactory-project-zuul bot commented Dec 4, 2019

Uh oh!

debarshiray left a comment

Choose a reason for hiding this comment

Uh oh!

martymichal commented Jan 16, 2020

Uh oh!

softwarefactory-project-zuul bot commented Jan 16, 2020

Uh oh!

debarshiray left a comment

Choose a reason for hiding this comment

Uh oh!

debarshiray Jan 22, 2020

Choose a reason for hiding this comment

Uh oh!

Uh oh!

debarshiray Jan 22, 2020

Choose a reason for hiding this comment

Uh oh!

martymichal Jan 22, 2020

Choose a reason for hiding this comment

Uh oh!

softwarefactory-project-zuul bot commented Jan 22, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

martymichal commented Sep 3, 2019 •

edited

Loading