Revert "playbooks: Add workaround for Fedora Rawhide" by debarshiray · Pull Request #1344 · containers/toolbox

debarshiray · 2023-08-03T16:02:24Z

The DNF5 Change [1] was dropped from Fedora 39 (and Rawhide) [2] and postponed for a later Fedora. Therefore, there's no need for this workaround.

This reverts commit 9679172.

[1] https://fedoraproject.org/wiki/Changes/ReplaceDnfWithDnf5

[2] https://pagure.io/fesco/issue/3039

The DNF5 Change [1] was dropped from Fedora 39 (and Rawhide) [2] and postponed for a later Fedora. Therefore, there's no need for this workaround. This reverts commit 9679172. [1] https://fedoraproject.org/wiki/Changes/ReplaceDnfWithDnf5 [2] https://pagure.io/fesco/issue/3039 containers#1344

softwarefactory-project-zuul · 2023-08-03T16:31:08Z

Build failed.
https://softwarefactory-project.io/zuul/t/local/buildset/36878fd028e841e9a36b290e2f0ab63c

❌ unit-test RETRY_LIMIT in 33s
✔️ unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 17s
❌ unit-test-restricted RETRY_LIMIT in 33s
❌ system-test-fedora-rawhide RETRY_LIMIT in 33s
✔️ system-test-fedora-38 SUCCESS in 24m 56s
✔️ system-test-fedora-37 SUCCESS in 24m 25s

debarshiray · 2023-08-03T16:46:57Z

recheck

debarshiray · 2023-08-03T16:54:26Z

@danpawlik @TristanCacqueray All the tests running on Fedora Rawhide hosts are failing in the same way as #1331 :

LOOP [fetch-output : Collect logs, artifacts and docs]
fedora-rawhide | ERROR: Item: {'dest': '/var/lib/zuul/builds/39a62ce6d3df4d45a74db1b5a43bb75b/work/logs', 'src': 'logs'}
fedora-rawhide | {
fedora-rawhide |   "ansible_loop_var": "zj_output",
fedora-rawhide |   "msg": "rsync: [sender] change_dir \"/home/zuul-worker/zuul-output/logs\" failed: No such file or directory (2)\nrsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1670) [Receiver=3.1.3]\nrsync: [Receiver] write error: Broken pipe (32)\n",
fedora-rawhide |   "rc": 23,
fedora-rawhide |   "zj_output": {
fedora-rawhide |     "dest": "/var/lib/zuul/builds/39a62ce6d3df4d45a74db1b5a43bb75b/work/logs",
fedora-rawhide |     "src": "logs"
fedora-rawhide |   }
fedora-rawhide | }
fedora-rawhide | changed: All items complete

fedora-rawhide | ERROR: Item: {'dest': '/var/lib/zuul/builds/39a62ce6d3df4d45a74db1b5a43bb75b/work/artifacts', 'src': 'artifacts'}
fedora-rawhide | {
fedora-rawhide |   "ansible_loop_var": "zj_output",
fedora-rawhide |   "msg": "rsync: [sender] change_dir \"/home/zuul-worker/zuul-output/artifacts\" failed: No such file or directory (2)\nrsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1670) [Receiver=3.1.3]\nrsync: [Receiver] write error: Broken pipe (32)\n",
fedora-rawhide |   "rc": 23,
fedora-rawhide |   "zj_output": {
fedora-rawhide |     "dest": "/var/lib/zuul/builds/39a62ce6d3df4d45a74db1b5a43bb75b/work/artifacts",
fedora-rawhide |     "src": "artifacts"
fedora-rawhide |   }
fedora-rawhide | }
fedora-rawhide | ERROR: Item: {'dest': '/var/lib/zuul/builds/39a62ce6d3df4d45a74db1b5a43bb75b/work/docs', 'src': 'docs'}
fedora-rawhide | {
fedora-rawhide |   "ansible_loop_var": "zj_output",
fedora-rawhide |   "msg": "rsync: [sender] change_dir \"/home/zuul-worker/zuul-output/docs\" failed: No such file or directory (2)\nrsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1670) [Receiver=3.1.3]\nrsync: [Receiver] write error: Broken pipe (32)\n",
fedora-rawhide |   "rc": 23,
fedora-rawhide |   "zj_output": {
fedora-rawhide |     "dest": "/var/lib/zuul/builds/39a62ce6d3df4d45a74db1b5a43bb75b/work/docs",
fedora-rawhide |     "src": "docs"
fedora-rawhide |   }
fedora-rawhide | }

My first guess was that this was because of the DNF5 Change getting dropped from Fedora 39, but that's not the case. Any ideas?

softwarefactory-project-zuul · 2023-08-03T17:14:00Z

Build failed.
https://softwarefactory-project.io/zuul/t/local/buildset/f1a190582c604912af4bb22c07980978

❌ unit-test RETRY_LIMIT in 32s
✔️ unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 43s
❌ unit-test-restricted RETRY_LIMIT in 33s
❌ system-test-fedora-rawhide RETRY_LIMIT in 32s
✔️ system-test-fedora-38 SUCCESS in 25m 14s
✔️ system-test-fedora-37 SUCCESS in 24m 40s

TristanCacqueray · 2023-08-03T17:43:25Z

In the executor.log there is this error when trying to copy the source to the node:

TASK [prepare-workspace : Synchronize src repos to workspace directory.] *******
fatal: [fedora-rawhide]: FAILED! => {"censored": "the output has been hidden due to the fact that \'no_log: true\' was specified for this result", "changed": false}

Then the upload-log roles fails because it can't fetch the logs, hence the error you observed. Perhaps the image is not booting, maybe this is caused by this recent change https://softwarefactory-project.io/r/c/config/+/28969 . @danpawlik did that produced a new rawhide image?

danpawlik · 2023-08-07T06:12:12Z

recheck

danpawlik · 2023-08-07T06:29:29Z

@TristanCacqueray Image for rawhide as usual each day is re-created. Image is booting normally, can ssh to it.
Even rsync works well (some time ago was an issue, that rsync command was not working). Will dig more in that.

softwarefactory-project-zuul · 2023-08-07T06:39:13Z

Build failed.
https://softwarefactory-project.io/zuul/t/local/buildset/c3fee442349446769264eab0cf4dcce3

❌ unit-test RETRY_LIMIT in 33s
✔️ unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 10s
❌ unit-test-restricted RETRY_LIMIT in 32s
❌ system-test-fedora-rawhide RETRY_LIMIT in 35s
✔️ system-test-fedora-38 SUCCESS in 24m 41s
✔️ system-test-fedora-37 SUCCESS in 23m 33s

danpawlik · 2023-08-07T06:56:59Z

recheck

danpawlik · 2023-08-07T07:11:37Z

So I have done tests from zuul-executor host to the remote vm. Simple task:

---
- name: test
  hosts: test.dev
  tasks:
   - name: Synchronize src repos to workspace directory.
     synchronize:
       dest: "~/src/"
       src: "/tmp/toolbox"

This task works well, but on executing tasks similar what is done: https://softwarefactory-project.io/r/plugins/gitiles/zuul-jobs/+/refs/heads/master/roles/prepare-workspace/tasks/main.yaml

---
- name: test
  hosts: test.dev
  gather_facts: false
  tasks:
  - name: Synchronize src repos to workspace directory.
    synchronize:
      delete: true
      dest: "~/src/"
      recursive: true
      src: "/tmp/toolbox"
      owner: no
      group: no

Raises an error:

fatal: [test.dev]: FAILED! => {"changed": false, "cmd": "/usr/bin/rsync --delay-updates -F --compress --delete-after --archive --no-owner --no-group --rsh=/usr/bin/ssh -S none -o Port=22 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null --out-format=<<CHANGED>>%i %n%L /tmp/toolbox zuul-worker@XXXX:~/src/", "msg": "Warning: Permanently added 'XXXX' (ECDSA) to the list of known hosts.\r\n*** buffer overflow detected ***: terminated\nrsync: connection unexpectedly closed (202380 bytes received so far) [sender]\nrsync error: unexplained error (code 255) at io.c(226) [sender=3.1.3]\n", "rc": 255}

softwarefactory-project-zuul · 2023-08-07T07:22:36Z

Build failed.
https://softwarefactory-project.io/zuul/t/local/buildset/7fab37d71e9945589917c14e30b89f61

❌ unit-test RETRY_LIMIT in 32s
✔️ unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 15s
❌ unit-test-restricted RETRY_LIMIT in 32s
❌ system-test-fedora-rawhide RETRY_LIMIT in 31s
✔️ system-test-fedora-38 SUCCESS in 23m 57s
✔️ system-test-fedora-37 SUCCESS in 23m 32s

danpawlik · 2023-08-07T08:43:46Z

Created an issue on BZ https://bugzilla.redhat.com/show_bug.cgi?id=2229654

debarshiray · 2023-08-10T12:43:59Z

Thanks for looking into this, @danpawlik & @TristanCacqueray

Is that RHEL 8 rsync bug the root cause of this problem? I am asking to be sure, before I try to see if it can be moved forward.

debarshiray · 2023-08-24T14:00:44Z

recheck

The DNF5 Change [1] was dropped from Fedora 39 (and Rawhide) [2] and postponed for a later Fedora. Therefore, there's no need for this workaround. This reverts commit 9679172. [1] https://fedoraproject.org/wiki/Changes/ReplaceDnfWithDnf5 [2] https://pagure.io/fesco/issue/3039 containers#1344

softwarefactory-project-zuul · 2023-08-24T14:37:53Z

Build failed.
https://softwarefactory-project.io/zuul/t/local/buildset/dbe2b1d7987349b3827df8cc799e857e

✔️ unit-test SUCCESS in 8m 40s
✔️ unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 39s
✔️ unit-test-restricted SUCCESS in 7m 32s
❌ system-test-fedora-rawhide FAILURE in 34m 56s
✔️ system-test-fedora-38 SUCCESS in 29m 28s
✔️ system-test-fedora-37 SUCCESS in 29m 25s

softwarefactory-project-zuul · 2023-08-24T15:18:55Z

Build failed.
https://softwarefactory-project.io/zuul/t/local/buildset/cff8f17f16274b98b27fac3a2f55fd27

✔️ unit-test SUCCESS in 8m 49s
✔️ unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 17s
✔️ unit-test-restricted SUCCESS in 7m 55s
❌ system-test-fedora-rawhide FAILURE in 37m 32s
✔️ system-test-fedora-38 SUCCESS in 36m 23s
✔️ system-test-fedora-37 SUCCESS in 36m 15s

debarshiray · 2023-08-24T19:01:48Z

Some of the test cases are failing because of changes in Fedora Rawhide. For example:

fedora-rawhide | not ok 3 help: Run command 'help' in 136ms
fedora-rawhide | # (from function `assert_line' in file test/system/libs/bats-assert/src/assert.bash, line 479,
fedora-rawhide | #  in test file test/system/002-help.bats, line 45)
fedora-rawhide | #   `assert_line --index 0 --partial "toolbox(1)"' failed
fedora-rawhide | # /usr/bin/man
fedora-rawhide | #
fedora-rawhide | # -- line does not contain substring --
fedora-rawhide | # index     : 0
fedora-rawhide | # substring : toolbox(1)
fedora-rawhide | # line      : troff:<standard input>:33: warning: cannot select font 'C'
fedora-rawhide | # --

However, that's unrelated to this pull request. The fact that we progressed far enough to run the tests, shows that this pull request is alright.

debarshiray · 2023-08-24T19:02:28Z

Also, yay, the Zuul CI breakage with Fedora Rawhide nodes got fixed!

debarshiray requested a review from martymichal as a code owner August 3, 2023 16:02

debarshiray force-pushed the wip/rishi/playbooks-rawhide-dnf5-revert branch from 2b21f70 to 3297c5b Compare August 3, 2023 16:04

This was referenced Aug 11, 2023

Be aware of security hardened mount points #1340

Merged

test/system: Silence some problems pointed out by ShellCheck #1347

Merged

test/system: Unbreak the line count checks with Bats >= 1.10.0 #1352

Merged

This was referenced Aug 22, 2023

Test that D-Bus works #1339

Merged

test/system: Test that group and user IDs work #1355

Merged

cmd/initContainer: Do not try to remount /etc/machine-id as ro #1354

Merged

cmd/initContainer: Simplify code by removing a function parameter #1356

Merged

debarshiray force-pushed the wip/rishi/playbooks-rawhide-dnf5-revert branch from 3297c5b to 983e07a Compare August 24, 2023 14:40

debarshiray merged commit 983e07a into containers:main Aug 24, 2023

debarshiray deleted the wip/rishi/playbooks-rawhide-dnf5-revert branch August 24, 2023 19:03

Conversation

debarshiray commented Aug 3, 2023

Uh oh!

softwarefactory-project-zuul bot commented Aug 3, 2023

Uh oh!

debarshiray commented Aug 3, 2023

Uh oh!

debarshiray commented Aug 3, 2023

Uh oh!

softwarefactory-project-zuul bot commented Aug 3, 2023

Uh oh!

TristanCacqueray commented Aug 3, 2023

Uh oh!

danpawlik commented Aug 7, 2023

Uh oh!

danpawlik commented Aug 7, 2023

Uh oh!

softwarefactory-project-zuul bot commented Aug 7, 2023

Uh oh!

danpawlik commented Aug 7, 2023

Uh oh!

danpawlik commented Aug 7, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

softwarefactory-project-zuul bot commented Aug 7, 2023

Uh oh!

danpawlik commented Aug 7, 2023

Uh oh!

debarshiray commented Aug 10, 2023

Uh oh!

debarshiray commented Aug 24, 2023

Uh oh!

softwarefactory-project-zuul bot commented Aug 24, 2023

Uh oh!

softwarefactory-project-zuul bot commented Aug 24, 2023

Uh oh!

debarshiray commented Aug 24, 2023

Uh oh!

debarshiray commented Aug 24, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

danpawlik commented Aug 7, 2023 •

edited

Loading