test/system: Unbreak the line count checks with Bats >= 1.10.0 by debarshiray · Pull Request #1352 · containers/toolbox

debarshiray · 2023-08-16T22:19:05Z

Until Bats 1.10.0, run --keep-empty-lines had a bug where it counted the trailing newline on the last line as a separate line [1]. However, Bats 1.10.0 is only available in Fedora >= 39 and is absent from Fedoras 37 and 38.

[1] Bats commit 6648e2143bffb933
bats-core/bats-core@6648e2143bffb933
bats-core/bats-core#708

Until Bats 1.10.0, 'run --keep-empty-lines' had a bug where it counted the trailing newline on the last line as a separate line [1]. However, Bats 1.10.0 is only available in Fedora >= 39 and is absent from Fedoras 37 and 38. [1] Bats commit 6648e2143bffb933 bats-core/bats-core@6648e2143bffb933 bats-core/bats-core#708 containers#1352

softwarefactory-project-zuul · 2023-08-16T22:50:33Z

Build failed.
https://softwarefactory-project.io/zuul/t/local/buildset/f799234f4d5b446aa2e0f68bb1843146

❌ unit-test RETRY_LIMIT in 34s
✔️ unit-test-migration-path-for-coreos-toolbox SUCCESS in 4m 36s
❌ unit-test-restricted RETRY_LIMIT in 33s
❌ system-test-fedora-rawhide RETRY_LIMIT in 33s
✔️ system-test-fedora-38 SUCCESS in 27m 51s
✔️ system-test-fedora-37 SUCCESS in 26m 56s

Until Bats 1.10.0, 'run --keep-empty-lines' had a bug where it counted the trailing newline on the last line as a separate line [1]. However, Bats 1.10.0 is only available in Fedora >= 39 and is absent from Fedoras 37 and 38. [1] Bats commit 6648e2143bffb933 bats-core/bats-core@6648e2143bffb933 bats-core/bats-core#708 containers#1352

softwarefactory-project-zuul · 2023-08-17T13:04:31Z

Build failed.
https://softwarefactory-project.io/zuul/t/local/buildset/b334c31cf4b849dcab7ef94781db86ad

❌ unit-test RETRY_LIMIT in 34s
✔️ unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 54s
❌ unit-test-restricted RETRY_LIMIT in 33s
❌ system-test-fedora-rawhide RETRY_LIMIT in 34s
✔️ system-test-fedora-38 SUCCESS in 30m 41s
✔️ system-test-fedora-37 SUCCESS in 30m 24s

Until Bats 1.10.0, 'run --keep-empty-lines' had a bug where it counted the trailing newline on the last line as a separate line [1]. However, Bats 1.10.0 is only available in Fedora >= 39 and is absent from Fedoras 37 and 38. [1] Bats commit 6648e2143bffb933 bats-core/bats-core@6648e2143bffb933 bats-core/bats-core#708 containers#1352

softwarefactory-project-zuul · 2023-08-17T14:19:04Z

Build failed.
https://softwarefactory-project.io/zuul/t/local/buildset/2c586bbc56fe45558b6307d6548e4d05

❌ unit-test RETRY_LIMIT in 34s
✔️ unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 33s
❌ unit-test-restricted RETRY_LIMIT in 33s
❌ system-test-fedora-rawhide RETRY_LIMIT in 33s
✔️ system-test-fedora-38 SUCCESS in 29m 07s
✔️ system-test-fedora-37 SUCCESS in 27m 51s

Until Bats 1.10.0, 'run --keep-empty-lines' had a bug where it counted the trailing newline on the last line as a separate line [1]. However, Bats 1.10.0 is only available in Fedora >= 39 and is absent from Fedoras 37 and 38. [1] Bats commit 6648e2143bffb933 bats-core/bats-core@6648e2143bffb933 bats-core/bats-core#708 containers#1352

softwarefactory-project-zuul · 2023-08-17T21:07:03Z

Build failed.
https://softwarefactory-project.io/zuul/t/local/buildset/33babad6abac4b78ba8fca2066ed7439

❌ unit-test RETRY_LIMIT in 32s
✔️ unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 04s
❌ unit-test-restricted RETRY_LIMIT in 33s
❌ system-test-fedora-rawhide RETRY_LIMIT in 32s
✔️ system-test-fedora-38 SUCCESS in 28m 36s
✔️ system-test-fedora-37 SUCCESS in 27m 45s

Until Bats 1.10.0, 'run --keep-empty-lines' had a bug where it counted the trailing newline on the last line as a separate line [1]. However, Bats 1.10.0 is only available in Fedora >= 39 and is absent from Fedoras 37 and 38. [1] Bats commit 6648e2143bffb933 bats-core/bats-core@6648e2143bffb933 bats-core/bats-core#708 containers#1352

softwarefactory-project-zuul · 2023-08-18T04:51:19Z

Build failed.
https://softwarefactory-project.io/zuul/t/local/buildset/1f8d44a7c83a404da7553912ad403f50

❌ unit-test RETRY_LIMIT in 33s
✔️ unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 17s
❌ unit-test-restricted RETRY_LIMIT in 33s
❌ system-test-fedora-rawhide RETRY_LIMIT in 33s
✔️ system-test-fedora-38 SUCCESS in 27m 42s
✔️ system-test-fedora-37 SUCCESS in 27m 09s

debarshiray · 2023-08-18T04:59:33Z

The tests run on Fedora Rawhide nodes are failing because of the same reasons as in #1344 and #1331 , and the root cause appears to be Zuul CI breakage due to rsync. Therefore, I set up a Fedora 39, because it has been branched from Rawhide, virtual machine to run the tests manually:

[rishi@fedora toolbox]$ meson test -C builddir
ninja: Entering directory `/home/rishi/devel/containers/git/toolbox/builddir'
ninja: no work to do.
1/8 go fmt                                 OK              0.06s
2/8 shellcheck src/go-build-wrapper        OK              0.07s
3/8 shellcheck profile.d/toolbox.sh        OK              0.09s
4/8 codespell                              OK              0.26s
5/8 go test                                OK              0.26s
6/8 go vet                                 OK              0.30s
7/8 shellcheck test/system                 OK              0.36s
8/8 shellcheck toolbox (deprecated)        OK              1.41s

Ok:                 8   
Expected Fail:      0   
Fail:               0   
Unexpected Pass:    0   
Skipped:            0   
Timeout:            0   

Full log written to /home/rishi/devel/containers/git/toolbox/builddir/meson-logs/testlog.txt

[rishi@fedora toolbox]$ TOOLBOX=/opt/bin/toolbox bats ./test/system
   test suite: Set up
001-version.bats
 ✓ version: Check version using option --version
002-help.bats
 ✓ help: Try to run toolbox with no command
 ✗ help: Run command 'help'
   (from function `assert_line' in file test/system/libs/bats-assert/src/assert.bash, line 479,
    in test file test/system/002-help.bats, line 45)
     `assert_line --index 0 --partial "toolbox(1)"' failed
   /usr/bin/man
   
   -- line does not contain substring --
   index     : 0
   substring : toolbox(1)
   line      : troff:<standard input>:33: warning: cannot select font 'C'
   --
   
 - help: Run command 'help' with no man present (skipped: Test works only if man is not in PATH)
 ✗ help: Use flag '--help' (it should show usage screen)
   (from function `assert_output' in file test/system/libs/bats-assert/src/assert.bash, line 247,
    in test file test/system/002-help.bats, line 69)
     `assert_output --partial "toolbox - Tool for containerized command line environments on Linux"' failed
   
   -- output does not contain substring --
   substring (1 lines):
     toolbox - Tool for containerized command line environments on Linux
   output (156 lines):
     troff:<standard input>:33: warning: cannot select font 'C'
     troff:<standard input>:43: warning: cannot select font 'C'
     troff:<standard input>:65: warning: cannot select font 'C'
     troff:<standard input>:65: warning: cannot select font 'C'
     troff:<standard input>:67: warning: cannot select font 'C'
     troff:<standard input>:67: warning: cannot select font 'C'
     troff:<standard input>:74: warning: cannot select font 'C'
     troff:<standard input>:74: warning: cannot select font 'C'
     troff:<standard input>:74: warning: cannot select font 'C'
     troff:<standard input>:74: warning: cannot select font 'C'
     troff:<standard input>:113: warning: cannot select font 'C'
     troff:<standard input>:113: warning: cannot select font 'C'
     troff:<standard input>:113: warning: cannot select font 'C'
     troff:<standard input>:176: warning: cannot select font 'C'
     toolbox(1)             General Commands Manual            toolbox(1)
   
     NAME
            toolbox ‐ Tool for containerized command line environments on
            Linux
   
     SYNOPSIS
            toolbox [‐‐assumeyes | ‐y]
                    [‐‐help | ‐h]
                    [‐‐log‐level LEVEL]
                    [‐‐log‐podman]
                    [‐‐verbose | ‐v]
                    COMMAND [ARGS...]
   
     DESCRIPTION
            Toolbox  is  a tool for Linux operating systems, which allows
            the use of containerized command  line  environments.  It  is
            built on top of Podman and other standard container technolo‐
            gies from OCI.
   
            This is particularly useful on OSTree based operating systems
            like  Fedora  CoreOS  and  Silverblue. The intention of these
            systems is to discourage  installation  of  software  on  the
            host,  and  instead  install software as (or in) containers —
            they mostly don’t even have package managers like DNF or YUM.
            This makes it difficult to set up a  development  environment
            or install tools for debugging in the usual way.
   
            Toolbox solves this problem by providing a fully mutable con‐
            tainer  within which one can install their favourite develop‐
            ment and debugging tools, editors and SDKs. For example, it’s
            possible to do yum install ansible without affecting the base
            operating system.
   
            However, this tool doesn’t require using an OSTree based sys‐
            tem. It works equally well on Fedora Workstation and  Server,
            and  that’s a useful way to incrementally adopt containeriza‐
            tion.
   
            The toolbox environment is based on an OCI image.  On  Fedora
            this  is the fedora‐toolbox image. This image is used to cre‐
            ate a toolbox container that seamlessly integrates  with  the
            rest  of  the  operating  system  by  providing access to the
            user’s home directory, the Wayland and X11 sockets,  network‐
            ing  (including  Avahi), removable devices (like USB sticks),
            systemd journal, SSH agent, D‐Bus, ulimits, /dev and the udev
            database, etc..
   
     Supported operating system distributions
            By default, Toolbox tries to use an image matching  the  host
            operating system distribution for creating containers. If the
            host  is not supported, then it falls back to a Fedora image.
            Supported host operating systems are:
   
                   • Fedora
   
                   • Red Hat Enterprise Linux >= 8.5
   
            However, it’s possible to create containers for  a  different
            distribution  through  the  use of the ‐‐distro and ‐‐release
            options that are accepted by the relevant commands, or  their
            counterparts  in  the  configuration  file. The ‐‐distro flag
            specifies the name of the distribution, and ‐‐release  speci‐
            fies its version. Supported combinations are:
   
            ┌────────┬────────────────────────────────────────┐
            │ Distro │ Release                                │
            ├────────┼────────────────────────────────────────┤
            │ fedora │ <release> or f<release> eg., 36 or f36 │
            ├────────┼────────────────────────────────────────┤
            │ rhel   │ <major>.<minor> eg., 8.5               │
            └────────┴────────────────────────────────────────┘
   
     GLOBAL OPTIONS
            The following options are understood:
   
            ‐‐assumeyes, ‐y
   
            Automatically answer yes for all questions.
   
            ‐‐help, ‐h
   
            Print a synopsis of this manual and exit.
   
            ‐‐log‐level=level
   
            Log messages above specified level: debug, info, warn, error,
            fatal or panic (default: error)
   
            ‐‐log‐podman
   
            Show  log messages of invocations of Podman based on the log‐
            ging level specified by option log‐level.
   
            ‐‐verbose, ‐v
   
            Same as ‐‐log‐level=debug. Use ‐vv to include ‐‐log‐podman.
   
     COMMANDS
            Commands for working with toolbox containers and images:
   
            toolbox‐create(1)
   
            Create a new toolbox container.
   
            toolbox‐enter(1)
   
            Enter a toolbox container for interactive use.
   
            toolbox‐help(1)
   
            Display help information about Toolbox.
   
            toolbox‐init‐container(1)
   
            Initialize a running container.
   
            toolbox‐list(1)
   
            List existing toolbox containers and images.
   
            toolbox‐rm(1)
   
            Remove one or more toolbox containers.
   
            toolbox‐rmi(1)
   
            Remove one or more toolbox images.
   
            toolbox‐run(1)
   
            Run a command in an existing toolbox container.
   
     FILES
            toolbox.conf(5)
   
            Toolbox configuration file.
   
     SEE ALSO
            podman(1), https://github.com/containers/toolbox
   
                                                               toolbox(1)
   --
   
 ✓ help: Try to run toolbox with non-existent command (shows usage screen)
 ✓ help: Try to run toolbox with non-existent flag (shows usage screen)
 ✓ help: Try to run 'toolbox create' with non-existent flag (shows usage screen)
 ✓ help: Try to run 'toolbox enter' with non-existent flag (shows usage screen)
 ✓ help: Try to run 'toolbox help' with non-existent flag (shows usage screen)
 ✓ help: Try to run 'toolbox init-container' with non-existent flag (shows usage screen)
 ✓ help: Try to run 'toolbox list' with non-existent flag (shows usage screen)
 ✓ help: Try to run 'toolbox rm' with non-existent flag (shows usage screen)
 ✓ help: Try to run 'toolbox rmi' with non-existent flag (shows usage screen)
 ✓ help: Try to run 'toolbox run' with non-existent flag (shows usage screen)
101-create.bats
 ✓ create: Smoke test
 ✓ create: With a custom name (using option --container)
 ✓ create: With a custom image and name (using option --container)
 ✓ create: Try with an invalid custom name (using positional argument)
 ✓ create: Try with an invalid custom name (using option --container)
 ✓ create: Try with an invalid custom image
 ✓ create: Arch Linux
 ✓ create: Arch Linux ('--release latest')
 ✓ create: Arch Linux ('--release rolling')
 ✓ create: Fedora 34
 ✓ create: RHEL 8.7
 ✗ create: Ubuntu 16.04
   (from function `assert_success' in file test/system/libs/bats-assert/src/assert.bash, line 114,
    from function `pull_distro_image' in file test/system/libs/helpers.bash, line 327,
    in test file test/system/101-create.bats, line 157)
     `pull_distro_image ubuntu 16.04' failed
   Failed to load image quay.io/toolbx/ubuntu-toolbox:16.04 from cache /tmp/bats-run-HSHXdp/image-cache/ubuntu-toolbox-16.04
   
   -- command failed --
   status : 1
   output : time="2023-08-18T06:29:29+02:00" level=fatal msg="determining manifest MIME type for dir:/tmp/bats-run-HSHXdp/image-cache/ubuntu-toolbox-16.04: open /tmp/bats-run-HSHXdp/image-cache/ubuntu-toolbox-16.04/manifest.json: no such file or directory"
   --
   
 ✗ create: Ubuntu 18.04
   (from function `assert_success' in file test/system/libs/bats-assert/src/assert.bash, line 114,
    from function `pull_distro_image' in file test/system/libs/helpers.bash, line 327,
    in test file test/system/101-create.bats, line 172)
     `pull_distro_image ubuntu 18.04' failed
   Failed to load image quay.io/toolbx/ubuntu-toolbox:18.04 from cache /tmp/bats-run-HSHXdp/image-cache/ubuntu-toolbox-18.04
   
   -- command failed --
   status : 1
   output : time="2023-08-18T06:29:29+02:00" level=fatal msg="determining manifest MIME type for dir:/tmp/bats-run-HSHXdp/image-cache/ubuntu-toolbox-18.04: open /tmp/bats-run-HSHXdp/image-cache/ubuntu-toolbox-18.04/manifest.json: no such file or directory"
   --
   
 ✗ create: Ubuntu 20.04
   (from function `assert_success' in file test/system/libs/bats-assert/src/assert.bash, line 114,
    from function `pull_distro_image' in file test/system/libs/helpers.bash, line 327,
    in test file test/system/101-create.bats, line 187)
     `pull_distro_image ubuntu 20.04' failed
   Failed to load image quay.io/toolbx/ubuntu-toolbox:20.04 from cache /tmp/bats-run-HSHXdp/image-cache/ubuntu-toolbox-20.04
   
   -- command failed --
   status : 1
   output : time="2023-08-18T06:29:29+02:00" level=fatal msg="determining manifest MIME type for dir:/tmp/bats-run-HSHXdp/image-cache/ubuntu-toolbox-20.04: open /tmp/bats-run-HSHXdp/image-cache/ubuntu-toolbox-20.04/manifest.json: no such file or directory"
   --
   
 ✓ create: Try an unsupported distribution
 ✓ create: Try a non-existent image
 ✓ create: Try Arch Linux with an invalid release ('--release foo')
 ✓ create: Try Fedora with an invalid release ('--release -3')
 ✓ create: Try Fedora with an invalid release ('--release -3.0')
 ✓ create: Try Fedora with an invalid release ('--release -3.1')
 ✓ create: Try Fedora with an invalid release ('--release 0')
 ✓ create: Try Fedora with an invalid release ('--release 0.0')
 ✓ create: Try Fedora with an invalid release ('--release 0.1')
 ✓ create: Try Fedora with an invalid release ('--release 3.0')
 ✓ create: Try Fedora with an invalid release ('--release 3.1')
 ✓ create: Try Fedora with an invalid release ('--release foo')
 ✓ create: Try Fedora with an invalid release ('--release 3foo')
 ✓ create: Try RHEL with an invalid release ('--release 8')
 ✓ create: Try RHEL with an invalid release ('--release 8.0.0')
 ✓ create: Try RHEL with an invalid release ('--release 8.0.1')
 ✓ create: Try RHEL with an invalid release ('--release 8.3.0')
 ✓ create: Try RHEL with an invalid release ('--release 8.3.1')
 ✓ create: Try RHEL with an invalid release ('--release foo')
 ✓ create: Try RHEL with an invalid release ('--release 8.2foo')
 ✓ create: Try RHEL with an invalid release ('--release -2.1')
 ✓ create: Try RHEL with an invalid release ('--release -2.-1')
 ✓ create: Try RHEL with an invalid release ('--release 2.-1')
 ✓ create: Try Ubuntu with an invalid release ('--release 20')
 ✓ create: Try Ubuntu with an invalid release ('--release 20.04.0')
 ✓ create: Try Ubuntu with an invalid release ('--release 20.04.1')
 ✓ create: Try Ubuntu with an invalid release ('--release foo')
 ✓ create: Try Ubuntu with an invalid release ('--release 20foo')
 ✓ create: Try Ubuntu with an invalid release ('--release foo.bar')
 ✓ create: Try Ubuntu with an invalid release ('--release foo.bar.baz')
 ✓ create: Try Ubuntu with an invalid release ('--release 3.10')
 ✓ create: Try Ubuntu with an invalid release ('--release 202.4')
 ✓ create: Try Ubuntu with an invalid release ('--release 202.04')
 ✓ create: Try Ubuntu with an invalid release ('--release 2020.4')
 ✓ create: Try Ubuntu with an invalid release ('--release 2020.04')
 ✓ create: Try Ubuntu with an invalid release ('--release 04.10')
 ✓ create: Try Ubuntu with an invalid release ('--release 4.bar')
 ✓ create: Try Ubuntu with an invalid release ('--release 4.bar.baz')
 ✓ create: Try Ubuntu with an invalid release ('--release 4.0')
 ✓ create: Try Ubuntu with an invalid release ('--release 4.00')
 ✓ create: Try Ubuntu with an invalid release ('--release 4.13')
 ✓ create: Try Ubuntu with an invalid release ('--release 20.4')
 ✓ create: Try a non-default distro without a release
 ✓ create: Try using both --distro and --image
 ✓ create: Try using both --image and --release
 ✓ create: Try a non-existent authentication file
 ✓ create: With a custom image that needs an authentication file
102-list.bats
 ✓ list: Smoke test
 ✓ list: Smoke test (using --containers)
 ✓ list: Smoke test (using --images)
 ✓ list: With just one non-Toolbx image
 ✓ list: With just one non-Toolbx image (using --images)
 ✗ list: Default image
   (from function `assert_line' in file test/system/libs/bats-assert/src/assert.bash, line 479,
    in test file test/system/102-list.bats, line 97)
     `assert_line --index 1 --partial "$default_image"' failed
   
   -- line does not contain substring --
   index     : 1
   substring : registry.fedoraproject.org/fedora-toolbox:39
   line      :
   --
   
 ✗ list: Default image (using --images)
   (from function `assert_line' in file test/system/libs/bats-assert/src/assert.bash, line 479,
    in test file test/system/102-list.bats, line 115)
     `assert_line --index 1 --partial "$default_image"' failed
   
   -- line does not contain substring --
   index     : 1
   substring : registry.fedoraproject.org/fedora-toolbox:39
   line      :
   --
   
 ✓ list: Arch Linux image
 ✓ list: Arch Linux image (using --images)
 ✓ list: Fedora 34 image
 ✓ list: Fedora 34 image (using --images)
 ✓ list: RHEL 8.7 image
 ✓ list: RHEL 8.7 image (using --images)
 ✗ list: Ubuntu 16.04 image
   (from function `assert_success' in file test/system/libs/bats-assert/src/assert.bash, line 114,
    from function `pull_distro_image' in file test/system/libs/helpers.bash, line 327,
    in test file test/system/102-list.bats, line 247)
     `pull_distro_image ubuntu 16.04' failed
   Failed to load image quay.io/toolbx/ubuntu-toolbox:16.04 from cache /tmp/bats-run-HSHXdp/image-cache/ubuntu-toolbox-16.04
   
   -- command failed --
   status : 1
   output : time="2023-08-18T06:31:28+02:00" level=fatal msg="determining manifest MIME type for dir:/tmp/bats-run-HSHXdp/image-cache/ubuntu-toolbox-16.04: open /tmp/bats-run-HSHXdp/image-cache/ubuntu-toolbox-16.04/manifest.json: no such file or directory"
   --
   
 ✗ list: Ubuntu 16.04 image (using --images)
   (from function `assert_success' in file test/system/libs/bats-assert/src/assert.bash, line 114,
    from function `pull_distro_image' in file test/system/libs/helpers.bash, line 327,
    in test file test/system/102-list.bats, line 268)
     `pull_distro_image ubuntu 16.04' failed
   Failed to load image quay.io/toolbx/ubuntu-toolbox:16.04 from cache /tmp/bats-run-HSHXdp/image-cache/ubuntu-toolbox-16.04
   
   -- command failed --
   status : 1
   output : time="2023-08-18T06:31:28+02:00" level=fatal msg="determining manifest MIME type for dir:/tmp/bats-run-HSHXdp/image-cache/ubuntu-toolbox-16.04: open /tmp/bats-run-HSHXdp/image-cache/ubuntu-toolbox-16.04/manifest.json: no such file or directory"
   --
   
 ✗ list: Ubuntu 18.04 image
   (from function `assert_success' in file test/system/libs/bats-assert/src/assert.bash, line 114,
    from function `pull_distro_image' in file test/system/libs/helpers.bash, line 327,
    in test file test/system/102-list.bats, line 289)
     `pull_distro_image ubuntu 18.04' failed
   Failed to load image quay.io/toolbx/ubuntu-toolbox:18.04 from cache /tmp/bats-run-HSHXdp/image-cache/ubuntu-toolbox-18.04
   
   -- command failed --
   status : 1
   output : time="2023-08-18T06:31:28+02:00" level=fatal msg="determining manifest MIME type for dir:/tmp/bats-run-HSHXdp/image-cache/ubuntu-toolbox-18.04: open /tmp/bats-run-HSHXdp/image-cache/ubuntu-toolbox-18.04/manifest.json: no such file or directory"
   --
   
 ✗ list: Ubuntu 18.04 image (using --images)
   (from function `assert_success' in file test/system/libs/bats-assert/src/assert.bash, line 114,
    from function `pull_distro_image' in file test/system/libs/helpers.bash, line 327,
    in test file test/system/102-list.bats, line 310)
     `pull_distro_image ubuntu 18.04' failed
   Failed to load image quay.io/toolbx/ubuntu-toolbox:18.04 from cache /tmp/bats-run-HSHXdp/image-cache/ubuntu-toolbox-18.04
   
   -- command failed --
   status : 1
   output : time="2023-08-18T06:31:29+02:00" level=fatal msg="determining manifest MIME type for dir:/tmp/bats-run-HSHXdp/image-cache/ubuntu-toolbox-18.04: open /tmp/bats-run-HSHXdp/image-cache/ubuntu-toolbox-18.04/manifest.json: no such file or directory"
   --
   
 ✗ list: Ubuntu 20.04 image
   (from function `assert_success' in file test/system/libs/bats-assert/src/assert.bash, line 114,
    from function `pull_distro_image' in file test/system/libs/helpers.bash, line 327,
    in test file test/system/102-list.bats, line 331)
     `pull_distro_image ubuntu 20.04' failed
   Failed to load image quay.io/toolbx/ubuntu-toolbox:20.04 from cache /tmp/bats-run-HSHXdp/image-cache/ubuntu-toolbox-20.04
   
   -- command failed --
   status : 1
   output : time="2023-08-18T06:31:29+02:00" level=fatal msg="determining manifest MIME type for dir:/tmp/bats-run-HSHXdp/image-cache/ubuntu-toolbox-20.04: open /tmp/bats-run-HSHXdp/image-cache/ubuntu-toolbox-20.04/manifest.json: no such file or directory"
   --
   
 ✗ list: Ubuntu 20.04 image (using --images)
   (from function `assert_success' in file test/system/libs/bats-assert/src/assert.bash, line 114,
    from function `pull_distro_image' in file test/system/libs/helpers.bash, line 327,
    in test file test/system/102-list.bats, line 352)
     `pull_distro_image ubuntu 20.04' failed
   Failed to load image quay.io/toolbx/ubuntu-toolbox:20.04 from cache /tmp/bats-run-HSHXdp/image-cache/ubuntu-toolbox-20.04
   
   -- command failed --
   status : 1
   output : time="2023-08-18T06:31:29+02:00" level=fatal msg="determining manifest MIME type for dir:/tmp/bats-run-HSHXdp/image-cache/ubuntu-toolbox-20.04: open /tmp/bats-run-HSHXdp/image-cache/ubuntu-toolbox-20.04/manifest.json: no such file or directory"
   --
   
 ✓ list: An image without a name
 ✓ list: An image without a name (using --images)
 ✗ list: Image and its copy
   (from function `assert_line' in file test/system/libs/bats-assert/src/assert.bash, line 479,
    in test file test/system/102-list.bats, line 427)
     `assert_line --index 1 --partial "$default_image"' failed
   
   -- line does not contain substring --
   index     : 1
   substring : registry.fedoraproject.org/fedora-toolbox:39
   line      :
   --
   
 ✗ list: Image and its copy (using --images)
   (from function `assert_line' in file test/system/libs/bats-assert/src/assert.bash, line 479,
    in test file test/system/102-list.bats, line 446)
     `assert_line --index 1 --partial "$default_image"' failed
   
   -- line does not contain substring --
   index     : 1
   substring : registry.fedoraproject.org/fedora-toolbox:39
   line      :
   --
   
 ✗ list: Containers and images
   (from function `assert_line' in file test/system/libs/bats-assert/src/assert.bash, line 479,
    in test file test/system/102-list.bats, line 477)
     `assert_line --index 2 --partial "$default_image"' failed
   
   -- line does not contain substring --
   index     : 2
   substring : registry.fedoraproject.org/fedora-toolbox:39
   line      :
   --
   
 ✗ list: Images with and without names
   (from function `assert_line' in file test/system/libs/bats-assert/src/assert.bash, line 479,
    in test file test/system/102-list.bats, line 521)
     `assert_line --index 3 --partial "$default_image"' failed
   
   -- line does not contain substring --
   index     : 3
   substring : registry.fedoraproject.org/fedora-toolbox:39
   line      :
   --
   
 ✗ list: Images with and without names (using --images)
   (from function `assert_line' in file test/system/libs/bats-assert/src/assert.bash, line 479,
    in test file test/system/102-list.bats, line 539)
     `assert_line --index 3 --partial "$default_image"' failed
   
   -- line does not contain substring --
   index     : 3
   substring : registry.fedoraproject.org/fedora-toolbox:39
   line      :
   --
   
 ✓ list: With just one non-Toolbx container and one non-Toolbx image
 ✓ list: With just one non-Toolbx container and one non-Toolbx image (using --containers)
 ✓ list: With just one non-Toolbx container and one non-Toolbx image (using --images)
103-container.bats
 ✓ container: Check container starts without issues
 ✓ container: Start with an old forward incompatible runtime
 - container(Fedora Rawhide): Containers with supported versions start without issues (skipped: This test is only for Fedora Rawhide)
104-run.bats
 ✓ run: Smoke test with true(1)
 ✓ run: Smoke test with false(1)
 ✓ run: Smoke test with Arch Linux
 ✓ run: Smoke test with Arch Linux ('--release latest')
 ✓ run: Smoke test with Arch Linux ('--release rolling')
 ✓ run: Smoke test with Fedora 34
 ✓ run: Smoke test with RHEL 8.7
 ✗ run: Smoke test with Ubuntu 16.04
   (from function `assert_success' in file test/system/libs/bats-assert/src/assert.bash, line 114,
    from function `pull_distro_image' in file test/system/libs/helpers.bash, line 327,
    from function `create_distro_container' in file test/system/libs/helpers.bash, line 381,
    in test file test/system/104-run.bats, line 101)
     `create_distro_container ubuntu 16.04 ubuntu-toolbox-16.04' failed
   Failed to load image quay.io/toolbx/ubuntu-toolbox:16.04 from cache /tmp/bats-run-HSHXdp/image-cache/ubuntu-toolbox-16.04
   
   -- command failed --
   status : 1
   output : time="2023-08-18T06:35:01+02:00" level=fatal msg="determining manifest MIME type for dir:/tmp/bats-run-HSHXdp/image-cache/ubuntu-toolbox-16.04: open /tmp/bats-run-HSHXdp/image-cache/ubuntu-toolbox-16.04/manifest.json: no such file or directory"
   --
   
 ✗ run: Smoke test with Ubuntu 18.04
   (from function `assert_success' in file test/system/libs/bats-assert/src/assert.bash, line 114,
    from function `pull_distro_image' in file test/system/libs/helpers.bash, line 327,
    from function `create_distro_container' in file test/system/libs/helpers.bash, line 381,
    in test file test/system/104-run.bats, line 111)
     `create_distro_container ubuntu 18.04 ubuntu-toolbox-18.04' failed
   Failed to load image quay.io/toolbx/ubuntu-toolbox:18.04 from cache /tmp/bats-run-HSHXdp/image-cache/ubuntu-toolbox-18.04
   
   -- command failed --
   status : 1
   output : time="2023-08-18T06:35:01+02:00" level=fatal msg="determining manifest MIME type for dir:/tmp/bats-run-HSHXdp/image-cache/ubuntu-toolbox-18.04: open /tmp/bats-run-HSHXdp/image-cache/ubuntu-toolbox-18.04/manifest.json: no such file or directory"
   --
   
 ✗ run: Smoke test with Ubuntu 20.04
   (from function `assert_success' in file test/system/libs/bats-assert/src/assert.bash, line 114,
    from function `pull_distro_image' in file test/system/libs/helpers.bash, line 327,
    from function `create_distro_container' in file test/system/libs/helpers.bash, line 381,
    in test file test/system/104-run.bats, line 121)
     `create_distro_container ubuntu 20.04 ubuntu-toolbox-20.04' failed
   Failed to load image quay.io/toolbx/ubuntu-toolbox:20.04 from cache /tmp/bats-run-HSHXdp/image-cache/ubuntu-toolbox-20.04
   
   -- command failed --
   status : 1
   output : time="2023-08-18T06:35:01+02:00" level=fatal msg="determining manifest MIME type for dir:/tmp/bats-run-HSHXdp/image-cache/ubuntu-toolbox-20.04: open /tmp/bats-run-HSHXdp/image-cache/ubuntu-toolbox-20.04/manifest.json: no such file or directory"
   --
   
 ✓ run: Ensure that a login shell is used to invoke the command
 ✓ run: 'echo "Hello World"' inside the default container
 ✓ run: 'echo "Hello World"' inside a restarted container
 ✓ run: 'sudo id' inside the default container
 ✓ run: Ensure that /run/.containerenv exists
 ✓ run: Ensure that /run/.toolboxenv exists
 ✓ run: Ensure that the default container is used
 ✓ run: Ensure that a specific container is used
 ✓ run: Ensure that /home/rishi is used as a fallback working directory
 ✓ run: Pass down 1 additional file descriptor
 ✓ run: Try the non-existent default container with none other present
 ✓ run: Try the non-existent default container with another present
 ✓ run: Try a specific non-existent container with another present
 ✓ run: Try an unsupported distribution
 ✓ run: Try Fedora with an invalid release ('--release -3')
 ✓ run: Try Fedora with an invalid release ('--release -3.0')
 ✓ run: Try Fedora with an invalid release ('--release -3.1')
 ✓ run: Try Fedora with an invalid release ('--release 0')
 ✓ run: Try Fedora with an invalid release ('--release 0.0')
 ✓ run: Try Fedora with an invalid release ('--release 0.1')
 ✓ run: Try Fedora with an invalid release ('--release 3.0')
 ✓ run: Try Fedora with an invalid release ('--release 3.1')
 ✓ run: Try Fedora with an invalid release ('--release foo')
 ✓ run: Try Fedora with an invalid release ('--release 3foo')
 ✓ run: Try RHEL with an invalid release ('--release 8')
 ✓ run: Try RHEL with an invalid release ('--release 8.0.0')
 ✓ run: Try RHEL with an invalid release ('--release 8.0.1')
 ✓ run: Try RHEL with an invalid release ('--release 8.3.0')
 ✓ run: Try RHEL with an invalid release ('--release 8.3.1')
 ✓ run: Try RHEL with an invalid release ('--release foo')
 ✓ run: Try RHEL with an invalid release ('--release 8.2foo')
 ✓ run: Try RHEL with an invalid release ('--release -2.1')
 ✓ run: Try RHEL with an invalid release ('--release -2.-1')
 ✓ run: Try RHEL with an invalid release ('--release 2.-1')
 ✓ run: Try a non-default distro without a release
 ✓ run: Smoke test with 'exit 2'
 ✓ run: Pass down 1 invalid file descriptor
 ✓ run: Try /etc as a command
 ✓ run: Try a non-existent command
105-enter.bats
 ✓ enter: Try to enter the default container with no containers created
 ✓ enter: Try to enter the default container with more than 1 other containers present
 ✓ enter: Try to enter a specific container with no containers created 
 ✓ enter: Try to enter a specific non-existent container with other containers present
 ✓ enter: Try to enter a container based on unsupported distribution
 ✓ enter: Try to enter a container based on Fedora but with wrong version
 ✓ enter: Try to enter a container based on RHEL but with wrong version
 ✓ enter: Try to enter a container based on non-default distro without providing a version
 - enter: Enter the default toolbox (skipped: Testing of entering toolboxes is not implemented)
 - enter: Enter the default toolbox when only 1 non-default toolbox is present (skipped: Testing of entering toolboxes is not implemented)
 - enter: Enter a specific toolbox (skipped: Testing of entering toolboxes is not implemented)
106-rm.bats
 ✓ rm: Try to remove a non-existent container
 - rm: Try to remove a running container (skipped: Bug: Fail in 'toolbox rm' does not return non-zero value)
 ✓ rm: Remove a not running container
 ✓ rm: Force remove a running container
 ✓ rm: Force remove all containers (with 2 containers created and 1 running)
107-rmi.bats
 ✓ rmi: --all without any images
 ✗ rmi: --all with the default image
   (from function `assert_equal' in file test/system/libs/bats-assert/src/assert.bash, line 91,
    in test file test/system/107-rmi.bats, line 61)
     `assert_equal "$new_num_of_images" "$num_of_images"' failed
   
   -- values do not equal --
   expected : 0
   actual   : 1
   --
   
 ✗ rmi: An image by name
   (from function `assert' in file test/system/libs/bats-assert/src/assert.bash, line 46,
    in test file test/system/107-rmi.bats, line 81)
     `assert [ ${#stderr_lines[@]} -eq 0 ]' failed
   
   -- assertion failed --
   expression : [ 1 -eq 0 ]
   --
   
 ✓ rmi: --all with an image without a name
 ✓ rmi: An image without a name
 ✗ rmi: An image and its copy by name, separately
   (from function `assert' in file test/system/libs/bats-assert/src/assert.bash, line 46,
    in test file test/system/107-rmi.bats, line 144)
     `assert [ ${#stderr_lines[@]} -eq 0 ]' failed
   
   -- assertion failed --
   expression : [ 1 -eq 0 ]
   --
   
 ✗ rmi: An image and its copy by name, separately (reverse order)
   (from function `assert' in file test/system/libs/bats-assert/src/assert.bash, line 46,
    in test file test/system/107-rmi.bats, line 173)
     `assert [ ${#stderr_lines[@]} -eq 0 ]' failed
   
   -- assertion failed --
   expression : [ 1 -eq 0 ]
   --
   
 ✗ rmi: An image and its copy by name, together
   (from function `assert' in file test/system/libs/bats-assert/src/assert.bash, line 46,
    in test file test/system/107-rmi.bats, line 202)
     `assert [ ${#stderr_lines[@]} -eq 0 ]' failed
   
   -- assertion failed --
   expression : [ 2 -eq 0 ]
   --
   
 ✗ rmi: An image and its copy by name, together (reverse order)
   (from function `assert' in file test/system/libs/bats-assert/src/assert.bash, line 46,
    in test file test/system/107-rmi.bats, line 225)
     `assert [ ${#stderr_lines[@]} -eq 0 ]' failed
   
   -- assertion failed --
   expression : [ 2 -eq 0 ]
   --
   
 - rmi: Try --all with a running container (skipped: Bug: Fail in 'toolbox rmi' does not return non-zero value)
 ✗ rmi: '--all --force' with a running container
   (from function `assert_equal' in file test/system/libs/bats-assert/src/assert.bash, line 91,
    in test file test/system/107-rmi.bats, line 264)
     `assert_equal "$new_num_of_images" "$num_of_images"' failed
   
   -- values do not equal --
   expected : 0
   actual   : 1
   --
   
108-completion.bats
 ✓ completion: Smoke test with 'bash'
 ✓ completion: Smoke test with 'fish'
 ✓ completion: Smoke test with 'zsh'
 ✓ completion: Try without any arguments
 ✓ completion: Try with invalid arguments
 ✓ completion: Try with unknown flag
 ✓ completion: Try with unsupported shell
201-ipc.bats
 ✓ ipc: no namespace
203-network.bats
 ✓ network: no namespace
 ✓ network: ping(8) inside the default container
 ✓ network: ping(8) inside Arch Linux
 ✓ network: ping(8) inside Fedora 34
 ✓ network: ping(8) inside RHEL 8.7
 ✗ network: ping(8) inside Ubuntu 16.04
   (from function `assert_success' in file test/system/libs/bats-assert/src/assert.bash, line 114,
    from function `pull_distro_image' in file test/system/libs/helpers.bash, line 327,
    from function `create_distro_container' in file test/system/libs/helpers.bash, line 381,
    in test file test/system/203-network.bats, line 118)
     `create_distro_container ubuntu 16.04 ubuntu-toolbox-16.04' failed
   Failed to load image quay.io/toolbx/ubuntu-toolbox:16.04 from cache /tmp/bats-run-HSHXdp/image-cache/ubuntu-toolbox-16.04
   
   -- command failed --
   status : 1
   output : time="2023-08-18T06:39:55+02:00" level=fatal msg="determining manifest MIME type for dir:/tmp/bats-run-HSHXdp/image-cache/ubuntu-toolbox-16.04: open /tmp/bats-run-HSHXdp/image-cache/ubuntu-toolbox-16.04/manifest.json: no such file or directory"
   --
   
 ✗ network: ping(8) inside Ubuntu 18.04
   (from function `assert_success' in file test/system/libs/bats-assert/src/assert.bash, line 114,
    from function `pull_distro_image' in file test/system/libs/helpers.bash, line 327,
    from function `create_distro_container' in file test/system/libs/helpers.bash, line 381,
    in test file test/system/203-network.bats, line 132)
     `create_distro_container ubuntu 18.04 ubuntu-toolbox-18.04' failed
   Failed to load image quay.io/toolbx/ubuntu-toolbox:18.04 from cache /tmp/bats-run-HSHXdp/image-cache/ubuntu-toolbox-18.04
   
   -- command failed --
   status : 1
   output : time="2023-08-18T06:39:55+02:00" level=fatal msg="determining manifest MIME type for dir:/tmp/bats-run-HSHXdp/image-cache/ubuntu-toolbox-18.04: open /tmp/bats-run-HSHXdp/image-cache/ubuntu-toolbox-18.04/manifest.json: no such file or directory"
   --
   
 ✗ network: ping(8) inside Ubuntu 20.04
   (from function `assert_success' in file test/system/libs/bats-assert/src/assert.bash, line 114,
    from function `pull_distro_image' in file test/system/libs/helpers.bash, line 327,
    from function `create_distro_container' in file test/system/libs/helpers.bash, line 381,
    in test file test/system/203-network.bats, line 148)
     `create_distro_container ubuntu 20.04 ubuntu-toolbox-20.04' failed
   Failed to load image quay.io/toolbx/ubuntu-toolbox:20.04 from cache /tmp/bats-run-HSHXdp/image-cache/ubuntu-toolbox-20.04
   
   -- command failed --
   status : 1
   output : time="2023-08-18T06:39:55+02:00" level=fatal msg="determining manifest MIME type for dir:/tmp/bats-run-HSHXdp/image-cache/ubuntu-toolbox-20.04: open /tmp/bats-run-HSHXdp/image-cache/ubuntu-toolbox-20.04/manifest.json: no such file or directory"
   --
   
210-ulimit.bats
 ✓ ulimit: real-time non-blocking time (hard)
 ✓ ulimit: real-time non-blocking time (soft)
 ✓ ulimit: core file size (hard)
 ✓ ulimit: core file size (soft)
 ✓ ulimit: data segment size (hard)
 ✓ ulimit: data segment size (soft)
 ✓ ulimit: scheduling priority (hard)
 ✓ ulimit: scheduling priority (soft)
 ✓ ulimit: file size (hard)
 ✓ ulimit: file size (soft)
 ✓ ulimit: number of pending signals (hard)
 ✓ ulimit: number of pending signals (soft)
 ✓ ulimit: locked memory size (hard)
 ✓ ulimit: locked memory size (soft)
 ✓ ulimit: resident memory size (hard)
 ✓ ulimit: resident memory size (soft)
 ✓ ulimit: number of open files (hard)
 ✓ ulimit: number of open files (soft)
 ✓ ulimit: pipe size (hard)
 ✓ ulimit: pipe size (soft)
 ✓ ulimit: POSIX message queue size (hard)
 ✓ ulimit: POSIX message queue size (soft)
 ✓ ulimit: real-time scheduling priority (hard)
 ✓ ulimit: real-time scheduling priority (soft)
 ✓ ulimit: stack size (hard)
 ✓ ulimit: stack size (soft)
 ✓ ulimit: CPU time (hard)
 ✓ ulimit: CPU time (soft)
 ✓ ulimit: number of user processes (hard)
 ✓ ulimit: number of user processes (soft)
 ✓ ulimit: virtual memory size (hard)
 ✓ ulimit: virtual memory size (soft)
 ✓ ulimit: number of file locks (hard)
 ✓ ulimit: number of file locks (soft)
   test suite: Tear down

234 tests, 31 failures, 7 skipped

I will continue to investigate those 31 test failures. Fortunately, they were not caused by the changes in this pull request. They are likely failing because of reasons other than the 'run --keep-empty-lines' line counting changes in Bats 1.10.0.

containers#1352

softwarefactory-project-zuul · 2023-08-18T05:32:30Z

Build failed.
https://softwarefactory-project.io/zuul/t/local/buildset/488dcdc19abe497c82b5fc886073b025

❌ unit-test RETRY_LIMIT in 33s
✔️ unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 30s
❌ unit-test-restricted RETRY_LIMIT in 35s
❌ system-test-fedora-rawhide RETRY_LIMIT in 33s
✔️ system-test-fedora-38 SUCCESS in 27m 29s
✔️ system-test-fedora-37 SUCCESS in 26m 52s

Until Bats 1.10.0, 'run --keep-empty-lines' had a bug where it counted the trailing newline on the last line as a separate line [1]. However, Bats 1.10.0 is only available in Fedora >= 39 and is absent from Fedoras 37 and 38. [1] Bats commit 6648e2143bffb933 bats-core/bats-core@6648e2143bffb933 bats-core/bats-core#708 containers#1352 containers#1743 (backported from commit f716b23)

Until Bats 1.10.0, 'run --keep-empty-lines' had a bug where it counted the trailing newline on the last line as a separate line [1]. However, Bats 1.10.0 is only available in Fedora >= 39 and is absent from Fedoras 37 and 38. [1] Bats commit 6648e2143bffb933 bats-core/bats-core@6648e2143bffb933 bats-core/bats-core#708 containers#1352 containers#1744 (backported from commit f716b23)

Until Bats 1.10.0, 'run --keep-empty-lines' had a bug where it counted the trailing newline on the last line as a separate line [1]. However, Bats 1.10.0 is only available in Fedora >= 39 and is absent from Fedoras 37 and 38. [1] Bats commit 6648e2143bffb933 bats-core/bats-core@6648e2143bffb933 bats-core/bats-core#708 containers#1352 containers#1387 containers#1744 (backported from commits f716b23 and 7abfa70)

debarshiray requested a review from martymichal as a code owner August 16, 2023 22:19

debarshiray force-pushed the wip/rishi/test-system-run-keep-empty-lines-extra-line branch from 6f8b8ba to e0f0427 Compare August 16, 2023 22:19

debarshiray changed the title ~~test/system: Unbreak line count checks with Bats >= 1.10.0~~ test/system: Unbreak the line count checks with Bats >= 1.10.0 Aug 16, 2023

debarshiray force-pushed the wip/rishi/test-system-run-keep-empty-lines-extra-line branch from e0f0427 to abe320f Compare August 17, 2023 12:31

debarshiray force-pushed the wip/rishi/test-system-run-keep-empty-lines-extra-line branch from abe320f to 2d12794 Compare August 17, 2023 13:48

debarshiray force-pushed the wip/rishi/test-system-run-keep-empty-lines-extra-line branch from 2d12794 to 6081653 Compare August 17, 2023 20:37

debarshiray force-pushed the wip/rishi/test-system-run-keep-empty-lines-extra-line branch from 6081653 to f716b23 Compare August 18, 2023 04:22

.github/workflows: Bump Bats to 1.10.0 for CI on Ubuntu 22.04

8ef3dd9

containers#1352

debarshiray merged commit 8ef3dd9 into containers:main Aug 18, 2023

debarshiray deleted the wip/rishi/test-system-run-keep-empty-lines-extra-line branch August 18, 2023 05:39

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test/system: Unbreak the line count checks with Bats >= 1.10.0#1352

test/system: Unbreak the line count checks with Bats >= 1.10.0#1352
debarshiray merged 2 commits intocontainers:mainfrom
debarshiray:wip/rishi/test-system-run-keep-empty-lines-extra-line

debarshiray commented Aug 16, 2023 •

edited

Loading

Uh oh!

softwarefactory-project-zuul bot commented Aug 16, 2023

Uh oh!

softwarefactory-project-zuul bot commented Aug 17, 2023

Uh oh!

softwarefactory-project-zuul bot commented Aug 17, 2023

Uh oh!

softwarefactory-project-zuul bot commented Aug 17, 2023

Uh oh!

softwarefactory-project-zuul bot commented Aug 18, 2023

Uh oh!

debarshiray commented Aug 18, 2023

Uh oh!

softwarefactory-project-zuul bot commented Aug 18, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

debarshiray commented Aug 16, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

softwarefactory-project-zuul bot commented Aug 16, 2023

Uh oh!

softwarefactory-project-zuul bot commented Aug 17, 2023

Uh oh!

softwarefactory-project-zuul bot commented Aug 17, 2023

Uh oh!

softwarefactory-project-zuul bot commented Aug 17, 2023

Uh oh!

softwarefactory-project-zuul bot commented Aug 18, 2023

Uh oh!

debarshiray commented Aug 18, 2023

Uh oh!

softwarefactory-project-zuul bot commented Aug 18, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

debarshiray commented Aug 16, 2023 •

edited

Loading