Skip to content

CI: test overlay and vfs#20161

Merged
openshift-merge-bot[bot] merged 1 commit intocontainers:mainfrom
edsantiago:ci_desired_storage
Nov 18, 2023
Merged

CI: test overlay and vfs#20161
openshift-merge-bot[bot] merged 1 commit intocontainers:mainfrom
edsantiago:ci_desired_storage

Conversation

@edsantiago
Copy link
Copy Markdown
Member

@edsantiago edsantiago commented Sep 26, 2023

We're only testing vfs in CI. That's bad. #18822 tried to
remedy that but that only worked for system tests, not e2e.

Here we introduce CI_DESIRED_STORAGE, to be set in .cirrus.yml
in the same vein as all the other CI_DESIRED_X. Since it's 2023
we default to overlay, testing vfs only in priorfedora.

Skip the "split imagestore" test under overlay (#19748)

Signed-off-by: Ed Santiago santiago@redhat.com

None

@openshift-ci openshift-ci bot added release-note-none approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Sep 26, 2023
@edsantiago edsantiago force-pushed the ci_desired_storage branch 2 times, most recently from 00ab112 to 1246120 Compare September 26, 2023 22:43
@edsantiago edsantiago marked this pull request as draft September 26, 2023 22:43
@openshift-ci openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Sep 26, 2023
@edsantiago
Copy link
Copy Markdown
Member Author

edsantiago commented Sep 26, 2023

Draft because I'm pretty sure this is going to fail due to missing settings in .cirrus.yml

@edsantiago
Copy link
Copy Markdown
Member Author

edsantiago commented Sep 27, 2023

Awright, who wants to speculate on why overlay make (only) podman-remote slower?

@vrothberg
Copy link
Copy Markdown
Member

vrothberg commented Sep 27, 2023

@edsantiago can you share a link to the timing results?

@edsantiago
Copy link
Copy Markdown
Member Author

edsantiago commented Sep 27, 2023

Durrr. Sorry.

type distro user DB local remote container
int debian-13 root 33:40 !50:03
int fedora-37 root 39:28 41:15 36:24
int fedora-38 root 34:32 !50:02 33:15
int fedora-38 root sqlite 34:30 !50:02
int rawhide root sqlite 33:15 !50:02
int debian-13 rootless 32:28
int fedora-37 rootless 34:21
int fedora-38 rootless 33:34
int fedora-38 rootless sqlite 33:58
int rawhide rootless sqlite 34:10
sys debian-13 root 30:00 20:48
sys fedora-37 root !36:00 !23:48
sys fedora-38 root 30:36 21:17
sys fedora-38 root sqlite 32:35 21:05
sys fedora-38-aarch64 root 27:22 19:02
sys rawhide root sqlite 28:34 19:28
sys debian-13 rootless 29:41
sys fedora-37 rootless !36:02
sys fedora-38 rootless 32:48 19:54
sys fedora-38 rootless sqlite 31:48
sys rawhide rootless sqlite 31:36
bud fedora-38 root 30:58 32:13

@vrothberg
Copy link
Copy Markdown
Member

vrothberg commented Sep 27, 2023

Are you sure they're slower and did not hang+timeout?

@edsantiago
Copy link
Copy Markdown
Member Author

edsantiago commented Sep 27, 2023

Good question. No, I'm not sure, but my reasoning is that all the test logs end at about 2900s, which is ~48m, which is close to the 50m limit. And the tail of each log is different (different tests). Granted, buffering means that we probably don't get to see the very very tail ... but in non-overlay tests, we never get to 2900s, so I still think something is slowing down. If I were more talented/hardworking/capable I'd write a minitool to check timestamps on each line and look for big jumps; or to compare against a regular vfs run. Maybe I'll do so today.

@edsantiago
Copy link
Copy Markdown
Member Author

edsantiago commented Sep 28, 2023

Here we go. This is a comparison of a good (standard) CI run against a bad one (this PR). Some tests run faster with overlay, but the huge majority run slower. I'm still trying to chase this down, but ITM would anyone like to look at the table below and see if they can figure out a common factor among the slow tests?

test name good bad
Podman pod create podman pod container can override pod pid NS 2.03 2.01
Podman start podman start container sets HOME to home of execUser 2.36 2.02
Podman login and logout podman login and logout with flag --authfile 3.29 2.14
Podman run entrypoint podman run user entrypoint overrides image entrypoint and image cmd 2.21 2.19
Podman exec podman exec --privileged container not running as root 2.36 2.21
Podman exec podman exec preserves --group-add groups 5.50 2.30
Podman run with --sig-proxy signals are forwarded to container using sig-proxy 2.11 2.32
Podman network podman network remove after disconnect when container initially created with the network 2.08 2.33
Podman search podman search with wildcards 2.35 2.35
Podman import podman import with source and reference 2.40 2.43
Podman build podman build and check identity with always 2.38 2.43
Podman import podman import with custom os, arch and variant 2.09 2.43
Podman import podman import with change flag CMD 2.02 2.47
Podman import podman import with message flag 2.02 2.47
Podman start podman start container retains the HOME env if present 2.19 2.48
Podman pod create podman pod container can override pod net NS 2.63 2.48
Podman import podman import with change flag CMD= 2.02 2.53
Podman exec podman exec with user with cap-add 2.83 2.56
Podman save podman save --multi-image-archive (tagged images) 2.19 2.63
Podman attach podman attach to a container with --sig-proxy set to false 2.64 2.64
Podman pod restart podman pod restart latest pod 2.29 2.65
Podman images podman pull by digest and list --all 2.44 2.67
Podman exec podman exec --privileged 2.95 2.67
Podman prune podman system prune - pod,container stopped 2.15 2.67
Podman commit podman commit single letter container 2.59 2.68
Podman create create use local store image if input image contains a manifest list 2.46 2.73
Podman logs streaming output: k8s-file 2.30 2.73
Podman ps podman pod ps filter ctr attributes 2.69 2.75
Podman create podman create --platform 2.35 2.77
Podman kube generate on pod with restartPolicy 2.93 2.78
Podman run ns podman run pidns test 5.18 2.82
Podman run networking podman attempt to ping container name and hostname --net=private 2.13 2.83
Podman start podman start container --filter 2.99 2.84
Podman run ns podman run ipcns ipcmk container test 3.43 2.85
Podman run networking podman run --uidmap /etc/hosts contains --hostname 2.15 2.85
Podman create create container in pod with network should not fail 2.14 2.86
Podman prune podman system prune - with dangling images true 2.55 2.87
Podman prune podman system prune pods 2.12 2.92
Podman version podman version --format json 2.68 2.97
Podman commit podman commit container check env variables 2.40 2.97
Podman attach podman attach to a running container 2.97 2.99
Podman run networking podman run with new:pod and static-ip 2.34 2.99
Podman Info podman info --format json 2.75 2.99
Podman attach podman attach to the latest container 3.02 3.10
Podman commit podman commit container with change CMD flag 2.29 3.10
Podman login and logout podman login and logout with multi registry 5.16 3.12
Podman create podman container create container based on a remote image 3.63 3.12
Podman logs streaming output: json-file 2.13 3.13
Podman kube generate on pod with restartPolicy set for container in a pod 2.97 3.14
Podman pull podman pull by instance digest (image list) 2.08 3.14
Podman exec podman exec with user with and cap-drop cap-add 3.48 3.18
Podman cp podman cp file 2.26 3.26
Podman rmi podman rmi image that is created from another named imaged 3.28 3.32
Verify podman containers.conf usage using journald for container with container log_tag 2.42 3.33
Podman run networking podman do not tamper with joined network ns interfaces 2.13 3.34
Podman rmi podman rmi with short name 2.70 3.35
Podman search podman search format flag 3.52 3.38
Podman logs tail zero lines: journald 2.20 3.43
Podman search podman search with filter is-automated 3.35 3.43
Podman search podman search format json 3.61 3.43
Podman prune podman image prune dangling images 5.40 3.43
Podman run with --ip flag Podman run two containers with the same IP 2.36 3.44
Podman search podman search image with description 3.68 3.46
Podman build podman build with a secret from file 2.39 3.47
Podman search podman search image with --compatible 3.38 3.49
Podman search podman search no-trunc=false flag 3.41 3.49
Podman search podman search with filter stars 3.32 3.50
Podman pod restart podman pod restart all pods 3.26 3.50
Podman login and logout podman login and logout 2.92 3.52
Podman network podman inspect container two CNI networks 2.47 3.54
Podman search podman search with filter is-official 3.28 3.62
Podman run networking podman run --publish-all with EXPOSE port ranges in Dockerfile 2.81 3.62
Podman search podman search 3.51 3.63
Podman untag podman tag/untag - tag normalization 2.46 3.66
Podman run podman test selinux --privileged label hosts 2.05 3.74
Podman search podman search with limit over 100 3.51 3.75
Podman run podman test selinux label /run/secrets 2.07 3.75
Podman search podman search single registry flag 3.79 3.77
Podman run podman test selinux --privileged label /run/secrets 2.03 3.81
Podman run podman test selinux label hostname 2.07 3.81
Podman run podman test selinux label hosts 2.08 3.82
Podman run podman test selinux --privileged label resolv.conf 2.08 3.83
Podman rmi podman rmi --no-prune with undangling parents 3.41 3.85
Podman prune podman image prune - remove only dangling images 5.36 3.87
Podman kube generate based on user in container 3.28 3.87
Podman rmi podman rmi all images 2.77 3.88
Podman run podman test selinux --privileged label hostname 2.13 3.88
Podman run podman test selinux label resolv.conf 2.28 3.88
Verify podman containers.conf usage oom-score-adj 2.32 4.06
Podman diff podman diff container and image with same name 2.43 4.07
Podman run device podman run device host device with --privileged 2.24 4.15
Verify podman containers.conf usage no-hosts=true /etc/hosts does not include hostname 2.19 4.15
Podman pod create podman pod correctly sets up NetNS 6.60 4.22
Podman run podman run selinux file type setup test 3.39 4.26
Podman exec podman exec preserves container groups with --user and --group-add 8.78 4.29
Podman rmi podman rmi --no-prune with dangling parents 3.92 4.47
Podman pause Unpause a bunch of running containers 2.32 4.51
Podman checkpoint podman run with checkpoint image 3.81 4.52
Podman create podman create with --mount flag 3.62 4.53
Podman prune podman system prune with running, exited pod and volume prune set true 4.08 4.54
Podman pause Pause a bunch of running containers 2.69 4.54
Podman kube generate - --privileged container 2.18 4.55
Verify podman containers.conf usage sysctl test 3.79 4.65
Podman network connect and disconnect podman network disconnect 3.10 4.66
Podman pull podman pull --platform 2.39 4.72
Podman network connect and disconnect podman network connect when not running 3.46 4.74
Verify podman containers.conf usage add capabilities 2.10 4.74
Podman pull podman pull --arch 2.67 4.76
Podman kube generate --podman-only on container with --publish-all 6.68 4.79
Podman init containers podman make sure init container runs before pod containers 2.03 4.83
Podman prune podman system image prune unused images 5.98 4.83
Verify podman containers.conf usage shm-size 2.16 4.91
Podman top podman top with ps(1) options 3.48 4.94
Podman kube generate with pods and containers 2.31 4.94
Podman run networking podman network works across user ns 5.04 4.96
Podman network connect and disconnect podman network disconnect when not running 3.37 4.97
Podman run with --cgroup-parent valid --cgroup-parent using slice 2.01 5.04
Verify podman containers.conf usage add timezone 2.25 5.05
Podman pull podman pull check all tags 4.09 5.07
Podman run with --cgroup-parent no --cgroup-parent 2.12 5.17
Podman commit podman commit should not commit env secret 2.39 5.22
Podman pull podman pull by tag (image list) 2.23 5.29
Podman network connect and disconnect podman network connect and run with network ID 3.47 5.29
Podman pod restart podman pod restart multiple pods 5.53 5.29
Verify podman containers.conf usage cgroup_conf in containers.conf 2.32 5.30
Podman run podman run findmnt nothing shared 2.25 5.34
Podman commit podman commit should not commit secret 2.49 5.34
Podman run podman run with --env-merge 3.67 5.35
Podman kube generate on pod with auto update labels in all containers 2.17 5.35
Podman run podman run --mount type=bind,bind-nonrecursive 2.02 5.36
Podman load podman load multiple tags 3.19 5.42
Podman pod create podman pod create --infra-image w/untagged image 2.27 5.46
Podman run networking podman run network bind to 127.0.0.1 4.50 5.48
Podman checkpoint podman checkpoint container with export and statistics 2.12 5.48
Podman kube generate sharing pid namespace 2.48 5.63
Podman systemd podman systemd in command triggers systemd mode 2.45 5.65
Podman push push test --force-compression 3.95 5.68
Podman port podman port -l nginx 2.74 5.69
Podman systemd podman run container with systemd PID1 4.22 5.81
Podman search podman search in local registry 2.21 5.90
Podman build podman remote build must not allow symlink for ignore files 2.47 5.94
Podman kube generate generate and reimport kube on pod 2.86 5.97
Podman kube generate with --ulimit set 2.28 6.02
Podman stop podman stop --all 2.24 6.10
Podman pause podman pause --filter 2.53 6.18
Podman build podman build with multiple secrets from files 3.84 6.34
Podman init containers podman make sure once container is removed 3.19 6.39
Podman checkpoint podman checkpoint --create-image with running container 5.34 6.48
Podman healthcheck run Verify default time is used and no utf-8 escapes 3.00 6.56
Podman images podman images filter before image 2.45 6.64
Podman kube play multiple publish ports 2.08 6.77
Podman kube play with privileged containers ports and publish in command line - curl should succeed 2.14 6.78
Podman kube play replace non-existing pod 2.10 6.86
Podman wait podman container wait on latest container with --interval flag 5.58 6.89
Podman build podman remote test container/docker file is not at root of context dir 5.28 6.96
Podman search podman search doesn't attempt HTTP if registry is not listed as insecure 2.42 6.96
Podman build podman build --from, --add-host, --cap-drop, --cap-add 11.36 6.98
Podman build podman build relay exit code to process 7.39 7.03
Podman search podman search attempts HTTP if registry is in registries.insecure and force secure is false 2.58 7.05
Podman kube play with Host Ports - curl should succeed 2.10 7.07
Podman network connect and disconnect podman network connect 3.84 7.13
Podman search podman search doesn't attempt HTTP if force secure is true 2.21 7.15
Podman run podman run with restart-policy always restarts containers 2.11 7.21
Podman kube play expose character device inside container 2.01 7.24
Verify podman containers.conf usage limits test 6.52 7.25
Podman kube play with Host Ports and publish in command line - curl should succeed only on overriding port 2.04 7.29
Podman kube play test correct command with both set args and cmd in yaml file 2.46 7.32
Podman build podman build device test 3.49 7.50
Podman images podman image prune --filter 5.47 7.67
Podman kube generate generate with user and reimport kube on pod 3.19 7.68
Podman pull podman pull + inspect from unqualified-search registry 2.69 7.75
Podman push podman push to local registry with authorization 2.82 7.78
Podman save podman save remove signature 3.57 7.80
Podman restart podman container restart running container 2.08 7.83
Podman kube generate on pod with init containers 3.86 7.84
Podman kube play test with infra name annotation set 2.48 7.87
Podman kube generate multiple pods 2.35 7.91
Podman commit podman commit adds exposed ports 2.94 7.91
Podman run podman run --seccomp-policy image (block all syscalls) 2.08 8.07
Podman kube play with TerminationGracePeriodSeconds set 2.93 8.08
Podman pod start multiple pods in conflict 3.14 8.14
Podman stop podman stop all containers with one stopped 2.01 8.18
Podman privileged container tests run no-new-privileges test 2.24 8.20
Podman healthcheck run podman healthcheck that should fail 3.20 8.21
Podman privileged container tests podman privileged should restart after host devices change 2.97 8.23
Podman checkpoint podman restore multiple containers from multiple checkpoint images 6.98 8.27
Podman UserNS support podman --userns=keep-id 2.13 8.29
Podman kube play test correct command with only set args in yaml file 2.11 8.31
Podman kube play test correct command with only set command in yaml file 2.38 8.40
Podman run passwd podman run --group-entry flag 2.16 8.53
Podman wait podman wait on three containers 2.76 8.59
Podman UserNS support podman --user with volume 2.37 8.63
Podman run passwd podman run numeric group from image and no group file 2.25 8.66
Podman run podman pod container --infra=false doesn't share SELinux labels 2.22 8.71
Podman ps podman ps filter test 3.55 8.73
Podman kube play should not rename pod if container in pod has same name 2.29 8.80
Podman kube generate with persistent volume claim 3.24 8.91
Podman kube play without Ports - curl should fail 2.16 8.91
Podman kube play with image data 3.48 8.93
Podman build podman build verify explicit cache use with squash-all and --layers 6.54 8.93
Podman init containers podman ensure always init containers always run 4.25 8.96
Podman run podman run makes workdir from image 2.14 8.97
Podman kube play test with hostIPC 2.34 8.99
Podman run podman run --http-proxy test 3.73 9.02
Podman kube play test correct output 2.90 9.05
Podman kube play test with default infra name 2.54 9.09
Podman kube play test with sysctl defined 2.24 9.10
Podman kube play with ctrName should be in network alias 2.31 9.15
Podman kube play test with init containers and annotation set 2.62 9.26
Podman pod rm podman pod rm with exited containers 2.17 9.26
Podman kube play test with valid Umask value 2.11 9.36
Podman network podman network with multiple aliases 5.44 9.39
Podman kube play support container startup probe 4.12 9.41
Podman restart podman restart stopped container by ID 2.33 9.46
Podman build podman remote test container/docker file is not inside context dir 3.73 9.49
Podman kube play ConfigMap volume with items 2.07 9.53
Podman kube play test with init container type set to default value 2.36 9.62
Podman run with volumes podman run with mount flag and boolean options 2.11 9.64
Podman build podman remote test .dockerignore 4.85 9.64
Podman pod inspect podman inspect a pod 2.20 9.75
Podman kube generate with containers in pods should fail 2.37 9.75
Podman run podman run from manifest list 4.51 9.76
Podman pod create podman pod create --restart set to default 2.75 9.84
Podman UserNS support podman --userns=container:CTR 2.57 9.87
Podman kube play liveness probe should fail 3.62 9.89
Podman kube play override with tcp should keep udp from YAML file 2.06 9.97
Podman build podman build device rename test 3.73 10.02
Podman kube play cap add 2.20 10.11
Podman kube generate with volume 3.26 10.15
Podman checkpoint podman checkpoint container with --pre-checkpoint 4.07 10.15
Podman kube play --ip and --mac-address 2.83 10.19
Podman checkpoint podman checkpoint a container started with --rm 3.51 10.22
Podman push podman push to local registry 5.52 10.24
Podman run with volumes podman named volume copyup empty directory 2.04 10.27
Podman kube play test with --no-trunc 2.69 10.30
Podman run podman run --seccomp-policy '' 2.58 10.33
Podman checkpoint podman restore multiple containers from single checkpoint image 8.78 10.38
Podman kube play with auto update annotations for all containers 2.15 10.40
Podman checkpoint podman checkpoint and restore container with different port mappings 5.55 10.48
Podman pod kill podman pod kill a pod by id 2.09 10.49
Podman restart podman restart non-stop container with short timeout 3.26 10.59
Podman run podman run with named volume 2.10 10.60
Podman run podman run --tz 2.16 10.65
Podman checkpoint podman checkpoint and restore container with --file-locks 3.83 10.78
Podman run podman run findmnt shared 3.91 10.82
Podman build podman build with a secret from file and verify if secret file is not leaked into image 4.66 10.88
Podman restart podman restart --all --running 2.32 10.88
Podman restart podman restart a container in a pod and hosts should not duplicated 2.02 10.94
Podman run podman relabels named volume with :Z 2.02 11.00
Podman kube play test with hostPID 2.09 11.01
Podman stop podman stop --filter 3.16 11.03
Podman pod rm podman pod start/remove single pod via --pod-id-file 2.47 11.10
Podman build podman build --isolation && --arch 10.51 11.14
Podman kube play with multiple networks 3.71 11.25
Podman run podman run --seccomp-policy default 2.49 11.34
Podman run podman run a container with a --rootfs 2.54 11.75
Podman stop podman stop all containers -t 2.70 11.81
Podman kube play with latest image should always pull 3.88 11.83
Podman kube play secret as volume support - simple 2.86 11.84
Podman checkpoint podman pause a checkpointed container by id 3.93 11.87
Podman diff podman image diff 11.13 11.92
podman system df podman system df 2.74 11.93
Podman run with volumes podman named volume copyup of /var 4.35 11.93
Podman kube play with auto update annotations for first container only 2.07 11.94
Podman rm podman rm --filter 3.32 12.01
Podman images podman images workingdir from image 2.44 12.04
Podman kube play secret as volume with items 2.06 12.10
Podman events podman events with an event filter and container=cid 7.06 12.16
Podman run podman run user capabilities test with image 3.21 12.28
Podman rmi podman rmi -a with parent child images 14.72
Podman kube play test with reserved Seccomp annotation in yaml 2.52 12.48
Podman kube play override with udp should keep tcp from YAML file 2.26 12.48
Podman checkpoint podman checkpoint and restore container with same IP 3.58 12.49
Podman kube play with configMap subpaths 2.43 12.56
Podman checkpoint podman checkpoint and restore container with root file-system changes using --ignore-rootfs during res 2.85 12.65
Podman kube play test duplicate volume destination between host path and image volumes 2.79 12.67
Podman checkpoint podman checkpoint container with export (migration) and --ipc host 3.24 12.71
Podman run podman run --secret mount with uid, gid, mode options 2.10 12.72
Podman kube play test with reserved privileged annotation in yaml 2.26 12.75
Podman kube play test with reserved CIDFile annotation in yaml 2.06 12.80
Podman checkpoint podman checkpoint with --leave-running 3.07 12.80
Podman kube play with pull always 3.49 12.82
Podman kube play test with reserved init annotation in yaml 2.56 12.97
Podman images podman images --all flag 4.68 12.97
Podman kube play test with reserved autoremove annotation in yaml 2.41 13.01
Podman systemd podman run --systemd arg is case insensitive 3.18 13.08
Podman run passwd podman can run container without /etc/passwd 3.14 13.29
Podman kube play test with reserved Apparmor annotation in yaml 2.22 13.29
Podman checkpoint podman checkpoint and run exec in restored container 3.74 13.33
Podman build podman build where workdir is a symlink and run without creating new workdir 6.24 13.37
Podman kube play RunAsUser 2.31 13.39
Podman kube play verify environment variables values containing equal sign 3.21 13.44
Podman checkpoint podman checkpoint and restore dev/shm content with --export and --import 3.63 13.70
Podman pod rm podman pod rm pod with infra container and running container 2.41 14.15
Podman pod stop podman pod start/stop single pod via --pod-id-file 2.18 14.17
Podman restart podman restart the latest container 3.14 14.26
Podman kube play test with reserved Label annotation in yaml 2.24 14.59
Podman pod create podman create pod with --no-hosts 2.84 14.59
Podman kube play teardown 2.50 14.61
Podman build podman build basic alpine with squash 7.63 14.71
Podman pod create podman pod create --shm-size-systemd 2.88 14.72
Podman restart podman restart all stopped containers with --all 2.81 14.72
Podman run with volumes podman run with conflict between image volume and user mount succeeds 4.31 14.91
Podman restart podman restart multiple --cidfile 2.75 14.92
Podman checkpoint podman checkpoint and restore container with root file-system changes using --ignore-rootfs during che 3.41 14.94
Podman pod stop podman pod stop multiple pods 2.74 14.99
Podman manifest authenticated push 20.80 15.03
Podman checkpoint podman checkpoint and restore container with root file-system changes 3.87 15.15
Podman load podman load xz compressed image 11.21 15.40
Podman checkpoint podman checkpoint and restore dev/shm content 3.35 15.47
Podman restart podman restart --all 2.38 15.53
Podman kube play test with reserved volumes-from annotation in yaml 3.57 15.78
Podman ps podman ps filter pod 3.08 15.83
Podman run podman run a container on an image with a workdir 5.12 15.83
Podman run podman run --volumes-from flag 2.91 15.83
Podman kube play secret as volume support - multiple volumes 3.00 15.86
Podman run podman run --pod automatically 2.23 15.89
Podman pod stop podman pod stop all pods 2.12 15.90
Podman pod kill podman pod kill latest pod 2.72 15.94
Podman pod create podman pod create infra inheritance test 2.16 15.96
Podman run podman run with restart policy does not restart on manual stop 11.06 15.98
Podman kube play should be able to run image where workdir is a symlink 5.82 15.98
Podman run podman pod container share SELinux labels 3.28 16.11
Podman pod kill podman pod kill all 3.26 16.12
Podman run podman run --volumes-from flag options 3.43 16.18
Podman run podman run container with --pull missing and only pull once 4.10 16.39
Podman kube play with emptyDir volume 3.29 16.44
Podman pod create podman pod create --security-opt 2.16 16.47
Podman run podman run --rm with --restart 3.25 16.49
Podman volume create image-backed volume force removal 11.18 16.50
Verify podman containers.conf usage podman-remote test localcontainers.conf 7.69 16.53
Podman pod create podman pod create --share-parent test 2.91 16.55
Podman kube play replace 3.50 16.62
Podman checkpoint podman checkpoint latest running container 4.59 16.86
Podman pod create podman pod create --uts test 2.05 17.16
Podman pull podman pull multiple images with/without tag/digest 7.80 17.17
Podman pod rm podman pod rm -fa removes everything 3.72 17.29
Podman restart podman restart multiple containers 3.75 17.36
Podman checkpoint podman checkpoint container with export (migration) 4.40 17.54
Podman checkpoint podman checkpoint a running container by id 4.91 17.57
Podman run podman run with built-in volume image 4.41 17.69
Podman pod create podman pod create --volume 3.60 18.20
Podman run podman run container with --pull missing should pull image multiple times 4.76 18.25
Podman pod create podman pod create --volumes-from 2.75 18.56
Podman run podman run --volumes-from flag mount conflicts with image volume 3.68 18.71
Podman volume create image-backed volume basic functionality 10.14 18.74
Podman build podman remote test context dir contains empty dirs and symlinks 6.58 19.20
Podman kube play test env value from configmap and --replace should reuse the configmap volume 3.28 19.22
Podman checkpoint podman checkpoint container with established tcp connections 6.61 19.59
Podman events podman events --until future 20.15 19.71
Podman kube play test restartPolicy 4.41 19.85
Podman kube play test with reserved PublishAll annotation in yaml 8.49 19.96
Podman build podman build http proxy test 14.73 20.00
Podman run podman run --requires 2.60 20.49
Podman run with volumes volume permissions after run 4.36 20.51
Podman run podman run capabilities test 3.99 20.66
Podman run podman run a container based on remote image 4.72 20.81
Podman login and logout podman login and logout without registry parameter 2.07 21.42
Podman build Remote build .containerignore filtering embedded directory (#13535) 18.65 21.50
Podman update podman update container all options v2 2.51 21.53
Podman restart podman restart --filter 3.98 21.64
Podman run podman run umask 4.63 22.38
Podman run with volumes podman volume with uid and gid works 2.93 22.43
Podman run with volumes podman named volume copyup symlink 5.41 22.50
Podman run podman run check /run/.containerenv 3.27 23.36
Podman checkpoint podman checkpoint and restore containers with --print-stats 5.61 23.61
Podman checkpoint podman checkpoint all running container 4.58 23.66
Podman run with volumes podman run with volume flag 5.01 24.16
Podman images podman builder prune 5.25 24.54
Podman network create podman network create with name and IPv6 flag (dual-stack) 3.97 24.55
Podman kube play multi doc yaml with multiple services, pods and deployments 4.04 24.64
Podman run with volumes podman named volume copyup 4.88 24.73
Podman checkpoint podman checkpoint a running container by name 5.37 25.59
Podman run podman run --replace 3.14 26.55
Podman checkpoint podman checkpoint container with export and different compression algorithms 8.34 26.88
Podman run podman run environment test 9.42 26.92
Podman rmi podman image rm - concurrent with shared layers 57.63 28.45
Podman checkpoint podman checkpoint a container with volumes 4.88 28.72
Podman manifest push with --add-compression and --force-compression 38.70 30.00
Podman run podman run security-opt unmask on /sys/fs/cgroup 5.81 30.40
Podman run podman run limits test 11.18 30.80
Podman kube play secret as volume support - optional field 6.16 34.67
Podman pod create podman pod create --restart=no/never 6.27 34.78
Podman run podman run mask and unmask path test 6.30 35.95
Podman pod start podman pod start multiple pods via --pod-id-file 9.13 36.67
Podman rmi podman rmi with cached images 76.47 37.70
Podman pod create podman pod create --sysctl test 5.66 39.19
Podman run with volumes podman run with --mount flag 8.22 39.22
Podman run podman run user capabilities test 13.36 74.21
Podman pod create podman pod create --restart=on-failure 4.08 80.94
Podman pod rm podman pod start/remove multiple pods via --pod-id-file 14.26 120.33
Podman pod stop podman pod start/stop multiple pods via --pod-id-file 12.30 133.71
__total 2980.35 8139.74

Methodology: I grabbed timing results from main.log on f38 remote root. I'm only showing results where both CI jobs took more than 2 seconds, i.e. quick jobs are ignored.

@edsantiago edsantiago force-pushed the ci_desired_storage branch 3 times, most recently from bb006e9 to f0f3b0b Compare October 2, 2023 14:54
@packit-as-a-service
Copy link
Copy Markdown

packit-as-a-service bot commented Oct 2, 2023

Cockpit tests failed for commit f0f3b0be736b96993c2d720ebc30ccad9a03fa32. @martinpitt, @jelly, @mvollmer please check.

@edsantiago
Copy link
Copy Markdown
Member Author

edsantiago commented Oct 2, 2023

I need help with this permissions-check failure:

[+1576s] not ok 444 podman container storage is not accessible by unprivileged users
...
         # #/vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
         # #| FAIL: Able to run 'ls /var/lib/containers/storage/vfs/dir/7b378ec3d3cac68b661b495a6c48785a10d4fafe1252194652c3afae288074b5/tmp' without error
         # #\^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
         # #|  drwxrwxrwt. 1 root root 0 Aug  9  2022 /var/lib/containers/storage/vfs/dir/7b378ec3d3cac68b661b495a6c48785a10d4fafe1252194652c3afae288074b5/tmp
         # #|  dr-xr-xr-x. 1 root root 114 Oct  2 14:03 /var/lib/containers/storage/vfs/dir/7b378ec3d3cac68b661b495a6c48785a10d4fafe1252194652c3afae288074b5
         # #|  drwx--x--x. 1 root root 1024 Oct  2 14:15 /var/lib/containers/storage/vfs/dir
                   ^^^^^^ <----- THIS IS THE PROBLEM
         # #|  drwx--x--x. 1 root root 6 Oct  2 13:48 /var/lib/containers/storage/vfs

I cannot reproduce on 1minutetip, because on a default 1minutetip the permissions are correct:

drwx------. 5 root root 4096 Oct  2 14:05 /var/lib/containers/storage/vfs/dir
    ^^^^^^ <--- All dashes. Nothing open. THIS IS GOOD.

Somehow, under Cirrus, the vfs/dir directory is getting created 0711 instead of 0700. Is this a CI-setup bug? Podman? Elsewhere? Any hints welcome.

There's also a cp failure but I'm ignoring that for now.

edsantiago
edsantiago commented Oct 2, 2023
View reviewed changes
test/e2e/common_test.go Outdated
Comment on lines 976 to 997
Copy link
Copy Markdown
Member Author

@edsantiago edsantiago Oct 2, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This fixes the test hang.

We actually have a long history with this issue of leaked mounts: https://github.com/containers/buildah/blob/0d717cd0a52376c80cb527a344a1d7446b0d56e7/tests/helpers.bash#L116-L123

That dates to 2020. See containers/buildah#1991 .

Is there really no way to fix this leak?

@edsantiago edsantiago force-pushed the ci_desired_storage branch 2 times, most recently from 8e93f67 to 6d2e4b6 Compare October 3, 2023 16:41
@rhatdan
Copy link
Copy Markdown
Member

rhatdan commented Oct 3, 2023

@giuseppe PTAL

@edsantiago
Copy link
Copy Markdown
Member Author

edsantiago commented Oct 3, 2023

Eyeballs welcome, but this is not ready to merge. I'm seeing a LOT of new flakes, like this new container-cp one.

@edsantiago edsantiago force-pushed the ci_desired_storage branch 3 times, most recently from e33f221 to 3a2dea7 Compare October 5, 2023 19:19
@edsantiago edsantiago mentioned this pull request Oct 5, 2023
Closed
@edsantiago edsantiago mentioned this pull request Oct 25, 2023
Closed
@edsantiago edsantiago force-pushed the ci_desired_storage branch 5 times, most recently from 185b678 to 559d688 Compare October 26, 2023 02:30
@edsantiago
Copy link
Copy Markdown
Member Author

edsantiago commented Oct 26, 2023

As a fun side note, vfs seems to be slower than overlay. Is this expected?

type distro user DB local remote container
int debian-13 root 31:25 31:59
int fedora-38 root 37:52 36:35 35:56
int fedora-39β root 33:15 32:36 32:39
int fedora-39β root sqlite 32:45 36:11
int rawhide root sqlite 34:48 33:10
int debian-13 rootless 31:18
int fedora-38 rootless 33:59
int fedora-39β rootless 31:53
int fedora-39β rootless sqlite 36:27
int rawhide rootless sqlite 33:31
sys debian-13 root 28:41 22:34
sys fedora-38 root 36:45 24:07
sys fedora-39β root 31:32 20:53
sys fedora-39β root sqlite 32:08 22:47
sys fedora-39β-aarch64 root 27:55 19:37
sys rawhide root sqlite 30:57 21:33
sys debian-13 rootless 28:21
sys fedora-38 rootless 37:24
sys fedora-39β rootless 34:01 19:32
sys fedora-39β rootless sqlite 33:37
sys rawhide rootless sqlite 33:27

Summary: all int tests in this PR run faster than usual, except f38. Reason: we only use vfs in int tests, that's the way the world has always been. In this PR I switch int tests to overlay, except f38. sys tests, which use system default (overlay), are unaffected... except f38, which now runs slower.

Anyhow, everyone probably knew that already, this is just confirmation that overlay is better. The horrible #20282 flake is another confirmation.

This was referenced Oct 26, 2023
Merged
Closed
@packit-as-a-service
Copy link
Copy Markdown

packit-as-a-service bot commented Nov 5, 2023

Cockpit tests failed for commit 5123917085ccc86f6e959c95ebac43bda9a79126. @martinpitt, @jelly, @mvollmer please check.

@packit-as-a-service
Copy link
Copy Markdown

packit-as-a-service bot commented Nov 6, 2023

Cockpit tests failed for commit 55c4ed8c84e2889879debce9bc7bfbd8c1729ce4. @martinpitt, @jelly, @mvollmer please check.

@edsantiago
Copy link
Copy Markdown
Member Author

edsantiago commented Nov 6, 2023

@containers/podman-maintainers I am marking this as ready for review... with one big fat warning: #20282. That has gone from failing every other run, to failing not at all in one week. That concerns me. I see two choices:

  1. Merge as-is, and suffer when flakes happen, and hope that it incentivizes someone to fix them; or
  2. Edit my PR, adding a skip if vfs to all cp tests.

Luap99
Luap99 reviewed Nov 6, 2023
View reviewed changes
@packit-as-a-service
Copy link
Copy Markdown

packit-as-a-service bot commented Nov 6, 2023

Cockpit tests failed for commit 881cdae821d5974522f390f407c3e8ff8dd651c4. @martinpitt, @jelly, @mvollmer please check.

@packit-as-a-service
Copy link
Copy Markdown

packit-as-a-service bot commented Nov 6, 2023

Cockpit tests failed for commit 2bbf47854143075fe75674c7d68917e510224b21. @martinpitt, @jelly, @mvollmer please check.

@packit-as-a-service
Copy link
Copy Markdown

packit-as-a-service bot commented Nov 6, 2023

Ephemeral COPR build failed. @containers/packit-build please check.

@packit-as-a-service
Copy link
Copy Markdown

packit-as-a-service bot commented Nov 6, 2023

Cockpit tests failed for commit 254b9e59427822d4b4f040931a585329f332563b. @martinpitt, @jelly, @mvollmer please check.

@packit-as-a-service
Copy link
Copy Markdown

packit-as-a-service bot commented Nov 6, 2023

Ephemeral COPR build failed. @containers/packit-build please check.

@packit-as-a-service
Copy link
Copy Markdown

packit-as-a-service bot commented Nov 6, 2023

Cockpit tests failed for commit fe0b71afabe599b17ebfbeba5fb7473d899b7ade. @martinpitt, @jelly, @mvollmer please check.

Luap99
Luap99 reviewed Nov 7, 2023
View reviewed changes
@packit-as-a-service
Copy link
Copy Markdown

packit-as-a-service bot commented Nov 8, 2023

Ephemeral COPR build failed. @containers/packit-build please check.

@edsantiago
Copy link
Copy Markdown
Member Author

edsantiago commented Nov 14, 2023

This is as ready as I can make it. Any feedback on my flake question?

@containers/podman-maintainers I am marking this as ready for review... with one big fat warning: #20282. That has gone from failing every other run, to failing not at all in one week. That concerns me. I see two choices:

1. Merge as-is, and suffer when flakes happen, and hope that it incentivizes someone to fix them; or

2. Edit my PR, adding a `skip if vfs` to all `cp` tests.

Luap99
Luap99 approved these changes Nov 14, 2023
View reviewed changes
Copy link
Copy Markdown
Member

@Luap99 Luap99 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, I fine to merge is as. If it flakes to often we can always skip later.

@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci bot commented Nov 14, 2023

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: edsantiago, Luap99

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@edsantiago
CI: test overlay and vfs
a10b88c 
We're only testing vfs in CI. That's bad. containers#18822 tried to
remedy that but that only worked on system tests, not e2e.

Here we introduce CI_DESIRED_STORAGE, to be set in .cirrus.yml
in the same vein as all the other CI_DESIRED_X. Since it's 2023
we default to overlay, testing vfs only in priorfedora.

Fixes required:
 - e2e tests:
   - in cleanup, umount ROOT/overlay to avoid leaking mounts

 - system tests:
   - fix a few badly-written tests that assumed/hardcoded overlay
   - buildx test: add weird exception to device-number test
   - mount tests: add special case code for vfs
   - unprivileged test: disable one section that is N/A on vfs

Signed-off-by: Ed Santiago <santiago@redhat.com>
@rhatdan
Copy link
Copy Markdown
Member

rhatdan commented Nov 18, 2023

/lgtm

@edsantiago edsantiago mentioned this pull request Dec 6, 2023
Closed
@edsantiago edsantiago mentioned this pull request Dec 18, 2023
Open
@edsantiago edsantiago mentioned this pull request Feb 8, 2024
Closed
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

@Luap99 Luap99 Luap99 approved these changes

Assignees

@rhatdan rhatdan

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. lgtm Indicates that a PR is ready to be merged. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. release-note-none

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@edsantiago @vrothberg @rhatdan @martinpitt @Luap99 @openshift-merge-robot