CI: Use local cache registry by edsantiago · Pull Request #22726 · containers/podman

edsantiago · 2024-05-16T13:16:37Z

As of containers/automation_images#357 our CI VMs include a local registry preloaded with all(*) images used in tests.

(*) where "all" means "most".

This PR sets up CI such that it will use that registry.

Signed-off-by: Ed Santiago santiago@redhat.com

None

packit-as-a-service · 2024-05-16T13:20:49Z

Ephemeral COPR build failed. @containers/packit-build please check.

Luap99 · 2024-05-16T16:20:01Z

test/e2e/pull_test.go

heads up, tests should still pass locally and I don't think we want to setup a local registry there right?
Thus I would think we need a regex or a Or() matcher to match both strings anyway

This is thorny. As tests are currently written, e2e tests hard-force the use of test/registries.conf. No matter where they're run (CI, laptop, anywhere). This may need to be reevaluated, but I'm not bothering with any of that until I find out if this approach is viable.

yes sure keep testing, just keep in mind that the end result must still work locally

Luap99 · 2024-05-16T16:20:52Z

contrib/cirrus/runner.sh

this does not do anything, not sure if you were expecting anything with that or if this serves documentation purposes.

I need the container to be able to talk to host:56789. --port does the opposite, IIRC: host can talk to container. I'll look at logs and see what happens.

The container uses --network=host so it shares the network namespace with the host so from a network POV there should be no functional difference

Duh. Thanks, I missed that.

contrib/cirrus/local-cache-registry

...to minimize hiccups. RUN-2091 in Jira. Network registries are too unreliable; they cause too many flakes in CI. Here we set up a registry running on each VM, prepopulated with all container images used in podman and buildah tests. Related PRs: containers/podman#22726 containers/buildah#5584 Once those merge, podman and buildah CI tests will fetch images from this local registry. Signed-off-by: Ed Santiago <santiago@redhat.com>

Luap99

LGTM

Thinking about this should we set up a new cron run that runs without the cache registry to make sure local tests will keep working? (not blocking of course)

edsantiago · 2024-07-08T13:44:31Z

Good idea.

/hold

Let's not merge this yet. I saw this bud flake over the weekend and I want to understand it better.

...to minimize hiccups. RUN-2091 in Jira. Network registries are too unreliable; they cause too many flakes in CI. Here we set up a registry running on each VM, prepopulated with all container images used in podman and buildah tests. Related PRs: containers/podman#22726 containers/buildah#5584 Once those merge, podman and buildah CI tests will fetch images from this local registry. Signed-off-by: Ed Santiago <santiago@redhat.com>

As of containers/automation_images#357 our CI VMs include a local registry preloaded with all(*) images used in tests. * where "all" means "most". This commit installs a new registries.conf that redirects docker and quay to the new local registry. The hope is that this will reduce CI flakes. Since tests change over time, and new tests may require new images, this commit also adds a mechanism for pulling in remote images at test run time. Obviously this negates the purpose of the cache, since it introduces a flake pain point. The idea is: DO NOT DO THIS UNLESS ABSOLUTELY NECESSARY, and then, if we have to do this, hurry up and spin new CI VMs that include the new image(s). Signed-off-by: Ed Santiago <santiago@redhat.com>

This commit gets tests working under the new local-registry system: * amend a few image names, mostly just sticking to a consistent list of those images in our registry cache. Mostly minor tag updates. * trickier: pull_test: change some error messages, and remove a test that's now a NOP. Basically, with a local (unprotected) registry we always get "404 manifest unknown"; with a real registry we'll get "403 I can't tell you". * trickiest: seccomp_test: build our own images at run time, with our desired labels. Until now we've been pulling prebuilt images, but those will not copy to the local cache registry. Something about v1? Anyhow, I gave up trying to cache them, and the workaround is straightforward. Also took the liberty of strengthening a few error-message checks Signed-off-by: Ed Santiago <santiago@redhat.com>

New tool, get-local-registry-script, intended for developers to get a local registry running in their environment. This is not necessary for any tests, but may be desirable for performance reasons and/or to recreate the CI environment. Signed-off-by: Ed Santiago <santiago@redhat.com>

edsantiago

Once again I think this is ready for review. My thanks for everyone's patience.

edsantiago · 2024-07-11T12:07:16Z

test/e2e/info_test.go

+			session = podmanTest.Podman([]string{"info", "--format", "{{.Host.RemoteSocket.Exists}}"})
+			session.WaitWithDefaultTimeout()
+			Expect(session).Should(ExitCleanly())
+			Expect(session.OutputToString()).To(Equal("true"))


This is, I think, the only change since last review. The latest Debian VM image creates and recreates /run/user/XXX/podman/podman.sock at (seemingly) random times. I've given up trying to understand what is creating it, and honestly I choose to decide that this test was broken. There is no harm in the socket existing in a podman-local test. The only necessary test is that if podman-remote, then socket-must-exist.

Luap99

LGTM

openshift-ci · 2024-07-11T12:16:00Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: edsantiago, Luap99

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [Luap99,edsantiago]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

rhatdan · 2024-07-11T12:37:57Z

/lgtm
I will let @edsantiago cancel the hold

edsantiago · 2024-07-11T12:39:43Z

/hold cancel

(I forgot about that). Thanks.

openshift-ci bot added release-note-none approved Indicates a PR has been approved by an approver from all required OWNERS files. labels May 16, 2024

edsantiago force-pushed the pull-from-local-registry branch 5 times, most recently from 684512b to c82b31a Compare May 16, 2024 16:14

Luap99 reviewed May 16, 2024

View reviewed changes

edsantiago force-pushed the pull-from-local-registry branch 7 times, most recently from 03825bc to 48e506f Compare May 22, 2024 14:40

Luap99 reviewed May 22, 2024

View reviewed changes

contrib/cirrus/local-cache-registry Outdated Show resolved Hide resolved

edsantiago force-pushed the pull-from-local-registry branch 10 times, most recently from 293a60b to 88fa517 Compare May 28, 2024 13:54

edsantiago force-pushed the pull-from-local-registry branch 3 times, most recently from 3bc7dbf to 80ed2a7 Compare May 30, 2024 01:57

edsantiago force-pushed the pull-from-local-registry branch from 644ae51 to edde737 Compare July 2, 2024 18:37

edsantiago force-pushed the pull-from-local-registry branch from edde737 to 75f8520 Compare July 2, 2024 21:23

edsantiago mentioned this pull request Jul 3, 2024

Windows: install dotnet and latest wix containers/automation_images#354

Merged

edsantiago force-pushed the pull-from-local-registry branch 2 times, most recently from 202cf63 to 61053e5 Compare July 8, 2024 13:29

Luap99 reviewed Jul 8, 2024

View reviewed changes

openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 8, 2024

edsantiago marked this pull request as draft July 8, 2024 13:44

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 8, 2024

edsantiago force-pushed the pull-from-local-registry branch from 61053e5 to 21f73f2 Compare July 8, 2024 14:52

edsantiago force-pushed the pull-from-local-registry branch 2 times, most recently from afc6438 to 9c82836 Compare July 8, 2024 17:13

edsantiago added 3 commits July 11, 2024 04:39

edsantiago commented Jul 11, 2024

View reviewed changes

Luap99 approved these changes Jul 11, 2024

View reviewed changes

This was referenced Jul 11, 2024

Create a local registry containers/automation_images#357

Merged

CI: use local registry containers/buildah#5584

Merged

Conversation

edsantiago commented May 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

packit-as-a-service bot commented May 16, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Luap99 left a comment

Choose a reason for hiding this comment

Uh oh!

edsantiago commented Jul 8, 2024

Uh oh!

edsantiago left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Luap99 left a comment

Choose a reason for hiding this comment

Uh oh!

openshift-ci bot commented Jul 11, 2024

Uh oh!

rhatdan commented Jul 11, 2024

Uh oh!

edsantiago commented Jul 11, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

edsantiago commented May 16, 2024 •

edited

Loading