Add support to migrate containers by adrianreber · Pull Request #2272 · containers/podman

adrianreber · 2019-02-05T19:08:09Z

This series adds container migration support to Podman.

The basic steps to migrate containers are:

 Source system:
  * podman container checkpoint -l -e /tmp/checkpoint.tar.gz
  * scp /tmp/checkpoint.tar.gz destination:/tmp

 Destination system:
  * podman pull 'container-image-as-on-source-system'
  * podman container restore -i /tmp/checkpoint.tar.gz

For the newly added test to work an updated runc is required, which is still under review: opencontainers/runc#1968

Tries to solve: #1618

This PR includes the actual code, man-pages, bash-completion, tests, tutorial update.

Once this is merged I would also like to publish a related article on podman.io.

cmd/podman/restore.go

libpod/container_internal_linux.go

docs/podman-container-restore.1.md

rst0git · 2019-02-05T22:27:44Z

The restore of a looper example fails for me. Still investigating why...
restore.log

libpod/container_api.go

libpod/container_internal.go

libpod/container_internal_linux.go

libpod/runtime_ctr.go

cmd/podman/restore.go

openshift-merge-robot · 2019-02-06T06:29:30Z

/retest

adrianreber · 2019-02-06T07:15:56Z

The restore of a looper example fails for me. Still investigating why...
restore.log

Are you using runc with the changes from opencontainers/runc#1968? Could be there is some runc code missing to create /proc just as it did not create bind mount mountpoints before opencontainers/runc#1968. Please let me know how you are testing it so I can try to recreate.

rst0git · 2019-02-06T09:07:45Z

Are you using runc with the changes from opencontainers/runc#1968?

Yes, I installed runc from opencontainers/runc#1968, criu from the criu-dev branch and podman from this PR.

Could be there is some runc code missing to create /proc just as it did not create bind mount mountpoints before opencontainers/runc#1968. Please let me know how you are testing it so I can try to recreate.

I followed these steps to create, checkpoint, and restore a container:

sudo podman run -d --name looper busybox /bin/sh -c 'i=0; while true; do echo $i; i=$(expr $i + 1); sleep 1; done'
sudo podman container checkpoint looper -e /tmp/checkpoint.tar.gz
sudo podman container rm looper
sudo podman container restore -i /tmp/checkpoint.tar.gz

Even though the restore has failed in the last step, a looper container is still created (with status Created). Therefore, I can still run sudo podman container start looper to start the container.

adrianreber · 2019-02-06T10:47:20Z

@rst0git This is the same bug as in my runc pull request except that it is also true for non-bind mount mountpoints. It seems runc just creates all missing mountpoints even for read-only root filesystems. If you use a container image which includes all required mountpoints (in this case /run /proc /sys are missing) it should work. I will update my runc PR to handle also missing non-bind mount mountpoints.

adrianreber · 2019-02-06T10:50:17Z

@mheon Thanks for the review. At first I was unsure if I can do all the necessary changes, but now, after I have thought about it, I think I can actually do everything you suggested and the result will be better. I hope. So thanks for pointing it out.

openshift-merge-robot · 2019-02-06T14:49:39Z

/retest

avagin · 2019-02-06T18:33:25Z

scp /tmp/checkpoint.tar.gz destination:/tmp
What is inside /tmp/checkpoint.tar.gz? Do you snapshot a container rootfs?

adrianreber · 2019-02-06T18:46:02Z

scp /tmp/checkpoint.tar.gz destination:/tmp
What is inside /tmp/checkpoint.tar.gz? Do you snapshot a container rootfs?

The checkpoint directory and the container definition (spec, config, network). No filesystem (yet). I was thinking about doing an automatic commit of the highest layer and including it also. But right now it is only the output of CRIU and some json files.

adrianreber · 2019-02-06T19:29:26Z

@mheon I tried to rework my changes to the newContainer() function. There is now a RestoreContainer() function. Please have a look if this is now a better approach.

adrianreber · 2019-02-06T20:35:08Z

All CI errors are the expected errors as long as the necessary runc patches have not been merged.

libpod/runtime_ctr.go

mheon · 2019-02-06T21:56:33Z

I'll do a more thorough review tomorrow, but I'm generally in favor of the way NewContainer was split up.

Still a little iffy on copying over the OCI config... I need to check, but there shouldn't be much in there that isn't deterministically generated, so it might not be necessary.

We do need to make sure that the ContainerConfig we copied over makes sense before we use it - if the container is in a pod, we need a pod with the same ID present on the remote host, and the same holds for named volumes. We might want to make a generic sanity checker for ContainerConfig that we can use for both NewContainer and this.

rh-atomic-bot · 2019-02-06T22:04:56Z

☔ The latest upstream changes (presumably #2252) made this pull request unmergeable. Please resolve the merge conflicts.

cevich · 2019-05-28T19:09:34Z

Nothing here and tests seem happy 😄

rhatdan · 2019-05-28T21:33:45Z

@mheon PTAL

adrianreber · 2019-06-03T17:44:37Z

@mheon any more comments on this PR?

mheon · 2019-06-03T17:49:43Z

Sorry, been distracted by other things. Let me do a final review.

cmd/podman/restore.go

libpod/container_internal_linux.go

libpod/runtime_ctr.go

mheon · 2019-06-03T18:23:24Z

Few things, but overall looks solid.

@rhatdan @vrothberg @giuseppe @TomSweeneyRedHat PTAL

adrianreber · 2019-06-03T18:50:41Z

@mheon , thanks a lot. I will add the checks for named volumes and dependencies and re-push.

Signed-off-by: Adrian Reber <areber@redhat.com>

This adds a couple of function in structure members needed in the next commit to make container migration actually work. This just splits of the function which are not modifying existing code. Signed-off-by: Adrian Reber <areber@redhat.com>

This commit adds an option to the checkpoint command to export a checkpoint into a tar.gz file as well as importing a checkpoint tar.gz file during restore. With all checkpoint artifacts in one file it is possible to easily transfer a checkpoint and thus enabling container migration in Podman. With the following steps it is possible to migrate a running container from one system (source) to another (destination). Source system: * podman container checkpoint -l -e /tmp/checkpoint.tar.gz * scp /tmp/checkpoint.tar.gz destination:/tmp Destination system: * podman pull 'container-image-as-on-source-system' * podman container restore -i /tmp/checkpoint.tar.gz The exported tar.gz file contains the checkpoint image as created by CRIU and a few additional JSON files describing the state of the checkpointed container. Now the container is running on the destination system with the same state just as during checkpointing. If the container is kept running on the source system with the checkpoint flag '-R', the result will be that the same container is running on two different hosts. Signed-off-by: Adrian Reber <areber@redhat.com>

The difference between container checkpoint/restore and container migration is that for migration the container which was checkpointed must not exist during restore. To simulate migration the container is remove ('podman rm -fa') before being restored. The migration test does following steps: * podman run * podman container checkpoint -l -e /tmp/checkpoint.tar.gz * podman rm -fa * podman container restore -i /tmp/checkpoint.tar.gz Signed-off-by: Adrian Reber <areber@redhat.com>

Signed-off-by: Adrian Reber <areber@redhat.com>

If restoring a container from a checkpoint it was necessary that the image the container is based was already available (podman pull). This commit adds the image download to podman container restore if it does not exist. Signed-off-by: Adrian Reber <areber@redhat.com>

adrianreber · 2019-06-03T20:13:42Z

I think I was able to implememt all the changes from the latest review, let's see if the tests still pass.

The option to restore a container from an external checkpoint archive (podman container restore -i /tmp/checkpoint.tar.gz) restores a container with the same name and same ID as id had before checkpointing. This commit adds the option '--name,-n' to 'podman container restore'. With this option the restored container gets the name specified after '--name,-n' and a new ID. This way it is possible to restore one container multiple times. If a container is restored with a new name Podman will not try to request the same IP address for the container as it had during checkpointing. This implicitly assumes that if a container is restored from a checkpoint archive with a different name, that it will be restored multiple times and restoring a container multiple times with the same IP address will fail as each IP address can only be used once. Signed-off-by: Adrian Reber <areber@redhat.com>

adrianreber · 2019-06-04T12:58:53Z

All tests green again (after a few retries).

adrianreber · 2019-06-06T20:20:16Z

Any further comments regarding this PR?

mheon · 2019-06-06T21:35:11Z

Sorry, we've been in a bit of a rush trying to get the recent CVE patched.

I'm good to merge with one more LGTM

rhatdan · 2019-06-07T12:18:14Z

/lgtm

adrianreber · 2019-06-07T20:41:25Z

Thanks everyone for the reviews and the patience getting this merged.

rst0git · 2019-06-08T22:36:00Z

test/e2e/checkpoint_test.go

+	// a container from one host to another
+	It("podman checkpoint container with export (migration)", func() {
+		// CRIU does not work with seccomp correctly on RHEL7
+		session := podmanTest.Podman([]string{"run", "-it", "--security-opt", "seccomp=unconfined", "-d", ALPINE, "top"})


Hi @adrianreber, I was wondering what is the reason for seccomp=unconfined, is there a GitHub issue for it?

@rst0git : This is a RHEL7 - CRIU limitation. CRIU cannot handle seccomp with the RHEL7 kernel. I never really tried to understand why it does not work, but is does not work on the RHEL7 kernel. I was using RHEL7 as a development platform initially, that is why I worked around the seccomp limitations there. Not sure it is necessary to fix it.

Thank you for the explanation Adrian! I was curious because the seccomp support seems to work on Fedora.

openshift-ci-robot requested review from baude and rhatdan February 5, 2019 19:08

openshift-ci-robot added the size/L label Feb 5, 2019

adrianreber force-pushed the migration branch 4 times, most recently from 331cae2 to 4d3fa26 Compare February 5, 2019 19:35

rst0git reviewed Feb 5, 2019

View reviewed changes

cmd/podman/restore.go Outdated Show resolved Hide resolved

rst0git reviewed Feb 5, 2019

View reviewed changes

libpod/container_internal_linux.go Outdated Show resolved Hide resolved

rst0git reviewed Feb 5, 2019

View reviewed changes

docs/podman-container-restore.1.md Outdated Show resolved Hide resolved

mheon reviewed Feb 6, 2019

View reviewed changes

adrianreber mentioned this pull request Feb 6, 2019

Create bind mount mountpoints during restore opencontainers/runc#1968

Merged

adrianreber force-pushed the migration branch 2 times, most recently from 74ab7b8 to 0ba0da7 Compare February 6, 2019 19:26

mheon reviewed Feb 6, 2019

View reviewed changes

libpod/runtime_ctr.go Outdated Show resolved Hide resolved

openshift-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Feb 7, 2019

adrianreber force-pushed the migration branch from 0ba0da7 to af784d5 Compare February 7, 2019 07:24

mheon reviewed Jun 3, 2019

View reviewed changes

cmd/podman/restore.go Outdated Show resolved Hide resolved

mheon reviewed Jun 3, 2019

View reviewed changes

cmd/podman/restore.go Outdated Show resolved Hide resolved

mheon reviewed Jun 3, 2019

View reviewed changes

libpod/container_internal_linux.go Outdated Show resolved Hide resolved

mheon reviewed Jun 3, 2019

View reviewed changes

libpod/runtime_ctr.go Outdated Show resolved Hide resolved

adrianreber added 8 commits June 3, 2019 22:05

Fix restore options help text and comments

e0c8c14

Signed-off-by: Adrian Reber <areber@redhat.com>

Added bash completion for container migration

7b1ab8a

Signed-off-by: Adrian Reber <areber@redhat.com>

Add man-pages for container migration

42e903d

Signed-off-by: Adrian Reber <areber@redhat.com>

Include container migration into tutorial

a4d3333

Signed-off-by: Adrian Reber <areber@redhat.com>

rh-atomic-bot mentioned this pull request Jun 7, 2019

use conmon for exec #3143

Merged

7 tasks

mheon mentioned this pull request Jun 7, 2019

RFE: the ability to checkpoint on a host and restore on another #1618

Closed

adrianreber mentioned this pull request Jun 8, 2019

docker checkpoint an experimental feature checkpoint-restore/criu#718

Closed

rst0git reviewed Jun 8, 2019

View reviewed changes

edsantiago mentioned this pull request Jun 10, 2019

BATS tests - get working again #3290

Merged

Conversation

adrianreber commented Feb 5, 2019

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rst0git commented Feb 5, 2019

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

openshift-merge-robot commented Feb 6, 2019

Uh oh!

adrianreber commented Feb 6, 2019

Uh oh!

rst0git commented Feb 6, 2019

Uh oh!

adrianreber commented Feb 6, 2019

Uh oh!

adrianreber commented Feb 6, 2019

Uh oh!

openshift-merge-robot commented Feb 6, 2019

Uh oh!

avagin commented Feb 6, 2019

Uh oh!

adrianreber commented Feb 6, 2019

Uh oh!

adrianreber commented Feb 6, 2019

Uh oh!

adrianreber commented Feb 6, 2019

Uh oh!

Uh oh!

mheon commented Feb 6, 2019

Uh oh!

rh-atomic-bot commented Feb 6, 2019

Uh oh!

cevich commented May 28, 2019

Uh oh!

rhatdan commented May 28, 2019

Uh oh!

adrianreber commented Jun 3, 2019

Uh oh!

mheon commented Jun 3, 2019

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mheon commented Jun 3, 2019

Uh oh!

adrianreber commented Jun 3, 2019

Uh oh!

adrianreber commented Jun 3, 2019

Uh oh!

adrianreber commented Jun 4, 2019

Uh oh!

adrianreber commented Jun 6, 2019

Uh oh!

mheon commented Jun 6, 2019

Uh oh!

rhatdan commented Jun 7, 2019

Uh oh!

adrianreber commented Jun 7, 2019

Uh oh!

rst0git Jun 8, 2019

Choose a reason for hiding this comment

Uh oh!

adrianreber Jun 19, 2019

Choose a reason for hiding this comment

Uh oh!

rst0git Jun 20, 2019

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels