Add OCI/Image Volume Source support #10579

wzshiming · 2024-08-12T05:44:16Z

k8s-ci-robot · 2024-08-12T05:44:26Z

Hi @wzshiming. Thanks for your PR.

I'm waiting for a containerd member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

mikebrow · 2025-02-17T21:15:38Z

FYI cri adds the subpath support for image volume type: kubernetes/kubernetes#130135

nod .. let's get this one merged and follow up with subpath, unless subpath has already merged and the test is required.

mikebrow · 2025-02-17T21:48:32Z

=== RUN TestImageMount
main_test.go:786: Image "registry.k8s.io/pause:3.10" already exists, not pulling.
main_test.go:790: Pull test image "ghcr.io/containerd/alpine:3.14.0"
image_mount_test.go:165:
Error Trace: /home/runner/work/containerd/containerd/integration/image_mount_test.go:165
/home/runner/work/containerd/containerd/integration/image_mount_test.go:44
Error: Received unexpected error:
timeout exceeded
Test: TestImageMount
--- FAIL: TestImageMount (31.71s)

hmm.. a timeout .. maybe use BusyBox it should always be smaller..

fuweid · 2025-02-18T02:53:17Z

   default: time="2025-02-17T12:02:03.557783499Z" level=debug msg="remove snapshot" key=/run/containerd-test/io.containerd.grpc.v1.cri/image-volumes/e282eb3b5a1a8337b5e9016253b41f3359460325ec91e4d0e0dd966ad5187bca/ee6521f290b2168b6e0935a181d4cff9be1ac3f505666ef0e3c98fae8199917a snapshotter=overlayfs
    default: time="2025-02-17T12:02:03.558221517Z" level=info msg="Container df376860f476b3d601d8c82e6ab95adcd1b7e49688fe937fd688ed8b54feccab: CDI devices from CRI Config.CDIDevices: []"
    default: time="2025-02-17T12:02:03.559726745Z" level=debug msg="schedule snapshotter cleanup" snapshotter=overlayfs
    default: time="2025-02-17T12:02:03.560854785Z" level=debug msg="removed snapshot" key=k8s.io/95/e3c6fc9bf3c590cd333f266b2379748796bc7e9f1ea2ed840f06a018f19ab15e snapshotter=overlayfs
    default: time="2025-02-17T12:02:03.561669736Z" level=debug msg="removed snapshot" key=k8s.io/99//run/containerd-test/io.containerd.grpc.v1.cri/image-volumes/e282eb3b5a1a8337b5e9016253b41f3359460325ec91e4d0e0dd966ad5187bca/ee6521f290b2168b6e0935a181d4cff9be1ac3f505666ef0e3c98fae8199917a snapshotter=overlayfs
    default: time="2025-02-17T12:02:03.562498973Z" level=debug msg="removed snapshot" key=k8s.io/97/dbe9f8f1545805758f5aefc6b671fb8f320c8e1a2bcab1f9156e527d9552d7af snapshotter=overlayfs
    default: time="2025-02-17T12:02:03.563695813Z" level=info msg="CreateContainer within sandbox \"e282eb3b5a1a8337b5e9016253b41f3359460325ec91e4d0e0dd966ad5187bca\" for &ContainerMetadata{Name:test-image-mount-container,Attempt:0,} returns container id \"df376860f476b3d601d8c82e6ab95adcd1b7e49688fe937fd688ed8b54feccab\""

Checked the code and found that snapshot creation doesn't use lease so that containerd GC cleanups the volume after creation. That's why the log content is empty. Since the GC cleanups snapshots async, CI could be happy if ls -Z executes before GC.

please consider using the following code to fix.

diff --git a/internal/cri/server/container_image_mount.go b/internal/cri/server/container_image_mount.go
index 4e84da212..b8d9cf59c 100644
--- a/internal/cri/server/container_image_mount.go
+++ b/internal/cri/server/container_image_mount.go
@@ -23,6 +23,7 @@ import (
        "path/filepath"

        containerd "github.com/containerd/containerd/v2/client"
+       "github.com/containerd/containerd/v2/core/leases"
        "github.com/containerd/containerd/v2/core/mount"
        "github.com/containerd/errdefs"
        "github.com/containerd/log"
@@ -39,6 +40,11 @@ func (c *criService) mutateMounts(
        sandboxID string,
        platform imagespec.Platform,
 ) error {
+       if err := c.ensureLeaseExist(ctx, sandboxID); err != nil {
+               return fmt.Errorf("failed to ensure lease %v for sandbox: %w", sandboxID, err)
+       }
+
+       ctx = leases.WithLease(ctx, sandboxID)
        for _, m := range extraMounts {
                err := c.mutateImageMount(ctx, m, snapshotter, sandboxID, platform)
                if err != nil {
@@ -48,6 +54,17 @@ func (c *criService) mutateMounts(
        return nil
 }

+func (c *criService) ensureLeaseExist(ctx context.Context, sandboxID string) error {
+       leaseSvc := c.client.LeasesService()
+       _, err := leaseSvc.Create(ctx, leases.WithID(sandboxID))
+       if err != nil {
+               if errdefs.IsAlreadyExists(err) {
+                       err = nil
+               }
+       }
+       return err
+}
+
 func (c *criService) mutateImageMount(
        ctx context.Context,
        extraMount *runtime.Mount,
diff --git a/internal/cri/server/sandbox_remove.go b/internal/cri/server/sandbox_remove.go
index f7e1ed880..b2f728e6d 100644
--- a/internal/cri/server/sandbox_remove.go
+++ b/internal/cri/server/sandbox_remove.go
@@ -21,6 +21,7 @@ import (
        "fmt"
        "time"

+       "github.com/containerd/containerd/v2/core/leases"
        "github.com/containerd/containerd/v2/pkg/tracing"
        "github.com/containerd/errdefs"
        "github.com/containerd/log"
@@ -56,6 +57,12 @@ func (c *criService) RemovePodSandbox(ctx context.Context, r *runtime.RemovePodS
                return nil, fmt.Errorf("failed to forcibly stop sandbox %q: %w", id, err)
        }

+       if err := c.client.LeasesService().Delete(ctx, leases.Lease{ID: id}); err != nil {
+               if !errdefs.IsNotFound(err) {
+                       return nil, fmt.Errorf("failed to delete lease for sandbox %q: %w", id, err)
+               }
+       }
+
        // Return error if sandbox network namespace is not closed yet.
        if sandbox.NetNS != nil {
                nsPath := sandbox.NetNS.GetPath()
diff --git a/internal/cri/server/sandbox_run.go b/internal/cri/server/sandbox_run.go
index e895d66b3..5e25a007f 100644
--- a/internal/cri/server/sandbox_run.go
+++ b/internal/cri/server/sandbox_run.go
@@ -31,6 +31,7 @@ import (
        "github.com/containerd/typeurl/v2"
        runtime "k8s.io/cri-api/pkg/apis/runtime/v1"

+       "github.com/containerd/containerd/v2/core/leases"
        sb "github.com/containerd/containerd/v2/core/sandbox"
        "github.com/containerd/containerd/v2/internal/cri/annotations"
        "github.com/containerd/containerd/v2/internal/cri/bandwidth"
@@ -87,6 +88,22 @@ func (c *criService) RunPodSandbox(ctx context.Context, r *runtime.RunPodSandbox
                }
        }()

+       leaseSvc := c.client.LeasesService()
+       ls, lerr := leaseSvc.Create(ctx, leases.WithID(id))
+       if lerr != nil {
+               return nil, fmt.Errorf("failed to create lease for sandbox name %q: %w", name, lerr)
+       }
+       defer func() {
+               if retErr != nil {
+                       deferCtx, deferCancel := util.DeferContext()
+                       defer deferCancel()
+
+                       if derr := leaseSvc.Delete(deferCtx, ls); derr != nil {
+                               log.G(deferCtx).WithError(derr).Error("failed to delete lease during cleanup")
+                       }
+               }
+       }()
+
        var (
                err         error
                sandboxInfo = sb.Sandbox{ID: id}

Signed-off-by: Shiming Zhang <wzshiming@hotmail.com>

wzshiming · 2025-02-18T12:30:26Z

@fuweid it works, thank you!!!

fuweid

LGTM

let's get this one merged first. Thanks @wzshiming!

…lpha-features` We now explicitly enable the image volume tests in the job, because it should be now supported after the merge of containerd/containerd#10579. We also still keep it in the alpha tests, because with the beta graduation it will purposely disabled by default. Signed-off-by: Sascha Grunert <sgrunert@redhat.com>

containerd 2.1.0 Welcome to the v2.1.0 release of containerd! The first minor release of containerd 2.x focuses on continued stability alongside new features and improvements. This is the first time-based released for containerd. Most the feature set and core functionality has long been stable and harderened in production environments, so now we transition to a balance of timely delivery of new functionality with the same high confidence in stability and performance. * Add no_sync option to boost boltDB performance on ephemeral environments ([containerd#10745](containerd#10745)) * Add content create event ([containerd#11006](containerd#11006)) * Erofs snapshotter and differ ([containerd#10705](containerd#10705)) * Update CRI to use transfer service for image pull by default ([containerd#8515](containerd#8515)) * Support multiple cni plugin bin dirs ([containerd#11311](containerd#11311)) * Support container restore through CRI/Kubernetes ([containerd#10365](containerd#10365)) * Add OCI/Image Volume Source support ([containerd#10579](containerd#10579)) * Enable Writable cgroups for unprivileged containers ([containerd#11131](containerd#11131)) * Fix recursive RLock() mutex acquisition ([containerd/go-cni#126](containerd/go-cni#126)) * Support CNI STATUS Verb ([containerd/go-cni#123](containerd/go-cni#123)) * Retry last registry host on 50x responses ([containerd#11484](containerd#11484)) * Multipart layer fetch ([containerd#10177](containerd#10177)) * Enable HTTP debug and trace for transfer based puller ([containerd#10762](containerd#10762)) * Add support for unpacking custom media types ([containerd#11744](containerd#11744)) * Add dial timeout field to hosts toml configuration ([containerd#11106](containerd#11106)) * Expose Pod assigned IPs to NRI plugins ([containerd#10921](containerd#10921)) * Support multiple uid/gid mappings ([containerd#10722](containerd#10722)) * Fix race between serve and immediate shutdown on the server ([containerd/ttrpc#175](containerd/ttrpc#175)) * Update FreeBSD defaults and re-organize platform defaults ([containerd#11017](containerd#11017)) * Postpone cri config deprecations to v2.2 ([containerd#11684](containerd#11684)) * Remove deprecated dynamic library plugins ([containerd#11683](containerd#11683)) * Remove the support for Schema 1 images ([containerd#11681](containerd#11681)) Please try out the release binaries and report any issues at https://github.com/containerd/containerd/issues. * Derek McGowan * Phil Estes * Akihiro Suda * Maksym Pavlenko * Jin Dong * Wei Fu * Sebastiaan van Stijn * Samuel Karp * Mike Brown * Adrien Delorme * Austin Vazquez * Akhil Mohan * Kazuyoshi Kato * Henry Wang * Gao Xiang * ningmingxiao * Krisztian Litkey * Yang Yang * Archit Kulkarni * Chris Henzie * Iceber Gu * Alexey Lunev * Antonio Ojea * Davanum Srinivas * Marat Radchenko * Michael Zappa * Paweł Gronowski * Rodrigo Campos * Alberto Garcia Hierro * Amit Barve * Andrey Smirnov * Divya * Etienne Champetier * Kirtana Ashok * Philip Laine * QiPing Wan * fengwei0328 * zounengren * Adrian Reber * Alfred Wingate * Amal Thundiyil * Athos Ribeiro * Brian Goff * Cesar Talledo * ChengyuZhu6 * Chongyi Zheng * Craig Ingram * Danny Canter * David Son * Fupan Li * HirazawaUi * Jing Xu * Jonathan A. Sternberg * Jose Fernandez * Kaita Nakamura * Kohei Tokunaga * Lei Liu * Marco Visin * Mike Baynton * Qiyuan Liang * Sameer * Shiming Zhang * Swagat Bora * Teresaliu * Tony Fang * Tõnis Tiigi * Vered Rosen * Vinayak Goyal * bo.jiang * chriskery * luchenhan * mahmut * zhaixiaojuan * **github.com/Microsoft/hcsshim** v0.12.9 -> v0.13.0-rc.3 * **github.com/cilium/ebpf** v0.11.0 -> v0.16.0 * **github.com/containerd/cgroups/v3** v3.0.3 -> v3.0.5 * **github.com/containerd/containerd/api** v1.8.0 -> v1.9.0 * **github.com/containerd/continuity** v0.4.4 -> v0.4.5 * **github.com/containerd/go-cni** v1.1.10 -> v1.1.12 * **github.com/containerd/imgcrypt/v2** v2.0.0-rc.1 -> v2.0.1 * **github.com/containerd/otelttrpc** ea5083fda723 -> v0.1.0 * **github.com/containerd/platforms** v1.0.0-rc.0 -> v1.0.0-rc.1 * **github.com/containerd/ttrpc** v1.2.6 -> v1.2.7 * **github.com/containerd/typeurl/v2** v2.2.2 -> v2.2.3 * **github.com/containernetworking/cni** v1.2.3 -> v1.3.0 * **github.com/containernetworking/plugins** v1.5.1 -> v1.7.1 * **github.com/containers/ocicrypt** v1.2.0 -> v1.2.1 * **github.com/davecgh/go-spew** d8f796af33cc -> v1.1.1 * **github.com/fsnotify/fsnotify** v1.7.0 -> v1.9.0 * **github.com/go-jose/go-jose/v4** v4.0.4 -> v4.0.5 * **github.com/google/go-cmp** v0.6.0 -> v0.7.0 * **github.com/grpc-ecosystem/grpc-gateway/v2** v2.22.0 -> v2.26.1 * **github.com/klauspost/compress** v1.17.11 -> v1.18.0 * **github.com/mdlayher/socket** v0.4.1 -> v0.5.1 * **github.com/moby/spdystream** v0.4.0 -> v0.5.0 * **github.com/moby/sys/user** v0.3.0 -> v0.4.0 * **github.com/opencontainers/image-spec** v1.1.0 -> v1.1.1 * **github.com/opencontainers/runtime-spec** v1.2.0 -> v1.2.1 * **github.com/opencontainers/selinux** v1.11.1 -> v1.12.0 * **github.com/pelletier/go-toml/v2** v2.2.3 -> v2.2.4 * **github.com/petermattis/goid** 4fcff4a6cae7 **_new_** * **github.com/pmezard/go-difflib** 5d4384ee4fb2 -> v1.0.0 * **github.com/prometheus/client_golang** v1.20.5 -> v1.22.0 * **github.com/prometheus/common** v0.55.0 -> v0.62.0 * **github.com/sasha-s/go-deadlock** v0.3.5 **_new_** * **github.com/smallstep/pkcs7** v0.1.1 **_new_** * **github.com/stretchr/testify** v1.9.0 -> v1.10.0 * **github.com/tchap/go-patricia/v2** v2.3.1 -> v2.3.2 * **github.com/urfave/cli/v2** v2.27.5 -> v2.27.6 * **github.com/vishvananda/netlink** v1.3.0 -> 0e7078ed04c8 * **github.com/vishvananda/netns** v0.0.4 -> v0.0.5 * **go.etcd.io/bbolt** v1.3.11 -> v1.4.0 * **go.opentelemetry.io/auto/sdk** v1.1.0 **_new_** * **go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc** v0.56.0 -> v0.60.0 * **go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp** v0.56.0 -> v0.60.0 * **go.opentelemetry.io/otel** v1.31.0 -> v1.35.0 * **go.opentelemetry.io/otel/exporters/otlp/otlptrace** v1.31.0 -> v1.35.0 * **go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc** v1.31.0 -> v1.35.0 * **go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp** v1.31.0 -> v1.35.0 * **go.opentelemetry.io/otel/metric** v1.31.0 -> v1.35.0 * **go.opentelemetry.io/otel/sdk** v1.31.0 -> v1.35.0 * **go.opentelemetry.io/otel/trace** v1.31.0 -> v1.35.0 * **go.opentelemetry.io/proto/otlp** v1.3.1 -> v1.5.0 * **golang.org/x/crypto** v0.28.0 -> v0.36.0 * **golang.org/x/exp** aacd6d4b4611 -> 2d47ceb2692f * **golang.org/x/mod** v0.21.0 -> v0.24.0 * **golang.org/x/net** v0.30.0 -> v0.38.0 * **golang.org/x/oauth2** v0.22.0 -> v0.27.0 * **golang.org/x/sync** v0.8.0 -> v0.14.0 * **golang.org/x/sys** v0.26.0 -> v0.33.0 * **golang.org/x/term** v0.25.0 -> v0.30.0 * **golang.org/x/text** v0.19.0 -> v0.23.0 * **golang.org/x/time** v0.3.0 -> v0.7.0 * **google.golang.org/genproto/googleapis/api** 5fefd90f89a9 -> 56aae31c358a * **google.golang.org/genproto/googleapis/rpc** 324edc3d5d38 -> 56aae31c358a * **google.golang.org/grpc** v1.67.1 -> v1.72.0 * **google.golang.org/protobuf** v1.35.1 -> v1.36.6 * **k8s.io/api** v0.31.2 -> v0.32.3 * **k8s.io/apimachinery** v0.31.2 -> v0.32.3 * **k8s.io/apiserver** v0.31.2 -> v0.32.3 * **k8s.io/client-go** v0.31.2 -> v0.32.3 * **k8s.io/cri-api** v0.31.2 -> v0.32.3 * **k8s.io/kubelet** v0.31.2 -> v0.32.3 * **k8s.io/utils** 18e509b52bc8 -> 3ea5e8cea738 * **sigs.k8s.io/json** bc3834ca7abd -> 9aa6b5e7a4b3 * **sigs.k8s.io/structured-merge-diff/v4** v4.4.1 -> v4.4.2 * **tags.cncf.io/container-device-interface** v0.8.0 -> v1.0.1 * **tags.cncf.io/container-device-interface/specs-go** v0.8.0 -> v1.0.0 Previous release can be found at [v2.0.0](https://github.com/containerd/containerd/releases/tag/v2.0.0) * `containerd-<VERSION>-<OS>-<ARCH>.tar.gz`: ✅Recommended. Dynamically linked with glibc 2.35 (Ubuntu 22.04). * `containerd-static-<VERSION>-<OS>-<ARCH>.tar.gz`: Statically linked. Expected to be used on Linux distributions that do not use glibc >= 2.35. Not position-independent. In addition to containerd, typically you will have to install [runc](https://github.com/opencontainers/runc/releases) and [CNI plugins](https://github.com/containernetworking/plugins/releases) from their official sites too. See also the [Getting Started](https://github.com/containerd/containerd/blob/main/docs/getting-started.md) documentation.

k8s-ci-robot added do-not-merge/work-in-progress needs-ok-to-test size/M labels Aug 12, 2024

wzshiming force-pushed the feat/oci-image-mount branch 2 times, most recently from 4446688 to fdc84b3 Compare August 12, 2024 06:38

k8s-ci-robot added size/L and removed size/M labels Aug 12, 2024

wzshiming force-pushed the feat/oci-image-mount branch 22 times, most recently from 8de3699 to 1589b41 Compare August 12, 2024 16:40

wzshiming force-pushed the feat/oci-image-mount branch from 8bda7ec to bca0a3c Compare February 17, 2025 11:40

wzshiming force-pushed the feat/oci-image-mount branch 2 times, most recently from b3e5b26 to c812018 Compare February 18, 2025 06:08

Add OCI/Image Volume Source support

1ec10d9

Signed-off-by: Shiming Zhang <wzshiming@hotmail.com>

wzshiming force-pushed the feat/oci-image-mount branch from c812018 to 1ec10d9 Compare February 18, 2025 07:59

fuweid approved these changes Feb 18, 2025

View reviewed changes

fuweid added this pull request to the merge queue Feb 18, 2025

Merged via the queue into containerd:main with commit e6ecbdd Feb 18, 2025
59 checks passed

pacoxu mentioned this pull request Feb 24, 2025

[KEP-4639] Graduate image volume sources to beta kubernetes/kubernetes#130135

Merged

saschagrunert mentioned this pull request Feb 24, 2025

Enable ImageVolume tests for pull-kubernetes-node-e2e-containerd-alpha-features kubernetes/test-infra#34398

Merged

dmcgowan added the impact/changelog label Mar 8, 2025

dmcgowan mentioned this pull request Mar 8, 2025

Prepare release notes for v2.1.0-beta.0 #11510

Merged

pacoxu mentioned this pull request Mar 10, 2025

pull-kubernetes-node-e2e-containerd-alpha-features: ImageVolume/Device Plugin test failed kubernetes/kubernetes#130669

Closed

djdongjin mentioned this pull request Mar 22, 2025

[KEP-4639] Support image volume mount subpath #11578

Merged

fuweid mentioned this pull request Mar 25, 2025

*: image volume feature's follow-up #11605

Merged

dmcgowan mentioned this pull request Apr 23, 2025

Add release notes for v2.1.0-rc.0 #11752

Merged

dmcgowan mentioned this pull request May 7, 2025

Prepare release notes for v2.1.0 #11819

Merged

creydr mentioned this pull request May 12, 2025

ImageVolume support incomplete? kubernetes-sigs/kind#3745

Closed

melinda-mytra mentioned this pull request May 23, 2025

Bump Containerd to 2.1 to support OCI image volumes k3s-io/k3s#12390

Closed

SergeyKanzhelev mentioned this pull request Sep 4, 2025

[KEP-4639] Graduate image volumes to GA kubernetes/enhancements#5450

Merged

carlory mentioned this pull request Nov 11, 2025

[Failing Test] [sig-node] ImageVolume should succeed with pod and pull policy of Always [MinimumKubeletVersion:1.35] kubernetes/kubernetes#135247

Closed

This was referenced Dec 10, 2025

Support ImageVolume feature Mirantis/cri-dockerd#526

Open

KEP-4639: OCI VolumeSource Mirantis/cri-dockerd#528

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add OCI/Image Volume Source support #10579

Add OCI/Image Volume Source support #10579

Uh oh!

wzshiming commented Aug 12, 2024

Uh oh!

k8s-ci-robot commented Aug 12, 2024

Uh oh!

mikebrow commented Feb 17, 2025

Uh oh!

mikebrow commented Feb 17, 2025

Uh oh!

fuweid commented Feb 18, 2025 •

edited

Loading

Uh oh!

wzshiming commented Feb 18, 2025

Uh oh!

fuweid left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

14 participants

Add OCI/Image Volume Source support #10579

Add OCI/Image Volume Source support #10579

Uh oh!

Conversation

wzshiming commented Aug 12, 2024

Uh oh!

k8s-ci-robot commented Aug 12, 2024

Uh oh!

mikebrow commented Feb 17, 2025

Uh oh!

mikebrow commented Feb 17, 2025

Uh oh!

fuweid commented Feb 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wzshiming commented Feb 18, 2025

Uh oh!

fuweid left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

14 participants

fuweid commented Feb 18, 2025 •

edited

Loading