Add parallel unpack support #12332

henry118 · 2025-09-25T19:10:37Z

Implement #8881 based on the discussions in the thread.

New config

Add max_concurrent_unpacks config to transfer service. If this value is >1, parallel unpacking feature could be enabled if the snapshotter has "rebase" capability.

[plugins]
  [plugins.'io.containerd.transfer.v1.local']
    max_concurrent_downloads = 20
    concurrent_layer_fetch_buffer = 16777216
    max_concurrent_unpacks = 5   <--- NEW!

New snapshotter capability

Added rebase capability to snapshotters. A snapshotter can declare this capability to allow parallel unpacking. The overlayfs snapshotter is updated in this PR to announce this capability.

The "rebase" is performed in the Commit snapshotter API. The caller will pass in the WithParent snapshot option. The snapshotter can rebase the current snapshot onto the parent during the commit operation.

This rebase capability requires a update to the API protobuf to add a parent field into the CommitSnapshotRequest message.

Updated unpacker logic

The existing doUnpackFn is split to two halves.

The top half performs the regular snapshot preparing and layer fetching. Instead of "applying" the layer contents synchronously, the updated logic does it in a go routine.
The bottom half performs the snapshot commit depending on the result of top half. There are two modes:
- Sequential: when the feature is off. It commits each snapshot before moving to the next layer.
- Parallel: the new logic collects the top half's result and immediately moves to the next layer. At the end, commits are performed sequentially from the collected results.

* **Parallel Unpack**  ([#12332](https://github.com/containerd/containerd/pull/12332))

  Adds support for unpacking layers in parallel during pull operations. This feature is supported with overlayfs and EROFS snapshotters.

dmcgowan · 2025-09-25T23:24:56Z

This would need to be handled in the proxied snapshotters as well, in core/snapshots/proxy and core/metadata/snapshots.go

dmcgowan

One little change, I still want to take another look through the unpack code, it was dense before and probably more so now. The core unpack code has always been like that though, but the other parts look good.

core/metadata/snapshot.go

henry118 · 2025-10-17T21:46:06Z

One little change, I still want to take another look through the unpack code, it was dense before and probably more so now. The core unpack code has always been like that though, but the other parts look good.

Yep the unpack code isn't trivial to follow or make changes to. When I am working on this patch my primary focus was keeping the existing logic without too much refactoring.

dmcgowan · 2025-10-23T19:58:11Z

core/unpack/unpacker.go

+				errs = errors.Join(errs, err)
+			}
+		}
+		if errs != nil {


Should this return errs and not err?

dmcgowan · 2025-10-23T20:00:41Z

plugins/snapshots/overlay/overlay.go

 		}

+		if info.Parent != "" && base.Parent != "" && base.Parent != info.Parent {
+			return fmt.Errorf("unabled to change parent of snapshot %q from %q to %q", key, info.Parent, base.Parent)


typo unable

dmcgowan · 2025-10-23T20:06:57Z

core/snapshots/storage/bolt.go


+		// If the snapshot didn't have a parent when created, allow it
+		// to be rebased on a parent on commit.
+		if si.Parent == "" && si.Parent != base.Parent {


Maybe the logic would be better as first checking si.Parent != base.Parent. If si.Parent is empty, set it, otherwise, we should probably throw an invalid argument error.

dmcgowan · 2025-10-23T20:09:52Z

core/unpack/unpacker.go

+		statusCh, err := topHalf(i, desc, layerSpan, unpackLayerStart)
+		if err != nil {
+			if parallel {
+				break


Should we close out the status channels here or will the context cancellation propagate?

The context cancellation will propagate and cause all goroutines created by top halves to return. On return the defer func will close the status channels.

Signed-off-by: Henry Wang <henwang@amazon.com>

dmcgowan · 2025-10-24T18:58:45Z

plugins/snapshots/overlay/overlay.go

 			return err
 		}

+		if len(info.Parent) > 0 && len(base.Parent) > 0 && info.Parent != base.Parent {


Don't need to make the change in this PR, but I think we could just remove this logic now and let the storage error handle this. This also means there is no other change needed in other snapshotters other than to set the capability.

Yep. I can change it in a follow-up PR.

github-project-automation bot added this to Pull Request Review Sep 25, 2025

github-project-automation bot moved this to Needs Triage in Pull Request Review Sep 25, 2025

k8s-ci-robot added the size/L label Sep 25, 2025

dosubot bot added the area/runtime Runtime label Sep 25, 2025

henry118 force-pushed the parallel-unpack branch from 1a777f2 to 7b5fd4f Compare September 25, 2025 19:17

henry118 marked this pull request as draft September 25, 2025 20:07

k8s-ci-robot added the do-not-merge/work-in-progress label Sep 25, 2025

henry118 force-pushed the parallel-unpack branch from 7b5fd4f to 7f78056 Compare September 25, 2025 20:24

henry118 marked this pull request as ready for review September 25, 2025 22:19

k8s-ci-robot removed the do-not-merge/work-in-progress label Sep 25, 2025

henry118 force-pushed the parallel-unpack branch from 7f78056 to 70737f4 Compare September 26, 2025 18:58

k8s-ci-robot added size/XXL and removed size/L labels Sep 26, 2025

henry118 force-pushed the parallel-unpack branch from 70737f4 to 4ccc52f Compare September 26, 2025 19:02

henry118 force-pushed the parallel-unpack branch 4 times, most recently from e885452 to 50b6dea Compare October 9, 2025 20:17

dmcgowan moved this from Needs Triage to Needs Reviewers in Pull Request Review Oct 15, 2025

dmcgowan added this to the 2.2 milestone Oct 15, 2025

dmcgowan assigned dmcgowan and fuweid Oct 15, 2025

dmcgowan reviewed Oct 17, 2025

View reviewed changes

core/metadata/snapshot.go Show resolved Hide resolved

dmcgowan moved this from Needs Reviewers to Review In Progress in Pull Request Review Oct 17, 2025

dmcgowan moved this from Review In Progress to Needs Update in Pull Request Review Oct 17, 2025

dmcgowan assigned henry118 Oct 17, 2025

henry118 force-pushed the parallel-unpack branch from 50b6dea to 7795e89 Compare October 17, 2025 21:39

henry118 force-pushed the parallel-unpack branch from 7795e89 to 9f6792b Compare October 17, 2025 22:32

henry118 force-pushed the parallel-unpack branch 2 times, most recently from f1660de to 9da85bf Compare October 18, 2025 06:56

dmcgowan moved this from Needs Update to Needs Reviewers in Pull Request Review Oct 20, 2025

dmcgowan reviewed Oct 23, 2025

View reviewed changes

dmcgowan moved this from Needs Reviewers to Needs Update in Pull Request Review Oct 24, 2025

henry118 force-pushed the parallel-unpack branch from 9da85bf to f5fc44b Compare October 24, 2025 17:19

Implement parallel unpack

0198b87

Signed-off-by: Henry Wang <henwang@amazon.com>

henry118 force-pushed the parallel-unpack branch from f5fc44b to 0198b87 Compare October 24, 2025 17:54

dmcgowan approved these changes Oct 24, 2025

View reviewed changes

dmcgowan reviewed Oct 24, 2025

View reviewed changes

fuweid approved these changes Oct 24, 2025

View reviewed changes

github-project-automation bot moved this from Needs Update to Review In Progress in Pull Request Review Oct 24, 2025

dmcgowan added this pull request to the merge queue Oct 24, 2025

dmcgowan changed the title ~~Implement parallel unpack~~ Parallel unpack Oct 24, 2025

dmcgowan added the impact/changelog label Oct 24, 2025

dmcgowan moved this from Review In Progress to Merge on Green in Pull Request Review Oct 24, 2025

Merged via the queue into containerd:main with commit 2e747a0 Oct 24, 2025
94 of 96 checks passed

github-project-automation bot moved this from Merge on Green to Done in Pull Request Review Oct 24, 2025

dmcgowan added area/distribution Image Distribution and removed area/runtime Runtime labels Oct 24, 2025

dmcgowan changed the title ~~Parallel unpack~~ Add parallel unpack support Oct 24, 2025

dmcgowan mentioned this pull request Oct 24, 2025

Prepare release notes for api/v1.10.0-rc.0 #12408

Merged

henry118 deleted the parallel-unpack branch October 24, 2025 21:27

henry118 mentioned this pull request Oct 24, 2025

Remove rebase validation logic from overlay snapshotter #12409

Merged

dmcgowan mentioned this pull request Oct 26, 2025

Prepare release notes for v2.2.0-rc.0 #12413

Merged

henry118 mentioned this pull request Oct 29, 2025

Parallel Container Layer Unpacking #8881

Closed

This was referenced Nov 5, 2025

Prepare release notes for api/v1.10.0 #12472

Merged

Prepare release notes for v2.2.0 #12473

Merged

Add parallel unpack support #12332

Add parallel unpack support #12332

Uh oh!

Conversation

henry118 commented Sep 25, 2025 • edited by dmcgowan Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

New config

New snapshotter capability

Updated unpacker logic

Uh oh!

dmcgowan commented Sep 25, 2025

Uh oh!

dmcgowan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

henry118 commented Oct 17, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

henry118 commented Sep 25, 2025 •

edited by dmcgowan

Loading