Add simultaneous unpack support by Random-Liu · Pull Request #2918 · containerd/containerd

Random-Liu · 2019-01-09T02:30:01Z

Time consumed to pull k8s.gcr.io/kube-cross:v1.10.3-1 on my machine:

Containerd HEAD: 40.347s
Containerd HEAD + Add direct unpack support for overlay and aufs #2913: 35.65s
Containerd HEAD + this change: 37.86s
Containerd HEAD + Add direct unpack support for overlay and aufs #2913 + this change: 32.95s

codecov-io · 2019-01-09T10:10:40Z

Codecov Report

Merging #2918 into master will decrease coverage by 0.87%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #2918      +/-   ##
==========================================
- Coverage   42.29%   41.41%   -0.88%     
==========================================
  Files         126       70      -56     
  Lines       13871     9457    -4414     
==========================================
- Hits         5867     3917    -1950     
+ Misses       7118     4979    -2139     
+ Partials      886      561     -325

Flag	Coverage Δ
#linux	`?`
#windows	`41.41% <ø> (+4.15%)`	⬆️

Impacted Files	Coverage Δ
snapshots/native/native.go	`1.78% <0%> (-41.27%)`	⬇️
archive/tar.go	`16.99% <0%> (-30.97%)`	⬇️
archive/tar_opts.go	`28.57% <0%> (-30.26%)`	⬇️
metadata/snapshot.go	`21.53% <0%> (-26.38%)`	⬇️
metadata/leases.go	`44.27% <0%> (-6.98%)`	⬇️
metadata/gc.go	`61.35% <0%> (-5.41%)`	⬇️
errdefs/grpc.go	`75% <0%> (-2.5%)`	⬇️
oci/spec_opts.go	`26.8% <0%> (-2.41%)`	⬇️
remotes/docker/authorizer.go	`68.2% <0%> (-2.33%)`	⬇️
content/local/writer.go	`56.86% <0%> (-1.8%)`	⬇️
... and 85 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bd46ea5...03aafaa. Read the comment docs.

fuweid · 2019-01-09T15:05:14Z

pull.go

how about return NewImageWithPlatform(c, img, pullCtx.PlatformMatcher), nil

fuweid · 2019-01-09T16:49:19Z

pull.go

it seems that it doesn't consider the manifest list(index) case, for example, pull hello-world:latest with --all-platform, if my understanding is correct

Pull only supports one platform, see https://github.com/containerd/containerd/blob/master/client.go#L377.

I can add a comment to explain this.

fuweid · 2019-01-10T06:11:28Z

pull.go

how about ctx.Err() instead of errors.New("context cancelled")? it might be closed by deadline.

fuweid · 2019-01-10T07:22:26Z

pull.go

could we filter by the mediatype? it is easy to break if dependence on the order.

Good idea, will do.

AkihiroSuda · 2019-01-31T15:13:11Z

needs rebase

AkihiroSuda · 2019-04-04T01:22:16Z

Any update?

AkihiroSuda · 2019-07-18T04:12:35Z

needs rebase

theopenlab-ci · 2019-07-31T21:56:35Z

Build succeeded.

containerd-build-arm64 : SUCCESS in 8m 44s (non-voting)

theopenlab-ci · 2019-07-31T22:33:41Z

Build succeeded.

containerd-build-arm64 : SUCCESS in 8m 19s (non-voting)

theopenlab-ci · 2019-08-01T00:19:48Z

Build succeeded.

containerd-build-arm64 : SUCCESS in 8m 18s (non-voting)

theopenlab-ci · 2019-08-09T22:01:34Z

Build succeeded.

containerd-build-arm64 : SUCCESS in 7m 54s (non-voting)

dmcgowan · 2019-08-12T18:07:57Z

unpacker.go

There should be some check here that the image and chain ID are aligned. I think my preference would be that instead of just returning the chain, return a combination of chain and config, then use the config directly from there.

I think the problem is that we don't have config for schema1 until convert is done.

Added validation.

This is bitter sweet to me. I hate the code structure, but to support schema1 I have to do it in this way. :(

Added a TODO for the cleanup after schema1 support removal.

theopenlab-ci · 2019-08-12T22:32:10Z

Build succeeded.

containerd-build-arm64 : SUCCESS in 8m 07s (non-voting)

theopenlab-ci · 2019-08-12T22:40:26Z

Build succeeded.

containerd-build-arm64 : SUCCESS in 7m 29s (non-voting)

fuweid · 2019-08-13T06:37:12Z

unpacker.go

I think we can do nothing for manifest list instead of sending invalid layer.

We can skip the manifest list.

fuweid

LGTM

Random-Liu · 2019-08-16T17:25:36Z

Offline discussed with @dmcgowan, I'm going to drop the schema1 support and cleanup the code.

As a followup, we can also add manifest list support, but that would be a following PR I think @dmcgowan

Random-Liu · 2019-08-20T08:13:01Z

@dmcgowan Removed schema 1 support, and cleaned up the code.

theopenlab-ci · 2019-08-20T09:07:43Z

Build succeeded.

containerd-build-arm64 : SUCCESS in 7m 38s (non-voting)

dmcgowan · 2019-08-21T22:48:40Z

pull.go

The target shouldn't have this media type. Instead is there a way to just see if the simultaneous unpacker didn't perform any unpacks, then try and do it synchronously

Oops, the image was converted already. My bad.

Let me see.

You can use an atomic counter on the number of unpack calls

Done. Added an atomic int unpacks.

Signed-off-by: Lantao Liu <lantaol@google.com>

theopenlab-ci · 2019-08-21T23:41:50Z

Build succeeded.

containerd-build-arm64 : SUCCESS in 7m 38s (non-voting)

dmcgowan · 2019-08-23T17:40:41Z

LGTM

I had to make some changes to ctr to test it there. There needs to be more thought to getting this working correctly in ctr as well as supporting multiple platforms. We can focus on this for 1.4.

crosbymichael · 2019-08-23T19:05:52Z

LGTM

Random-Liu · 2019-08-23T20:54:49Z

Thanks for reviewing!

yxxhero · 2025-05-22T14:41:26Z

What are the basic conditions needed to use this feature? Thank you so much. @Random-Liu

Random-Liu assigned dmcgowan and AkihiroSuda Jan 9, 2019

Random-Liu added this to the 1.3 milestone Jan 9, 2019

Random-Liu mentioned this pull request Jan 9, 2019

Optimize Image Pull #2886

Closed

Random-Liu force-pushed the parellel-unpack branch from 5990b7f to 31cc342 Compare January 9, 2019 09:25

fuweid reviewed Jan 9, 2019

View reviewed changes

Random-Liu force-pushed the parellel-unpack branch from 31cc342 to 92538f7 Compare January 9, 2019 18:22

Random-Liu mentioned this pull request Jan 9, 2019

Add image pull concurrency limit. #2920

Merged

Random-Liu force-pushed the parellel-unpack branch from 92538f7 to e5c4dca Compare January 9, 2019 22:12

fuweid reviewed Jan 10, 2019

View reviewed changes

Random-Liu force-pushed the parellel-unpack branch from e5c4dca to ec09808 Compare January 10, 2019 09:01

dmcgowan mentioned this pull request Feb 13, 2019

Add image handler wrapper #3010

Merged

estesp mentioned this pull request Feb 22, 2019

Content store to snapshotter performance #3042

Closed

Random-Liu force-pushed the parellel-unpack branch 4 times, most recently from f581cc6 to 94d0a60 Compare July 31, 2019 21:45

Random-Liu force-pushed the parellel-unpack branch from 94d0a60 to b96e746 Compare July 31, 2019 22:23

Random-Liu force-pushed the parellel-unpack branch from b96e746 to 0081ba4 Compare August 1, 2019 00:09

Random-Liu force-pushed the parellel-unpack branch from 0081ba4 to 570c195 Compare August 9, 2019 21:52

AkihiroSuda approved these changes Aug 10, 2019

View reviewed changes

crosbymichael requested a review from dmcgowan August 12, 2019 16:58

dmcgowan reviewed Aug 12, 2019

View reviewed changes

Random-Liu force-pushed the parellel-unpack branch 2 times, most recently from 2fdfa51 to 10c756b Compare August 12, 2019 22:32

fuweid reviewed Aug 13, 2019

View reviewed changes

fuweid reviewed Aug 15, 2019

View reviewed changes

Random-Liu force-pushed the parellel-unpack branch from 10c756b to 8ece078 Compare August 20, 2019 08:12

Random-Liu force-pushed the parellel-unpack branch from 8ece078 to f6f56bf Compare August 20, 2019 08:58

dmcgowan reviewed Aug 21, 2019

View reviewed changes

Random-Liu added 2 commits August 21, 2019 16:31

Support simultaneous image unpack.

60f73b6

Signed-off-by: Lantao Liu <lantaol@google.com>

Update the integration test.

03aafaa

Signed-off-by: Lantao Liu <lantaol@google.com>

Random-Liu force-pushed the parellel-unpack branch from f6f56bf to 03aafaa Compare August 21, 2019 23:31

crosbymichael merged commit 4a2f61c into containerd:master Aug 23, 2019

Random-Liu deleted the parellel-unpack branch August 23, 2019 20:54

yxxhero mentioned this pull request May 24, 2025

promoting use of parallel image pulls kubernetes/kubernetes#108405

Open

7 tasks

Conversation

Random-Liu commented Jan 9, 2019

Uh oh!

codecov-io commented Jan 9, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AkihiroSuda commented Jan 31, 2019

Uh oh!

AkihiroSuda commented Apr 4, 2019

Uh oh!

AkihiroSuda commented Jul 18, 2019

Uh oh!

theopenlab-ci bot commented Jul 31, 2019

Uh oh!

theopenlab-ci bot commented Jul 31, 2019

Uh oh!

theopenlab-ci bot commented Aug 1, 2019

Uh oh!

theopenlab-ci bot commented Aug 9, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Random-Liu Aug 12, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Random-Liu Aug 12, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

theopenlab-ci bot commented Aug 12, 2019

Uh oh!

theopenlab-ci bot commented Aug 12, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Random-Liu Aug 13, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fuweid left a comment

Choose a reason for hiding this comment

Uh oh!

Random-Liu commented Aug 16, 2019

Uh oh!

Random-Liu commented Aug 20, 2019

Uh oh!

theopenlab-ci bot commented Aug 20, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov-io commented Jan 9, 2019 •

edited

Loading

Random-Liu Aug 12, 2019 •

edited

Loading

Random-Liu Aug 12, 2019 •

edited

Loading

Random-Liu Aug 13, 2019 •

edited

Loading

dmcgowan commented Aug 23, 2019 •

edited

Loading