I recently did some benchmark to compare the pull image performance between containerd and docker.

• Test environment:
  • Machine Type: n1-standard-1
  • Disk Size: 50GB
• Image information:
  • k8s.gcr.io/kube-cross:v1.10.3-1
  • Uncompressed size: 2.08 GB
  • Compressed size: 728 MB
  • Largest layer compressed size: 184 MB
  • Layer count: 14

It turns out that containerd image pull is significantly slower than Docker:

• With pigz: ~70s vs. ~57s (+23%)
• Without pigz: ~75s vs. 65s (+15%)

After some investigation, I find that this is mainly caused by 2 issues:

1. Containerd only starts unpacking the image after all contents are downloaded. (Issue: Optimize unpacking on pulling #2737)
2. Containerd temporarily mount/umount all parent snapshots when unpacking each layer. The mount/umount can take 1~2 seconds for large layers. @dmcgowan told me that Docker has an "optimization" (dangerous hack) where it directly converts tars to overlayfs layers on disk without mounting. We could add this ability on unpack to optimize pull.

I made some local change to resolve 1) and 2) on my desktop, and here is the performance difference:

1. Docker 18.06: 28.033s
2. Containerd HEAD: 40.347s
3. Containerd HEAD - mount/umount: 36.521s
4. Containerd HEAD - mount/umount + simultaneous unpack: 29.898s

Pull image performance is important for some use case, e.g. fast auto scale. And the improvement is significant. We may want to do the optimization to avoid regression.

/cc @containerd/containerd-maintainers