Skip to content

[WIP] buildkit integration#52

Closed
ktock wants to merge 1 commit intomasterfrom
buildkit
Closed

[WIP] buildkit integration#52
ktock wants to merge 1 commit intomasterfrom
buildkit

Conversation

@ktock
Copy link
Member

@ktock ktock commented Mar 4, 2020 •
edited
Loading

This is an experimental integration with buildkit for speeding up fetching base image.

Our patched version of buildkit is here.

This commit includes benchmark scripts to measure the time for buliding sample images.
See this doc for manual testing.

Though it seems good for the lazy distribution of the base images, we currently have two problems for exporting output image.

  1. Archiving layers takes a long time. It is because of the low READ performance including fetching contents from the registry.
  2. We only support exporting type tar. It is because when we use remote snapshotters, containerd doesn't store the image contents in the content store but buildkit needs to use these contents for exporting images.

@github-actions
Copy link

github-actions bot commented Mar 4, 2020

Benchmarking Result

Runs on the ubuntu-18.04 runner on Github Actions.

python:3.7

mode pull(sec) create(sec) run(sec)
legacy 30.80867099761963 0.02688884735107422 0.1361839771270752
stargz 2.2660980224609375 0.05379605293273926 1.0895819664001465
estargz 2.6346559524536133 0.31299591064453125 0.17602109909057617

gcc:9.2.0

mode pull(sec) create(sec) run(sec)
legacy 33.24489402770996 0.032514095306396484 0.14731407165527344
stargz 2.3038840293884277 0.05231308937072754 2.3267719745635986
estargz 2.2181098461151123 0.03937196731567383 2.003551959991455

rethinkdb:2.3.6

mode pull(sec) create(sec) run(sec)
legacy 8.111081838607788 0.02593088150024414 0.12709808349609375
stargz 1.1628179550170898 0.04550909996032715 1.7875571250915527
estargz 1.4818990230560303 1.3497328758239746 0.17242002487182617

glassfish:4.1-jdk8

mode pull(sec) create(sec) run(sec)
legacy 27.240201950073242 0.02878880500793457 2.4394888877868652
stargz 2.4255881309509277 0.05092787742614746 16.81278896331787
estargz 2.556001901626587 7.158114194869995 2.92494797706604

@github-actions
Copy link

github-actions bot commented Mar 4, 2020

Buildkit integration performance(without exporting)

OCI Worker

With original baseimage

Arguments for buildkitd: --oci-worker-snapshotter=overlayfs
Context for buildctl: ./script/buildkit/sample_org

real	0m16.974s
user	0m0.018s
sys	0m0.018s

With stargz baseimage

Arguments for buildkitd: --oci-worker-snapshotter=stargz --oci-worker-proxy-snapshotter-address=/run/containerd-stargz-grpc/containerd-stargz-grpc.sock
Context for buildctl: ./script/buildkit/sample_sgz

containerd Worker

real 0m3.044s
user 0m0.011s
sys 0m0.011s

With original baseimage

Arguments for buildkitd: --oci-worker=false --containerd-worker=true
Context for buildctl: ./script/buildkit/sample_org

real 0m17.914s
user 0m0.029s
sys 0m0.017s

With stargz baseimage

Arguments for buildkitd: --oci-worker=false --containerd-worker=true --containerd-worker-snapshotter=stargz
Context for buildctl: ./script/buildkit/sample_sgz

real 0m2.976s
user 0m0.000s
sys 0m0.018s

Buildkit integration performance(with exporting: --output type=tar)

OCI Worker

With original baseimage

Arguments for buildkitd: --oci-worker-snapshotter=overlayfs
Context for buildctl: ./script/buildkit/sample_org

real	0m24.823s
user	0m4.136s
sys	0m1.756s

With stargz baseimage

Arguments for buildkitd: --oci-worker-snapshotter=stargz --oci-worker-proxy-snapshotter-address=/run/containerd-stargz-grpc/containerd-stargz-grpc.sock
Context for buildctl: ./script/buildkit/sample_sgz

real	1m35.111s
user	0m11.298s
sys	0m6.532s

containerd Worker

With original baseimage

Arguments for buildkitd: --oci-worker=false --containerd-worker=true
Context for buildctl: ./script/buildkit/sample_org

real	0m34.437s
user	0m4.173s
sys	0m1.891s

With stargz baseimage

Arguments for buildkitd: --oci-worker=false --containerd-worker=true --containerd-worker-snapshotter=stargz
Context for buildctl: ./script/buildkit/sample_sgz

real	1m26.735s
user	0m10.159s
sys	0m6.171s

@ktock
[WIP] buildkit integration
da181c0 
Signed-off-by: Kohei Tokunaga <ktokunaga.mail@gmail.com>
@github-actions
Copy link

github-actions bot commented Mar 4, 2020

Benchmarking Result

Runs on the ubuntu-18.04 runner on Github Actions.

python:3.7

mode pull(sec) create(sec) run(sec)
legacy 31.67469096183777 0.0321660041809082 0.15041399002075195
stargz 2.2904369831085205 0.06085491180419922 1.0506479740142822
estargz 2.5134129524230957 0.1449909210205078 0.17807698249816895

gcc:9.2.0

mode pull(sec) create(sec) run(sec)
legacy 32.86303901672363 0.0327608585357666 0.13144803047180176
stargz 2.5307531356811523 0.04351210594177246 2.359808921813965
estargz 2.3148438930511475 0.04931497573852539 2.002725124359131

rethinkdb:2.3.6

mode pull(sec) create(sec) run(sec)
legacy 8.167922973632812 0.040946006774902344 0.14784908294677734
stargz 1.871750831604004 0.05150914192199707 1.833923101425171
estargz 1.6254642009735107 1.3515660762786865 0.19742989540100098

glassfish:4.1-jdk8

mode pull(sec) create(sec) run(sec)
legacy 27.817287921905518 0.026812076568603516 2.305588960647583
stargz 2.4905929565429688 0.04720902442932129 14.54265809059143
estargz 2.451666831970215 7.0523388385772705 2.938326120376587

@github-actions
Copy link

github-actions bot commented Mar 4, 2020

Buildkit integration performance(without exporting)

OCI Worker

With original baseimage

Arguments for buildkitd: --oci-worker-snapshotter=overlayfs
Context for buildctl: ./script/buildkit/sample_org

real	0m16.473s
user	0m0.024s
sys	0m0.012s

With stargz baseimage

Arguments for buildkitd: --oci-worker-snapshotter=stargz --oci-worker-proxy-snapshotter-address=/run/containerd-stargz-grpc/containerd-stargz-grpc.sock
Context for buildctl: ./script/buildkit/sample_sgz

real	0m3.054s
user	0m0.015s
sys	0m0.007s

containerd Worker

With original baseimage

Arguments for buildkitd: --oci-worker=false --containerd-worker=true
Context for buildctl: ./script/buildkit/sample_org

real	0m17.628s
user	0m0.016s
sys	0m0.032s

With stargz baseimage

Arguments for buildkitd: --oci-worker=false --containerd-worker=true --containerd-worker-snapshotter=stargz
Context for buildctl: ./script/buildkit/sample_sgz

real	0m2.944s
user	0m0.006s
sys	0m0.012s

Buildkit integration performance(with exporting: --output type=tar)

OCI Worker

With original baseimage

Arguments for buildkitd: --oci-worker-snapshotter=overlayfs
Context for buildctl: ./script/buildkit/sample_org

real	0m25.007s
user	0m4.507s
sys	0m1.571s

With stargz baseimage

Arguments for buildkitd: --oci-worker-snapshotter=stargz --oci-worker-proxy-snapshotter-address=/run/containerd-stargz-grpc/containerd-stargz-grpc.sock
Context for buildctl: ./script/buildkit/sample_sgz

real	1m31.731s
user	0m9.909s
sys	0m6.174s

containerd Worker

With original baseimage

Arguments for buildkitd: --oci-worker=false --containerd-worker=true
Context for buildctl: ./script/buildkit/sample_org

real	0m31.812s
user	0m4.004s
sys	0m1.555s

With stargz baseimage

Arguments for buildkitd: --oci-worker=false --containerd-worker=true --containerd-worker-snapshotter=stargz
Context for buildctl: ./script/buildkit/sample_sgz

real	1m23.942s
user	0m9.094s
sys	0m5.968s

@AkihiroSuda
Copy link
Member

AkihiroSuda commented Mar 4, 2020

Can you open a proposal for BuildKit changes in https://github.com/moby/buildkit/issues

cc @tonistiigi

@ktock
Copy link
Member Author

ktock commented Mar 4, 2020 •
edited
Loading

Yes. Before that, please let me re-open this PR from my forked repo.

@ktock ktock mentioned this pull request Mar 4, 2020
Closed
@ktock
Copy link
Member Author

ktock commented Mar 4, 2020

Reopened on #53.

@ktock ktock closed this Mar 4, 2020
@ktock ktock deleted the buildkit branch March 4, 2020 08:42
@strigazi strigazi mentioned this pull request Sep 30, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@ktock @AkihiroSuda