Skip to content

Generic layer support#3649

Merged
dmcgowan merged 1 commit intocontainerd:masterfrom
dmcgowan:generic-layer-support
Sep 20, 2019
Merged

Generic layer support#3649
dmcgowan merged 1 commit intocontainerd:masterfrom
dmcgowan:generic-layer-support

Conversation

@dmcgowan
Copy link
Copy Markdown
Member

Updates layer support to generically handle the OCI layer types, removing the need to continue to add to a long list of possible values. For example, encrypted containers are using "+enc", however this is not yet standardized but support is desirable. Additionally, OCI has added "+zstd" to define zstd compression. This change ensures "+zstd" defines the correct compression algorithm and adds it to the compression library.

@dmcgowan dmcgowan added this to the 1.3 milestone Sep 12, 2019
@theopenlab-ci
Copy link
Copy Markdown

theopenlab-ci bot commented Sep 12, 2019

Build succeeded.

@dmcgowan
Copy link
Copy Markdown
Member Author

dmcgowan commented Sep 12, 2019

@AkihiroSuda can you help me dig up some images using this compression? You proposed having it adding to the image-spec, but looks like it was rushed through without even showing examples of where it is being used.

@codecov-io
Copy link
Copy Markdown

Codecov Report

Merging #3649 into master will increase coverage by <.01%.
The diff coverage is 47.05%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #3649      +/-   ##
==========================================
+ Coverage   42.39%   42.39%   +<.01%     
==========================================
  Files         127      127              
  Lines       14048    14065      +17     
==========================================
+ Hits         5955     5963       +8     
- Misses       7193     7200       +7     
- Partials      900      902       +2
Flag Coverage Δ
#linux 45.91% <50%> (ø) ⬆️
#windows 37.28% <47.05%> (+0.01%) ⬆️
Impacted Files Coverage Δ
archive/compression/compression.go 57.41% <47.05%> (-1.28%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9741f03...741a99e. Read the comment docs.

@codecov-io
Copy link
Copy Markdown

codecov-io commented Sep 13, 2019

Codecov Report

Merging #3649 into master will not change coverage.
The diff coverage is 50%.

Impacted file tree graph

@@          Coverage Diff           @@
##           master   #3649   +/-   ##
======================================
  Coverage      42%     42%           
======================================
  Files         129     129           
  Lines       14275   14275           
======================================
  Hits         5996    5996           
  Misses       7379    7379           
  Partials      900     900
Flag Coverage Δ
#linux 45.51% <50%> (ø) ⬆️
#windows 36.91% <50%> (ø) ⬆️
Impacted Files Coverage Δ
remotes/handlers.go 10.88% <50%> (ø) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d4802a6...6f31417. Read the comment docs.

@AkihiroSuda
Copy link
Copy Markdown
Member

Support for zstd layers are implemented by Red Hat containers (containers/image#639 containers/storage#363) but I don't think there are publicly available images with zstd layers.

Their work seems mostly motivated for CRFS integration containers/fuse-overlayfs#79

@AkihiroSuda
Copy link
Copy Markdown
Member

I agree the merge was too much rushed, and the media type string may change before the next formal release of OCI Image Spec opencontainers/image-spec#791

@dmcgowan dmcgowan force-pushed the generic-layer-support branch from 741a99e to 2aa8f4a Compare September 13, 2019 18:28
@theopenlab-ci
Copy link
Copy Markdown

theopenlab-ci bot commented Sep 13, 2019

Build succeeded.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wouldn't it be better to return compression.XYZ directly from DiffCompression to avoid extra switch? compression package expects compression.Compression type anyway.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we were to return that, it would be better to move it out of the images package to limit its dependency graph. Right now they are intended to be different though as there may be more compression types from the media types than supported in the compression package. Likewise handling the "unknown" logic is kind of weird and would prefer to handle it outside the compression package. What might make more sense though is to have a package just for handling the media types.

@dmcgowan
Copy link
Copy Markdown
Member Author

I agree the merge was too much rushed, and the media type string may change before the next formal release of OCI Image Spec

zstd is pretty safe, this is unlikely to change considering the name is already widely used outside of OCI and there is not much reason for OCI maintainers to wait for IANA or push for something different. It is the +enc that I am more worried about the motivation for the generic aspect of this PR.

If you think I should split these up into separate PRs, I can do that. I think the generic part is easier to accept. Even the test repo that I saw used by RH in their PR comments didn't seem to still be using zstd, so verifying that is kind of tricky.

@AkihiroSuda
Copy link
Copy Markdown
Member

f you think I should split these up into separate PRs, I can do that. I think the generic part is easier to accept.

Yes, it seems better for 1.3, and I think we can revisit support for zstd in 1.4.

If somebody wants to decompress zstd layers with 1.3, they should be able to use stream processors?
I'm also wondering stream processor might be better for performance, but didn't do benchmark

@dmcgowan dmcgowan force-pushed the generic-layer-support branch from 2aa8f4a to 30354f5 Compare September 17, 2019 18:21
@theopenlab-ci
Copy link
Copy Markdown

theopenlab-ci bot commented Sep 17, 2019

Build succeeded.

@dmcgowan
Copy link
Copy Markdown
Member Author

Removed zstd from this PR. I will add it in a separate PR.

I think we can revisit support for zstd in 1.4.

I like getting support for stuff like this in as early as possible to reduce the support matrix in the future. Moving it out of this release is fine though, I just want to make sure it at least works through the stream processors.

@dmcgowan dmcgowan changed the title Generic layer support with zstd support Generic layer support Sep 17, 2019
@dmcgowan dmcgowan force-pushed the generic-layer-support branch from 30354f5 to c1b12ae Compare September 17, 2019 20:20
@theopenlab-ci
Copy link
Copy Markdown

theopenlab-ci bot commented Sep 17, 2019

Build succeeded.

@crosbymichael
Copy link
Copy Markdown
Member

LGTM

@fuweid
Copy link
Copy Markdown
Member

fuweid commented Sep 19, 2019

need to rebase.

Avoid directly handling media types with "+" attributes,
instead handling the base and passing through the full
media type to the appropriate stream processor or decompression.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
@dmcgowan dmcgowan force-pushed the generic-layer-support branch from c1b12ae to 6f31417 Compare September 19, 2019 23:14
@theopenlab-ci
Copy link
Copy Markdown

theopenlab-ci bot commented Sep 19, 2019

Build succeeded.

@dmcgowan dmcgowan deleted the generic-layer-support branch March 23, 2022 22:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants