Zstd(:chunked) work tracking checklist #205

New issue

Copy link

New issue

Copy link

Open

Open

Zstd(:chunked) work tracking checklist#205

Copy link

Labels

imageRelated to "image" packageRelated to "image" package

Description

mtrmac

opened on Nov 13, 2023

Issue body actions

• Zstd in general:
  • Switch to Zstd default, earlier than a Zstd:chunked default? Testing in DO NOT MERGE: Run tests with Zstd compression default podman#21571 ; Adjust some tests to work with Zstd in containers.conf podman#21793
  • Not a bug: pull of gzip + push of zstd changes image ID due to triggering a conversion to OCI: [rawhide+podman-next] podman push/pull registry roundtrip changes the image podman#20611
  • Pushes with Zstd compression prefer using schema1 manifests: Conversion to schema1 does not fail with Zstd layers, making it uncertain we correctly convert to OCI image#2181 , Refuse compression to zstd when using schema1 image#2196
  • Pushes with Zstd compression to schema-agnostic transports don’t trigger a conversion: Copies with Zstd compression to schema-agnostic transports don’t trigger schema conversion image#2195 , Trigger a conversion to OCI when compressing to Zstd image#2204 Zstd:chunked:
  • Blob reuse is not restricted by possible manifest schema options: Blob reuse decisions do not take into account manifest support image#2205 , Make blob reuse choices manifest-format-sensitive, and allow conversions when writing to format-agnostic transports image#2213
  • Uncompressed pushes of originally-compressed layers don’t correctly update MIME types (always a bug, but now breaks pushes of originally-Zstd to docker-archive: destinations): Copies of originally-compressed images from c/storage to uncompressed destinations don’t trigger MIME type updates image#2182
• Zstd:chunked:
  • copy.Options.EnsureCompressionVariantsExist would never stop adding zstd:chunked variants: copy.Options.EnsureCompressionVariantsExist doesn’t detect existing variants with zstd:chunked #201
  • See if the BlobCache needs updating for Zstd / Zstd:chunked: https://issues.redhat.com/browse/RUN-1124
  • The chunked manifest size has a hard-coded limit, and pulling large layers outright fails: pulling image with zstd:chunked compression is returning manifest too big error podman#24885
  • Exercise Podman tests with Zstd:chunked default: DO NOT MERGE: Run tests with Zstd:chunked compression default podman#21903
  • “view ambiguity”: https://issues.redhat.com/browse/RHEL-66492 , Compute an uncompressed digest for chunked layers storage#2155 , Expect UncompressedDigest to be set for partial pulls, enforce DiffID match image#2613 , Zstd:chunked podman-side tests podman#25007
  • Pulls with missing parent directories fail: https://issues.redhat.com/browse/RUN-2364: Fixed in chunked: handle creating root directory storage#2194
  • Reusing already-present data can be optimized: Extend PutLayer to optimize reusing data from existing layers storage#1830, Update a comment after a c/storage update image#2299, Propagate CompressedDigest/CompressedSize when reusing data from another layer image#2583
  • Podman’s zstd:chunked-specific tests are not representative: Fix apparent typos in zstd:chunked tests podman#24686
  • Pulls on VFS fail outright: podman pull zstd:chunked-image, with vfs: not supported podman#24308 , Fall back from partial pull when on VFS storage#2140
  • expectedLayerDiffIDFlag seems to use mismatching types: expectedLayerDiffIDFlag expectation mismatch image#2602 , expectedLayerDiffIDFlag expectation mismatch image#2602
  • Document zstd:chunked and encryption interaction: Document that zstd:chunked doesn’t make sense with encryption common#2117, Document that zstd:chunked is downgraded to zstd when encrypting common#2176, Document that zstd:chunked is downgraded to zstd when encrypting skopeo#2427, Document that zstd:chunked is downgraded to zstd when encrypting buildah#5759, Document that zstd:chunked is downgraded to zstd when encrypting podman#24113
  • Propose iterateTarSplit to upstream tar-split: Ensure chunked TOC and tar-split metadata are consistent storage#2035 (comment) . Filed Add tar/asm.IterateHeaders vbatts/tar-split#71 , Use tar-split/tar/asm.IterateHeaders now that it has been accepted storage#2116 .
  • blobPipelineDetectCompressionStep detects zstd:chunked as zstd, causing unnecessary recompression: around copy: do not fail if digest mismatches image#1980 (comment) , Record zstd:chunked format, and annotations, in BlobInfoCache image#2487 .
  • Pushes reusing chunked layers don’t create the required annotations, making the layers impossible to pull chunked: [WIP] compression: add specific prefix for zstd:chunked image#2183 (review) / WIP HACK: Do not reuse zstd:chunked blobs image#2185 , Record zstd:chunked format, and annotations, in BlobInfoCache image#2487
  • zstd:chunked and layer encryption don’t make sense together: zstd:chunked and layer encryption don’t make sense together image#2485
  • Blocker? TOC data and tar-split may be ambiguous: zstd:chunked metadata ambiguity storage#2014 , Ensure chunked TOC and tar-split metadata are consistent storage#2035
  • c/storage AdditionalLayerStore seems to be used for TOC-identified layers, incorrectly: Support additional layer store (patch for containers/storage) storage#795 (comment) , [Additional Layer Store] Use TOCDigest as ID of each layer (patch for c/storage) storage#1924
  • TarSplitChecksumKey not used in a layer ID: zstd:chunked blocker: TarSplitChecksumKey not used in a layer ID storage#1888 , Move the tar-split digest into the TOC storage#1902
  • blobPipelineCompressionStep would trigger a recompression of zstd:chunked if the user asks for zstd (except that we don’t currently detect zstd:chunked): around copy: do not fail if digest mismatches image#1980 (comment) , Improve handling of zstd vs. zstd:chunked matching image#2317
  • The c/storage “binary footer” code path does not work: zstd:chunked binary footer format is broken storage#1886
  • Outstanding items from chunked: generate tar-split as part of zstd:chunked storage#1627 (review) : Chunked cleanups storage#1844
  • Copies of chunked layers don’t mark the image as requiring Zstd support: Copies don’t set OCI1InstanceAnnotationCompressionZSTD on Zstd:chunked image#2077 , fix minor review comment about the driver mutex and a minor lint issue storage#2302
  • Chunked layers are visible in c/storage, and usable by other processes, while they are still being populated: Creation of Zstd:chunked layers seems racy image#1979, store: new API ApplyStagedLayer  storage#1826 + idmap: fix first argument to open_tree storage#2301 .
  • PutBlobPartial with a non-chunked input and conversion in c/storage enabled doesn’t work: Fix c/storage destination with partial pulls image#2288 (comment) , fixed there
  • Pushes of chunked layers fail because the UncompressedDigest fields stores a TOC digest, and layer digest validation fails: Fixed in copy: do not fail if digest mismatches image#1980
  • Pulls of chunked layers do not reuse locally-existing layers: Part of copy: do not fail if digest mismatches image#1980
  • In c/storage, we don’t sufficiently differentiate layers pulled by blob digest vs. TOC digest, and reuse is unclear: Part of copy: do not fail if digest mismatches image#1980 , and chunked: generate tar-split as part of zstd:chunked storage#1627 (comment) . Fixed in RFC: Use Go 1.23 iterators for locking+traversing stores. storage#2288 .
  • Pulls may reuse layers pulled by blob digests vs. TOC digests: Part of copy: do not fail if digest mismatches image#1980 . Fixed in RFC: Use Go 1.23 iterators for locking+traversing stores. storage#2288 .
  • Image deduplication contains a (required!) sanity check that TopLayer matches, but that might not be true if layers pulled by blob vs. TOC digests have different IDs: Part of copy: do not fail if digest mismatches image#1980 . Fixed in RFC: Use Go 1.23 iterators for locking+traversing stores. storage#2288 .
  • Reuse of local layers by TOC may trigger inconsistent metadata updates: [crio-1.31] chore(deps): update dependency golangci/golangci-lint to v1.64.8 storage#2294
  • A pull by TOC + conversion to non-OCI might parse data incorrectly: chore(deps): update dependency golangci/golangci-lint to v2 storage#2295

Reactions are currently unavailable

Metadata

Metadata

Assignees

No one assigned

Labels

imageRelated to "image" packageRelated to "image" package

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions