Skip to content

Digest of blobs changes after transfer #797

@frewilhelm

Description

@frewilhelm

Context
The replication controller unit test showed that the hardcoded localImagePath changed after upgrading ocm.software/ocm from v0.32.0 to v0.34.1:

-	localImagePath = "blobs/sha256.4b93359cc643b5d8575d4f96c2d107b4512675dcfee1fa035d0c44a00b9c027c"
+	localImagePath = "blobs/sha256.1bcf96a2451e3d44117d1b31e5bfa15dd326e1393d3434be3e74863ea4f380fc"

The content of the blob has no noteworthy changes, which is why we suspected a change in the compression. We were able to pin the digest change down to the version upgrade of github.com/klauspost/compress to v1.18.1 in the ocm.software/ocm release v0.33.0.

Version
> v0.33.0

To Reproduce

# Download ocm release v0.32 (adjust OS and ARCH)
mkdir ocm32
cd ocm32
wget https://github.com/open-component-model/ocm/releases/download/v0.32.0/ocm-0.32.0-darwin-arm64.tar.gz
tar xzf ocm-0.32.0-darwin-arm64.tar.gz

# Transfer component version to 
./ocm transfer cv --copy-resources ghcr.io/open-component-model/ocm//ocm.software/toi/demo/helmdemo:0.12.0 ./ctf

# Display blobs
ls -la ctf/blobs
total 45136
drwx------ 10 user group      320 Dec 16 11:19 .
drwx------  4 user group      128 Dec 16 11:19 ..
-rw-------  1 user group      201 Dec 16 11:19 sha256.316e1bdff7b7e3e041f67118fff250dea204525cf27c0f3c3bea62b1a0209594
-rw-------  1 user group 46181313 Dec 16 11:19 sha256.4b93359cc643b5d8575d4f96c2d107b4512675dcfee1fa035d0c44a00b9c027c
-rw-------  1 user group     1905 Dec 16 11:19 sha256.5f804d7f6e8e7e8f599fc6ce413af0f5de9cc354af13ae335eb9a7470d7c77ff
-rw-------  1 user group       23 Dec 16 11:19 sha256.87cef1e2233bf5591030ac854e2556fbe6a00a28bb5640e25a9cb69ece519c5a
-rw-------  1 user group      747 Dec 16 11:18 sha256.8a2fe6af4ce56249094622c9d618e24b4cfb461a7dfa6a42cce31749189bc499
-rw-------  1 user group       91 Dec 16 11:19 sha256.e790920a11de2016de64225280efcf062e14b767955f7508de64fd5192e3fb3a
-rw-------  1 user group     5120 Dec 16 11:19 sha256.fa166a15156ac88c4f9235c94318822129ae461a36d9bd34ea77edf83ce99757
-rw-------  1 user group     4707 Dec 16 11:18 sha256.fbbc1a667c469da8e93886653d98625cb715cd009cbc9b172f33b17e81b97515

Then, do the same thing with ocm release v0.33.0:

# Download ocm release v0.33 (adjust OS and ARCH)
mkdir ocm33
cd ocm33
wget https://github.com/open-component-model/ocm/releases/download/v0.33.0/ocm-0.33.0-darwin-arm64.tar.gz
tar xzf ocm-0.33.0-darwin-arm64.tar.gz

# Transfer component version to 
./ocm transfer cv --copy-resources ghcr.io/open-component-model/ocm//ocm.software/toi/demo/helmdemo:0.12.0 ./ctf

# Display blobs
ls -la ctf/blobs
total 45136
drwx------ 10 user group      320 Dec 16 11:20 .
drwx------  4 user group      128 Dec 16 11:20 ..
-rw-------  1 user group 46181312 Dec 16 11:20 sha256.1bcf96a2451e3d44117d1b31e5bfa15dd326e1393d3434be3e74863ea4f380fc
-rw-------  1 user group     5120 Dec 16 11:20 sha256.27924f431a8daf7afb7b3c9ffd79b5aec3c8cd48ab08b2b971c365cb2a71ad40
-rw-------  1 user group       23 Dec 16 11:20 sha256.87cef1e2233bf5591030ac854e2556fbe6a00a28bb5640e25a9cb69ece519c5a
-rw-------  1 user group      747 Dec 16 11:20 sha256.8a2fe6af4ce56249094622c9d618e24b4cfb461a7dfa6a42cce31749189bc499
-rw-------  1 user group     1905 Dec 16 11:20 sha256.b34d3087c6dc42a72fcbc2f47aa3b40ed63ad5b3f63dc421219836fb6921a790
-rw-------  1 user group      201 Dec 16 11:20 sha256.b49d4f04a480039fa7b48bbb60b5af39ae9cf0e0eebd313eb8d8187d682feefb
-rw-------  1 user group       91 Dec 16 11:20 sha256.e790920a11de2016de64225280efcf062e14b767955f7508de64fd5192e3fb3a
-rw-------  1 user group     4707 Dec 16 11:20 sha256.fbbc1a667c469da8e93886653d98625cb715cd009cbc9b172f33b17e81b97515

Some of the downloaded blobs have the same SHA and size, others differ, for example:

# ocm 0.32.0
-rw-------  1 user group 46181313 Dec 16 11:19 sha256.4b93359cc643b5d8575d4f96c2d107b4512675dcfee1fa035d0c44a00b9c027c

# ocm 0.33.0
-rw-------  1 user group 46181312 Dec 16 11:20 sha256.1bcf96a2451e3d44117d1b31e5bfa15dd326e1393d3434be3e74863ea4f380fc

Actual behavior
The SHA and size of transferred blobs do differ.

Expected behavior
The SHA and size of transferred blobs should not differ.

Environment

  • windows
  • linux
  • mac

Additional Comments
Currently, this transfer behaviour was only reproduced when transferring a component version from an OCI registry to a local CTF.

TODOs:

  • Pin the version of github.com/klauspost/compress to v1.18.0 until we found out what causes the change
  • Backport the change to the release branches v0.33, v0.34, and v0.35
  • Create patch-releases for the release branches
  • Retract any release that contains the bug (v0.33.0, v0.34.1)
  • Create tests that test such scenarios
  • Find the root cause of the problem or create an issue to address it

Metadata

Metadata

Assignees

Labels

area/ipceiImportant Project of Common European Interestkind/tasksmall task, normally part of feature or epic

Type

No fields configured for Bug.

Projects

Status
🔍 Review

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions