-
Notifications
You must be signed in to change notification settings - Fork 240
Option to set custom layer size via environment variables #374
Description
A lot of the storage providers put restrictions on the minimum size of a single chunk (or part) that can be uploaded via something like a multipart upload. For example, AWS S3 says for a multipart upload, each part must be at least 5MB except for the last part, which can be any size. This scenario is true for most of the storage providers (for a good reason).
Inside the conformance testing suite, currently, we're using fixed length strings as chunks for the blobs. I was wondering if something like setting the size of the blobs would be possible via environment variables? Here's why:
Now, in our (as in opencontainers/distribution-spec) current implementation of conformance suite, tests usually have blobs (which really are just layers) as small as 23Bytes, and it is then chunked to even smaller sizes to be used in chunked layer uploading. This lead to a certain problem for us since one of our storage providers didn't support a chunk/part of size smaller than 5MB and that caused our Push Conformance Tests to fail, but when I tried uploading the hello-world and busybox images (two of the smallest container images I know of), they both worked just fine and container image push was successful.
So finally I had to go on and take a look at the conformance setup.go file and tweak a bit to set layer/blob size larger than 5MB and distribute them in two chunks:
- Use
math/randto fill the layer/blob with random data (I hope that's okay) - Part/chunk A, at least 5MB
- Part/chunk B, rest of the data
After this change, the conformance tests started passing. So this lead to me thinking about two scenarios:
- One, Would it be a wise idea to have the option to configure the size of the layer being used for tests set via some environment variable? If this is the case, I'd love to submit a PR for this and I'll make sure the existing behaviour doesn't break.
- Two, It breaks the rules for conformance and it's upto the specification implementors to take care nuances like this?
Both of these situations seem right to me. But of course as a registry developer and maintainer, it's easier for me to just set an environment variable and get away with it (I also feel like it should be okay to set this size via environment variable?)
Please let me know what you guys think about this :)