Conversation
2a89e0f to
3215330
Compare
|
created a skopeo PR to validate this containers/skopeo#294 |
3215330 to
15621ae
Compare
mtrmac
left a comment
There was a problem hiding this comment.
I very much like moving the conversion to image/; the piece in docker.newImageSource really does not belong there I think, though.
docker/docker_image_src.go
Outdated
| // and OCI doesn't understand it (it cannot convert from v2s1 to OCI) | ||
| for _, mt := range requestedManifestMIMETypes { | ||
| if mt == imgspecv1.MediaTypeImageManifest { | ||
| requestedManifestMIMETypes = append(requestedManifestMIMETypes, manifest.DockerV2Schema2MediaType) |
There was a problem hiding this comment.
This is ugly and suprising for an ImageSource to do (consider a caller who does not expect to use image.Image). Can we instead add the v2s1→OCI conversion? It might well be possible to do by creating a temporary v2s2 memoryImage (using the existing code) and then using the existing v2s2→OCI conversion.
There was a problem hiding this comment.
(We haven’t supported a v2s1→OCI conversion before, so AFAICT this is not an essential part of this PR.)
There was a problem hiding this comment.
makes sense to have v2s1 -> OCI at this point, so yeah, I'll implement this.
There was a problem hiding this comment.
Considering the failing tests, could we merge a version without that conversion now, and deal with the v2s1→OCI conversion as a separate issue or PR?
There was a problem hiding this comment.
wait, tests will fail w/o this mime type check, shall I leave this here?
There was a problem hiding this comment.
What is the failure / code path?
There was a problem hiding this comment.
basic skopeo copy docker://busybox oci:something, we need this to ask for at least v2s2 if we want OCI conversion (since we cannot convert from v2s1 which is the default when the registry doesn't understand the media type, see the comment, the test failure is in skopeo tests)
There was a problem hiding this comment.
What is the failure / code path?
Let me guess: ociDest.SupportedManifestMIMETypes returns OCI only, and so copy.Image asks for unsupported manifest MIME type, and that gets sent to the registry.
Adding the OCI conversion is the long-term fix.
Short-term, can we:
- move the array from
dockerImageDestinationSupportedManifestMIMETypesinto a global variable (shared by src and dest) - in
newImageSource, ignorerequestedManifestMIMETypesif it has no value common with the known supported MIME types (as defined by the variable from above)?
That’s still kind of a hack, but at least it is conceptually clean: dockerImage* knows what it supports/accepts, and does not need to know about OCI.
There was a problem hiding this comment.
(*shrug*, if you want to merge it with the existing hack instead, feel free. Just please make sure to file an issue to have this cleaned up in the future.)
image/docker_schema2.go
Outdated
| // The only difference between OCI and DockerSchema2 is the mediatypes. | ||
| config.MediaType = imgspecv1.MediaTypeImageConfig | ||
|
|
||
| configBytes, err := m.ConfigBlob() |
There was a problem hiding this comment.
Is this really necessary to read the config here? It would perhaps be preferable to preserve the ImageSource and pass a nil configBlob, the way manifestOCI1.convertToManifestSchema2 does it.
| om.MediaType = imgspecv1.MediaTypeImageManifest | ||
| for i, l := range om.Layers { | ||
| if l.MediaType == manifest.DockerV2Schema2ForeignLayerMediaType { | ||
| om.Layers[i].MediaType = imgspecv1.MediaTypeImageLayerNonDistributable |
There was a problem hiding this comment.
This …NonDistributable conversion is not present in the new conversion code, is that intentional?
oci/layout/oci_dest.go
Outdated
| } | ||
| // TODO(runcom): not fully sure how to prevent a docker v2s2 from slipping in | ||
| // any idea?!?! | ||
| if ociMan.SchemaVersion != 2 { |
There was a problem hiding this comment.
I am tempted no to try at all… But if we should, I think it would be much cleaner to have the caller supply a mimeType as a parameter than guessing by parsing contents in an unknown format in here.
There was a problem hiding this comment.
let's skip this altogether for now and re-iterate on adding an optional mime type, but then, ppl might cheat on mime type as well no?
There was a problem hiding this comment.
A lying caller will get what is coming to it :)
Right now we do do this wrong, because copy.determineManifestConversion can decide to keep an unsupported type anyway (it is treating SupportedManifestMIMETypes as a best-effort request), and PutManifest isn’t told about that. But we would be sending the correct MIME type to PutManifest if there was a way to do that.
image/docker_schema2.go
Outdated
| return memoryImageFromManifest(©), nil | ||
| } | ||
|
|
||
| func (m *manifestSchema2) convertToOCIManifest() (types.Image, error) { |
There was a problem hiding this comment.
Also, docker_schema2.go has (unlike …schema1.go) a reasonable test coverage, and it would be nice to preserve that (the conversions happen frequently enough for users to care but perhaps not frequently enough to prevent us from breaking them without noticing). This should be just a pretty trivial variant of TestConvertToManifestSchema2.
There was a problem hiding this comment.
yup, have it on my todo, just wanted to bring this PR up for general consensus on the way I'm fixing this.
24efd99 to
a5c3380
Compare
|
|
|
No, wait, we need the conversion, otherwise, the old |
| manifest.DockerV2Schema1MediaType, | ||
| } | ||
|
|
||
| func supportedManifestMIMETypesMap() map[string]bool { |
There was a problem hiding this comment.
I think a map[string]struct{} is more idiomatic, but I really don’t care enough.
There was a problem hiding this comment.
arg, that's how it was, then changed my mind, I'll put it back with struct{}
There was a problem hiding this comment.
on a second though, I like comparing with just ! - we change this any time though...
docker/docker_image_src.go
Outdated
| requestedManifestMIMETypes = manifest.DefaultRequestedManifestMIMETypes | ||
| } | ||
| for _, mtrequested := range requestedManifestMIMETypes { | ||
| if !supportedManifestMIMETypesMap()[mtrequested] { |
There was a problem hiding this comment.
This is still O(N^2): move the call supportedManifestMIMETypesMap outside the loop, save it in a variable. Then it is O(N).
(Or don’t bother and just write two nested loops. The arrays are small enough.)
There was a problem hiding this comment.
This is still O(N^2): move the call supportedManifestMIMETypesMap outside the loop, save it in a variable. Then it is O(N).
silly me, ETOOLATE
| for _, mtrequested := range requestedManifestMIMETypes { | ||
| if !supportedManifestMIMETypesMap()[mtrequested] { | ||
| requestedManifestMIMETypes = manifest.DefaultRequestedManifestMIMETypes | ||
| break |
There was a problem hiding this comment.
This would ignore the input value if any value is unrecognized; I think it should ignore input only if all values are unrecognized (so that a caller can say “i prefer v2s1, then OCI, then v2s2 as a last resort”.)
e19c1ba to
8246d68
Compare
|
@mtrmac PTAL |
8246d68 to
ab0bcd5
Compare
docker/docker_image_src.go
Outdated
| if !mts[mtrequested] { | ||
| continue | ||
| } | ||
| reset = false |
There was a problem hiding this comment.
break here, and then it’s more natural to reword as if mts[mtrequested] { EDIT reset = false;break }.
docker/docker_image_src.go
Outdated
| } | ||
| var ( | ||
| mts = supportedManifestMIMETypesMap() | ||
| reset = true |
There was a problem hiding this comment.
(and as long you are changing this) this would benefit from either clearer variable names, or a comment explicitly describing what this loop intends to do. Preferably the former, perhaps naming them supportedMIMEs and acceptableRequestedMIMEs (with the opposite semantics to reset, starting as false).
There was a problem hiding this comment.
…also, no need for the var() here AFAICS, just use :=. Or does that have an idiomatic meaning?
Instead, copy the error.NewAggregate implementation inside. This allows use of this package in OpenShift, which uses an older version of k8s.io packages. Signed-off-by: Miloslav Trmač <mitr@redhat.com>
Instead, inline net.SetTransportDefaults and copy net.NewProxierWithNoProxyCIDR. This allows use of this package in OpenShift, which uses an older version of k8s.io packages. Signed-off-by: Miloslav Trmač <mitr@redhat.com>
…o/client-go Because this is an entirely new package, depending on any version of it should hopefully not be problematic for downstreams. Signed-off-by: Miloslav Trmač <mitr@redhat.com>
ab0bcd5 to
df1e240
Compare
docker/docker_image_src.go
Outdated
| continue | ||
| } | ||
| acceptableRequestedMIMEs = false | ||
| break |
There was a problem hiding this comment.
This rejects requestedManifestMIMETypes if any content is unrecognized; shouldn’t it reject them only if none are recognized?
There was a problem hiding this comment.
(i.e. when reaming reset to acceptableRequestedMIMEs, only the last if was flipped AFAICS.)
There was a problem hiding this comment.
I believe I've fixed it now - sorry I'm in a meeting but want to get this in asap to unblock other PRs
There was a problem hiding this comment.
I was thinking
supportedMIMEs := supportedManifestMIMETypesMap()
acceptableRequestedMIMEs := false
for _, mtrequested := range requestedManifestMIMETypes {
if supportedMIMEs[mtrequested] {
acceptableRequestedMIMEs = true
break
}
}
if !acceptableRequestedMIMEs {
requestedManifestMIMETypes = manifest.DefaultRequestedManifestMIMETypes
}Perhaps I am being an idiot?
df1e240 to
d982fea
Compare
d982fea to
6f10a87
Compare
|
another issue just popped up https://travis-ci.org/containers/image/builds/195004166#L346 - perhaps image-spec shouldn't really have a vendor directory at all. Commented here #223 |
Signed-off-by: Antonio Murdaca <runcom@redhat.com>
( opencontainers/image-spec#527 , for the record.) |
|
@mtrmac will we wait till the image-spec-vendor-dir is gone before merging this? |
I dunno, what other options do we have? Just keep our old checkouts on our development machines, and ignore Travis and keep merging other PRs? Ewww. Or, give up and set up Is there a chance that the vendoring will be fixed quickly? |
not sure, I hope it'll be really soon |
Due to the new vendoring logic in the golang compiler it can cause issues for projects that import a package that has vendored a package that is used locally. See containers/image#223 This change moves the vendored sources to the package that uses them, rather than for the whole project. Also is explictly is not vendoring code repos from "github.com/opencontainers/". For now we'll consider these non-remote. Though versioning may likely be future concern. Fixes opencontainers#527 Obsoletes opencontainers#528 Signed-off-by: Vincent Batts <vbatts@hashbangbash.com>
6f10a87 to
6a51d25
Compare
Signed-off-by: Antonio Murdaca <runcom@redhat.com>
|
seems like something else in OCI image-spec broke this.. |
Signed-off-by: Antonio Murdaca <runcom@redhat.com>
Signed-off-by: Antonio Murdaca <runcom@redhat.com>
|
alright, pretty sure I've this sorted out now, @mtrmac PTAL |
mtrmac
left a comment
There was a problem hiding this comment.
Yay!!
This looks really good, it seems a skopeo revendor will fix everything.
| if m.LayersDescriptors[idx].MediaType == manifest.DockerV2Schema2ForeignLayerMediaType { | ||
| layers[idx].MediaType = imgspecv1.MediaTypeImageLayerNonDistributable | ||
| } else { | ||
| layers[idx].MediaType = imgspecv1.MediaTypeImageLayerGzip |
There was a problem hiding this comment.
This works for now (i.e. at it was perfectly valid a week ago), longer-term we should actually know whether the input is gzipped. Filed #229.
|
alright let's merge this (and take care of this comment #229 (comment) later) |
|
👍 |
opencontainers/image-spec#411 removed the
MediaTypefield inManifest[List]causing builds error here. Fix that by teaching docker v2s2 to convert to OCI and remove that ugly manifest conversion in OCI image destination.@mtrmac @cyphar PTAL