conformance: initial commit for references test by rchincha · Pull Request #375 · opencontainers/distribution-spec

rchincha · 2023-01-19T18:10:46Z

Based off of 01_pull_test.go

Fixes #370

Signed-off-by: Ramkumar Chinchani rchincha@cisco.com

rchincha · 2023-01-24T01:25:08Z

Testing against zot ...

      you have skipped this test; if this is an error, check your environment variable settings:
        OCI_TEST_CONTENT_DISCOVERY=0
        OCI_TEST_CONTENT_MANAGEMENT=1
        OCI_REFERRERS=1
        OCI_TEST_PULL=0
        OCI_TEST_PUSH=0


      /local/rchincha/go/src/github.com/opencontainers/distribution-spec/conformance/setup.go:408
------------------------------
•••••••••••••••••••••••HTML report was created: /local/rchincha/go/src/github.com/opencontainers/distribution-spec/conformance/report.html
JUnit report was created: /local/rchincha/go/src/github.com/opencontainers/distribution-spec/conformance/junit.xml

Ran 23 of 78 Specs in 0.085 seconds
SUCCESS! -- 23 Passed | 0 Failed | 0 Pending | 55 Skipped

jdolitsky · 2023-04-19T12:58:00Z

can you rebase?

rchincha · 2023-05-26T20:21:29Z

@jdolitsky @sudo-bmitch @sajayantony
Some eyes on this pls.

FYI, top of zot/main passes.

sudo-bmitch

@rchincha thanks for getting this started! I've got a bunch of feedback, but we're on the right path with this.

conformance/05_referrers.go

sudo-bmitch · 2023-05-28T13:26:18Z

conformance/05_referrers.go

+				// also check resp header "OCI-Filters-Applied: artifactType" denoting that an artifactType filter was applied
+				Expect(resp.Header().Get("OCI-Filters-Applied")).ToNot(BeEmpty())
+				Expect(resp.Header().Get("OCI-Filters-Applied")).To(Equal("application/vnd.oci.conformance"))


Filter support is a SHOULD, so at most we can warn. I'd prefer we push a third referrer with a different type, and then conditionally handle the response based on if the registry returns the OCI-Filters-Applied header. If it's missing, all 3 referrers should be returned. Otherwise if the header is defined with the requested value, we should verify the two requested referrers were returned and not the 3rd.

WARN: you have failed this test, but ignoring Expected <string>: application/vnd.nhl.peanut.butter.bagel to equal <string>:

^ warn support added

Modified implementation.

conformance/05_referrers.go

sudo-bmitch · 2023-05-28T13:34:49Z

conformance/setup.go

+	scratchDescriptor                 = Descriptor{
+		MediaType: "application/vnd.oci.scratch.v1+json",
+		Size:      2,
+		Digest:    godigest.FromString("sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a"),
+	}


Pending the result of opencontainers/image-spec#1068

Used same empty JSON descriptor

conformance/05_referrers.go

sudo-bmitch · 2023-06-02T15:30:23Z

conformance/05_referrers.go

+				Expect(len(index.Manifests)).To(Equal(2))
+
+				// also check resp header "OCI-Filters-Applied: artifactType" denoting that an artifactType filter was applied
+				Expect(resp.Header().Get("OCI-Filters-Applied")).To(EqualOrWarn(testRefArtifactTypeA))


Since filter support is a "SHOULD", the length and header check here needs to be conditional. If the header is set, then we expect 2 entries and the header must match the expected value. Else, we'd expect 4 manifests.

Maybe we don't need the "warn" after all.

sudo-bmitch

Couple minor updates. Does Zot still pass after the latest changes?

sudo-bmitch · 2023-06-04T14:22:19Z

conformance/image.go

+// Artifact describes an artifact manifest.
+// This structure provides `application/vnd.oci.artifact.manifest.v1+json` mediatype when marshalled to JSON.
+type Artifact struct {
+	// MediaType is the media type of the object this schema refers to.
+	MediaType string `json:"mediaType"`
+
+	// ArtifactType is the IANA media type of the artifact this schema refers to.
+	ArtifactType string `json:"artifactType,omitempty"`
+
+	// Blobs is a collection of blobs referenced by this manifest.
+	Blobs []Descriptor `json:"blobs,omitempty"`
+
+	// Subject (reference) is an optional link from the artifact to another manifest forming an association between the artifact and the other manifest.
+	Subject *Descriptor `json:"subject,omitempty"`
+
+	// Annotations contains arbitrary metadata for the artifact manifest.
+	Annotations map[string]string `json:"annotations,omitempty"`
+}


conformance/05_referrers.go

rchincha · 2023-06-05T02:46:21Z

Couple minor updates. Does Zot still pass after the latest changes?

Yes, top of zot/main does.

      /local/rchincha/go/src/github.com/opencontainers/distribution-spec/conformance/setup.go:491
------------------------------
••••••••••••••••••••••••••••••••••HTML report was created: /local/rchincha/go/src/github.com/opencontainers/distribution-spec/conformance/report.html
JUnit report was created: /local/rchincha/go/src/github.com/opencontainers/distribution-spec/conformance/junit.xml

Ran 81 of 86 Specs in 0.152 seconds
SUCCESS! -- 81 Passed | 0 Failed | 0 Pending | 5 Skipped

rchincha · 2023-06-09T19:36:17Z

bin/zot-linux-amd64 serve examples/config-conformance.json
^ to test

Made the warning code/output a little "nicer"

sudo-bmitch · 2023-06-15T17:15:33Z

conformance/setup.go

+func Warn(message string) {
+	fmt.Fprintf(colorable.NewColorableStderr(), "\n")
+	// print file:line
+	_, file, line, _ := runtime.Caller(1)
+	warncolor := color.New(color.FgMagenta).FprintfFunc()
+	warncolor(colorable.NewColorableStderr(), formatter.Fi(2, "\nWARNING: "+message))
+	fmt.Fprintf(colorable.NewColorableStderr(), "\n")
+	// print message
+	msgcolor := color.New(color.FgWhite).FprintfFunc()
+	msgcolor(colorable.NewColorableStderr(), formatter.Fi(2, "\n"))
+	msgcolor(colorable.NewColorableStderr(), formatter.Fi(2, "%s:%d\n", file, line))
+}


@jdolitsky is there any issue generating warnings in the conformance test this way?

Rebased and updated after recent merges.

https://github.com/opencontainers/distribution-spec/blob/main/spec.md#enabling-the-referrers-api Based off of 01_pull_test.go Fixes opencontainers#370 https://github.com/opencontainers/image-spec/blob/main/manifest.md#guidelines-for-artifact-usage Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>

rchincha · 2023-06-21T23:24:28Z

As a data point, top of zot/main passes

••••••••••••••••••••••••••••••••••HTML report was created: /local/rchincha/go/src/github.com/opencontainers/distribution-spec/conformance/report.html
JUnit report was created: /local/rchincha/go/src/github.com/opencontainers/distribution-spec/conformance/junit.xml

Ran 81 of 86 Specs in 0.255 seconds
SUCCESS! -- 81 Passed | 0 Failed | 0 Pending | 5 Skipped

rchincha · 2023-06-21T23:25:54Z

Instrumented zot warns like so,


      /local/rchincha/go/src/github.com/opencontainers/distribution-spec/conformance/setup.go:490
------------------------------
••••••••••••••••••••••••••••
    WARNING: filtering by artifact-type is not implemented                                                                                                                                                                                                                                                                                              /local/rchincha/go/src/github.com/opencontainers/distribution-spec/conformance/05_referrers.go:240

rchincha · 2023-06-21T23:34:15Z

opencontainers/image-spec#1077
^ tests for this in a separate PR after it lands

outdated

https://github.com/opencontainers/distribution-spec/blob/main/spec.md#enabling-the-referrers-api Based off of 01_pull_test.go Fixes opencontainers#370 https://github.com/opencontainers/image-spec/blob/main/manifest.md#guidelines-for-artifact-usage Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>

rchincha force-pushed the references branch 2 times, most recently from e28419c to eb5ccad Compare January 24, 2023 00:59

jdolitsky previously approved these changes Jan 24, 2023

View reviewed changes

sudo-bmitch added this to the v1.1.0 milestone Mar 14, 2023

rchincha dismissed jdolitsky’s stale review via d526d56 May 26, 2023 19:59

rchincha force-pushed the references branch from eb5ccad to d526d56 Compare May 26, 2023 19:59

rchincha marked this pull request as ready for review May 26, 2023 20:00

rchincha requested review from dmcgowan, jonjohnsonjr, jzelinskie, mikebrow, stevvooe, sudo-bmitch and vbatts as code owners May 26, 2023 20:00

rchincha force-pushed the references branch 2 times, most recently from 180974d to 18ab2cb Compare May 26, 2023 20:17

rchincha requested a review from jdolitsky May 26, 2023 20:20

rchincha force-pushed the references branch from 18ab2cb to 151f848 Compare May 26, 2023 20:24

sudo-bmitch requested changes May 28, 2023

View reviewed changes

sudo-bmitch reviewed May 29, 2023

View reviewed changes

conformance/05_referrers.go Outdated Show resolved Hide resolved

rchincha force-pushed the references branch 3 times, most recently from 4683bf1 to 2e3cfc1 Compare June 2, 2023 05:13

rchincha requested a review from sudo-bmitch June 2, 2023 05:15

sudo-bmitch reviewed Jun 2, 2023

View reviewed changes

rchincha force-pushed the references branch 2 times, most recently from 9769a96 to 627be76 Compare June 2, 2023 23:44

rchincha force-pushed the references branch from 627be76 to 9b233d5 Compare June 2, 2023 23:48

rchincha requested a review from sudo-bmitch June 2, 2023 23:56

rchincha force-pushed the references branch from 9b233d5 to da259b1 Compare June 3, 2023 17:04

sudo-bmitch previously requested changes Jun 4, 2023

View reviewed changes

rchincha force-pushed the references branch from da259b1 to af46363 Compare June 5, 2023 02:45

rchincha requested a review from sudo-bmitch June 5, 2023 06:00

rchincha force-pushed the references branch from af46363 to d705dff Compare June 9, 2023 19:35

sudo-bmitch reviewed Jun 15, 2023

View reviewed changes

rchincha force-pushed the references branch from d705dff to e586e62 Compare June 16, 2023 20:14

rchincha mentioned this pull request Jun 17, 2023

conformance: skip check (by default) for zero layers in image manifest #421

Merged

jdolitsky mentioned this pull request Jun 21, 2023

Reference types conformance tests #370

Closed

jdolitsky added the conformance label Jun 21, 2023

rchincha force-pushed the references branch from e586e62 to 68fb0ce Compare June 21, 2023 23:17

rchincha force-pushed the references branch from 68fb0ce to 3af85ef Compare June 21, 2023 23:21

rchincha requested a review from sudo-bmitch June 21, 2023 23:31

jonjohnsonjr approved these changes Jun 22, 2023

View reviewed changes

jdolitsky approved these changes Jun 22, 2023

View reviewed changes

jdolitsky merged commit 99aba31 into opencontainers:main Jun 22, 2023

jdolitsky mentioned this pull request Jun 27, 2023

tag 1.1.0-rc3 #433

Closed

8 tasks

jdolitsky mentioned this pull request Jul 6, 2023

tag 1.1.0-rc3 (part 2) #440

Merged

8 tasks

andaaron mentioned this pull request Jul 11, 2023

conformance: make references tests part of content discovery workflow #430

Merged

sudo-bmitch mentioned this pull request Feb 1, 2024

Release v1.1.0 #507

Merged

8 tasks

Conversation

rchincha commented Jan 19, 2023

Uh oh!

rchincha commented Jan 24, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jdolitsky commented Apr 19, 2023

Uh oh!

rchincha commented May 26, 2023

Uh oh!

sudo-bmitch left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sudo-bmitch left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

rchincha commented Jun 5, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rchincha commented Jun 9, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rchincha Jun 21, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rchincha commented Jun 21, 2023

Uh oh!

rchincha commented Jun 21, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rchincha commented Jun 21, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

rchincha commented Jan 24, 2023 •

edited

Loading

rchincha commented Jun 5, 2023 •

edited

Loading

rchincha Jun 21, 2023 •

edited

Loading

rchincha commented Jun 21, 2023 •

edited

Loading