Modernize Crossplane conformance suite by jbw976 · Pull Request #31 · crossplane/conformance

jbw976 · 2024-10-09T16:52:10Z

Description of your changes

This PR updates the Crossplane conformance test suite and build logic to be aligned and compliant with Crossplane v2.0 The tests are all now compliant with a Crossplane v2.0 control plane. The current builds in this repo were last verified to work with Crossplane v1.2, so there were a number of updates to incorporate from those ~20 intermediary minor releases and 1 major release 😇

Major changes to call out:

Conformance test cases target API behavior from Crossplane v2.0, the most recently released upstream Crossplane version
Provider conformance tests have been removed, the main focus of this suite will now be core Crossplane itself
go module dependencies updates to modern versions, e.g. crossplane-runtime:v2.0.0
Build submodule switched from https://github.com/upbound/build to https://github.com/crossplane/build
Build and CI versions updated to similar versions used in more recent Crossplane ecosystem projects, e.g. golang 1.23.2, ubuntu-24.04 runners, etc.
Additional CI checks have been added such as codeql and trivy-scan-fs
Added many common test helpers like CreateNamespace and TestResourceRefs

Test case high level areas of coverage:

API Extensions - Modern (v2)
- TestCompositeResourceDefinitionNamespace: Creates and validates a namespaced XRD and
  verifies it generates conformant CRDs for composite resources
- TestCompositeResourceDefinitionCluster: Creates and validates a cluster-scoped XRD and
  verifies it generates conformant CRDs for composite resources
- TestCompositeResourceNamespace: Tests end-to-end namespaced composite resource
  functionality with a function pipeline
- TestCompositeResourceCluster: Tests end-to-end cluster-scoped composite resource
  functionality with a function pipeline
API Extensions - Legacy (v1)
- TestCompositeResourceDefinitionLegacy: Validates legacy XRDs create conformant CRDs for
  both composite resources and claims
- TestCompositeResourceLegacy: Tests end-to-end legacy composite resource and claim
  functionality using a function pipeline
Packages
- Configurations
  - TestConfiguration: Validates Configuration package installation, health status, and expected resources
- Providers
  - TestProvider: Validates Provider package installation, health status, expected resources (CRDs via MRDs)
- Functions
  - TestFunction: Validates Function package installation, health status, and expected resources
Crossplane CRDs
- TestCustomResourceDefinitions: Verifies all core Crossplane CRDs exist with proper
  structure and categorization
Usages
- TestUsageProtectionNamespaced: Validates namespaced Usage objects properly protect
  resources from deletion while in use
- TestUsageProtectionCluster: Validates ClusterUsage objects properly protect cluster-scoped
  resources from deletion while in use
- TestUsageLegacy: Validates legacy Usage objects properly protect cluster-scoped resources from deletion
Operations
- TestOperation: Validates Operation execution and verifies applied resource references
- TestWatchOperation: Tests WatchOperation functionality to detect resource changes and
  trigger operation pipelines
- TestCronOperation: Tests CronOperation functionality to schedule and execute operation
  pipelines on a cron schedule

I have:

Read and followed conformance's contribution process.
Run make reviewable test to ensure this PR is ready for review.

How has this code been tested

I have run all tests locally against a v2.0 Crossplane control plane with the following command:

❯ go test -v -run ^Test tests/*

I have also built the plugin and executed it against a v2.0 control plane with sonobouy:

CONFORMANCE_VERSION=v2.0.0
VERSION=${CONFORMANCE_VERSION} make build.all
docker tag build-ae964580/conformance-amd64 xpkg.crossplane.io/crossplane/conformance:${CONFORMANCE_VERSION}
kind load docker-image xpkg.crossplane.io/crossplane/conformance:${CONFORMANCE_VERSION} 

# update plugin-crossplane.yaml with:
# image: xpkg.crossplane.io/crossplane/conformance:v2.0.0-rc.0
# imagePullPolicy: Never

sonobuoy run --wait --plugin ./plugin-crossplane.yaml

Everything passes with the exception of TestConfiguration, which will not pass until we update index.docker.io/crossplane/conformance-testdata-configuration:latest after merging the CI changes included in the PR. I have gotten that test case to pass locally though by temporarily using xpkg.upbound.io/jaredorg/conformance-testdata-configuration:v0.0.1, which is essentially the same thing.

Future improvements to consider

Some improvements that are not included in this PR, but could be considered as follow-ups:

setup Renovate to keep all dependencies and build tools up to date
port the tests to use the same sigs.k8s.io/e2e-framework that crossplane uses for consistency and maintainability sake

…onformance testing Signed-off-by: Jared Watts <jbw976@gmail.com>

Signed-off-by: Jared Watts <jbw976@gmail.com>

jbw976 · 2024-10-09T16:54:33Z

oh wow, off to a pretty poor start for the CI in https://github.com/crossplane/conformance/actions/runs/11259504031/job/31308621246?pr=31 🙄

GitHub Actions has encountered an internal error when running your job.

Signed-off-by: Jared Watts <jbw976@gmail.com>

- this is to avoid the conflict with xpkg-action's and crossplane's install.sh download of the crank binary to a file called crossplane Signed-off-by: Jared Watts <jbw976@gmail.com>

Signed-off-by: Jared Watts <jbw976@gmail.com>

github-advanced-security · 2024-10-14T10:54:41Z

This pull request sets up GitHub code scanning for this repository. Once the scans have completed and the checks have passed, the analysis results for this pull request branch will appear on this overview. Once you merge this pull request, the 'Security' tab will show more code scanning analysis results (for example, for the default branch). Depending on your configuration and choice of analysis tool, future pull requests will be annotated with code scanning analysis results. For more information about GitHub code scanning, check out the documentation.

Signed-off-by: Jared Watts <jbw976@gmail.com>

- install a function and verify it becomes installed and healthy - create a composition that uses pipeline mode to compose resources with functions Signed-off-by: Jared Watts <jbw976@gmail.com>

jbw976 · 2024-10-15T14:49:02Z

@negz I've moved this PR out of draft after getting it to state I'm feeling decent about. I'll continue some more clean up work, but the bulk of what I wanted to accomplish here is now completed.

Let me know if you'd like a sync session to walk through these changes effectively together, or how I can help make your review easier. Thank you! 🙇

Signed-off-by: Jared Watts <jbw976@gmail.com>

- create set of legacy tests that verify legacy XRDs and XRs - create new v2 tests that verify cluster and namespaced scoped XRs - update provider/function tests Signed-off-by: Jared Watts <jbw976@gmail.com>

Signed-off-by: Jared Watts <jbw976@gmail.com>

negz

LGTM overall. None of my comments are blocking - I think this is worth merging to modernize these ancient tests.

port the tests to use the same sigs.k8s.io/e2e-framework that crossplane uses for consistency and maintainability sake

I think we should consider deduping these conformance tests with the E2E tests. I could imagine keeping both under c/c test/e2e directory. I suspect we could use labels such that conformance tests only ran a subset of the E2E tests (plus some conformance specific tests)?

The conformance test for distros of Crossplane would then become "must pass the c/c E2E suite with --label type=conformance. I suspect this'd reduce duplication and ongoing maintenance burden.

negz · 2025-09-02T22:16:24Z

tests/apiextensions_test.go

+	// verify the XRD becomes established
+	testXRDIsEstablished(ctx, t, kube, xrd)
+
+	t.Run("CRDIsCreatedForXR", func(t *testing.T) {


I wonder if we should test that you can't do legacy stuff (e.g. use claims) when using v2 style XRs. e.g. Test hat only LegacyCluster XRs make claim CRDs.

good idea! i'll include a test case for this as well, to ensure that modern XRDs do not produce claim CRDs.

negz · 2025-09-02T22:18:43Z

tests/apiextensions_legacy_test.go

+	})
+}
+
+// TestCompositeResourcePipelineModLegacy tests the creation of a legacy XRD and


Nit: This doesn't match the actual test name.

negz · 2025-09-02T22:21:13Z

tests/customresources_test.go

+			continue
+		}
+
+		// TODO(negz): Just cmp.Diff the entire CRD spec?


This particular test does seem a little low value without actually doing this. 🤔

The CRD could be completely different schematically, as long as it was e.g. cluster scoped and in the right categories.

That's a fair point - I do like though that it is at least verifying the entire expected API surface area. This test will fail if a distribution doesn't have all the required APIs, which has some value in testing.

jbw976 · 2025-09-06T01:07:05Z

I think we should consider deduping these conformance tests with the E2E tests.

this sounds like a good idea for long term maintenance. We would lose (i think) the current UX that is standard for Kubernetes conformance too where folks basically just have to execute sonobuoy run and everything is deployed/executed/processed with a nice results format output for them. For a distributor, that is a nice experience.

I'm not sure exactly what the experience would be for running conformance if it was part of e2e tests, but it probably has a slightly higher burden, like maybe checking out the c/c repo and then an earthly +e2e with relevant arguments? Haven't thought about it much yet, but I will capture this in an issue on this repo 🤓

edit: ah, we have a tracking issue already with #15

Signed-off-by: Jared Watts <jbw976@gmail.com>

jbw976 · 2025-09-06T23:43:24Z

I pushed another commit to add a new verification that modern XRDs do not create claim CRDs. I've incoporated/responded to all the feedback here, so I'll go ahead and merge now if CI is green 🙏

Thanks for the review @negz!

jbw976 added 2 commits September 28, 2024 08:55

Modernize Crossplane conformance tests to v1.17 and remove provider c…

18a2a71

…onformance testing Signed-off-by: Jared Watts <jbw976@gmail.com>

build: update build and CI to use modern versions and processes

c9cf91e

Signed-off-by: Jared Watts <jbw976@gmail.com>

jbw976 added 2 commits October 9, 2024 20:50

build: add go.cachedir target to makefile

4d6ff88

Signed-off-by: Jared Watts <jbw976@gmail.com>

build: adopt same linters as crossplane and fix all linting issues

e34f343

Signed-off-by: Jared Watts <jbw976@gmail.com>

This was referenced Oct 11, 2024

Update Crossplane CLI install script variable names crossplane-contrib/xpkg-action#19

Closed

bug: update crossplane CLI install script env var names crossplane-contrib/xpkg-action#20

Merged

jbw976 added 4 commits October 11, 2024 09:22

build: update xpkg-action version and usage

41f26cd

Signed-off-by: Jared Watts <jbw976@gmail.com>

build: use stable channel for xpkg-action

c7439cc

Signed-off-by: Jared Watts <jbw976@gmail.com>

build: rename crossplane tests directory to just tests

055f8fc

- this is to avoid the conflict with xpkg-action's and crossplane's install.sh download of the crank binary to a file called crossplane Signed-off-by: Jared Watts <jbw976@gmail.com>

build: add codeql and trivy scans

4ac8565

Signed-off-by: Jared Watts <jbw976@gmail.com>

jbw976 added 2 commits October 14, 2024 12:59

build: use specific cache for codeql

2ef0973

Signed-off-by: Jared Watts <jbw976@gmail.com>

test: add conformance tests for Functions

a43c53c

- install a function and verify it becomes installed and healthy - create a composition that uses pipeline mode to compose resources with functions Signed-off-by: Jared Watts <jbw976@gmail.com>

jbw976 marked this pull request as ready for review October 15, 2024 14:45

jbw976 added 2 commits October 15, 2024 21:44

test: use t.Helper() for test helper functions

a8ea8e8

Signed-off-by: Jared Watts <jbw976@gmail.com>

build: update build submodule to latest

d987f76

Signed-off-by: Jared Watts <jbw976@gmail.com>

jbw976 mentioned this pull request Jul 16, 2025

[v2] Update Conformance tests for v2 crossplane/crossplane#6635

Closed

jbw976 added 5 commits July 16, 2025 18:22

update XRD/XR tests to be v2 compatible

bac733b

- create set of legacy tests that verify legacy XRDs and XRs - create new v2 tests that verify cluster and namespaced scoped XRs - update provider/function tests Signed-off-by: Jared Watts <jbw976@gmail.com>

Test Operations functionality

bcf9d89

Signed-off-by: Jared Watts <jbw976@gmail.com>

test all v2 Crossplane CRDs

17341f8

Signed-off-by: Jared Watts <jbw976@gmail.com>

Usage conformance test verification (namespaced, cluster, and legacy)

e5ece80

Signed-off-by: Jared Watts <jbw976@gmail.com>

chore(deps): bump to apiextensions/v2

19ca7e1

Signed-off-by: Jared Watts <jbw976@gmail.com>

jbw976 mentioned this pull request Jul 26, 2025

Remove .status.crossplane.connectionDetails from modern XR schema crossplane/crossplane#6655

Merged

7 tasks

jbw976 added 4 commits July 26, 2025 22:38

test: WatchOperation conformance test

837e979

Signed-off-by: Jared Watts <jbw976@gmail.com>

test: CronOperation conformance test

dd8c5aa

Signed-off-by: Jared Watts <jbw976@gmail.com>

chore(deps): bump crossplane and crossplane-runtime to v2 modules

b60c2a3

Signed-off-by: Jared Watts <jbw976@gmail.com>

test: compose a secret also in TestCompositeResourceNamespace

feef199

Signed-off-by: Jared Watts <jbw976@gmail.com>

jbw976 force-pushed the modernize branch from f50c13e to feef199 Compare August 1, 2025 19:00

jbw976 added 7 commits August 6, 2025 16:38

test: Verify Provider installs an activated MRD

5e3f0e2

Signed-off-by: Jared Watts <jbw976@gmail.com>

chore(deps): bump crossplane and crossplane-runtime to v2.0.0-rc.1

79ee7fc

Signed-off-by: Jared Watts <jbw976@gmail.com>

Make namespace creation and clean up more reliable across multiple tests

689a5cf

Signed-off-by: Jared Watts <jbw976@gmail.com>

chore(deps): bump crossplane dependencies to v2 releases

a8c2347

Signed-off-by: Jared Watts <jbw976@gmail.com>

build: download crossplane CLI to build test config package

c6f9ce6

Signed-off-by: Jared Watts <jbw976@gmail.com>

build: bump go to 1.24.5, golangci to v2.2.2, docker buildx to v0.23.0

5d58c6c

Signed-off-by: Jared Watts <jbw976@gmail.com>

lint: linter fixes

0d92149

Signed-off-by: Jared Watts <jbw976@gmail.com>

negz reviewed Sep 2, 2025

View reviewed changes

negz approved these changes Sep 2, 2025

View reviewed changes

jbw976 mentioned this pull request Sep 6, 2025

Merge conformance tests into crossplane/crossplane? #15

Open

test: add verification that modern XRDs do not create claim CRDs

475a02c

Signed-off-by: Jared Watts <jbw976@gmail.com>

jbw976 merged commit e9a40e7 into crossplane:main Sep 6, 2025
9 checks passed

jbw976 mentioned this pull request Sep 8, 2025

build: fixes for make build and make publish after migrating to imagelight.mk #32

Merged

jbw976 mentioned this pull request Sep 17, 2025

update conformance-testdata-configuration for v2 #35

Merged

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Modernize Crossplane conformance suite#31

Modernize Crossplane conformance suite#31
jbw976 merged 29 commits intocrossplane:mainfrom
jbw976:modernize

jbw976 commented Oct 9, 2024 •

edited

Loading

Uh oh!

jbw976 commented Oct 9, 2024

Uh oh!

github-advanced-security bot commented Oct 14, 2024

Uh oh!

jbw976 commented Oct 15, 2024

Uh oh!

negz left a comment

Uh oh!

negz Sep 2, 2025

Uh oh!

jbw976 Sep 6, 2025

Uh oh!

negz Sep 2, 2025

Uh oh!

negz Sep 2, 2025

Uh oh!

jbw976 Sep 6, 2025

Uh oh!

jbw976 commented Sep 6, 2025 •

edited

Loading

Uh oh!

jbw976 commented Sep 6, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

jbw976 commented Oct 9, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of your changes

How has this code been tested

Future improvements to consider

Uh oh!

jbw976 commented Oct 9, 2024

Uh oh!

github-advanced-security bot commented Oct 14, 2024

Uh oh!

jbw976 commented Oct 15, 2024

Uh oh!

negz left a comment

Choose a reason for hiding this comment

Uh oh!

negz Sep 2, 2025

Choose a reason for hiding this comment

Uh oh!

jbw976 Sep 6, 2025

Choose a reason for hiding this comment

Uh oh!

negz Sep 2, 2025

Choose a reason for hiding this comment

Uh oh!

negz Sep 2, 2025

Choose a reason for hiding this comment

Uh oh!

jbw976 Sep 6, 2025

Choose a reason for hiding this comment

Uh oh!

jbw976 commented Sep 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jbw976 commented Sep 6, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jbw976 commented Oct 9, 2024 •

edited

Loading

jbw976 commented Sep 6, 2025 •

edited

Loading