chore: add conformance to PRs using uploaded artifacts#2005
Conversation
…uild and load them into the kind cluster before tests. On-behalf-of: Gerald Morrison (SAP) <gerald.morrison@sap.com> Signed-off-by: Gerald Morrison (SAP) <gerald.morrison@sap.com>
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughAdds PR-specific conformance testing that consumes build artifacts: build job now exposes Changes
Sequence DiagramsequenceDiagram
participant GH as GitHub Actions
participant Build as Build Job
participant Artifacts as Artifact Storage
participant Conformance as Conformance (PR) Job
participant Skopeo as Skopeo/Docker
participant Kind as Kind Cluster
rect rgba(100,150,255,0.5)
Note over GH,Build: PR triggers build
GH->>Build: Run build job
Build->>Artifacts: Upload CLI/chart/image artifacts and set outputs (artifact_name, version)
end
rect rgba(100,200,150,0.5)
Note over Conformance,Kind: PR conformance consumes artifacts
GH->>Conformance: Trigger conformance-pr (needs: build)
Conformance->>Artifacts: Download CLI/chart/image artifacts
Conformance->>Skopeo: Convert OCI layout -> docker-archive
Skopeo->>Conformance: Provide docker-archive
Conformance->>Kind: Load images via kind load image-archive
Conformance->>Kind: Run conformance scenarios (uses CLI_IMAGE, TOOLKIT_IMAGE, PRELOAD_IMAGES)
end
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Possibly related PRs
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
📝 Coding Plan
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…t-to-prs On-behalf-of: Gerald Morrison (SAP) <gerald.morrison@sap.com> Signed-off-by: Gerald Morrison (SAP) <gerald.morrison@sap.com>
On-behalf-of: Gerald Morrison (SAP) <gerald.morrison@sap.com> Signed-off-by: Gerald Morrison (SAP) <gerald.morrison@sap.com>
There was a problem hiding this comment.
🧹 Nitpick comments (2)
.github/workflows/kubernetes-controller.yml (1)
220-225: Keep the PR chart artifact name in one place.
controller-chart-pris now the contract between the upload step andconformance-pr. Extracting it to a shared env var or build output would make the next rename a one-line change instead of a runtime break.Also applies to: 473-475
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In @.github/workflows/kubernetes-controller.yml around lines 220 - 225, The workflow hardcodes the artifact name "controller-chart-pr" in the "Upload Chart Artifact" step, creating a fragile contract with the later "conformance-pr" consumer; extract that artifact name into a single shared value (e.g., a workflow env var like CHART_ARTIFACT_NAME or a build output) and replace the literal name in the Upload Chart Artifact step and any consumers (including the conformance-pr references) with that variable so future renames are a single change..github/workflows/conformance.yml (1)
21-38: Validate artifact/version pairs before theskopeosteps.
cli_versionandtoolkit_versionare only documented as conditionally required, but Line 113 and Line 136 consume them unguarded. If a caller forgets one, this reusable workflow fails later with a fairly opaqueoci:...:/skopeoerror instead of a direct input validation failure.Proposed guard step
- name: Install kind run: go install sigs.k8s.io/kind@v0.31.0 + - name: Validate artifact inputs + env: + CLI_ARTIFACT: ${{ inputs.cli_artifact }} + CLI_VERSION: ${{ inputs.cli_version }} + TOOLKIT_IMAGE_ARTIFACT: ${{ inputs.toolkit_image_artifact }} + TOOLKIT_VERSION: ${{ inputs.toolkit_version }} + run: | + if [ -n "${CLI_ARTIFACT}" ] && [ -z "${CLI_VERSION}" ]; then + echo "::error::cli_version is required when cli_artifact is set" + exit 1 + fi + if [ -n "${TOOLKIT_IMAGE_ARTIFACT}" ] && [ -z "${TOOLKIT_VERSION}" ]; then + echo "::error::toolkit_version is required when toolkit_image_artifact is set" + exit 1 + fi + # Artifact mode: Download and prepare CLI image - name: Download CLI artifactAlso applies to: 95-137
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In @.github/workflows/conformance.yml around lines 21 - 38, Add explicit validation before any skopeo steps to check that artifact/version input pairs are provided together: if toolkit_image_artifact is set then toolkit_version must be non-empty, and if cli_artifact is set then cli_version must be non-empty; when a pair is missing fail fast with a clear error message. Implement this as a dedicated workflow step (e.g., "Validate artifact/version inputs") that inspects the inputs toolkit_image_artifact, toolkit_version, cli_artifact, and cli_version and uses a failing exit (with a descriptive echo) when the conditional requirement is not met so downstream skopeo invocations are never reached with missing inputs.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Nitpick comments:
In @.github/workflows/conformance.yml:
- Around line 21-38: Add explicit validation before any skopeo steps to check
that artifact/version input pairs are provided together: if
toolkit_image_artifact is set then toolkit_version must be non-empty, and if
cli_artifact is set then cli_version must be non-empty; when a pair is missing
fail fast with a clear error message. Implement this as a dedicated workflow
step (e.g., "Validate artifact/version inputs") that inspects the inputs
toolkit_image_artifact, toolkit_version, cli_artifact, and cli_version and uses
a failing exit (with a descriptive echo) when the conditional requirement is not
met so downstream skopeo invocations are never reached with missing inputs.
In @.github/workflows/kubernetes-controller.yml:
- Around line 220-225: The workflow hardcodes the artifact name
"controller-chart-pr" in the "Upload Chart Artifact" step, creating a fragile
contract with the later "conformance-pr" consumer; extract that artifact name
into a single shared value (e.g., a workflow env var like CHART_ARTIFACT_NAME or
a build output) and replace the literal name in the Upload Chart Artifact step
and any consumers (including the conformance-pr references) with that variable
so future renames are a single change.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: bdafc597-86b1-4cc5-aa98-b783c7369224
📒 Files selected for processing (3)
.github/workflows/conformance.yml.github/workflows/controller-release.yml.github/workflows/kubernetes-controller.yml
…default plugin verification (is not signed) On-behalf-of: Gerald Morrison (SAP) <gerald.morrison@sap.com> Signed-off-by: Gerald Morrison (SAP) <gerald.morrison@sap.com>
On-behalf-of: Gerald Morrison (SAP) <gerald.morrison@sap.com> Signed-off-by: Gerald Morrison (SAP) <gerald.morrison@sap.com>
… tested On-behalf-of: Gerald Morrison (SAP) <gerald.morrison@sap.com> Signed-off-by: Gerald Morrison (SAP) <gerald.morrison@sap.com>
On-behalf-of: Gerald Morrison (SAP) <gerald.morrison@sap.com> Signed-off-by: Gerald Morrison (SAP) <gerald.morrison@sap.com>
There was a problem hiding this comment.
🧹 Nitpick comments (2)
.github/workflows/conformance.yml (2)
147-151: Consider using a more robust glob expansion.The
lscommand with a glob pattern could capture multiple files if the artifact contains more than one matching chart. While unlikely in practice, this would causeTOOLKIT_IMAGEto contain multiple paths.🔧 Safer single-file extraction
- name: Set toolkit image to local chart if: ${{ inputs.toolkit_chart_artifact != '' }} run: | - CHART_PATH=$(ls tmp/chart-artifact/chart-*.tgz) + CHART_PATH=$(ls tmp/chart-artifact/chart-*.tgz | head -n 1) + if [ -z "$CHART_PATH" ]; then + echo "::error::No chart found in artifact" + exit 1 + fi echo "TOOLKIT_IMAGE=${{ github.workspace }}/${CHART_PATH}" >> "$GITHUB_ENV"🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In @.github/workflows/conformance.yml around lines 147 - 151, The CHART_PATH assignment uses `ls` with a glob which can expand to multiple files and produce multiple paths in TOOLKIT_IMAGE; update the step that sets CHART_PATH so it uses shell globbing to pick a single artifact (e.g., populate an array from tmp/chart-artifact/chart-*.tgz and assign CHART_PATH to the first element) and add a guard that fails the job if no match is found, then export TOOLKIT_IMAGE using that single CHART_PATH value.
16-40: Consider adding validation for required input combinations.The input descriptions state that
cli_versionis "required when cli_artifact is set" andtoolkit_versionis "required when toolkit_image_artifact is set", but there's no enforcement. If artifact inputs are provided without their corresponding version inputs, the skopeo commands will use empty tags, potentially causing silent failures.Since callers are internal reusable workflows, you could either:
- Add a validation step at the start of the job
- Document this as a caller responsibility
🔧 Optional validation step
- name: Validate artifact inputs if: ${{ inputs.cli_artifact != '' || inputs.toolkit_image_artifact != '' }} run: | if [ -n "${{ inputs.cli_artifact }}" ] && [ -z "${{ inputs.cli_version }}" ]; then echo "::error::cli_version is required when cli_artifact is set" exit 1 fi if [ -n "${{ inputs.toolkit_image_artifact }}" ] && [ -z "${{ inputs.toolkit_version }}" ]; then echo "::error::toolkit_version is required when toolkit_image_artifact is set" exit 1 fi🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In @.github/workflows/conformance.yml around lines 16 - 40, Add an input validation step at the start of the workflow/job to enforce the stated input combinations: when cli_artifact is non-empty require cli_version, and when toolkit_image_artifact is non-empty require toolkit_version; implement the check using the workflow's shell step (e.g., a step named "Validate artifact inputs") that reads inputs.cli_artifact, inputs.cli_version, inputs.toolkit_image_artifact and inputs.toolkit_version, emits clear error messages like "cli_version is required when cli_artifact is set" or "toolkit_version is required when toolkit_image_artifact is set", and exits non-zero on failure so the job fails fast rather than allowing skopeo to run with empty tags.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Nitpick comments:
In @.github/workflows/conformance.yml:
- Around line 147-151: The CHART_PATH assignment uses `ls` with a glob which can
expand to multiple files and produce multiple paths in TOOLKIT_IMAGE; update the
step that sets CHART_PATH so it uses shell globbing to pick a single artifact
(e.g., populate an array from tmp/chart-artifact/chart-*.tgz and assign
CHART_PATH to the first element) and add a guard that fails the job if no match
is found, then export TOOLKIT_IMAGE using that single CHART_PATH value.
- Around line 16-40: Add an input validation step at the start of the
workflow/job to enforce the stated input combinations: when cli_artifact is
non-empty require cli_version, and when toolkit_image_artifact is non-empty
require toolkit_version; implement the check using the workflow's shell step
(e.g., a step named "Validate artifact inputs") that reads inputs.cli_artifact,
inputs.cli_version, inputs.toolkit_image_artifact and inputs.toolkit_version,
emits clear error messages like "cli_version is required when cli_artifact is
set" or "toolkit_version is required when toolkit_image_artifact is set", and
exits non-zero on failure so the job fails fast rather than allowing skopeo to
run with empty tags.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: fcf5eea0-b04a-44e6-8417-a9f3089ae105
📒 Files selected for processing (1)
.github/workflows/conformance.yml
|
So just to confirm. We are now unconditionally always pushing artifacts on every build. Do we risk hitting some quota somewhere if we build too much? Do we need to clean up? |
fabianburth
left a comment
There was a problem hiding this comment.
Should we set a shorter artifact retention policy, especially for the artifacts uploaded for PR conformance tests?
|
we can set retention to 1 day |
|
That sounds good 👍 |
…t-to-prs On-behalf-of: Gerald Morrison (SAP) <gerald.morrison@sap.com> Signed-off-by: Gerald Morrison (SAP) <gerald.morrison@sap.com>
…en-component-model into add-conformance-test-to-prs On-behalf-of: Gerald Morrison (SAP) <gerald.morrison@sap.com> Signed-off-by: Gerald Morrison (SAP) <gerald.morrison@sap.com>
|
For the retention period, I added 1 day. In the cli the upload is done for all triggers, PR and push, but the short retention period should be fine without condition as in the non-PR case the uploaded artifacts are directly used by the publish step. |
…n provided and retention period of 1 day for artifact uploads. On-behalf-of: Gerald Morrison (SAP) <gerald.morrison@sap.com> Signed-off-by: Gerald Morrison (SAP) <gerald.morrison@sap.com>
There was a problem hiding this comment.
🧹 Nitpick comments (2)
.github/workflows/conformance.yml (2)
147-151: Consider using a safer glob expansion.Using
lsto capture the chart path works when exactly one file matches, but will break if multiple.tgzfiles exist (e.g., from a previous failed run or accidental extra files).🔧 Suggested safer approach
- name: Set toolkit image to local chart if: ${{ inputs.toolkit_chart_artifact != '' }} run: | - CHART_PATH=$(ls tmp/chart-artifact/chart-*.tgz) + shopt -s nullglob + CHARTS=(tmp/chart-artifact/chart-*.tgz) + if [ ${`#CHARTS`[@]} -ne 1 ]; then + echo "::error::Expected exactly 1 chart file, found ${`#CHARTS`[@]}" + exit 1 + fi + CHART_PATH=${CHARTS[0]} echo "TOOLKIT_IMAGE=${{ github.workspace }}/${CHART_PATH}" >> "$GITHUB_ENV"🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In @.github/workflows/conformance.yml around lines 147 - 151, The step "Set toolkit image to local chart" uses `ls` to populate CHART_PATH which breaks if zero or multiple chart-*.tgz files exist; change the shell logic that sets CHART_PATH to use a safe glob or find (e.g., expand tmp/chart-artifact/chart-*.tgz into an array and verify exactly one match or use find ... -print -quit) and fail with a clear error if none or more than one file is found so CHART_PATH is always a single, deterministic path.
167-178: Validation step is a good addition; error message could be clarified.The validation ensures both required images are set before running scenarios. However, the error message on line 175 mentions
'toolkit_version'as a required input alongsidetoolkit_chart_artifact, buttoolkit_versionis only needed when usingtoolkit_image_artifact(to tag the controller image), not for the chart itself.📝 Suggested message refinement
if [ -z "$TOOLKIT_IMAGE" ]; then - echo "::error::TOOLKIT_IMAGE is not set. Provide either 'toolkit_image' or 'toolkit_chart_artifact' + 'toolkit_version' inputs." + echo "::error::TOOLKIT_IMAGE is not set. Provide either 'toolkit_image' or 'toolkit_chart_artifact' inputs." err=1 fi🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In @.github/workflows/conformance.yml around lines 167 - 178, Update the validation error text for TOOLKIT_IMAGE to accurately reflect required inputs: change the echo in the TOOLKIT_IMAGE check (the line that currently mentions "'toolkit_chart_artifact' + 'toolkit_version'") to state that either 'toolkit_image' or 'toolkit_chart_artifact' must be provided, and separately note that 'toolkit_version' is only required when using the toolkit image artifact (e.g., 'toolkit_image_artifact' + 'toolkit_version'), not for chart-only usage; edit the echo message referencing TOOLKIT_IMAGE and the inputs 'toolkit_chart_artifact' and 'toolkit_version' to convey this distinction.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Nitpick comments:
In @.github/workflows/conformance.yml:
- Around line 147-151: The step "Set toolkit image to local chart" uses `ls` to
populate CHART_PATH which breaks if zero or multiple chart-*.tgz files exist;
change the shell logic that sets CHART_PATH to use a safe glob or find (e.g.,
expand tmp/chart-artifact/chart-*.tgz into an array and verify exactly one match
or use find ... -print -quit) and fail with a clear error if none or more than
one file is found so CHART_PATH is always a single, deterministic path.
- Around line 167-178: Update the validation error text for TOOLKIT_IMAGE to
accurately reflect required inputs: change the echo in the TOOLKIT_IMAGE check
(the line that currently mentions "'toolkit_chart_artifact' +
'toolkit_version'") to state that either 'toolkit_image' or
'toolkit_chart_artifact' must be provided, and separately note that
'toolkit_version' is only required when using the toolkit image artifact (e.g.,
'toolkit_image_artifact' + 'toolkit_version'), not for chart-only usage; edit
the echo message referencing TOOLKIT_IMAGE and the inputs
'toolkit_chart_artifact' and 'toolkit_version' to convey this distinction.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 4d138474-1f56-434c-aafd-f6fd64359771
📒 Files selected for processing (3)
.github/workflows/cli.yml.github/workflows/conformance.yml.github/workflows/kubernetes-controller.yml
On-behalf-of: Gerald Morrison (SAP) <gerald.morrison@sap.com> Signed-off-by: Gerald Morrison (SAP) <gerald.morrison@sap.com>
<!-- markdownlint-disable MD041 --> Upgrades Helm from v3.14.0 (and v4.0.0 in conformance) to v4.1.3 across all CI workflows and the controller Taskfile: - `.github/workflows/kubernetes-controller.yml` — update all 3 Install Helm steps from v3.14.0 to v4.1.3, standardize action comment to `# v4.3.1` - `.github/workflows/controller-release.yml` — update Install Helm step from v3.14.0 to v4.1.3, standardize action comment to `# v4.3.1` - `.github/workflows/conformance.yml` — update Install Helm step from v4.0.0 to v4.1.3 - `kubernetes/controller/Taskfile.yml` — update helm-schema-plugin install to use `.git` suffix and `--verify=false` for Helm v4 compatibility <!-- Usage: `Fixes #<issue number>`, or `Fixes (paste link of issue)`. --> Relates to: open-component-model#2005 Signed-off-by: Piotr Janik <piotr.janik@sap.com>
<!-- markdownlint-disable MD041 --> Upgrades Helm from v3.14.0 (and v4.0.0 in conformance) to v4.1.3 across all CI workflows and the controller Taskfile: - `.github/workflows/kubernetes-controller.yml` — update all 3 Install Helm steps from v3.14.0 to v4.1.3, standardize action comment to `# v4.3.1` - `.github/workflows/controller-release.yml` — update Install Helm step from v3.14.0 to v4.1.3, standardize action comment to `# v4.3.1` - `.github/workflows/conformance.yml` — update Install Helm step from v4.0.0 to v4.1.3 - `kubernetes/controller/Taskfile.yml` — update helm-schema-plugin install to use `.git` suffix and `--verify=false` for Helm v4 compatibility <!-- Usage: `Fixes #<issue number>`, or `Fixes (paste link of issue)`. --> Relates to: open-component-model#2005 Signed-off-by: Piotr Janik <piotr.janik@sap.com> # Conflicts: # .github/workflows/controller-release.yml # .github/workflows/kubernetes-controller.yml
…t-model#2005) On-behalf-of: Gerald Morrison (SAP) <gerald.morrison@sap.com> <!-- markdownlint-disable MD041 --> #### What this PR does / why we need it Add conformance test runs to PRs. This was restricted only to pushes to main up to now. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Conformance tests run for pull requests using PR-built artifacts via a new PR-specific conformance job. * Builds now publish a tracked CLI version that downstream workflows consume. * **Improvements** * Conformance workflows accept artifact-driven inputs (CLI, chart, image, version), validate inputs, and provide environment fallbacks. * Artifacts upload more reliably for PRs with short retention; added optional image preloading for air‑gapped testing. * Helm tooling upgraded to a newer v3.x release. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Gerald Morrison (SAP) <gerald.morrison@sap.com> Co-authored-by: Gerald Morrison (SAP) <gerald.morrison@sap.com> Co-authored-by: Matthias Bruns <github@matthiasbruns.com> Co-authored-by: Jakob Möller <jakob.moeller@sap.com>
#### What this PR does / why we need it <!-- markdownlint-disable MD041 --> Upgrades Helm from v3.14.0 (and v4.0.0 in conformance) to v4.1.3 across all CI workflows and the controller Taskfile: - `.github/workflows/kubernetes-controller.yml` — update all 3 Install Helm steps from v3.14.0 to v4.1.3, standardize action comment to `# v4.3.1` - `.github/workflows/controller-release.yml` — update Install Helm step from v3.14.0 to v4.1.3, standardize action comment to `# v4.3.1` - `.github/workflows/conformance.yml` — update Install Helm step from v4.0.0 to v4.1.3 - `kubernetes/controller/Taskfile.yml` — update helm-schema-plugin install to use `.git` suffix and `--verify=false` for Helm v4 compatibility <!-- Usage: `Fixes #<issue number>`, or `Fixes (paste link of issue)`. --> Relates to: #2005 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Updated Helm CLI versions in CI/CD workflows and build configurations for infrastructure improvements. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Signed-off-by: Piotr Janik <piotr.janik@sap.com>
On-behalf-of: Gerald Morrison (SAP) gerald.morrison@sap.com
What this PR does / why we need it
Add conformance test runs to PRs. This was restricted only to pushes to main up to now.
Summary by CodeRabbit
New Features
Improvements