feat(883): helm transfer cli#1846
Conversation
83ccd7e to
14e9bae
Compare
bee94d8 to
8a0a6cf
Compare
fc182e4 to
3690357
Compare
4effad9 to
91b19f1
Compare
|
TODOS: test that helm from oci repo to oci repo works - this is assumed to work with the current transformers - I just need to verify it |
|
verified oci helm -> oci transfer as well - just waiting for #1862 to be merged |
91b19f1 to
01af0c9
Compare
725be0f to
6092277
Compare
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughThe PR adds Helm chart resource support to the component-version transfer functionality by integrating new Helm transformers (GetHelmChart, ConvertHelmToOCI), registering the Helm scheme, and updating the transformation graph to handle Helm resources alongside OCI artifacts and local resources. Changes
Sequence Diagram(s)sequenceDiagram
participant Client
participant GetHelmChart as GetHelmChart<br/>Transformer
participant ConvertHelmToOCI as ConvertHelmToOCI<br/>Transformer
participant Upload as Upload<br/>Transformer
Client->>GetHelmChart: Helm chart resource
GetHelmChart->>GetHelmChart: Build unstructured spec
GetHelmChart->>GetHelmChart: Fetch chart & prov files
GetHelmChart-->>ConvertHelmToOCI: (resource, chartFile, provFile)
ConvertHelmToOCI->>ConvertHelmToOCI: Transform to OCI artifact
ConvertHelmToOCI-->>Upload: (converted resource)
alt uploadAsOCIArtifact = true
Upload->>Upload: Upload as OCI artifact
Upload-->>Client: OCI artifact reference
else uploadAsOCIArtifact = false
Upload->>Upload: Upload as local resource
Upload-->>Client: Local blob access
end
Estimated code review effort🎯 4 (Complex) | ⏱️ ~65 minutes Possibly related PRs
Suggested labels
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
6ceb790 to
75b5d0a
Compare
On-behalf-of: SAP <matthias.bruns@sap.com> Signed-off-by: Matthias Bruns <git@matthiasbruns.com>
On-behalf-of: SAP <matthias.bruns@sap.com> Signed-off-by: Matthias Bruns <git@matthiasbruns.com>
On-behalf-of: SAP <matthias.bruns@sap.com> Signed-off-by: Matthias Bruns <git@matthiasbruns.com>
On-behalf-of: SAP <matthias.bruns@sap.com> Signed-off-by: Matthias Bruns <git@matthiasbruns.com> # Conflicts: # cli/go.sum
On-behalf-of: SAP <matthias.bruns@sap.com> Signed-off-by: Matthias Bruns <git@matthiasbruns.com> # Conflicts: # cli/go.mod # cli/go.sum # cli/integration/go.mod # cli/integration/go.sum
c17bb2f to
5ca98bd
Compare
On-behalf-of: SAP <matthias.bruns@sap.com> Signed-off-by: Matthias Bruns <git@matthiasbruns.com>
5ca98bd to
58b4af6
Compare
…en-component-model#1889) #### What this PR does / why we need it This PR solves the issue that happens when you want to pass nil values for optional pointers through the transformation steps. If in a spec a type is annotated by `omitempty` and resolved to `nil` one would suspect that `nil` is a valid value. The problem is, that our schemagen does not generate a rule that allows `null` to be passed in such cases. Since we have no way to omitting keys based on the fact that values exist or not, we have to circumvent that in the transformer graph. This PR does that. Here is an example of what we are trying to solve with this change: ```go convertToOCITransform := transformv1alpha1.GenericTransformation{ TransformationMeta: meta.TransformationMeta{ Type: ociv1alpha1.ConvertHelmToOCIV1alpha1, ID: convertResourceID, }, Spec: &runtime.Unstructured{Data: map[string]any{ "resource": fmt.Sprintf("${%s.output.resource}", getResourceID), "chartFile": fmt.Sprintf("${%s.output.chartFile}", getResourceID), "provFile": fmt.Sprintf("${%[1]s.output.?provFile}", getResourceID), }}, } ``` `provFile` might be null, if not provided by the helm chart. We can only run checks on the values with CEL here. If we pass `nil` to the `provFile` json field, the schemavalidation will fail. open-component-model#1846 #### Which issue(s) this PR fixes Contributes: open-component-model/ocm-project#883 #### Testing ##### How to test the changes Fully tested in https://github.com/open-component-model/open-component-model/pull/1846/changes#diff-85d55581ce565d132e8a8b027169c162a024c7e25f6df4c5cc2ddbbfb1294d2fR356 ##### Verification - [x] I have tested the changes locally by running `ocm` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Schema-driven resolution: resolvers can accept optional schemas (including spec sub-schemas) to detect optional fields and guide processing. * **Bug Fixes** * Optional-field removal: optional fields that resolve to nil are removed from resources (no null retained). * Runtime processing no longer treats optional nil values as errors. * **Tests** * Expanded coverage for nil handling, nested/referenced schemas, array items, and targeted deletions. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Matthias Bruns <git@matthiasbruns.com> Signed-off-by: Matthias Bruns <github@matthiasbruns.com> Co-authored-by: Fabian Burth <fabian.burth@sap.com>
<!-- markdownlint-disable MD041 -->
#### What this PR does / why we need it
This pull-request adds the "download classic helm chart" to the
transformer api.
`This PR adds a bigger refactoring to the existing `input`and
`oci-image-creation` - the concepts stayed the same, just technical
changes.
`
**introduce helm access type from ocmv1**
It also adds the missing Helm access type to the new ocm.
```go
type Helm struct {
// +ocm:jsonschema-gen:enum=Helm/v1,helm/v1
// +ocm:jsonschema-gen:enum:deprecated=Helm,helm
Type runtime.Type `json:"type"`
// HelmRepository is the URL of the helm repository to load the chart from.
HelmRepository string `json:"helmRepository"`
// HelmChart if the name of the helm chart and its version separated by a colon.
HelmChart string `json:"helmChart"`
// Version can either be specified as part of the chart name or separately.
Version string `json:"version,omitempty"`
}
```
It is important to note what I dropped deprecated fields from the spec.
CA and Keyring related fields should be loaded from the credentials as
already done in the helm input.
**reuse blob download from helm input**
Additionally, this PR refactors the `download helm blob `from input to
generalize the logic and make it reusable for helm access as well.
**Reuse chart to OCI from help input**
I also generalized the chart->OCI logic from `input` and made it
reusable. It is now used in both `input` and the `convert transformer.`
**Refactor `CopyChartToOCILayout`**
I had to refactor `CopyChartToOCILayout` to be able to get the
`ociImageSpecV1.Descriptor`. The problem was that the whole impl was
async. Channels lead to blocking goroutines since `io.Pipe` was not done
writing at the point where I needed the descriptor. I had to create
```go
type Result struct {
*direct.Blob
desc chan descriptorOrError
}
type descriptorOrError struct {
Descriptor ociImageSpecV1.Descriptor
Err error
}
```
to delay the access to said data.
**GetHelmChart transformer implementation**
The get transformer does the following:
- validate input spec against new `v1alpha1.GetHelmChart`
- generate output paths for the spec output
- get credentials by requesting
`ResourceConsumerIdentityProvider.GetResourceCredentialConsumerIdentity`
which is implemented in the `HelmAccess`
- delegate download to generalized helm download from the input package
(now downloads)
- in theory, we could use this transformer also for oci helm download,
since the downloader supports the feature
- but we will use the native GetOCIArtifact from transformers (probably)
- resulting blobs (chart and optionally a prov) will be copied into
`*v1alpha1.File`
- the output contains
- the original `Resource`
- the chart file pointer with `v1alpha1.File`
- (optionally) the prov file pointer with `*v1alpha1.File`
**ConvertHelmChartToOCI transformer implementation**
The convertion transformer does the following:
- expect chart, prov and resources as input
- reuse the oci creation logic from input
- calculate the correct `ImageReference` for the `OCIImage` access
- push the artifact as localblob as an output back the the graph
**testing**
- The pr spawns a ~~repotest.NewTempServer()~~ mock http server pointing
to the shared helm `testdata/mychart-0.1.0.tgz` dir
- In the test we create Resources with several variations of the `Helm`
access
- Each test verifies that the files are correctly downloaded and equal
the original file
- provenance file download added
- **Open Question**: should we also verify during transform? Currently,
we only download the .prov file
- helm to oci tests and validated the created oci image and the prov
files
#### Which issue(s) this PR fixes
Contributes:
open-component-model/ocm-project#883
#### Testing
##### How to test the changes
See
open-component-model#1846
for testing
**make sure helm input did not break**
```bash
#!/bin/zsh
alias OCM='go run ../../main.go'
# Test helm input with local path
CHART_PATH="../../../bindings/go/helm/testdata/mychart"
# stat the chart path to ensure it exists
if [ ! -d "$CHART_PATH" ]; then
echo "Error: Chart path $CHART_PATH does not exist"
exit 1
fi
cat <<EOF > constructor-input-local.yaml
components:
- name: ocm.software/helm-input-local
version: 0.1.0
provider:
name: ocm.software
resources:
- name: mychart
version: 0.1.0
type: helmChart
input:
type: helm/v1
path: $CHART_PATH
EOF
# create dir relative to current directory to avoid issues with absolute paths in the constructor
INPUT_CTF_LOCAL="input-ctf-local-$(date +%s)"
mkdir $INPUT_CTF_LOCAL
echo "Using temporary directory for helm input (local): $INPUT_CTF_LOCAL"
OCM add cv --repository ctf::$INPUT_CTF_LOCAL --constructor constructor-input-local.yaml
echo "--- Local helm input test passed ---"
# Test helm input with remote repository (uses same podinfo chart)
cat <<EOF > constructor-input-remote.yaml
components:
- name: ocm.software/helm-input-remote
version: 6.9.1
provider:
name: ocm.software
resources:
- name: podinfo
version: 6.9.1
type: helmChart
input:
type: helm/v1
helmRepository: https://stefanprodan.github.io/podinfo/podinfo-6.9.1.tgz
EOF
INPUT_CTF_REMOTE="input-ctf-remote-$(date +%s)"
mkdir $INPUT_CTF_REMOTE
echo "Using temporary directory for helm input (remote): $INPUT_CTF_REMOTE"
OCM add cv --repository ctf::$INPUT_CTF_REMOTE --constructor constructor-input-remote.yaml
echo "--- Remote helm input test passed ---"
```
##### Verification
- [x] I have tested the changes locally by running `ocm`
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added Helm access spec, credential-aware identity resolution, and
options for TLS/credentials.
* Remote Helm chart retrieval (HTTP/OCI) with provenance support and
configurable buffering/output path handling.
* Conversion of Helm charts into OCI image layouts and transformers to
fetch charts and produce OCI artifacts.
* **Tests**
* Extensive unit and integration tests covering access parsing,
identity/error cases, downloads, provenance, OCI conversion, and
transformers.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Signed-off-by: Matthias Bruns <git@matthiasbruns.com>
<!-- markdownlint-disable MD041 --> #### What this PR does / why we need it This PR implements the transformers from open-component-model#1832 It creates a 3-step-transformation for the helm chart transformation-chain: ```mermaid flowchart TD A[Transfer Helm Chart] -->B(Get Helm Chart) B --> C{Download from Repo} C -->|Has Prov| D[Download from Repo] C --> E{Pack Helm LocalBlob} D --> E E -->|GetHelmOutput| F{Convert to OCI} F -->|ConverToHelm| G{Pack OCI from Helm} F -->|ConverToHelm with prov| G{Pack OCI from Helm} G -->|upload-as localBlob/default | I{Upload as localBlob} G -->|upload-as ociArtifact | H{Upload as OCI Image} ``` 1. get helm chart with `GetHelmChartV1alpha1` 2. convert helm to OCI with `ConvertHelmToOCIV1alpha1` 3. upload a. localBlobl: the oci artifact with `OCIAddLocalResourceV1alpha1` b. ociArtifact: the oci artifact with `AddOCIArtifactType` The [spec PR](open-component-model#1832) will be kept in sync with changed from here in the `bindings/go/helm` package. #### Which issue(s) this PR fixes Contributes: open-component-model/ocm-project#883 #### Testing ##### How to test the changes ```bash #!/bin/zsh alias OCM='go run ../../main.go' REGISTRY="ghcr.io/matthiasbruns/ocm-tutorials" REGISTRY2="ghcr.io/matthiasbruns/ocm-tutorials-2" pause() { echo "\n>>> Next: $1" echo "--- Press Enter to continue ---" read } # OCM --help # create constructor.yaml # https://stefanprodan.github.io/podinfo/podinfo-6.9.1.tgz cat <<EOF > constructor.yaml components: - name: ocm.software/podinfo version: 6.9.1 provider: name: ocm.software resources: - name: podinfo version: 6.9.1 type: helmChart access: type: helm/v1 helmRepository: https://stefanprodan.github.io/podinfo helmChart: podinfo-6.9.1.tgz EOF CTF_DIR=$(mktemp -d) echo "Using temporary directory: $CTF_DIR" pause "Add component version to CTF from constructor.yaml" # add cv command OCM add cv --repository ctf::$CTF_DIR --constructor constructor.yaml --skip-reference-digest-processing HELM_REF="ctf::$CTF_DIR//ocm.software/podinfo:6.9.1" pause "Create component version ($REGISTRY)" # transfer OCM transfer component-version $HELM_REF $REGISTRY --copy-resources pause "Transfer with --upload-as localBlob (OCI registry)" # transfer --upload-as localBlob OCM transfer component-version $HELM_REF $REGISTRY --copy-resources --upload-as localBlob pause "Transfer with --upload-as ociArtifact (OCI registry)" # transfer --upload-as ociArtifact OCM transfer component-version $HELM_REF $REGISTRY --copy-resources --upload-as ociArtifact pause "Download component descriptor with oras" # download with oras oras pull $REGISTRY/component-descriptors/ocm.software/podinfo:6.9.1 --output . pause "Download resource using OCM CLI" # rm downloaded if exists rm -rf downloaded # downloadCMD resource OCM download resource https://$REGISTRY//ocm.software/podinfo:6.9.1 --identity name=podinfo,version=6.9.1 --output ./downloaded ``` You can also unpack the blob and it should contain the `podinfo` chart contents # transfer oci helm to another oci `ocm transfer component-version http://$REGISTRY//ocm.software/podinfo:6.9.1 $REGISTRY2 --copy-resources --upload-as ociArtifact` ##### Verification - [x] I have tested the changes locally by running `ocm` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added support for transferring Helm charts as part of component version transfers * Helm charts can now be converted to OCI artifacts during transfer * Support for both local Helm chart paths and remote Helm repositories as transfer sources * **Documentation** * Updated transfer command documentation with Helm chart transfer capabilities and examples <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Matthias Bruns <git@matthiasbruns.com>
#### What this PR does / why we need it Adds a how-to guide for transferring component versions containing Helm chart resources (`helm/v1` access type) between OCI registries using the OCM CLI. Covers the `--upload-as localBlob` and `--upload-as ociArtifact` options. Related to open-component-model/open-component-model#1846 #### Which issue(s) this PR is related to Related to open-component-model/open-component-model#1846 (comment) #### Type of content - [ ] Tutorial (`getting-started/` or `tutorials/`) - [x] How-to Guide (`how-to/`) - [ ] Explanation / Concept (`concepts/`) - [ ] Reference (`reference/`) - [ ] Other (infrastructure, config, fixes) #### Checklist - [x] I have read and followed the [Contributing Guide](https://github.com/open-component-model/ocm-website/blob/main/CONTRIBUTING.md) - [x] All commands/code snippets are tested and can be copy-pasted #### Test ```bash #!/bin/zsh set -e REGISTRY="${REGISTRY:-ghcr.io/matthiasbruns/ocm-tutorials}" COMPONENT="ocm.software/podinfo" VERSION="6.9.1" CTF_DIR=$(mktemp -d) echo "CTF archive: $CTF_DIR" pause() { echo "\n>>> Next: $1" echo "--- Press Enter to continue ---" read } # Step 1: Create constructor.yaml cat <<EOF > constructor.yaml components: - name: ${COMPONENT} version: ${VERSION} provider: name: ocm.software resources: - name: podinfo version: ${VERSION} type: helmChart access: type: helm/v1 helmRepository: https://stefanprodan.github.io/podinfo helmChart: podinfo-${VERSION}.tgz EOF echo "Created constructor.yaml" pause "Add component version to CTF archive" # Step 2: Add component version to CTF ./ocm add cv \ --repository "ctf::${CTF_DIR}" \ --constructor constructor.yaml \ --skip-reference-digest-processing echo "Component version added to CTF" pause "Transfer with --upload-as ociArtifact" # Step 3: Transfer as ociArtifact ./ocm transfer cv \ --copy-resources \ --upload-as ociArtifact \ "ctf::${CTF_DIR}//${COMPONENT}:${VERSION}" \ "${REGISTRY}" echo "Transfer complete (ociArtifact)" pause "Verify: get component version from registry" # Step 4: Verify component version ./ocm get cv "${REGISTRY}//${COMPONENT}:${VERSION}" pause "Verify: inspect component descriptor to find imageReference" # Step 5: Inspect component descriptor to find imageReference ./ocm get cv "${REGISTRY}//${COMPONENT}:${VERSION}" -o yaml pause "Verify: download the chart resource" # Step 6: Download the chart resource rm -rf ./downloaded ./ocm download resource \ "${REGISTRY}//${COMPONENT}:${VERSION}" \ --identity "name=podinfo,version=${VERSION}" \ --output ./downloaded echo "\nDownloaded to ./downloaded:" ls -la ./downloaded # Cleanup pause "Clean up CTF archive" rm -rf "${CTF_DIR}" echo "Removed ${CTF_DIR}" echo "\nDone." ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Added a detailed how-to for transferring Helm-chart-backed component versions between OCI registries via the CLI, including prerequisites, step-by-step workflow, verification, storage modes, cross-registry examples, and relevant commands. * **Chores** * Expanded repository configuration wordlist with two additional entries to recognize more terms. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Matthias Bruns <git@matthiasbruns.com>
#### What this PR does / why we need it Adds a how-to guide for transferring component versions containing Helm chart resources (`helm/v1` access type) between OCI registries using the OCM CLI. Covers the `--upload-as localBlob` and `--upload-as ociArtifact` options. Related to open-component-model/open-component-model#1846 #### Which issue(s) this PR is related to Related to open-component-model/open-component-model#1846 (comment) #### Type of content - [ ] Tutorial (`getting-started/` or `tutorials/`) - [x] How-to Guide (`how-to/`) - [ ] Explanation / Concept (`concepts/`) - [ ] Reference (`reference/`) - [ ] Other (infrastructure, config, fixes) #### Checklist - [x] I have read and followed the [Contributing Guide](https://github.com/open-component-model/ocm-website/blob/main/CONTRIBUTING.md) - [x] All commands/code snippets are tested and can be copy-pasted #### Test ```bash #!/bin/zsh set -e REGISTRY="${REGISTRY:-ghcr.io/matthiasbruns/ocm-tutorials}" COMPONENT="ocm.software/podinfo" VERSION="6.9.1" CTF_DIR=$(mktemp -d) echo "CTF archive: $CTF_DIR" pause() { echo "\n>>> Next: $1" echo "--- Press Enter to continue ---" read } # Step 1: Create constructor.yaml cat <<EOF > constructor.yaml components: - name: ${COMPONENT} version: ${VERSION} provider: name: ocm.software resources: - name: podinfo version: ${VERSION} type: helmChart access: type: helm/v1 helmRepository: https://stefanprodan.github.io/podinfo helmChart: podinfo-${VERSION}.tgz EOF echo "Created constructor.yaml" pause "Add component version to CTF archive" # Step 2: Add component version to CTF ./ocm add cv \ --repository "ctf::${CTF_DIR}" \ --constructor constructor.yaml \ --skip-reference-digest-processing echo "Component version added to CTF" pause "Transfer with --upload-as ociArtifact" # Step 3: Transfer as ociArtifact ./ocm transfer cv \ --copy-resources \ --upload-as ociArtifact \ "ctf::${CTF_DIR}//${COMPONENT}:${VERSION}" \ "${REGISTRY}" echo "Transfer complete (ociArtifact)" pause "Verify: get component version from registry" # Step 4: Verify component version ./ocm get cv "${REGISTRY}//${COMPONENT}:${VERSION}" pause "Verify: inspect component descriptor to find imageReference" # Step 5: Inspect component descriptor to find imageReference ./ocm get cv "${REGISTRY}//${COMPONENT}:${VERSION}" -o yaml pause "Verify: download the chart resource" # Step 6: Download the chart resource rm -rf ./downloaded ./ocm download resource \ "${REGISTRY}//${COMPONENT}:${VERSION}" \ --identity "name=podinfo,version=${VERSION}" \ --output ./downloaded echo "\nDownloaded to ./downloaded:" ls -la ./downloaded # Cleanup pause "Clean up CTF archive" rm -rf "${CTF_DIR}" echo "Removed ${CTF_DIR}" echo "\nDone." ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Added a detailed how-to for transferring Helm-chart-backed component versions between OCI registries via the CLI, including prerequisites, step-by-step workflow, verification, storage modes, cross-registry examples, and relevant commands. * **Chores** * Expanded repository configuration wordlist with two additional entries to recognize more terms. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Matthias Bruns <git@matthiasbruns.com> ef663ee
…en-component-model#1889) #### What this PR does / why we need it This PR solves the issue that happens when you want to pass nil values for optional pointers through the transformation steps. If in a spec a type is annotated by `omitempty` and resolved to `nil` one would suspect that `nil` is a valid value. The problem is, that our schemagen does not generate a rule that allows `null` to be passed in such cases. Since we have no way to omitting keys based on the fact that values exist or not, we have to circumvent that in the transformer graph. This PR does that. Here is an example of what we are trying to solve with this change: ```go convertToOCITransform := transformv1alpha1.GenericTransformation{ TransformationMeta: meta.TransformationMeta{ Type: ociv1alpha1.ConvertHelmToOCIV1alpha1, ID: convertResourceID, }, Spec: &runtime.Unstructured{Data: map[string]any{ "resource": fmt.Sprintf("${%s.output.resource}", getResourceID), "chartFile": fmt.Sprintf("${%s.output.chartFile}", getResourceID), "provFile": fmt.Sprintf("${%[1]s.output.?provFile}", getResourceID), }}, } ``` `provFile` might be null, if not provided by the helm chart. We can only run checks on the values with CEL here. If we pass `nil` to the `provFile` json field, the schemavalidation will fail. open-component-model#1846 #### Which issue(s) this PR fixes Contributes: open-component-model/ocm-project#883 #### Testing ##### How to test the changes Fully tested in https://github.com/open-component-model/open-component-model/pull/1846/changes#diff-85d55581ce565d132e8a8b027169c162a024c7e25f6df4c5cc2ddbbfb1294d2fR356 ##### Verification - [x] I have tested the changes locally by running `ocm` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Schema-driven resolution: resolvers can accept optional schemas (including spec sub-schemas) to detect optional fields and guide processing. * **Bug Fixes** * Optional-field removal: optional fields that resolve to nil are removed from resources (no null retained). * Runtime processing no longer treats optional nil values as errors. * **Tests** * Expanded coverage for nil handling, nested/referenced schemas, array items, and targeted deletions. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Matthias Bruns <git@matthiasbruns.com> Signed-off-by: Matthias Bruns <github@matthiasbruns.com> Co-authored-by: Fabian Burth <fabian.burth@sap.com> Signed-off-by: Gerald Morrison (SAP) <gerald.morrison@sap.com>
<!-- markdownlint-disable MD041 -->
#### What this PR does / why we need it
This pull-request adds the "download classic helm chart" to the
transformer api.
`This PR adds a bigger refactoring to the existing `input`and
`oci-image-creation` - the concepts stayed the same, just technical
changes.
`
**introduce helm access type from ocmv1**
It also adds the missing Helm access type to the new ocm.
```go
type Helm struct {
// +ocm:jsonschema-gen:enum=Helm/v1,helm/v1
// +ocm:jsonschema-gen:enum:deprecated=Helm,helm
Type runtime.Type `json:"type"`
// HelmRepository is the URL of the helm repository to load the chart from.
HelmRepository string `json:"helmRepository"`
// HelmChart if the name of the helm chart and its version separated by a colon.
HelmChart string `json:"helmChart"`
// Version can either be specified as part of the chart name or separately.
Version string `json:"version,omitempty"`
}
```
It is important to note what I dropped deprecated fields from the spec.
CA and Keyring related fields should be loaded from the credentials as
already done in the helm input.
**reuse blob download from helm input**
Additionally, this PR refactors the `download helm blob `from input to
generalize the logic and make it reusable for helm access as well.
**Reuse chart to OCI from help input**
I also generalized the chart->OCI logic from `input` and made it
reusable. It is now used in both `input` and the `convert transformer.`
**Refactor `CopyChartToOCILayout`**
I had to refactor `CopyChartToOCILayout` to be able to get the
`ociImageSpecV1.Descriptor`. The problem was that the whole impl was
async. Channels lead to blocking goroutines since `io.Pipe` was not done
writing at the point where I needed the descriptor. I had to create
```go
type Result struct {
*direct.Blob
desc chan descriptorOrError
}
type descriptorOrError struct {
Descriptor ociImageSpecV1.Descriptor
Err error
}
```
to delay the access to said data.
**GetHelmChart transformer implementation**
The get transformer does the following:
- validate input spec against new `v1alpha1.GetHelmChart`
- generate output paths for the spec output
- get credentials by requesting
`ResourceConsumerIdentityProvider.GetResourceCredentialConsumerIdentity`
which is implemented in the `HelmAccess`
- delegate download to generalized helm download from the input package
(now downloads)
- in theory, we could use this transformer also for oci helm download,
since the downloader supports the feature
- but we will use the native GetOCIArtifact from transformers (probably)
- resulting blobs (chart and optionally a prov) will be copied into
`*v1alpha1.File`
- the output contains
- the original `Resource`
- the chart file pointer with `v1alpha1.File`
- (optionally) the prov file pointer with `*v1alpha1.File`
**ConvertHelmChartToOCI transformer implementation**
The convertion transformer does the following:
- expect chart, prov and resources as input
- reuse the oci creation logic from input
- calculate the correct `ImageReference` for the `OCIImage` access
- push the artifact as localblob as an output back the the graph
**testing**
- The pr spawns a ~~repotest.NewTempServer()~~ mock http server pointing
to the shared helm `testdata/mychart-0.1.0.tgz` dir
- In the test we create Resources with several variations of the `Helm`
access
- Each test verifies that the files are correctly downloaded and equal
the original file
- provenance file download added
- **Open Question**: should we also verify during transform? Currently,
we only download the .prov file
- helm to oci tests and validated the created oci image and the prov
files
#### Which issue(s) this PR fixes
Contributes:
open-component-model/ocm-project#883
#### Testing
##### How to test the changes
See
open-component-model#1846
for testing
**make sure helm input did not break**
```bash
#!/bin/zsh
alias OCM='go run ../../main.go'
# Test helm input with local path
CHART_PATH="../../../bindings/go/helm/testdata/mychart"
# stat the chart path to ensure it exists
if [ ! -d "$CHART_PATH" ]; then
echo "Error: Chart path $CHART_PATH does not exist"
exit 1
fi
cat <<EOF > constructor-input-local.yaml
components:
- name: ocm.software/helm-input-local
version: 0.1.0
provider:
name: ocm.software
resources:
- name: mychart
version: 0.1.0
type: helmChart
input:
type: helm/v1
path: $CHART_PATH
EOF
# create dir relative to current directory to avoid issues with absolute paths in the constructor
INPUT_CTF_LOCAL="input-ctf-local-$(date +%s)"
mkdir $INPUT_CTF_LOCAL
echo "Using temporary directory for helm input (local): $INPUT_CTF_LOCAL"
OCM add cv --repository ctf::$INPUT_CTF_LOCAL --constructor constructor-input-local.yaml
echo "--- Local helm input test passed ---"
# Test helm input with remote repository (uses same podinfo chart)
cat <<EOF > constructor-input-remote.yaml
components:
- name: ocm.software/helm-input-remote
version: 6.9.1
provider:
name: ocm.software
resources:
- name: podinfo
version: 6.9.1
type: helmChart
input:
type: helm/v1
helmRepository: https://stefanprodan.github.io/podinfo/podinfo-6.9.1.tgz
EOF
INPUT_CTF_REMOTE="input-ctf-remote-$(date +%s)"
mkdir $INPUT_CTF_REMOTE
echo "Using temporary directory for helm input (remote): $INPUT_CTF_REMOTE"
OCM add cv --repository ctf::$INPUT_CTF_REMOTE --constructor constructor-input-remote.yaml
echo "--- Remote helm input test passed ---"
```
##### Verification
- [x] I have tested the changes locally by running `ocm`
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added Helm access spec, credential-aware identity resolution, and
options for TLS/credentials.
* Remote Helm chart retrieval (HTTP/OCI) with provenance support and
configurable buffering/output path handling.
* Conversion of Helm charts into OCI image layouts and transformers to
fetch charts and produce OCI artifacts.
* **Tests**
* Extensive unit and integration tests covering access parsing,
identity/error cases, downloads, provenance, OCI conversion, and
transformers.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Signed-off-by: Matthias Bruns <git@matthiasbruns.com>
Signed-off-by: Gerald Morrison (SAP) <gerald.morrison@sap.com>
<!-- markdownlint-disable MD041 --> #### What this PR does / why we need it This PR implements the transformers from open-component-model#1832 It creates a 3-step-transformation for the helm chart transformation-chain: ```mermaid flowchart TD A[Transfer Helm Chart] -->B(Get Helm Chart) B --> C{Download from Repo} C -->|Has Prov| D[Download from Repo] C --> E{Pack Helm LocalBlob} D --> E E -->|GetHelmOutput| F{Convert to OCI} F -->|ConverToHelm| G{Pack OCI from Helm} F -->|ConverToHelm with prov| G{Pack OCI from Helm} G -->|upload-as localBlob/default | I{Upload as localBlob} G -->|upload-as ociArtifact | H{Upload as OCI Image} ``` 1. get helm chart with `GetHelmChartV1alpha1` 2. convert helm to OCI with `ConvertHelmToOCIV1alpha1` 3. upload a. localBlobl: the oci artifact with `OCIAddLocalResourceV1alpha1` b. ociArtifact: the oci artifact with `AddOCIArtifactType` The [spec PR](open-component-model#1832) will be kept in sync with changed from here in the `bindings/go/helm` package. #### Which issue(s) this PR fixes Contributes: open-component-model/ocm-project#883 #### Testing ##### How to test the changes ```bash #!/bin/zsh alias OCM='go run ../../main.go' REGISTRY="ghcr.io/matthiasbruns/ocm-tutorials" REGISTRY2="ghcr.io/matthiasbruns/ocm-tutorials-2" pause() { echo "\n>>> Next: $1" echo "--- Press Enter to continue ---" read } # OCM --help # create constructor.yaml # https://stefanprodan.github.io/podinfo/podinfo-6.9.1.tgz cat <<EOF > constructor.yaml components: - name: ocm.software/podinfo version: 6.9.1 provider: name: ocm.software resources: - name: podinfo version: 6.9.1 type: helmChart access: type: helm/v1 helmRepository: https://stefanprodan.github.io/podinfo helmChart: podinfo-6.9.1.tgz EOF CTF_DIR=$(mktemp -d) echo "Using temporary directory: $CTF_DIR" pause "Add component version to CTF from constructor.yaml" # add cv command OCM add cv --repository ctf::$CTF_DIR --constructor constructor.yaml --skip-reference-digest-processing HELM_REF="ctf::$CTF_DIR//ocm.software/podinfo:6.9.1" pause "Create component version ($REGISTRY)" # transfer OCM transfer component-version $HELM_REF $REGISTRY --copy-resources pause "Transfer with --upload-as localBlob (OCI registry)" # transfer --upload-as localBlob OCM transfer component-version $HELM_REF $REGISTRY --copy-resources --upload-as localBlob pause "Transfer with --upload-as ociArtifact (OCI registry)" # transfer --upload-as ociArtifact OCM transfer component-version $HELM_REF $REGISTRY --copy-resources --upload-as ociArtifact pause "Download component descriptor with oras" # download with oras oras pull $REGISTRY/component-descriptors/ocm.software/podinfo:6.9.1 --output . pause "Download resource using OCM CLI" # rm downloaded if exists rm -rf downloaded # downloadCMD resource OCM download resource https://$REGISTRY//ocm.software/podinfo:6.9.1 --identity name=podinfo,version=6.9.1 --output ./downloaded ``` You can also unpack the blob and it should contain the `podinfo` chart contents # transfer oci helm to another oci `ocm transfer component-version http://$REGISTRY//ocm.software/podinfo:6.9.1 $REGISTRY2 --copy-resources --upload-as ociArtifact` ##### Verification - [x] I have tested the changes locally by running `ocm` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added support for transferring Helm charts as part of component version transfers * Helm charts can now be converted to OCI artifacts during transfer * Support for both local Helm chart paths and remote Helm repositories as transfer sources * **Documentation** * Updated transfer command documentation with Helm chart transfer capabilities and examples <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Matthias Bruns <git@matthiasbruns.com> Signed-off-by: Gerald Morrison (SAP) <gerald.morrison@sap.com>
What this PR does / why we need it
This PR implements the transformers from #1832
It creates a 3-step-transformation for the helm chart transformation-chain:
flowchart TD A[Transfer Helm Chart] -->B(Get Helm Chart) B --> C{Download from Repo} C -->|Has Prov| D[Download from Repo] C --> E{Pack Helm LocalBlob} D --> E E -->|GetHelmOutput| F{Convert to OCI} F -->|ConverToHelm| G{Pack OCI from Helm} F -->|ConverToHelm with prov| G{Pack OCI from Helm} G -->|upload-as localBlob/default | I{Upload as localBlob} G -->|upload-as ociArtifact | H{Upload as OCI Image}GetHelmChartV1alpha1ConvertHelmToOCIV1alpha1a. localBlobl: the oci artifact with
OCIAddLocalResourceV1alpha1b. ociArtifact: the oci artifact with
AddOCIArtifactTypeThe spec PR will be kept in sync with changed from here in the
bindings/go/helmpackage.Which issue(s) this PR fixes
Contributes: open-component-model/ocm-project#883
Testing
How to test the changes
You can also unpack the blob and it should contain the
podinfochart contentstransfer oci helm to another oci
ocm transfer component-version http://$REGISTRY//ocm.software/podinfo:6.9.1 $REGISTRY2 --copy-resources --upload-as ociArtifactVerification
ocmSummary by CodeRabbit
New Features
Documentation