One comment from the dev call: Will we support download by digest? (i.e. @sha256:...) cc @jlegrone

This doesn't fit into this proposal very well, but its an important topic. Perhaps we allow this on helm pull, and support this on helm push using a boolean flag such as --no-tag?

Could the "pull a digest" case work with helm pull --devel --version sha256:abc1234? --devel seems like an odd way to word it in this case, but that could bypass the semver check and fetch the content directly.

I'm curious what would be the need for a --no-tag. The use case to me sounds like someone who wants to push a chart to a repository without the ability to pull it down. Can you elaborate what would be the use case to publish a chart without creating a tag? I think docker push supports this use case, but that goes back to the discussion of Helm supporting arbitrary tags. I'm curious whether if we can support the helm pull case, that would be enough.

Since we just had the discussion about provenance files, it seems that helm pull by digest would be useful there, since we know the hash we expect (per the provenance file), so it's one less indirection to follow. Of course, you still verify the hash after pulling, and I assume for local caching and general use you need to get the version out of the archive anyway (or trust the provenance file).

I wouldn't use it with --version though, but just with the oci-registry syntax, oci://r.myreg.io/mycharts/pepper@sha256:111112. A sha256 is not a version. ^_^

helm push I'm less sure about, but perhaps when publishing with a provenance file, it's useful? I'm somewhat askance about that though, it feels like it's wandering down a path towards future confusion.

Could the "pull a digest" case work with helm pull --devel --version sha256:abc1234?

That would work. As @TBBle mentioned, I think better UX might be:

helm pull --devel oci://r.myreg.io/mycharts/pepper@sha256:111112

The UX gets weird, as "@" is used when trying to get a manifest by SHA (vs ":"). Ultimately I think people want to copy-paste a full reference, whether by tag or digest.

Can you elaborate what would be the use case to publish a chart without creating a tag?

Manifests can be uploaded by tag or digest. See https://github.com/opencontainers/distribution-spec/blob/6a6ee23bbab1e5d07b45a10547265bc5249c3c4d/spec.md#put-manifest

So one could upload by digest, then download by digest. The version could be determined by unpacking the config blob (which is Chart.yaml JSONified). But this is essentially the same as supporting arbitrary tags...

Access by digest is different from supporting arbitrary tags, because a given chart object can only appear at one digest (the digest of the blob), and one tag (the version contained in Chart.yaml), both determined by the content of the blob. Arbitrary tags allow putting charts in places that cannot be tied back to the content of the Chart.

Whether you apply a tag or not, an OCI Registry is a content-addressable store, and so pulling by digest is naturally supported, as long as we support it in the tools. In Helm's case, we aren't a content-addressable store locally, so we need to determine the correct address (version) from the content we pull, whether it's by tag or by digest.

Effectively, pushing by digest is pushing untagged, which makes sense for Docker images, but I don't think it's awesome for Helm. The only use-cases I can see is if you're either pushing multiple instances of a Chart with the same version and don't want the latest push to win (bad idea! SemVer has + and - for this) or trying to publish a chart that isn't discoverable through tag-listing (seems like the wrong solution?).

Side-note: From memory, the Docker syntax allows repository:tag@digest, but at least one of the non-Docker container runtimes (I thought it was podman, but can't find the relevant ticket just now) rejects that format because they didn't have a good approach to reconciling when the two were out-of-sync, i.e. when repository:tag does not have digest digest, the semantics were unclear.

Another place where pulling by digest might be handy is if you have the version->digest mapping from elsewhere, it saves having to mangle the version to account for + in the version, or anything else that SemVer allows but the OCI Distribution spec tag does not.

I just noticed that we use application/vnd.cncf.helm.config.v1+json, but the OCI documents application/vnd.cncf.helm.chart.config.v1+json (and application/vnd.cncf.helm.chart.layer.v1.tar) in their example list.

Hello, everyone! Nice to see Helm consider signing/verification.

Unfortunately, the current "provenance" (a misleading term, considering that it actually does not consider the whole software supply chain) model seems to broken. This is because it looks like attackers who control the OCI registry can replace PGP/GPG signatures with any other signature they like, since there is no trusted information about which GPG public keys to trust.

Securing software supply chains is not a trivial problem, and I strongly recommend that the Helm community looks at how CNAB—which can include Helm charts—solved this problem in a much more secure manner.

Cc @radu-matei @jlegrone @SantiagoTorres

Hi @trishankatdatadog. Thanks for your response.

As an FYI, a few Helm core maintainers were part of the early discussions in CNAB and are quite aware of the spec - @radu-matei and I are co-workers, and we worked together on the initial draft for the CNAB repository specification, which was loosely based on the OCI distribution protcol. cnabio/cnab-spec@4515571

Good call-out on the security issue regarding PGP/GPG signatures being replaced on the server side. However, I do believe you may be mis-reading Helm's security model on chart provenance.

To clarify, the current security model handles cases where the server has been compromised. Public keys distributed by the chart authors are distributed separately from the OCI/chart repository API workflow, and are added to the user's local keychain. Therefore, if an attacker were to replace the PGP signatures and the blobs in the OCI registry, they would not have access to the private key that was used to originally sign those packages, and therefore package verification would fail as the user's local keyring does not trust this new key.

To be totally clear here, we've had one security audit performed by an external third party who was sponsored by the CNCF. You may read up on the audit here (have a look under subsection 2: signing and verification code), but the auditing team did not find any glaring loopholes in this process. We have another security audit ongoing at this very moment, and we've asked them to look over this part of the codebase as well.

In any case, the plan is to port over the current provenance process, as it is a known, audited, working process for signing and verifying charts. We have discussed moving to an external key server like Notary v2 once that becomes available, which can be introduced at a later time. That might align closer to CNAB's goals with TUF and in-toto.

Let us know what you think.

See also https://helm.sh/docs/topics/provenance/#establishing-authority-and-authenticity.

Let us know what you think.

I apologize if I seemed hostile, and also for claiming that the PGP model of Helm provenance is "broken." It is not: I read the security audit [1], and also verified that the code does indeed check signatures against the local keyring by default.

While the PGP model has some security issues [2] that are better fixed with something like TUF and in-toto [3], I agree that this is a good start. Again, please accept my apologies.

[1] By the same people who audited Notary-v1 BTW IIRC.
[2] For example, if attackers are able to control one or more private keys whose public keys are listed in the local keyring, they may be able to cause a private key meant to be associated with one Helm chart to be trusted for another.
[3] Support for which may be available in Notary-v2?

@trishankatdatadog We understand issues with PGP and Notary v2 would be great to use. Notary v1 has some shortcomings. When Notary v2 is ready for use we will definitely evaluate it.

On the PGP side, I would suggest people take the verification public key and pass that in using the --keyring flag. That way you can control the keys used for verification.

Notary v1 has some shortcomings.

Do you have a reference of which ones in particular got in the way? a usecase as simple as gpg signing seems like notary should have had no issues to provide.

@bacongobbler yes, can do!

FYI - this proposal has been cleaned up and converted to the new HIP format.
cc @bacongobbler @mattfarina

The first PR addressing this proposal is here: helm/helm#8843. It implements helm pull for use with OCI registries

@bacongobbler - thanks for detailed review. I've just made a bunch of updates to address your comments, and also modified to HIP 0006. Please let me know if any other changes necessary.

I don't see it mentioned in this PR so either it's obvious to those involved, confusion on my part, or it's missing: how will helm search an OCI registry for charts, if at all? Based on what I've read I assume it won't and that the plan to rely on an external hub? (artifact hub and the like).

I saw that the /v2/_catalog endpoint was removed from OCI, which may have helped with this (as a global end point across all repositories) although I'm not even sure about that. So I'm just keen for some clarification on whether the helm CLI will rely on helm search hub and not have a concept of an OCI registry, only repositories, or whether there is some future plan to enable adding a registry and searching across repositories within that registry?

Hi @absoludity,
There are a number of features we still need to add to OCI Based registries. As you noted, removing the inconsistently implemented _catalog API was part of clearing a path.
We have a number of things we're working on. Some more active than others:

• Artifact signing, through the Notary v2 effort
• Search & Eventing APIS
• meta-data services
• dependency management using some reverse lookup apis we're building for Notary v2

I've been trying to catch up on the latest Helm OCI support, but this happened to catch my eye, so figured I'd drop a quick note.

Steve

Thanks Steve, that certainly fills in some gaps on what's planned (or at least, hoped for) for the future with regards to search support in tho OCI distribution spec (eg. the examples included at tool-specific searches). But support for such searches in the OCI distribution spec appear to be further away even than a stable OCI support in helm. That helps clarify my question a little: Will the OCI support in helm only be considered stable when such searches are possible with an OCI registry, or will it be considered stable when you can install OCI-based charts from the artifact hub? (I realise you're not in a position to answer that, but I do think it's relevant for this PR)

I don't think we can hold back this feature for future OCI Distribution features, so we should be able to get it stabilised when Helm can pull and push charts by name, and (I guess) once tag-listing (where a Registry supports it) is used for fetching all known versions of a chart (single repository), so SemVer matching works.

That will be enough for charts registered on Helm Hub, and for charts referenced as dependencies of other charts in Charts.yaml.

Any more-generic searching (across multiple repositories, partial match, etc) should be able to be added later, and would not change how the existing pull/push/SemVer works.

Completely agree that folks should prioritize what's available today, vs. what we're hoping to get done tomorrow. Having a prioritized execution helps make continued progress.
The tag listing API is part of the spec, and is either supported across all registries or will be as registry operators align on the OCI conformance tests.

The thing to be aware of is this:
The tag listing API doesn't currently support a mediaType filter. Meaning, when you get a list of tags back, you don't know if these are helm charts, docker images, singularities, or someone's cat pictures. So, you will have to check the mediaType for each tag.
This is another gap we hope to address. Luckily, there's a general best practice to put like things in a repo, and use different repos for different things. So, this shouldn't be a problem.

I would also endorse the client/server request model for a list of versions of content. The index.yaml for the list of charts in a repo is great when it's small and a focused set of charts. When customers have large numbers of charts, or have automation that continually pushes new versions of the chart for each deployment, the list gets unmanageable. This has been one of the biggest scalability issues we've seen with helm repo support, as chart repo solutions must maintain and return a single index.yaml file, rather than have the client ask specific questions for what it's interested in. Imagine asking google for an index.yaml of everything you'd think about searching for?

But, we also realize the registry must provide better capabilities than tag listing, which gets us back to the search scenario.

But, what do we need now?
The thing that would be great to understand is how public charts are used, vs. private copies of charts?
When customers import public charts for usage in their environment, and when customers build their own charts for their internal deployment, do they really need a search API? Or, do they know the chart they need, as it's got a version associated with their build-id, and simply need a secure, reliable, scalable way to push/pull charts in their secured production environment?

Just suggesting there's a subset of capabilities internal teams need, and they need it lifted from the experimental flag, and would be super happy. We continue to hear the tension that large customers can't use features "in preview". OCI Artifacts is GA supported in ACR and other cloud providers. We just need to lift a subset of helm features so projects like Terraform can support it.

So, what is the minimum scoped features that could be added, with more to come later?

Search was intentionally left out of this HIP. I think it's safer to limit the scope of this for now, and add more features later. Otherwise we are perpetually caught in this loop of "what about X".

For now, specific chart versions must be specified, search and automatic semver resolution will not be supported.

Custom tags were ruled out and added to the "Rejected Ideas" section. But last I recall when we discussed this in September, the system was designed to handle automatic semver resolution. That's why we decided to implement an OCI Getter. The tag listing API allows us to fetch all available tags, which will allow us to determine which is "latest" similar to the chart repository API. We can then check the mediaType to ensure it's a Helm chart, if not fall over to the next latest version, etc.

We even ruled out the idea that docker images and Helm charts are to be stored in the same "repository", so I'm not sure why this is being brought up again.

What changed between September and today to rule out automatic semver lookup support?

@bacongobbler resolved.

Re: tag listing - this was part of the original discussion/PR but never put into this HIP. There is still some experimentation to do here, coupled with the fact that docker/distribution (backbone of most registries) still does not support pagination. See distribution/distribution#3143. I think the search/semver aspect can come in a subsequent HIP.

Search was intentionally left out of this HIP. I think it's safer to limit the scope of this for now, and add more features later. Otherwise we are perpetually caught in this loop of "what about X".

FWIW, I wasn't suggesting search should be included, just that IMO the position should be clear in this HIP, stating that any search or browser functionality is not supported by this proposal and so the helm client will be able to install only an explicitly provided chart and tag from a specific repository, if that's the case. And possibly what people's other options are for finding the explicit chart and tag (ie. mentioned above, internal systems you might know the exact tag, other public systems might want to populate some external searchable directory, such as a hub - just guessing with my own thoughts for the last point).

We even ruled out the idea that docker images and Helm charts are to be stored in the same "repository", so I'm not sure why this is being brought up again.

It's up to the user to segment the content they push to a repo. Just calling out that it's not something helm can limit, and it just needs to parse the results. Great to hear the logic is built-in.

Really great to see the progress happening. And, we're looking for any asks needed in the underlying distribution spec/oci artifacts spec that we should add to enable additional functionality.