Summary

This PR provides a make-based flow for building all cert-manager artifacts, with the intention of replacing bazel for deployments, with minimal changes. This isn't intended to fully remove bazel today, but instead to remove it for the development edit-build-run workflow and for cert-manager deployments, which are currently handled by cmrel.

Help for Reviewers

If you follow both "Comparison Against Old Releases" and "Building Everything + Signing" you should be able to easily compare an old bazel release and a "new" make release.

Comparison Against Old Releases

The goal here is that bin/release should contain everything needed for cmrel publish to be able to publish a release based on its contents, with minimal changes.

To compare bin/release with a previous release, the following may be helpful to check out an existing cert-manager release locally:

$ mkdir -p /tmp/maker/bzl
$ gsutil -m cp -r gs://cert-manager-release/stage/gcb/release/v1.6.0-beta.0-49914a057b39c887be0974c4657c095bd7724bc7 /tmp/maker/bzl/
$ ls /tmp/maker/bzl/v1.6.0-beta.0-49914a057b39c887be0974c4657c095bd7724bc7
...

Building Everything + Signing

Creating a full staged release includes signing the helm chart, which requires a GCP KMS key. I have a test one in my project and I can allowlist you to use it if you ask. Then, you should be able to run the following:

$ export CMREL_KEY="projects/jetstack-ashley-davis/locations/global/keyRings/testring/cryptoKeys/sha512key/cryptoKeyVersions/1"
# note you need to be allowlisted to sign using this test key! ask if you want that, or else you can create your own.
$ make -j8 -f make/Makefile staged-release

High Level Structure

The intention of this PR is that everything we need to stage a deployment can be built with exactly one command, including generating signatures. This build step should also be trivially run in parallel using make's -jN option.

$ make -j8 -f make/Makefile <target>
# lots of output...

The above command, starting from a clean repo with no bin/ dir but with a populated go build and module cache takes under 2 minutes to completely build and sign every artifact for a release when run on a developer laptop. The 2 minutes includes downloading and installing tools as dependencies (helm, cosign, etc).

Artifacts

Everything that's built is placed under bin/. Crucially, using make -f make/Makefile staged-release will create bin/release containing the same artifacts which bazel built, in the same format. In other words, bin/release should contain everything required to stage a complete release and side effect of this is that it should obsolete most of the logic in cmrel stage and allow a lot of cleanup there.

After this PR merges, cmrel stage will essentially become "clone repo, run make staged-release, upload to a bucket".

Notes / Differences from Bazel Artifacts

Lines prefixed with a ⚠️ denote functional changes which could affect others. Lines without a warning sign should be non-functional changes (e.g. a comment change)

Versioning

• If HEAD is not a tag, a version like v1.6.0-67-gf4e2b2c3c will be generated using git describe --tags. This should uniquely identify practically all development builds.
  • The format is v<last-tag>-<N>-<commit-sha> where N is the number of commits since <last-tag>
  • This format is directly from git.
• If HEAD is a tag, the tag will be used for the version.

Windows Client Binaries

• We add a zip file for windows client binaries since tar.gz is less common on Windows
• ⚠️ All windows client binaries now have a ".exe" extension

Helm Charts

• We now write helm chart filenames using the correct version in the filename, rather than just naming the chart cert-manager.tgz. This more closely matches how helm expects chart filenames.
• ⚠️ Helm charts are now packaged into cert-manager-manifests.tar.gz with their version in their filename
• The helm chart signing being moved into the cert-manager repo means that it's possible to embed the artifacthub signKey annotation directly into the actual helm chart, which should improve the ArtifactHub UX. Previously this annotation was only added in our charts repo's index.yaml where it seemed to largely be ignored.

Signing

• The make flow has support for signing helm charts (using cmrel as an installed standalone tool).
• Includes a commented-out template for generating and signing a SHA256SUMS file automatically generated by the build process.
• ⚠️ While not a breaking change, it's worth highlighting that the signing key information is currently hardcoded in a single file - deploy/charts/cert-manager/signkey_annotation.txt. This can obviously be improved later.

Containers

• Containers are built with docker by default, but this is configurable through the CTR environment variable and both podman and nerdctl can be used as drop-in replacements.
• ⚠️ When we build containers, the tag looks like cert-manager-controller-<arch>:<version> and doesn't include the registry (quay.io/...). This is to avoid creating a race condition if building containers in parallel. This will require a small change to cmrel publish, but seems worth to avoid this killing the ability to build containers in parallel.

Base Images

• Adds a script for automatically updating base images, both distroless/base and distroless/static across all relevant arches.
• All containers now use a distroless base image whose tag matches their arch (e.g. a s390x base for the s390x cert-manager containers). With bazel all containers used amd64, which never caused a problem with the distroless static images but would break if anyone tried to use dynamic or a debug distroless image on non-amd64 platforms.
• Using distroless/base or distroless/static is now a configurable through a single env var (BASE_IMAGE_TYPE)

Binary Locations

Background: Bazel adds binaries to containers at a long unusual path, along with a symlink:

# taken from a layer in one of our webhook docker images
$ tree .
app
└── cmd
    └── webhook
        ├── webhook -> /app/cmd/webhook/webhook.runfiles/com_github_jetstack_cert_manager/cmd/webhook/webhook_/webhook
        └── webhook.runfiles
            └── com_github_jetstack_cert_manager
                ├── cmd
                │   └── webhook
                │       └── webhook_
                │           └── webhook
                └── external -> /app/cmd/webhook/webhook.runfiles

In other words, the actual binary is /app/cmd/webhook/webhook.runfiles/com_github_jetstack_cert_manager/cmd/webhook/webhook_/webhook and /app/cmd/webhook/webhook exists as a symlink to that binary.

• ⚠️ The make version only adds the actual binary, this time at /app/cmd/webhook/webhook.
• ⚠️ The make version doesn't add the external symlink seen in the tree above, either.

Licenses in Containers

• ⚠️ The LICENSES file changed into a short string pointing people at our repo if they want to see licenses.
  • This is because LICENSES is 1.4MB on its own, which is absurdly wasteful for a file which nobody ever looks at or cares about.
  • This change alone shaves maybe ~3%-5% off the size of the cert-manager containers, basically for free.

Static manifests: cert-manager.yaml / cert-manager.crds.yaml

• Some comments are different.
• The static manifest now has the namespace resource at the top of the file rather than after the CRDs

Release Notes

/kind feature

Added a makefile based build workflow which doesn't depend on bazel