feat(acme): Add default feature gate for ACME HTTP01 Ingress pathType Exact#7795
Conversation
|
@sspreitzer: The label(s) DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
Hi @sspreitzer. Thanks for your PR. I'm waiting for a cert-manager member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
/kind feature |
903746d to
ba9be37
Compare
ba9be37 to
efecd30
Compare
c31df7f to
7a09b82
Compare
|
@wallrj Please have a look. There could be some parts missing, eg. tests and e2e tests. If they should be part of this PR. |
|
/ok-to-test @sspreitzer I expect the E2E tests will fail since #7792. I'll figure out how to work around that. |
|
I think what we need to do is --set config.strict-path-validation=false here: cert-manager/make/e2e-setup.mk Lines 369 to 388 in 12cad36 And then, ideally, figure out how we can set that to true if the new feature gate is disabled. cert-manager/make/e2e-setup.mk Line 228 in 12cad36
If you want to try running some of the E2E tests rather than the whole suite, you can do this: If you want to test the updated code on a real cluster (with cilium installed), you can publish the images to a public registry and deploy to the current cluster in your kubeconfig, as follows:
|
Signed-off-by: Sascha Spreitzer <sascha@spreitzer.ch>
ec4ceca to
965c1b4
Compare
|
/test pull-cert-manager-master-e2e-v1-33-upgrade |
| --set controller.service.clusterIP=${SERVICE_IP_PREFIX}.15 \ | ||
| --set controller.service.type=ClusterIP \ | ||
| --set controller.config.no-tls-redirect-locations= \ | ||
| --set-string controller.config.strict-validate-path-type=false \ |
|
@wallrj I believe we are good to go. Regarding the end to end testing idea of testing
Regarding adding cilium ingress as an end-to-test scenario, I would prefer creating an other PR for that implementation. Insofar, I believe we are ready to merge this contribution. Is the release note sufficient or shall we add some more context? |
|
Due to this change, NGINX Ingress Controller v1.12.3 admission validation web hook is throwing the below error which is preventing the ingress creation: admission webhook "validate.nginx.ingress.kubernetes.io" denied the request: ingress contains invalid paths: path /.well-known/acme-challenge/TOKEN cannot be used with pathType Exact |
|
@sspreitzer thank you for sharing. It is clear that this bug needs to be resolved by ingress Nginx maintainers. |
There was a problem hiding this comment.
Pull Request Overview
This PR adds a feature gate (ACMEHTTP01IngressPathTypeExact) to control whether ACME HTTP01 Ingress challenge solvers default to pathType Exact or ImplementationSpecific. It introduces new unit tests in the ingress_test.go file, updates ingress creation logic in ingress.go, tweaks e2e test configuration in make/e2e-setup.mk, and adds the new feature flag in the internal/controller/feature/features.go file.
Reviewed Changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| pkg/issuer/acme/http/ingress_test.go | Added tests to validate Ingress pathType behavior based on the feature gate |
| pkg/issuer/acme/http/ingress.go | Updated Ingress creation logic to select the correct pathType |
| make/e2e-setup.mk | Added configuration to disable strict path type validation for e2e tests |
| internal/controller/feature/features.go | Added and documented the new ACMEHTTP01IngressPathTypeExact feature flag |
Comments suppressed due to low confidence (1)
internal/controller/feature/features.go:181
- There appears to be an extra backtick in the documentation comment. Consider changing
pathType`` topathType` for clarity and consistency.
// `ACMEHTTP01IngressPathTypeExact` changes the default `pathType`` for ACME
wallrj
left a comment
There was a problem hiding this comment.
Thanks @sspreitzer
This looks good enough to me.
Let's merge it and tomorrow
I didn't mean that you should add automated cilium tests. I just meant that you should do a manual E2E test to verity that cert-manager HTTP01 can now be used with the cilium ingress.
Similarly, I'll go through the https://cert-manager.io/docs/tutorials/acme/nginx-ingress/ tutorial to see for myself that disabling the feature gate allows me to use ingress-nginx 4.12....we'll need to add a note to that tutorial.
There's a tiny typo in the feature gate docs which we can address in an another PR.
And I want to update this section in the helm chart before we release 1.18.1, with this and another new feature gate.
cert-manager/deploy/charts/cert-manager/values.yaml
Lines 237 to 253 in 12cad36
And I'll need to update the cert-manager documentation with this new featuregate.
/approve
/lgtm
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: wallrj The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
/cherry-pick release-1.18 |
|
@wallrj: #7795 failed to apply on top of branch "release-1.18": DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
/cherry-pick release-1.18 |
|
@wallrj: new pull request created: #7810 DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
TestingI temporarily re-enabled the ingress-nginx strict-validate-path feature Observed the E2E tests fail:
Verified that I can disable the acmesolver:
image:
repository: cert-manager-acmesolver-amd64
tag: v1.18.0-17-g379f43e3de2237
cainjector:
extraArgs:
- --feature-gates=ServerSideApply=true
image:
repository: cert-manager-cainjector-amd64
tag: v1.18.0-17-g379f43e3de2237
crds:
enabled: true
dns01RecursiveNameservers: 10.0.0.16:53
dns01RecursiveNameserversOnly: true
extraArgs:
- --kube-api-qps=9000
- --kube-api-burst=9000
- --concurrent-workers=200
- --enable-gateway-api
featureGates: ExperimentalCertificateSigningRequestControllers=true,ExperimentalGatewayAPISupport=true,ServerSideApply=true,LiteralCertificateSubject=true,UseCertificateRequestBasicConstraints=true,NameConstraints=true,OtherNames=true,ACMEHTTP01IngressPathTypeExact=false
image:
repository: cert-manager-controller-amd64
tag: v1.18.0-17-g379f43e3de2237
startupapicheck:
image:
repository: cert-manager-startupapicheck-amd64
tag: v1.18.0-17-g379f43e3de2237
webhook:
featureGates: LiteralCertificateSubject=true,NameConstraints=true,OtherNames=true
image:
repository: cert-manager-webhook-amd64
tag: v1.18.0-17-g379f43e3de2237
Now the E2E tests pass: |
|
Also checked that I can disable the feature via the config file |
| OtherNames: {Default: false, PreRelease: featuregate.Alpha}, | ||
| UseDomainQualifiedFinalizer: {Default: true, PreRelease: featuregate.GA}, | ||
| DefaultPrivateKeyRotationPolicyAlways: {Default: true, PreRelease: featuregate.Beta}, | ||
| ACMEHTTP01IngressPathTypeExact: {Default: true, PreRelease: featuregate.GA}, |
There was a problem hiding this comment.
This should be featuregate.Beta, like DefaultPrivateKeyRotationPolicyAlways above it.
It'll cause it to be turned on by default, but won't show a warning in the logs if you disable it.
There was a problem hiding this comment.
This PR contains the following updates: | Package | Update | Change | |---|---|---| | [cert-manager](https://cert-manager.io) ([source](https://github.com/cert-manager/cert-manager)) | minor | `v1.18.2` -> `v1.19.0` | --- ### Release Notes <details> <summary>cert-manager/cert-manager (cert-manager)</summary> ### [`v1.19.0`](https://github.com/cert-manager/cert-manager/releases/tag/v1.19.0) [Compare Source](cert-manager/cert-manager@v1.18.2...v1.19.0) cert-manager is the easiest way to automatically manage certificates in Kubernetes and OpenShift clusters. This release focuses on expanding platform compatibility, improving deployment flexibility, enhancing observability, and addressing key reliability issues. > 📖 Read the full release notes at cert-manager.io: <https://cert-manager.io/docs/releases/release-notes/release-notes-1.19> Changes since `v1.18.0`: #### Feature - Add IPv6 rules to the default network policy ([#​7726](cert-manager/cert-manager#7726), [@​jcpunk](https://github.com/jcpunk)) - Add `global.nodeSelector` to helm chart to allow for a single `nodeSelector` to be set across all services. ([#​7818](cert-manager/cert-manager#7818), [@​StingRayZA](https://github.com/StingRayZA)) - Add a feature gate to default to Ingress `pathType` `Exact` in ACME HTTP01 Ingress challenge solvers. ([#​7795](cert-manager/cert-manager#7795), [@​sspreitzer](https://github.com/sspreitzer)) - Add generated `applyconfigurations` allowing clients to make type-safe server-side apply requests for cert-manager resources. ([#​7866](cert-manager/cert-manager#7866), [@​erikgb](https://github.com/erikgb)) - Added API defaults to issuer references group (cert-manager.io) and kind (Issuer). ([#​7414](cert-manager/cert-manager#7414), [@​erikgb](https://github.com/erikgb)) - Added `certmanager_certificate_challenge_status` Prometheus metric. ([#​7736](cert-manager/cert-manager#7736), [@​hjoshi123](https://github.com/hjoshi123)) - Added `protocol` field for `rfc2136` DNS01 provider ([#​7881](cert-manager/cert-manager#7881), [@​hjoshi123](https://github.com/hjoshi123)) - Added experimental field `hostUsers` flag to all pods. Not set by default. ([#​7973](cert-manager/cert-manager#7973), [@​hjoshi123](https://github.com/hjoshi123)) - Support configurable resource requests and limits for ACME HTTP01 solver pods through ClusterIssuer and Issuer specifications, allowing granular resource management that overrides global `--acme-http01-solver-resource-*` settings. ([#​7972](cert-manager/cert-manager#7972), [@​lunarwhite](https://github.com/lunarwhite)) - The `CAInjectorMerging` feature has been promoted to BETA and is now enabled by default ([#​8017](cert-manager/cert-manager#8017), [@​ThatsMrTalbot](https://github.com/ThatsMrTalbot)) - The controller, webhook and ca-injector now log their version and git commit on startup for easier debugging and support. ([#​8072](cert-manager/cert-manager#8072), [@​prasad89](https://github.com/prasad89)) - Updated `certificate` metrics to the collector approach. ([#​7856](cert-manager/cert-manager#7856), [@​hjoshi123](https://github.com/hjoshi123)) #### Bug or Regression - ACME: Increased challenge authorization timeout to 2 minutes to fix `error waiting for authorization` ([#​7796](cert-manager/cert-manager#7796), [@​hjoshi123](https://github.com/hjoshi123)) - BUGFIX: permitted URI domains were incorrectly used to set the excluded URI domains in the CSR's name constraints ([#​7816](cert-manager/cert-manager#7816), [@​kinolaev](https://github.com/kinolaev)) - Enforced ACME HTTP-01 solver validation to properly reject configurations when multiple ingress options (`class`, `ingressClassName`, `name`) are specified simultaneously ([#​8021](cert-manager/cert-manager#8021), [@​lunarwhite](https://github.com/lunarwhite)) - Increase maximum sizes of PEM certificates and chains which can be parsed in cert-manager, to handle leaf certificates with large numbers of DNS names or other identities ([#​7961](cert-manager/cert-manager#7961), [@​SgtCoDFish](https://github.com/SgtCoDFish)) - Reverted adding the `global.rbac.disableHTTPChallengesRole` Helm option. ([#​7836](cert-manager/cert-manager#7836), [@​inteon](https://github.com/inteon)) - This change removes the `path` label of core ACME client metrics and will require users to update their monitoring dashboards and alerting rules if using those metrics. ([#​8109](cert-manager/cert-manager#8109), [@​mladen-rusev-cyberark](https://github.com/mladen-rusev-cyberark)) - Use the latest version of `ingress-nginx` in E2E tests to ensure compatibility ([#​7792](cert-manager/cert-manager#7792), [@​wallrj](https://github.com/wallrj)) #### Other (Cleanup or Flake) - Helm: Fix naming template of `tokenrequest` RoleBinding resource to improve consistency ([#​7761](cert-manager/cert-manager#7761), [@​lunarwhite](https://github.com/lunarwhite)) - Improve error messages when certificates, CRLs or private keys fail admission due to malformed or missing PEM data ([#​7928](cert-manager/cert-manager#7928), [@​SgtCoDFish](https://github.com/SgtCoDFish)) - Major upgrade of Akamai SDK. NOTE: The new version has not been fully tested end-to-end due to the lack of cloud infrastructure. ([#​8003](cert-manager/cert-manager#8003), [@​hjoshi123](https://github.com/hjoshi123)) - Update kind images to include the Kubernetes 1.33 node image ([#​7786](cert-manager/cert-manager#7786), [@​wallrj](https://github.com/wallrj)) - Use `maps.Copy` for cleaner map handling ([#​8092](cert-manager/cert-manager#8092), [@​quantpoet](https://github.com/quantpoet)) - Vault: Migrate Vault E2E add-on tests from deprecated `vault-client-go` to the new `vault/api` client. ([#​8059](cert-manager/cert-manager#8059), [@​armagankaratosun](https://github.com/armagankaratosun)) </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR is behind base branch, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0MS4xMzUuNCIsInVwZGF0ZWRJblZlciI6IjQxLjEzNS40IiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJjaGFydCJdfQ==--> Reviewed-on: https://gitea.alexlebens.dev/alexlebens/infrastructure/pulls/1711 Co-authored-by: Renovate Bot <renovate-bot@alexlebens.net> Co-committed-by: Renovate Bot <renovate-bot@alexlebens.net>
Pull Request Motivation
With the release of cert-manager version
1.18.1the pathType changes fromImplementationSpecifictoExact. This PR implements the feature gateACMEHTTP01IngressPathTypeExact.ACMEHTTP01IngressPathTypeExactchanges the defaultpathTypefor ACME HTTP01 Ingress based challenges toExact. This security feature ensures that the challenge path (which is an exact path) is not misinterpreted as a regular expression or some other Ingress specific (ImplementationSpecific) parsing. This allows HTTP01 challenges to be solved when using standards compliant Ingress controllers such as Cilium. The old defaultImplementationSpecificcan be reinstated by disabling this feature gate. You may need to disable the feature for compatibility withingress-nginx. See version 1.18 release notes.Kind
/kind feature
Release Note