feat: Add client verification for webhook server#8010
Conversation
|
Hi @shubham14bajpai. Thanks for your PR. I'm waiting for a cert-manager member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
c10380b to
e6782d7
Compare
|
👋 Thanks for updating, I've added a few more comments. I'll also add the ok-to-test flag so you can use CI to run tests /ok-to-test |
8c1dc36 to
4fcffce
Compare
9905b20 to
5ea366e
Compare
Hey, sorry for the delay. I was finally able to fix the test failures. Please take another look. |
There was a problem hiding this comment.
Pull Request Overview
This PR adds client verification capability to the cert-manager webhook server to authenticate API server requests using client certificates. This addresses security concerns when running cert-manager on hostNetwork where webhook endpoints are exposed to anyone with node IP access.
- Adds client certificate verification to webhook server using configurable CA and client CN validation
- Introduces new configuration options for enabling client verification, CA path, and client certificate CN
- Updates Helm chart with new configuration values and deployment template changes
Reviewed Changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| pkg/webhook/server/server.go | Core implementation of client certificate verification logic |
| pkg/webhook/options/options.go | CLI flag definitions for client verification options |
| pkg/apis/config/webhook/v1alpha1/types.go | Public API type definitions for client verification config |
| internal/webhook/webhook.go | Integration of client verification options into webhook server construction |
| internal/apis/config/webhook/v1alpha1/zz_generated.conversion.go | Generated conversion functions for new config fields |
| internal/apis/config/webhook/types.go | Internal API type definitions for client verification config |
| deploy/charts/cert-manager/values.yaml | Default Helm chart values for client verification |
| deploy/charts/cert-manager/values.schema.json | JSON schema definitions for new Helm values |
| deploy/charts/cert-manager/templates/webhook-deployment.yaml | Template updates to pass client verification flags and fix indentation |
| deploy/charts/cert-manager/README.template.md | Documentation for new configuration options |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
|
Hiya! I'll take a look this week as soon as I have time |
|
👋 Apologies for the delay - I've had a look at the code and it looks good. Given this area is not covered by tests I want to spin this up locally and do some manual testing before I approve. |
Hey I did some manual testing and here is the log 🙂 Created a webhook client Configured the kube-apiserver and cert-manager webhook Tried curling the endpoints with & without the client certificate Tested the certificate generation |
There was a problem hiding this comment.
Sorry about the delay - my time has been fairly limited so I have not had a chance to dive into testing this.
I did get this setup fully locally and proved it works, I have the setup and tests I ran at the bottom.
I have left a comment around the CN validation as the final thing I would like to address
Setup
-
I created a deployment of your branch using the
make e2e-setupcommand, this deployed your code via the helm chart into a new kind cluster -
I exec'd into the kind control-plane container and made the following changes:
- Created an admission config file / kubeconfig to tell the APIServer to present a certificate
apiVersion: apiserver.config.k8s.io/v1 kind: AdmissionConfiguration plugins: - name: ValidatingAdmissionWebhook configuration: apiVersion: apiserver.config.k8s.io/v1 kind: WebhookAdmissionConfiguration kubeConfigFile: /etc/kubernetes/admission/webhook-authn.kubeconfig - name: MutatingAdmissionWebhook configuration: apiVersion: apiserver.config.k8s.io/v1 kind: WebhookAdmissionConfiguration kubeConfigFile: /etc/kubernetes/admission/webhook-authn.kubeconfig
apiVersion: v1 kind: Config users: - name: '*' user: client-certificate: /etc/kubernetes/pki/apiserver-kubelet-client.crt client-key: /etc/kubernetes/pki/apiserver-kubelet-client.key
-
Updated the static manifest of the apiserver pod to mount
/etc/kubernetes/admission/and have the flag--admission-control-config-file=/etc/kubernetes/admission/admission-config.yaml -
I then grabbed copies of
/etc/kubernetes/pki/apiserver-kubelet-client.crtand/etc/kubernetes/pki/ca.crtand exited the container
-
I created a secret in the cert-manager namespace containing the
ca.crt -
I updated the helm values to contain:
webhook:
apiserverClientCertSubject: kube-apiserver-kubelet-client
clientCAFile: /apiserver-webhook-client-ca/ca.crt
enableClientVerification: true
volumeMounts:
- mountPath: /apiserver-webhook-client-ca
name: apiserver-webhook-client-ca
volumes:
- name: apiserver-webhook-client-ca
secret:
secretName: apiserver-webhook-client-caTests
My first test was running some dry-run=server operations:
cat <<EOF | kubectl apply -f - --dry-run=server
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: webhook-test-cert
namespace: default
spec:
secretName: webhook-test-tls
duration: 2160h # 90 days
renewBefore: 360h # 15 days
subject:
organizations:
- example-org
commonName: webhook-test.example.com
dnsNames:
- webhook-test.example.com
- alt-webhook-test.example.com
issuerRef:
name: test-issuer
kind: Issuer
EOF
Warning: spec.privateKey.rotationPolicy: In cert-manager >= v1.18.0, the default value changed from `Never` to `Always`.
certificate.cert-manager.io/webhook-test-cert created (server dry run)This hit the webhook correctly as evidenced by the warning the webhook emits.
I then tried to curl the webhook directly without a cert and got an error as expected:
root@kind-control-plane:/# curl -k https://10.0.243.85
curl: (56) OpenSSL SSL_read: OpenSSL/3.0.16: error:0A00045C:SSL routines::tlsv13 alert certificate required, errno 0I tried curling with a valid cert, which worked:
root@kind-control-plane:/# curl -k https://10.0.243.85 --cert /etc/kubernetes/pki/apiserver-kubelet-client.crt --key /etc/kubernetes/pki/apiserver-kubelet-client.key
404 page not foundI then tried curling with an invalid cert, which errored as expected:
root@kind-control-plane:/# curl -k https://10.0.243.85 --cert /etc/kubernetes/pki/apiserver-etcd-client.crt --key /etc/kubernetes/pki/apiserver-etcd-client.key
curl: (56) OpenSSL SSL_read: OpenSSL/3.0.16: error:0A000418:SSL routines::tlsv1 alert unknown ca, errno 000a7624 to
bda220a
Compare
99d9556 to
e50a258
Compare
ThatsMrTalbot
left a comment
There was a problem hiding this comment.
👋 I have re-completed my testing after these changes and it all looks great! I have left one minor documentation comment just to clarify the apiserverClientCertSubjects Helm value, since thats just a comment update I can merge as soon as thats fixed.
I have included my testing notes below for documentation:
Setup
The following certificates were created for this test:
- ca.crt - A self-signed CA certificate
- valid-cn.crt - A client cert signed by the CA, with the CN being valid-client-name and no SANs
- valid-cn.key - Key for the above
- valid-dns.crt - A client cert signed by the CA, with the CN being invalid-client-name and a SAN of valid-client-name
- valid-dns.key - Key for the above
- invalid.crt - A client cert signed by the CA, with the CN being invalid-client-name and no SANs
- invalid.key - Key for the above
A kind cluster was created with this branch deployed using the e2e tooling, the certificates and keys were duplicated into the kind container using docker cp, additionally the CA was created as a Kubernetes secret.
For changes to the cert-manager config the helm-edit plugin was used to quickly change the Helm values.
Testing
Test 1 - Using Curl
This is the most thorough test of all configurations to ensure the server behaves as expected
Config 1 - Validating CA and CN/SAN
Helm config added from baseline:
webhook:
apiserverClientCertSubjects: valid-client-name
clientCAFile: /apiserver-webhook-client-ca/ca.crt
enableClientVerification: true
volumeMounts:
- mountPath: /apiserver-webhook-client-ca
name: apiserver-webhook-client-ca
volumes:
- name: apiserver-webhook-client-ca
secret:
secretName: apiserver-webhook-client-caWithout certificate
root@kind-control-plane:/# curl -k https://10.0.207.228
curl: (56) OpenSSL SSL_read: OpenSSL/3.0.16: error:0A00045C:SSL routines::tlsv13 alert certificate required, errno 0Blocks the request as expected
With certificate signed by correct CA with correct CN and no SAN
root@kind-control-plane:/# curl -k https://10.0.207.228 --cert /testing/valid-cn.crt --key /testing/valid-cn.key
404 page not foundAllows the request as expected
With certificate signed by correct CA with incorrect CN correct SAN
root@kind-control-plane:/# curl -k https://10.0.207.228 --cert /testing/valid-dns.crt --key /testing/valid-dns.key
404 page not foundAllows the request as expected
With certificate signed by correct CA with incorrect CN and no SAN
root@kind-control-plane:/# curl -k https://10.0.207.228 --cert /testing/invalid.crt --key /testing/invalid.key
curl: (56) OpenSSL SSL_read: OpenSSL/3.0.16: error:0A000412:SSL routines::sslv3 alert bad certificate, errno 0Blocks the request as expected
With certificate signed by incorrect CA
root@kind-control-plane:/# curl -k https://10.0.207.228 --cert /etc/kubernetes/pki/apiserver-etcd-client.crt --key /etc/kubernetes/pki/apiserver-etcd-client.key
curl: (56) OpenSSL SSL_read: OpenSSL/3.0.16: error:0A000418:SSL routines::tlsv1 alert unknown ca, errno 0Blocks the request as expected
Config 2 - Validating CA with any CN
Helm config added from baseline:
webhook:
clientCAFile: /apiserver-webhook-client-ca/ca.crt
enableClientVerification: true
volumeMounts:
- mountPath: /apiserver-webhook-client-ca
name: apiserver-webhook-client-ca
volumes:
- name: apiserver-webhook-client-ca
secret:
secretName: apiserver-webhook-client-caWithout certificate
root@kind-control-plane:/# curl -k https://10.0.207.228
curl: (56) OpenSSL SSL_read: OpenSSL/3.0.16: error:0A00045C:SSL routines::tlsv13 alert certificate required, errno 0Blocks the request as expected
With certificate signed by correct CA with correct CN and no SAN
root@kind-control-plane:/# curl -k https://10.0.207.228 --cert /testing/valid-cn.crt --key /testing/valid-cn.key
404 page not foundAllows the request as expected
With certificate signed by correct CA with incorrect CN correct SAN
root@kind-control-plane:/# curl -k https://10.0.207.228 --cert /testing/valid-dns.crt --key /testing/valid-dns.key
404 page not foundAllows the request as expected
With certificate signed by correct CA with incorrect CN and no SAN
root@kind-control-plane:/# curl -k https://10.0.207.228 --cert /testing/invalid.crt --key /testing/invalid.key
404 page not foundAllows the request as expected
With certificate signed by incorrect CA
root@kind-control-plane:/# curl -k https://10.0.207.228 --cert /etc/kubernetes/pki/apiserver-etcd-client.crt --key /etc/kubernetes/pki/apiserver-etcd-client.key
curl: (56) OpenSSL SSL_read: OpenSSL/3.0.16: error:0A000418:SSL routines::tlsv1 alert unknown ca, errno 0Blocks the request as expected
Test 2 - Using the APIServer
This is a simple happy path test to ensure the APIServer can still hit the webhook, the APIServer is set up to use the valid-cn.crt as its client certificate using an AdmissionConfiguration:
cat <<EOF | kubectl apply -f - --dry-run=server
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: webhook-test-cert
namespace: default
spec:
secretName: webhook-test-tls
duration: 2160h # 90 days
renewBefore: 360h # 15 days
subject:
organizations:
- example-org
commonName: webhook-test.example.com
dnsNames:
- webhook-test.example.com
- alt-webhook-test.example.com
issuerRef:
name: test-issuer
kind: Issuer
EOF
Warning: spec.privateKey.rotationPolicy: In cert-manager >= v1.18.0, the default value changed from `Never` to `Always`.
certificate.cert-manager.io/webhook-test-cert created (server dry run)The webhook is the thing setting the "warning", proving it was invoked correctly.
This change adds client verification ability to webhook server using [documentation](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#authenticate-apiservers) Usecase: We are running cert-manager on hostNetwork due to the design of our system. This exposes the webhook endpoints to anyone with the access to node IP. We need to secure the endpoints. Testing done: Tested the helm-chart changes using template command Rendered the charts yaml using default values and custom values as follows: ```yaml # Additional volumes to add to the cert-manager controller pod. volumes: - configMap: items: - key: ca.crt path: ca.crt name: kube-root-ca.crt name: client-ca # Additional volume mounts to add to the cert-manager controller container. volumeMounts: - mountPath: /tmp/k8s-webhook-server/serving-certs/client-ca name: client-ca readOnly: true # enableClientVerification turns on client verification of requests # made to the webhook server enableClientVerification: true # the client CA file to be used for verification clientCAFile: "client-ca/ca.crt" # if provided the subject name to be verified for the given client cert apiserverClientCertSubject: "apiserver-webhook-client" ``` ```shell $ make helm-chart $ cp -f deploy/charts/cert-manager/templates/webhook-deployment.yaml _bin/helm/cert-manager/templates/webhook-deployment.yaml /Users/shbajpai/go/src/github/cert-manager/cert-manager/_bin/tools/helm package --app-version=v1.18.0-beta.0-215-ga8829d041bb809-dirty --version=v1.18.0-beta.0-215-ga8829d041bb809-dirty --destination "_bin/" ./_bin/helm/cert-manager Successfully packaged chart and saved it to: _bin/cert-manager-v1.18.0-beta.0-215-ga8829d041bb809-dirty.tgz $ helm template cert-manager _bin/cert-manager-v1.18.0-beta.0-215-ga8829d041bb809-dirty.tgz \ --namespace cert-manager \ --values values.yaml > test-rendered.yaml $ helm template cert-manager _bin/cert-manager-v1.18.0-beta.0-215-ga8829d041bb809-dirty.tgz \ --namespace cert-manager > original-rendered.yaml $ diff -u original-rendered.yaml test-rendered.yaml --- original-rendered.yaml 2025-08-26 14:09:28 +++ rendered.yaml 2025-08-26 14:01:55 @@ -1153,6 +1153,7 @@ args: - --v=2 - --secure-port=10250 + - --enable-webhook-client-verification=true - --dynamic-serving-ca-secret-namespace=$(POD_NAMESPACE) - --dynamic-serving-ca-secret-name=cert-manager-webhook-ca - --dynamic-serving-dns-names=cert-manager-webhook @@ -1200,8 +1201,23 @@ valueFrom: fieldRef: fieldPath: metadata.namespace + - name: WEBHOOK_CLIENT_CA_FILE + value: client-ca/ca.crt + - name: APISERVER_CLIENT_CERT_SUBJECT + value: apiserver-webhook-client + volumeMounts: + - mountPath: /tmp/k8s-webhook-server/serving-certs/client-ca + name: client-ca + readOnly: true nodeSelector: kubernetes.io/os: "linux" + volumes: + - configMap: + items: + - key: ca.crt + path: ca.crt + name: kube-root-ca.crt + name: client-ca --- # Source: cert-manager/templates/webhook-mutating-webhook.yaml apiVersion: admissionregistration.k8s.io/v1 ``` Signed-off-by: Shubham Bajpai <shubham14bajpai@gmail.com>
Signed-off-by: Shubham Bajpai <shubham14bajpai@gmail.com>
Signed-off-by: Shubham Bajpai <shubham14bajpai@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Shubham Bajpai <shbajpai@vmware.com>
Signed-off-by: Shubham Bajpai <shubham14bajpai@gmail.com>
Signed-off-by: Shubham Bajpai <shubham14bajpai@gmail.com>
Signed-off-by: Shubham Bajpai <shubham14bajpai@gmail.com>
e50a258 to
0484e5f
Compare
Thank you so much for the detailed testing 🙏 |
|
Thats great! Thanks for contributing this 🙂 /lgtm |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: ThatsMrTalbot The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Pull Request Motivation
This change adds client verification ability to webhook server using documentation
Usecase:
We are running cert-manager on hostNetwork due to the design of our system. This exposes the webhook endpoints to anyone with the access to node IP. We need to secure the endpoints.
Testing done:
Tested the helm-chart changes using template command
Rendered the charts yaml using default values and custom values as follows:
Kind
/kind feature
Release Note