Skip to content

feat: Add client verification for webhook server#8010

Merged
cert-manager-prow[bot] merged 7 commits into
cert-manager:masterfrom
shubham14bajpai:add/webhook-auth
Nov 25, 2025
Merged

feat: Add client verification for webhook server#8010
cert-manager-prow[bot] merged 7 commits into
cert-manager:masterfrom
shubham14bajpai:add/webhook-auth

Conversation

@shubham14bajpai

@shubham14bajpai shubham14bajpai commented Aug 26, 2025

Copy link
Copy Markdown
Contributor

Pull Request Motivation

This change adds client verification ability to webhook server using documentation

Usecase:
We are running cert-manager on hostNetwork due to the design of our system. This exposes the webhook endpoints to anyone with the access to node IP. We need to secure the endpoints.

Testing done:

Tested the helm-chart changes using template command
Rendered the charts yaml using default values and custom values as follows:

  # Additional volumes to add to the cert-manager controller pod.
  volumes:
  - configMap:
      items:
      - key: ca.crt
        path: ca.crt
      name: kube-root-ca.crt
    name: client-ca

  # Additional volume mounts to add to the cert-manager controller container.
  volumeMounts:
  - mountPath: /tmp/k8s-webhook-server/serving-certs/client-ca
    name: client-ca
    readOnly: true

  # enableClientVerification turns on client verification of requests
  # made to the webhook server
  enableClientVerification: true

  # the client CA file to be used for verification
  clientCAFile: "client-ca/ca.crt"

  # if provided the subject name to be verified for the given client cert
  apiserverClientCertSubject: "apiserver-webhook-client"
$ make helm-chart

$ cp -f deploy/charts/cert-manager/templates/webhook-deployment.yaml _bin/helm/cert-manager/templates/webhook-deployment.yaml
/Users/shbajpai/go/src/github/cert-manager/cert-manager/_bin/tools/helm package --app-version=v1.18.0-beta.0-215-ga8829d041bb809-dirty --version=v1.18.0-beta.0-215-ga8829d041bb809-dirty --destination "_bin/" ./_bin/helm/cert-manager
Successfully packaged chart and saved it to: _bin/cert-manager-v1.18.0-beta.0-215-ga8829d041bb809-dirty.tgz

$ helm template  cert-manager _bin/cert-manager-v1.18.0-beta.0-215-ga8829d041bb809-dirty.tgz \
  --namespace cert-manager \
  --values values.yaml > test-rendered.yaml

$ helm template  cert-manager _bin/cert-manager-v1.18.0-beta.0-215-ga8829d041bb809-dirty.tgz \
  --namespace cert-manager > original-rendered.yaml

$ diff -u original-rendered.yaml rendered.yaml
--- original-rendered.yaml      2025-08-29 09:50:22
+++ rendered.yaml       2025-08-29 09:49:27
@@ -1153,6 +1153,9 @@
           args:
           - --v=2
           - --secure-port=10250
+          - --enable-client-verification=true
+          - --client-ca-name=client-ca/ca.crt
+          - --client-certificate-cn=apiserver-webhook-client
           - --dynamic-serving-ca-secret-namespace=$(POD_NAMESPACE)
           - --dynamic-serving-ca-secret-name=cert-manager-webhook-ca
           - --dynamic-serving-dns-names=cert-manager-webhook
@@ -1200,8 +1203,19 @@
             valueFrom:
               fieldRef:
                 fieldPath: metadata.namespace
+          volumeMounts:
+          - mountPath: /tmp/k8s-webhook-server/serving-certs/client-ca
+            name: client-ca
+            readOnly: true
       nodeSelector:
         kubernetes.io/os: "linux"
+      volumes:
+      - configMap:
+          items:
+          - key: ca.crt
+            path: ca.crt
+          name: kube-root-ca.crt
+        name: client-ca
 ---
 # Source: cert-manager/templates/webhook-mutating-webhook.yaml
 apiVersion: admissionregistration.k8s.io/v1

Kind

/kind feature

Release Note

NONE

@cert-manager-prow cert-manager-prow Bot added kind/feature Categorizes issue or PR as related to a new feature. release-note-none Denotes a PR that doesn't merit a release note. dco-signoff: no Indicates that at least one commit in this pull request is missing the DCO sign-off message. area/deploy Indicates a PR modifies deployment configuration needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Aug 26, 2025
@cert-manager-prow

Copy link
Copy Markdown
Contributor

Hi @shubham14bajpai. Thanks for your PR.

I'm waiting for a cert-manager member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@cert-manager-prow cert-manager-prow Bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Aug 26, 2025
@cert-manager-prow cert-manager-prow Bot added dco-signoff: yes Indicates that all commits in the pull request have the valid DCO sign-off message. and removed dco-signoff: no Indicates that at least one commit in this pull request is missing the DCO sign-off message. labels Aug 26, 2025
Comment thread internal/webhook/webhook.go Outdated
Comment thread pkg/webhook/server/server.go Outdated
Comment thread pkg/webhook/server/server.go Outdated
@ThatsMrTalbot

Copy link
Copy Markdown
Contributor

👋 Thanks for updating, I've added a few more comments.

I'll also add the ok-to-test flag so you can use CI to run tests

/ok-to-test

@cert-manager-prow cert-manager-prow Bot added ok-to-test and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Sep 3, 2025
@cert-manager-prow cert-manager-prow Bot added the area/api Indicates a PR directly modifies the 'pkg/apis' directory label Oct 18, 2025
@shubham14bajpai shubham14bajpai force-pushed the add/webhook-auth branch 4 times, most recently from 9905b20 to 5ea366e Compare October 18, 2025 15:51
@shubham14bajpai

shubham14bajpai commented Oct 19, 2025

Copy link
Copy Markdown
Contributor Author

👋 Thanks for updating, I've added a few more comments.

I'll also add the ok-to-test flag so you can use CI to run tests

/ok-to-test

Hey, sorry for the delay. I was finally able to fix the test failures. Please take another look.

@erikgb erikgb requested a review from Copilot October 19, 2025 18:37

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds client verification capability to the cert-manager webhook server to authenticate API server requests using client certificates. This addresses security concerns when running cert-manager on hostNetwork where webhook endpoints are exposed to anyone with node IP access.

  • Adds client certificate verification to webhook server using configurable CA and client CN validation
  • Introduces new configuration options for enabling client verification, CA path, and client certificate CN
  • Updates Helm chart with new configuration values and deployment template changes

Reviewed Changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
pkg/webhook/server/server.go Core implementation of client certificate verification logic
pkg/webhook/options/options.go CLI flag definitions for client verification options
pkg/apis/config/webhook/v1alpha1/types.go Public API type definitions for client verification config
internal/webhook/webhook.go Integration of client verification options into webhook server construction
internal/apis/config/webhook/v1alpha1/zz_generated.conversion.go Generated conversion functions for new config fields
internal/apis/config/webhook/types.go Internal API type definitions for client verification config
deploy/charts/cert-manager/values.yaml Default Helm chart values for client verification
deploy/charts/cert-manager/values.schema.json JSON schema definitions for new Helm values
deploy/charts/cert-manager/templates/webhook-deployment.yaml Template updates to pass client verification flags and fix indentation
deploy/charts/cert-manager/README.template.md Documentation for new configuration options

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

Comment thread pkg/webhook/server/server.go Outdated
Comment thread pkg/apis/config/webhook/v1alpha1/types.go Outdated
Comment thread internal/apis/config/webhook/types.go Outdated
Comment thread pkg/webhook/server/server.go Outdated
@ThatsMrTalbot

Copy link
Copy Markdown
Contributor

Hiya! I'll take a look this week as soon as I have time

@ThatsMrTalbot

Copy link
Copy Markdown
Contributor

👋 Apologies for the delay - I've had a look at the code and it looks good. Given this area is not covered by tests I want to spin this up locally and do some manual testing before I approve.

@shubham14bajpai

Copy link
Copy Markdown
Contributor Author

👋 Apologies for the delay - I've had a look at the code and it looks good. Given this area is not covered by tests I want to spin this up locally and do some manual testing before I approve.

Hey I did some manual testing and here is the log 🙂

Created a webhook client

~/go/src/github/cert-manager on  add/webhook-auth! ⌚ 10:50:39
$ openssl x509 -text -noout -in /var/lib/minikube/certs/admission/webhook-client.crt 
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            6b:a3:56:1c:63:96:42:12:56:a3:92:20:9d:39:44:34:87:7d:12:5f
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN = minikubeCA
        Validity
            Not Before: Nov  8 04:58:00 2025 GMT
            Not After : Nov  8 04:58:00 2026 GMT
        Subject: CN = admission-webhook-client

Configured the kube-apiserver and cert-manager webhook

~/go/src/github/cert-manager on  add/webhook-auth! ⌚ 10:51:10
$ cat /etc/kubernetes/manifests/kube-apiserver.yaml | grep admission
    - --admission-control-config-file=/var/lib/minikube/certs/admission/admission-config.yaml

~/go/src/github/cert-manager on  add/webhook-auth! ⌚ 10:54:31
$ kubectl get -n cert-manager-dev deployment/cert-manager-dev-webhook -oyaml | grep client
        - --enable-client-verification=true
        - --client-ca-path=/var/lib/minikube/certs/ca.crt
        - --client-certificate-cn=admission-webhook-client

Tried curling the endpoints with & without the client certificate

~/go/src/github/cert-manager on  add/webhook-auth! ⌚ 10:56:20
$ kubectl exec -n cert-manager-dev curlpod2 -- curl -k --no-progress-meter --max-time 10 --cacert /tmp/ca.crt --cert /tmp/webhook-client.crt --key /tmp/webhook-client.key https://cert-manager-dev-webhook.cert-manager-dev.svc:443/mutate\?timeout\=1s
{"response":{"uid":"","allowed":false,"status":{"metadata":{},"message":"contentType=, expected application/json","code":400}}}

~/go/src/github/cert-manager on  add/webhook-auth! ⌚ 10:57:29
$ kubectl exec -n cert-manager-dev curlpod2 -- curl -k --no-progress-meter --max-time 10 https://cert-manager-dev-webhook.cert-manager-dev.svc:443/mutate\?timeout\=1s
curl: (56) SSL certificate OpenSSL verify result: unable to get local issuer certificate (20)
command terminated with exit code 56

Tested the certificate generation

~/go/src/github/cert-manager on  add/webhook-auth! ⌚ 17:18:27
$ kubectl -n test-ns get issuer,certificate,secret
NAME                                 READY   AGE
issuer.cert-manager.io/test-issuer   True    82s

NAME                                    READY   SECRET          AGE
certificate.cert-manager.io/test-cert   True    test-cert-tls   82s

NAME                   TYPE                DATA   AGE
secret/test-cert-tls   kubernetes.io/tls   3      81s

@ThatsMrTalbot ThatsMrTalbot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry about the delay - my time has been fairly limited so I have not had a chance to dive into testing this.

I did get this setup fully locally and proved it works, I have the setup and tests I ran at the bottom.

I have left a comment around the CN validation as the final thing I would like to address


Setup

  1. I created a deployment of your branch using the make e2e-setup command, this deployed your code via the helm chart into a new kind cluster

  2. I exec'd into the kind control-plane container and made the following changes:

    1. Created an admission config file / kubeconfig to tell the APIServer to present a certificate
    apiVersion: apiserver.config.k8s.io/v1
    kind: AdmissionConfiguration
    plugins:
      - name: ValidatingAdmissionWebhook
        configuration:
          apiVersion: apiserver.config.k8s.io/v1
          kind: WebhookAdmissionConfiguration
          kubeConfigFile: /etc/kubernetes/admission/webhook-authn.kubeconfig
      - name: MutatingAdmissionWebhook
        configuration:
          apiVersion: apiserver.config.k8s.io/v1
          kind: WebhookAdmissionConfiguration
          kubeConfigFile: /etc/kubernetes/admission/webhook-authn.kubeconfig
    apiVersion: v1
    kind: Config
    users:
      - name: '*'
        user:
          client-certificate: /etc/kubernetes/pki/apiserver-kubelet-client.crt
          client-key: /etc/kubernetes/pki/apiserver-kubelet-client.key
    1. Updated the static manifest of the apiserver pod to mount /etc/kubernetes/admission/ and have the flag --admission-control-config-file=/etc/kubernetes/admission/admission-config.yaml

    2. I then grabbed copies of /etc/kubernetes/pki/apiserver-kubelet-client.crt and /etc/kubernetes/pki/ca.crt and exited the container

  3. I created a secret in the cert-manager namespace containing the ca.crt

  4. I updated the helm values to contain:

webhook:
  apiserverClientCertSubject: kube-apiserver-kubelet-client
  clientCAFile: /apiserver-webhook-client-ca/ca.crt
  enableClientVerification: true
  volumeMounts:
  - mountPath: /apiserver-webhook-client-ca
    name: apiserver-webhook-client-ca
  volumes:
  - name: apiserver-webhook-client-ca
    secret:
      secretName: apiserver-webhook-client-ca

Tests

My first test was running some dry-run=server operations:

cat <<EOF | kubectl apply -f - --dry-run=server
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: webhook-test-cert
  namespace: default
spec:
  secretName: webhook-test-tls
  duration: 2160h    # 90 days
  renewBefore: 360h  # 15 days
  subject:
    organizations:
      - example-org
  commonName: webhook-test.example.com
  dnsNames:
    - webhook-test.example.com
    - alt-webhook-test.example.com
  issuerRef:
    name: test-issuer
    kind: Issuer
EOF
Warning: spec.privateKey.rotationPolicy: In cert-manager >= v1.18.0, the default value changed from `Never` to `Always`.
certificate.cert-manager.io/webhook-test-cert created (server dry run)

This hit the webhook correctly as evidenced by the warning the webhook emits.

I then tried to curl the webhook directly without a cert and got an error as expected:

root@kind-control-plane:/# curl -k https://10.0.243.85
curl: (56) OpenSSL SSL_read: OpenSSL/3.0.16: error:0A00045C:SSL routines::tlsv13 alert certificate required, errno 0

I tried curling with a valid cert, which worked:

root@kind-control-plane:/# curl -k https://10.0.243.85 --cert /etc/kubernetes/pki/apiserver-kubelet-client.crt --key /etc/kubernetes/pki/apiserver-kubelet-client.key
404 page not found

I then tried curling with an invalid cert, which errored as expected:

root@kind-control-plane:/# curl -k https://10.0.243.85 --cert /etc/kubernetes/pki/apiserver-etcd-client.crt --key  /etc/kubernetes/pki/apiserver-etcd-client.key
curl: (56) OpenSSL SSL_read: OpenSSL/3.0.16: error:0A000418:SSL routines::tlsv1 alert unknown ca, errno 0

Comment thread pkg/webhook/server/server.go Outdated

@ThatsMrTalbot ThatsMrTalbot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👋 I have re-completed my testing after these changes and it all looks great! I have left one minor documentation comment just to clarify the apiserverClientCertSubjects Helm value, since thats just a comment update I can merge as soon as thats fixed.

I have included my testing notes below for documentation:


Setup

The following certificates were created for this test:

  • ca.crt - A self-signed CA certificate
  • valid-cn.crt - A client cert signed by the CA, with the CN being valid-client-name and no SANs
  • valid-cn.key - Key for the above
  • valid-dns.crt - A client cert signed by the CA, with the CN being invalid-client-name and a SAN of valid-client-name
  • valid-dns.key - Key for the above
  • invalid.crt - A client cert signed by the CA, with the CN being invalid-client-name and no SANs
  • invalid.key - Key for the above

A kind cluster was created with this branch deployed using the e2e tooling, the certificates and keys were duplicated into the kind container using docker cp, additionally the CA was created as a Kubernetes secret.

For changes to the cert-manager config the helm-edit plugin was used to quickly change the Helm values.

Testing

Test 1 - Using Curl

This is the most thorough test of all configurations to ensure the server behaves as expected

Config 1 - Validating CA and CN/SAN

Helm config added from baseline:

webhook:
  apiserverClientCertSubjects: valid-client-name
  clientCAFile: /apiserver-webhook-client-ca/ca.crt
  enableClientVerification: true
  volumeMounts:
  - mountPath: /apiserver-webhook-client-ca
    name: apiserver-webhook-client-ca
  volumes:
  - name: apiserver-webhook-client-ca
    secret:
      secretName: apiserver-webhook-client-ca

Without certificate

root@kind-control-plane:/# curl -k https://10.0.207.228
curl: (56) OpenSSL SSL_read: OpenSSL/3.0.16: error:0A00045C:SSL routines::tlsv13 alert certificate required, errno 0

Blocks the request as expected

With certificate signed by correct CA with correct CN and no SAN

root@kind-control-plane:/# curl -k https://10.0.207.228 --cert /testing/valid-cn.crt --key  /testing/valid-cn.key
404 page not found

Allows the request as expected

With certificate signed by correct CA with incorrect CN correct SAN

root@kind-control-plane:/# curl -k https://10.0.207.228 --cert /testing/valid-dns.crt --key  /testing/valid-dns.key
404 page not found

Allows the request as expected

With certificate signed by correct CA with incorrect CN and no SAN

root@kind-control-plane:/# curl -k https://10.0.207.228 --cert /testing/invalid.crt --key  /testing/invalid.key
curl: (56) OpenSSL SSL_read: OpenSSL/3.0.16: error:0A000412:SSL routines::sslv3 alert bad certificate, errno 0

Blocks the request as expected

With certificate signed by incorrect CA

root@kind-control-plane:/# curl -k https://10.0.207.228 --cert /etc/kubernetes/pki/apiserver-etcd-client.crt --key  /etc/kubernetes/pki/apiserver-etcd-client.key
curl: (56) OpenSSL SSL_read: OpenSSL/3.0.16: error:0A000418:SSL routines::tlsv1 alert unknown ca, errno 0

Blocks the request as expected

Config 2 - Validating CA with any CN

Helm config added from baseline:

webhook:
  clientCAFile: /apiserver-webhook-client-ca/ca.crt
  enableClientVerification: true
  volumeMounts:
  - mountPath: /apiserver-webhook-client-ca
    name: apiserver-webhook-client-ca
  volumes:
  - name: apiserver-webhook-client-ca
    secret:
      secretName: apiserver-webhook-client-ca

Without certificate

root@kind-control-plane:/# curl -k https://10.0.207.228
curl: (56) OpenSSL SSL_read: OpenSSL/3.0.16: error:0A00045C:SSL routines::tlsv13 alert certificate required, errno 0

Blocks the request as expected

With certificate signed by correct CA with correct CN and no SAN

root@kind-control-plane:/# curl -k https://10.0.207.228 --cert /testing/valid-cn.crt --key  /testing/valid-cn.key
404 page not found

Allows the request as expected

With certificate signed by correct CA with incorrect CN correct SAN

root@kind-control-plane:/# curl -k https://10.0.207.228 --cert /testing/valid-dns.crt --key  /testing/valid-dns.key
404 page not found

Allows the request as expected

With certificate signed by correct CA with incorrect CN and no SAN

root@kind-control-plane:/# curl -k https://10.0.207.228 --cert /testing/invalid.crt --key  /testing/invalid.key
404 page not found

Allows the request as expected

With certificate signed by incorrect CA

root@kind-control-plane:/# curl -k https://10.0.207.228 --cert /etc/kubernetes/pki/apiserver-etcd-client.crt --key  /etc/kubernetes/pki/apiserver-etcd-client.key
curl: (56) OpenSSL SSL_read: OpenSSL/3.0.16: error:0A000418:SSL routines::tlsv1 alert unknown ca, errno 0

Blocks the request as expected

Test 2 - Using the APIServer

This is a simple happy path test to ensure the APIServer can still hit the webhook, the APIServer is set up to use the valid-cn.crt as its client certificate using an AdmissionConfiguration:

cat <<EOF | kubectl apply -f - --dry-run=server
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: webhook-test-cert
  namespace: default
spec:
  secretName: webhook-test-tls
  duration: 2160h    # 90 days
  renewBefore: 360h  # 15 days
  subject:
    organizations:
      - example-org
  commonName: webhook-test.example.com
  dnsNames:
    - webhook-test.example.com
    - alt-webhook-test.example.com
  issuerRef:
    name: test-issuer
    kind: Issuer
EOF
Warning: spec.privateKey.rotationPolicy: In cert-manager >= v1.18.0, the default value changed from `Never` to `Always`.
certificate.cert-manager.io/webhook-test-cert created (server dry run)

The webhook is the thing setting the "warning", proving it was invoked correctly.

Comment thread deploy/charts/cert-manager/README.template.md Outdated
shubham14bajpai and others added 7 commits November 24, 2025 20:07
This change adds client verification ability to webhook server using [documentation](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#authenticate-apiservers)

Usecase:
We are running cert-manager on hostNetwork due to the design of our system. This exposes the webhook endpoints to anyone with the access to node IP. We need to secure the endpoints.

Testing done:
Tested the helm-chart changes using template command
Rendered the charts yaml using default values and custom values as follows:

```yaml
  # Additional volumes to add to the cert-manager controller pod.
  volumes:
  - configMap:
      items:
      - key: ca.crt
        path: ca.crt
      name: kube-root-ca.crt
    name: client-ca

  # Additional volume mounts to add to the cert-manager controller container.
  volumeMounts:
  - mountPath: /tmp/k8s-webhook-server/serving-certs/client-ca
    name: client-ca
    readOnly: true

  # enableClientVerification turns on client verification of requests
  # made to the webhook server
  enableClientVerification: true

  # the client CA file to be used for verification
  clientCAFile: "client-ca/ca.crt"

  # if provided the subject name to be verified for the given client cert
  apiserverClientCertSubject: "apiserver-webhook-client"

```

```shell

$ make helm-chart

$ cp -f deploy/charts/cert-manager/templates/webhook-deployment.yaml _bin/helm/cert-manager/templates/webhook-deployment.yaml
/Users/shbajpai/go/src/github/cert-manager/cert-manager/_bin/tools/helm package --app-version=v1.18.0-beta.0-215-ga8829d041bb809-dirty --version=v1.18.0-beta.0-215-ga8829d041bb809-dirty --destination "_bin/" ./_bin/helm/cert-manager
Successfully packaged chart and saved it to: _bin/cert-manager-v1.18.0-beta.0-215-ga8829d041bb809-dirty.tgz

$ helm template  cert-manager _bin/cert-manager-v1.18.0-beta.0-215-ga8829d041bb809-dirty.tgz \
  --namespace cert-manager \
  --values values.yaml > test-rendered.yaml

$ helm template  cert-manager _bin/cert-manager-v1.18.0-beta.0-215-ga8829d041bb809-dirty.tgz \
  --namespace cert-manager > original-rendered.yaml

$ diff -u original-rendered.yaml test-rendered.yaml
--- original-rendered.yaml      2025-08-26 14:09:28
+++ rendered.yaml       2025-08-26 14:01:55
@@ -1153,6 +1153,7 @@
           args:
           - --v=2
           - --secure-port=10250
+          - --enable-webhook-client-verification=true
           - --dynamic-serving-ca-secret-namespace=$(POD_NAMESPACE)
           - --dynamic-serving-ca-secret-name=cert-manager-webhook-ca
           - --dynamic-serving-dns-names=cert-manager-webhook
@@ -1200,8 +1201,23 @@
             valueFrom:
               fieldRef:
                 fieldPath: metadata.namespace
+          - name: WEBHOOK_CLIENT_CA_FILE
+            value: client-ca/ca.crt
+          - name: APISERVER_CLIENT_CERT_SUBJECT
+            value: apiserver-webhook-client
+          volumeMounts:
+          - mountPath: /tmp/k8s-webhook-server/serving-certs/client-ca
+            name: client-ca
+            readOnly: true
       nodeSelector:
         kubernetes.io/os: "linux"
+      volumes:
+      - configMap:
+          items:
+          - key: ca.crt
+            path: ca.crt
+          name: kube-root-ca.crt
+        name: client-ca
 ---
 # Source: cert-manager/templates/webhook-mutating-webhook.yaml
 apiVersion: admissionregistration.k8s.io/v1

```

Signed-off-by: Shubham Bajpai <shubham14bajpai@gmail.com>
Signed-off-by: Shubham Bajpai <shubham14bajpai@gmail.com>
Signed-off-by: Shubham Bajpai <shubham14bajpai@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Shubham Bajpai <shbajpai@vmware.com>
Signed-off-by: Shubham Bajpai <shubham14bajpai@gmail.com>
Signed-off-by: Shubham Bajpai <shubham14bajpai@gmail.com>
Signed-off-by: Shubham Bajpai <shubham14bajpai@gmail.com>
@shubham14bajpai

Copy link
Copy Markdown
Contributor Author

👋 I have re-completed my testing after these changes and it all looks great! I have left one minor documentation comment just to clarify the apiserverClientCertSubjects Helm value, since thats just a comment update I can merge as soon as thats fixed.

I have included my testing notes below for documentation:

Setup

The following certificates were created for this test:

  • ca.crt - A self-signed CA certificate
  • valid-cn.crt - A client cert signed by the CA, with the CN being valid-client-name and no SANs
  • valid-cn.key - Key for the above
  • valid-dns.crt - A client cert signed by the CA, with the CN being invalid-client-name and a SAN of valid-client-name
  • valid-dns.key - Key for the above
  • invalid.crt - A client cert signed by the CA, with the CN being invalid-client-name and no SANs
  • invalid.key - Key for the above

A kind cluster was created with this branch deployed using the e2e tooling, the certificates and keys were duplicated into the kind container using docker cp, additionally the CA was created as a Kubernetes secret.

For changes to the cert-manager config the helm-edit plugin was used to quickly change the Helm values.

Testing

Test 1 - Using Curl

This is the most thorough test of all configurations to ensure the server behaves as expected

Config 1 - Validating CA and CN/SAN

Helm config added from baseline:

webhook:
  apiserverClientCertSubjects: valid-client-name
  clientCAFile: /apiserver-webhook-client-ca/ca.crt
  enableClientVerification: true
  volumeMounts:
  - mountPath: /apiserver-webhook-client-ca
    name: apiserver-webhook-client-ca
  volumes:
  - name: apiserver-webhook-client-ca
    secret:
      secretName: apiserver-webhook-client-ca

Without certificate

root@kind-control-plane:/# curl -k https://10.0.207.228
curl: (56) OpenSSL SSL_read: OpenSSL/3.0.16: error:0A00045C:SSL routines::tlsv13 alert certificate required, errno 0

Blocks the request as expected

With certificate signed by correct CA with correct CN and no SAN

root@kind-control-plane:/# curl -k https://10.0.207.228 --cert /testing/valid-cn.crt --key  /testing/valid-cn.key
404 page not found

Allows the request as expected

With certificate signed by correct CA with incorrect CN correct SAN

root@kind-control-plane:/# curl -k https://10.0.207.228 --cert /testing/valid-dns.crt --key  /testing/valid-dns.key
404 page not found

Allows the request as expected

With certificate signed by correct CA with incorrect CN and no SAN

root@kind-control-plane:/# curl -k https://10.0.207.228 --cert /testing/invalid.crt --key  /testing/invalid.key
curl: (56) OpenSSL SSL_read: OpenSSL/3.0.16: error:0A000412:SSL routines::sslv3 alert bad certificate, errno 0

Blocks the request as expected

With certificate signed by incorrect CA

root@kind-control-plane:/# curl -k https://10.0.207.228 --cert /etc/kubernetes/pki/apiserver-etcd-client.crt --key  /etc/kubernetes/pki/apiserver-etcd-client.key
curl: (56) OpenSSL SSL_read: OpenSSL/3.0.16: error:0A000418:SSL routines::tlsv1 alert unknown ca, errno 0

Blocks the request as expected

Config 2 - Validating CA with any CN

Helm config added from baseline:

webhook:
  clientCAFile: /apiserver-webhook-client-ca/ca.crt
  enableClientVerification: true
  volumeMounts:
  - mountPath: /apiserver-webhook-client-ca
    name: apiserver-webhook-client-ca
  volumes:
  - name: apiserver-webhook-client-ca
    secret:
      secretName: apiserver-webhook-client-ca

Without certificate

root@kind-control-plane:/# curl -k https://10.0.207.228
curl: (56) OpenSSL SSL_read: OpenSSL/3.0.16: error:0A00045C:SSL routines::tlsv13 alert certificate required, errno 0

Blocks the request as expected

With certificate signed by correct CA with correct CN and no SAN

root@kind-control-plane:/# curl -k https://10.0.207.228 --cert /testing/valid-cn.crt --key  /testing/valid-cn.key
404 page not found

Allows the request as expected

With certificate signed by correct CA with incorrect CN correct SAN

root@kind-control-plane:/# curl -k https://10.0.207.228 --cert /testing/valid-dns.crt --key  /testing/valid-dns.key
404 page not found

Allows the request as expected

With certificate signed by correct CA with incorrect CN and no SAN

root@kind-control-plane:/# curl -k https://10.0.207.228 --cert /testing/invalid.crt --key  /testing/invalid.key
404 page not found

Allows the request as expected

With certificate signed by incorrect CA

root@kind-control-plane:/# curl -k https://10.0.207.228 --cert /etc/kubernetes/pki/apiserver-etcd-client.crt --key  /etc/kubernetes/pki/apiserver-etcd-client.key
curl: (56) OpenSSL SSL_read: OpenSSL/3.0.16: error:0A000418:SSL routines::tlsv1 alert unknown ca, errno 0

Blocks the request as expected

Test 2 - Using the APIServer

This is a simple happy path test to ensure the APIServer can still hit the webhook, the APIServer is set up to use the valid-cn.crt as its client certificate using an AdmissionConfiguration:

cat <<EOF | kubectl apply -f - --dry-run=server
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: webhook-test-cert
  namespace: default
spec:
  secretName: webhook-test-tls
  duration: 2160h    # 90 days
  renewBefore: 360h  # 15 days
  subject:
    organizations:
      - example-org
  commonName: webhook-test.example.com
  dnsNames:
    - webhook-test.example.com
    - alt-webhook-test.example.com
  issuerRef:
    name: test-issuer
    kind: Issuer
EOF
Warning: spec.privateKey.rotationPolicy: In cert-manager >= v1.18.0, the default value changed from `Never` to `Always`.
certificate.cert-manager.io/webhook-test-cert created (server dry run)

The webhook is the thing setting the "warning", proving it was invoked correctly.

Thank you so much for the detailed testing 🙏

@ThatsMrTalbot

Copy link
Copy Markdown
Contributor

Thats great! Thanks for contributing this 🙂

/lgtm
/approve

@cert-manager-prow cert-manager-prow Bot added the lgtm Indicates that a PR is ready to be merged. label Nov 25, 2025
@cert-manager-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ThatsMrTalbot

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@cert-manager-prow cert-manager-prow Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 25, 2025
@cert-manager-prow cert-manager-prow Bot merged commit b097c97 into cert-manager:master Nov 25, 2025
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. area/api Indicates a PR directly modifies the 'pkg/apis' directory area/deploy Indicates a PR modifies deployment configuration dco-signoff: yes Indicates that all commits in the pull request have the valid DCO sign-off message. kind/feature Categorizes issue or PR as related to a new feature. lgtm Indicates that a PR is ready to be merged. ok-to-test release-note-none Denotes a PR that doesn't merit a release note. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants