-
Notifications
You must be signed in to change notification settings - Fork 2.4k
(Cluster)Issuer with vault auth and serviceAccountRef is not accepted by cluster due to audience #6150
Description
I have kubernetes 1.25, cert-manager 1.12.1, external Vault 1.12 and I am trying to use new feature from cert-manager 1.12: kubernetes auth in Vault without reviewer token.
I followed documentation how to set this up (https://cert-manager.io/docs/configuration/vault/#secretless-authentication-with-a-service-account).
I have ClusterIssuer named vault-issuer and service account name created named cert-manager-vault.
I created all RoleBindings just as in documentation.
Also, I added ClusterRole auth-delegator to cert-manager-vault so it can work without reviewer token configured in Vault. (https://developer.hashicorp.com/vault/docs/auth/kubernetes#use-the-vault-client-s-jwt-as-the-reviewer-jwt)
I am having following errors:
ClusterIssuer: Error initializing issuer: while requesting a Vault token using the Kubernetes auth: error calling Vault server: Error making API request. Code: 403. Errors: * permission denied
So, I captured request to Vault, decoded token, looks fine, maybe except "aud" field, but still this is fine according to docs.
Looks like Vault is trying to call back kubernetes API, lets see:
kube-api: [authentication.go:63] "Unable to authenticate the request" err="[invalid bearer token, token audiences ["vault://vault-issuer"] is invalid for the target audiences ["https://kubernetes.default.svc.cluster.my.domain"]]"
And here I am stuck. How to fix this?
By the way, I got same setup working for external-secrets project (https://external-secrets.io/v0.8.3/provider/hashicorp-vault/#kubernetes-authentication) - but there in jwt "aud" field is set to "https://kubernetes.default.svc.cluster.my.domain/" so kube-api accepts it and validates.
I think both Vault validation of audience and API validation could be satisfied if there were two audiences in JWT token:
- vault://namespace/issuer (this is for Vault to validate)
- https://kubernetes.default.svc.cluster.my.domain/ (this is for API to validate).
JWT tokens have "aud" as array, so both would fit.
kube-api docs state that only one of audiences must match cluster audience to be OK.
Not sure how Vault validates this though, but I suspect that only one from list is sufficient as well to pass validation.
@maelvls would this work?
My vault auth config:
disable_iss_validation: True
disable_local_ca_jwt: False
issuer:
kubernetes_ca_cert:
-----BEGIN CERTIFICATE-----
(...)
-----END CERTIFICATE-----
kubernetes_host: https://kubernetes.default.svc.cluster.my.domain
pem_keys: []
Cluster issuer config:
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: vault-issuer
namespace: cert-manager
spec:
vault:
server: http://active.vault.service.my.domain:8200
caBundle:
path: pki_int/sign/platform.role
auth:
# https://cert-manager.io/docs/configuration/vault/#secretless-authentication-with-a-service-account
kubernetes:
role: "k8s-cert-manager-role"
mountPath: "/v1/auth/k8s-sd"
serviceAccountRef:
name: "cert-manager-vault"