Skip to content

(Cluster)Issuer with vault auth and serviceAccountRef is not accepted by cluster due to audience #6150

@Garagoth

Description

@Garagoth

I have kubernetes 1.25, cert-manager 1.12.1, external Vault 1.12 and I am trying to use new feature from cert-manager 1.12: kubernetes auth in Vault without reviewer token.
I followed documentation how to set this up (https://cert-manager.io/docs/configuration/vault/#secretless-authentication-with-a-service-account).

I have ClusterIssuer named vault-issuer and service account name created named cert-manager-vault.
I created all RoleBindings just as in documentation.
Also, I added ClusterRole auth-delegator to cert-manager-vault so it can work without reviewer token configured in Vault. (https://developer.hashicorp.com/vault/docs/auth/kubernetes#use-the-vault-client-s-jwt-as-the-reviewer-jwt)

I am having following errors:

ClusterIssuer: Error initializing issuer: while requesting a Vault token using the Kubernetes auth: error calling Vault server: Error making API request. Code: 403. Errors: * permission denied

So, I captured request to Vault, decoded token, looks fine, maybe except "aud" field, but still this is fine according to docs.
Looks like Vault is trying to call back kubernetes API, lets see:

kube-api: [authentication.go:63] "Unable to authenticate the request" err="[invalid bearer token, token audiences ["vault://vault-issuer"] is invalid for the target audiences ["https://kubernetes.default.svc.cluster.my.domain"]]"

And here I am stuck. How to fix this?
By the way, I got same setup working for external-secrets project (https://external-secrets.io/v0.8.3/provider/hashicorp-vault/#kubernetes-authentication) - but there in jwt "aud" field is set to "https://kubernetes.default.svc.cluster.my.domain/" so kube-api accepts it and validates.

I think both Vault validation of audience and API validation could be satisfied if there were two audiences in JWT token:

JWT tokens have "aud" as array, so both would fit.
kube-api docs state that only one of audiences must match cluster audience to be OK.
Not sure how Vault validates this though, but I suspect that only one from list is sufficient as well to pass validation.

@maelvls would this work?

My vault auth config:

    disable_iss_validation:        True
    disable_local_ca_jwt:        False
    issuer:
    kubernetes_ca_cert:
        -----BEGIN CERTIFICATE-----
        (...)
        -----END CERTIFICATE-----
    kubernetes_host:        https://kubernetes.default.svc.cluster.my.domain
    pem_keys: []

Cluster issuer config:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: vault-issuer
  namespace: cert-manager
spec:
  vault:
    server: http://active.vault.service.my.domain:8200
    caBundle: 
    path: pki_int/sign/platform.role
    auth:
      # https://cert-manager.io/docs/configuration/vault/#secretless-authentication-with-a-service-account
      kubernetes:
        role: "k8s-cert-manager-role"
        mountPath: "/v1/auth/k8s-sd"
        serviceAccountRef:
          name: "cert-manager-vault"

Metadata

Metadata

Assignees

No one assigned

    Labels

    lifecycle/rottenDenotes an issue or PR that has aged beyond stale and will be auto-closed.priority/important-soonMust be staffed and worked on either currently, or very soon, ideally in time for the next release.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions