Overview

I get a lot of questions about how to setup mutual authentication for node-level cluster plugins. Examples include:

• Local volume provisioning
• Device plugins (modify node object)
• Device metrics (scraping)
• CRI Streaming
• Are there others I'm missing?

For the general case, our recommendation has been to use a service mesh (such as Istio), or NetworkPolicy (although this isn't really a full solution). But as both of those are optional features on Kubernetes, I don't think they are sufficient for core functionality.

We need some general guidelines for plugins, and may need to build some additional capabilities or libraries to support them.

The architecture we need to target includes:

• A central control component, running as a deployment in the cluster. Let's call this the "CCC".
• A DaemonSet that handles node operations, and communicates back to the controller. Let's call this the "DSP" (DaemonSet Pod).
• K8s API server

Things we might need to solve:

1. CCC -> apiserver (solved)
   a. [x] server (apiserver) authn -- cluster CA
   b. [x] client (CCC) authn -- Use service account tokens
   c. [x] authz -- RBAC
2. CCC -> DSP
   a. [ ] server (DSP) authn
   b. [x] client (CCC) authn -- Service account tokens + TokenReview
   c. [x] authz -- RBAC + SubjectAccessReview
3. DSP -> CCC
   a. [ ] server (CCC) authn
   b. [ ] client (DSP) authn
   c. [ ] authz
4. DSP -> apiserver
   a. [x] server (apiserver) authn -- cluster CA
   b. [ ] client (DSP) authn
   c. [ ] authz

Optionally, we could only solve one of (2) and (3), and just require communications to happen in a single direction (i.e. pull or push). We also might be able to avoid (4) by delegating to the CCC (i.e. require all DSP requests to the apiserver to go through the CCC).

/kind technical-debt
/sig auth

/cc @dashpole @vishh @davidz627 @Random-Liu