Issue description

In NixOS 18.03, with a default installation of an "all-in-one"
kubernetes cluster, the kubectl command is unable to run any "logs" or
"exec" subcommand on any of the cluster's pods.

Steps to reproduce

In a fresh installation of NixOS 18.03 add the following to
configuration.nix:

  services.kubernetes.roles = [ "master" "node"];

Then switch the configuration with:

# nixos-rebuild switch

When the new configuration is installed, create a new simple pod by
opening a terminal and typing:

# kubectl run --image=nginx nginx-app --port=80

... await until the pod reaches the "Running" state:

# kubectl get pods
NAME                        READY     STATUS    RESTARTS   AGE
nginx-app-b6559f86b-g5wtv   1/1       Running   0          6m

Then try one of the "logs" or "exec" commands on it:

# kubectl logs nginx-app-b6559f86b-g5wtv
Error from server (Forbidden): Forbidden (user=system:anonymous, verb=get, resource=nodes, subresource=proxy) ( pods/log nginx-app-b6559f86b-g5wtv)

The server answers with authentication/authorization error, mostly due
to some misconfiguration and the usage of the "Webhook" authorization
mode in the "kubelet" service, selected by using the
--authorization-mode=Webhook in the kubelet cmdline:

# systemctl cat kubelet
[...]
ExecStart=/nix/store/00wh6lsfqdbyc854aiznvvcils3xys8q-kubernetes-1.9.1/bin/kubelet \
            --pod-manifest-path=/nix/store/qlf2fd5xix7rs719r9n1jdrxcb0axz1c-kubernetes-manifests \
             \
            --kubeconfig=/nix/store/1ic5zs560q8gf34sa322sp40ncjn00z1-kubelet-kubeconfig \
            --require-kubeconfig \
            --address=0.0.0.0 \
            --port=10250 \
            --register-node=true \
             \
             \
             \
            --authentication-token-webhook \
            --authentication-token-webhook-cache-ttl="10s" \
            --authorization-mode=Webhook \
            --healthz-bind-address=127.0.0.1 \
            --healthz-port=10248 \
            --hostname-override=giskard.lan \
            --allow-privileged=true \
            --root-dir=/var/lib/kubernetes \
            --cadvisor_port=4194 \
            --cluster-dns=10.0.0.254 \
            --cluster-domain=cluster.local \
            --pod-infra-container-image=pause \
            --network-plugin=kubenet \
            --cni-conf-dir=/nix/store/39x16051zmhzxfpzaplpavn1arjg683b-kubernetes-cni-config \
            --hairpin-mode=hairpin-veth \
[...]

With this kind of authentication/authorization setup it seems from
the official docs
that the kubelet service will delegate any decision by calling some
apis on the kube-apiserver service. The same doc page mentions for a
non-stable api "authentication.k8s.io/v1beta1" to be enabled on the
apiserver for the system to work and from a check of the available
apis it seems to be enabled:

# kubectl api-versions | grep authentication\.k8s\.io/v1beta1
authentication.k8s.io/v1beta1

# 

Any further investigation hasn't given much of an explanation of this
issue. Clearly the RBAC system is involved, because by increasing the
kube-apiserver logging it's possible to see it when the command is
being run, but it's not clear who's calling who or why the user that's
recognized ends up being "system:anonymous". My impression is that
some pieces of the bootstrap RBAC policies are missing or
misconfigured.

Even trying to use well-defined user as explained in #40560 doesn't
change the outcome.

Looking on the net for references of the symptom hasn't produced any
meaningful result for this kind of configuration. the examples I found
fix the thing using x509 certs which from the docs seems to be the
other way of managing the auth/authz.

To circumvent this issue it's possible to use the "AlwaysAllow"
authorization policy by adding the --authorization-mode=AlwaysAllow
to the services.kubernetes.kubelet.extraOpts configuration entry,
but this will make the kubelet daemon to accept any request from any
client, leaving the node exposed to tampering.

Ah, as a final note, I must say I've tried both the packages of the
18.03 distro and the one in unstable as of now, that is still an 1.9.1
but with a fix of an issue with the kube-proxy and the
iptables-restore command (See #36739). The outcome it's always the
same.

Technical details

Please run nix-shell -p nix-info --run "nix-info -m" and paste the
results.

• system: "x86_64-linux"
• host os: Linux 4.15.18, NixOS, 18.03.132336.ef74cafd3e5 (Impala)
• multi-user?: no
• sandbox: yes
• version: nix-env (Nix) 2.0.1
• channels(root): "unstable, nixos-18.03.132336.ef74cafd3e5"
• nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs