I'm trying to set up AWS ELB attach volume access and keep getting an error on the kubelet. I have some bad feeling it might be related to #9801 as these nodes are CoreOS that were brought up by Terraform with custom service files but everything else is working fine (we're running cluster monitoring, dns, and a number of our own pods without issues). I've run awscli on that instance and the privileges definitely work. What am I doing wrong?
logs from kubelet service
I0719 02:22:59.549444 9662 factory.go:234] Registering Docker factory
I0719 02:22:59.549826 9662 factory.go:89] Registering Raw factory
I0719 02:22:59.638468 9662 manager.go:946] Started watching for new ooms in manager
I0719 02:22:59.639054 9662 oomparser.go:183] oomparser using systemd
I0719 02:22:59.640589 9662 manager.go:243] Starting recovery of all containers
E0719 02:22:59.679107 9662 kubelet.go:787] Unable to construct api.Node object for kubelet: failed to get external ID from cloud provider: instance not found
I0719 02:22:59.753816 9662 manager.go:248] Recovery completed
I0719 02:22:59.816369 9662 status_manager.go:76] Starting to sync pod status with apiserver
I0719 02:22:59.816426 9662 kubelet.go:1725] Starting kubelet main sync loop.
E0719 02:22:59.953337 9662 kubelet.go:787] Unable to construct api.Node object for kubelet: failed to get external ID from cloud provider: instance not found
E0719 02:23:01.047929 9662 kubelet.go:787] Unable to construct api.Node object for kubelet: failed to get external ID from cloud provider: instance not found
E0719 02:23:01.975409 9662 kubelet.go:787] Unable to construct api.Node object for kubelet: failed to get external ID from cloud provider: instance not found
E0719 02:23:03.645707 9662 kubelet.go:787] Unable to construct api.Node object for kubelet: failed to get external ID from cloud provider: instance not found
E0719 02:23:06.917503 9662 kubelet.go:787] Unable to construct api.Node object for kubelet: failed to get external ID from cloud provider: instance not found
I0719 02:23:07.790941 9662 server.go:635] POST /stats/container/: (46.254087ms) 0 [[Go 1.1 package http] 10.0.39.222:45497]
kube-kubelet.service
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
[Service]
Environment="KUBERNETES_BINARY_VERSION=1.0.0"
EnvironmentFile=/etc/environment
ExecStartPre=/usr/bin/curl -L -o /opt/bin/kubelet https://storage.googleapis.com/kubernetes-release/release/v${KUBERNETES_BINARY_VERSION}/bin/linux/amd64
ExecStartPre=/usr/bin/chmod +x /opt/bin/kubelet
ExecStart=/opt/bin/kubelet \
--address=0.0.0.0 \
--port=10250 \
--cloud-provider=aws \
--hostname-override=${COREOS_PRIVATE_IPV4} \
--api-servers=${KUBE_MASTER_IP}:8080 \
--allow-privileged=false \
--cluster_dns=10.2.0.2 \
--cluster_domain=cluster.local \
--cadvisor_port=4194 \
--healthz_bind_address=0.0.0.0 \
--healthz_port=10248 \
--v=2 \
--logtostderr=true
Restart=always
RestartSec=10
[Install]
WantedBy=multi-user.target
iam policy
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ec2:Describe*"
],
"Resource": [
"*"
]
},
{
"Effect": "Allow",
"Action": [
"ec2:AttachVolume",
"ec2:DetachVolume"
],
"Resource": [
"arn:aws:ec2:*:*:instance/*"
]
},
]
}
I'm trying to set up AWS ELB attach volume access and keep getting an error on the kubelet. I have some bad feeling it might be related to #9801 as these nodes are CoreOS that were brought up by Terraform with custom service files but everything else is working fine (we're running cluster monitoring, dns, and a number of our own pods without issues). I've run awscli on that instance and the privileges definitely work. What am I doing wrong?
logs from kubelet service
kube-kubelet.service
iam policy
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "ec2:Describe*" ], "Resource": [ "*" ] }, { "Effect": "Allow", "Action": [ "ec2:AttachVolume", "ec2:DetachVolume" ], "Resource": [ "arn:aws:ec2:*:*:instance/*" ] }, ] }