Is this a request for help?
No
What keywords did you search in Kubernetes issues before filing this one?
panic, findAndRemoveDeletedPods
Is this a BUG REPORT or FEATURE REQUEST?
BUG REPORT
Kubernetes version (use kubectl version): 1.5.1
Environment:
- Cloud provider or hardware configuration: GKE
- OS (e.g. from /etc/os-release): gci 56 (8938.0.0)
- Kernel (e.g.
uname -a): Linux gke-xxx 4.4.21+ #1 SMP Thu Oct 27 02:50:36 PDT 2016 x86_64 Intel(R) Xeon(R) CPU @ 2.30GHz GenuineIntel GNU/Linux
- Install tools:
- Others:
What happened:
I created a pod and it went into the failed state. I couldn't find any info about the failure, so I logged into the GKE node and ran "journalctl -u kubelet" and saw:
Dec 29 20:20:04 gke-xxx kubelet[1654]: I1229 20:20:04.906839 1654 kubelet.go:1816] SyncLoop (PLEG): "restore-bigeasy-02-b_grosskur(cfb7edba-ce03-11e6-9c17-42010af00156)", event: &pleg.PodLifecycleEvent{ID:"cfb7edba-ce03-11e6-9c17-42010af00156", Type:"ContainerDied", Data:"8a12a6c754e5cc7c7efe826a5af5d7376ec5557bb0648a5abe830a57c1843027"}
Dec 29 20:20:04 gke-xxx kubelet[1654]: I1229 20:20:04.968938 1654 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/cfb7edba-ce03-11e6-9c17-42010af00156-default-token-5llx5" (spec.Name: "default-token-5llx5") pod "cfb7edba-ce03-11e6-9c17-42010af00156" (UID: "cfb7edba-ce03-11e6-9c17-42010af00156").
Dec 29 20:20:05 gke-xxx kubelet[1654]: I1229 20:20:05.087282 1654 conversion.go:134] failed to handle multiple devices for container. Skipping Filesystem stats
Dec 29 20:20:05 gke-xxx kubelet[1654]: I1229 20:20:05.087316 1654 conversion.go:134] failed to handle multiple devices for container. Skipping Filesystem stats
Dec 29 20:20:05 gke-xxx kubelet[1654]: I1229 20:20:05.093216 1654 server.go:740] GET /stats/summary/: (6.167919ms) 200 [[Go-http-client/1.1] 10.52.6.7:42062]
Dec 29 20:20:05 gke-xxx kubelet[1654]: I1229 20:20:05.917924 1654 kubelet.go:1816] SyncLoop (PLEG): "restore-bigeasy-02-b_grosskur(cfb7edba-ce03-11e6-9c17-42010af00156)", event: &pleg.PodLifecycleEvent{ID:"cfb7edba-ce03-11e6-9c17-42010af00156", Type:"ContainerStarted", Data:"c1221640498373252af613914b3df52f0ebf867c78916a7e2c970296d2f98966"}
Dec 29 20:20:05 gke-xxx kubelet[1654]: I1229 20:20:05.972091 1654 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/cfb7edba-ce03-11e6-9c17-42010af00156-default-token-5llx5" (spec.Name: "default-token-5llx5") pod "cfb7edba-ce03-11e6-9c17-42010af00156" (UID: "cfb7edba-ce03-11e6-9c17-42010af00156").
Dec 29 20:20:06 gke-xxx kubelet[1654]: I1229 20:20:06.928592 1654 kubelet.go:1816] SyncLoop (PLEG): "restore-bigeasy-02-b_grosskur(cfb7edba-ce03-11e6-9c17-42010af00156)", event: &pleg.PodLifecycleEvent{ID:"cfb7edba-ce03-11e6-9c17-42010af00156", Type:"ContainerDied", Data:"c1221640498373252af613914b3df52f0ebf867c78916a7e2c970296d2f98966"}
Dec 29 20:20:06 gke-xxx kubelet[1654]: E1229 20:20:06.949922 1654 runtime.go:64] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
Dec 29 20:20:06 gke-xxx kubelet[1654]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/runtime/runtime.go:70
Dec 29 20:20:06 gke-xxx kubelet[1654]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/runtime/runtime.go:63
Dec 29 20:20:06 gke-xxx kubelet[1654]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/runtime/runtime.go:49
Dec 29 20:20:06 gke-xxx kubelet[1654]: /usr/local/go/src/runtime/asm_amd64.s:479
Dec 29 20:20:06 gke-xxx kubelet[1654]: /usr/local/go/src/runtime/panic.go:458
Dec 29 20:20:06 gke-xxx kubelet[1654]: /usr/local/go/src/runtime/panic.go:62
Dec 29 20:20:06 gke-xxx kubelet[1654]: /usr/local/go/src/runtime/sigpanic_unix.go:24
Dec 29 20:20:06 gke-xxx kubelet[1654]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/volumemanager/populator/desired_state_of_world_populator.go:163
Dec 29 20:20:06 gke-xxx kubelet[1654]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/volumemanager/populator/desired_state_of_world_populator.go:128
Dec 29 20:20:06 gke-xxx kubelet[1654]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:96
Dec 29 20:20:06 gke-xxx kubelet[1654]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:97
Dec 29 20:20:06 gke-xxx kubelet[1654]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:52
Dec 29 20:20:06 gke-xxx kubelet[1654]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/volumemanager/populator/desired_state_of_world_populator.go:102
Dec 29 20:20:06 gke-xxx kubelet[1654]: /usr/local/go/src/runtime/asm_amd64.s:2086
Dec 29 20:20:06 gke-xxx kubelet[1654]: I1229 20:20:06.975942 1654 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/cfb7edba-ce03-11e6-9c17-42010af00156-default-token-5llx5" (spec.Name: "default-token-5llx5") pod "cfb7edba-ce03-11e6-9c17-42010af00156" (UID: "cfb7edba-ce03-11e6-9c17-42010af00156").
Dec 29 20:20:07 gke-xxx kubelet[1654]: E1229 20:20:07.050393 1654 runtime.go:64] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
Dec 29 20:20:07 gke-xxx kubelet[1654]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/runtime/runtime.go:70
Dec 29 20:20:07 gke-xxx kubelet[1654]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/runtime/runtime.go:63
Dec 29 20:20:07 gke-xxx kubelet[1654]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/runtime/runtime.go:49
Dec 29 20:20:07 gke-xxx kubelet[1654]: /usr/local/go/src/runtime/asm_amd64.s:479
Dec 29 20:20:07 gke-xxx kubelet[1654]: /usr/local/go/src/runtime/panic.go:458
Dec 29 20:20:07 gke-xxx kubelet[1654]: /usr/local/go/src/runtime/panic.go:62
Dec 29 20:20:07 gke-xxx kubelet[1654]: /usr/local/go/src/runtime/sigpanic_unix.go:24
Dec 29 20:20:07 gke-xxx kubelet[1654]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/volumemanager/populator/desired_state_of_world_populator.go:163
Dec 29 20:20:07 gke-xxx kubelet[1654]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/volumemanager/populator/desired_state_of_world_populator.go:128
Dec 29 20:20:07 gke-xxx kubelet[1654]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:96
Dec 29 20:20:07 gke-xxx kubelet[1654]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:97
Dec 29 20:20:07 gke-xxx kubelet[1654]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:52
Dec 29 20:20:07 gke-xxx kubelet[1654]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/volumemanager/populator/desired_state_of_world_populator.go:102
Dec 29 20:20:07 gke-xxx kubelet[1654]: /usr/local/go/src/runtime/asm_amd64.s:2086
When I tail the log, the invalid memory address or nil pointer dereference error messages continue indefinitely with multiple messages printed every second.
Digging into pkg/kubelet/volumemanager/populator/desired_state_of_world_populator.go#L163 I see:
if (volume.EmptyDir == nil || volume.EmptyDir.Medium != api.StorageMediumMemory) &&
So I'm wondering if perhaps there is a bug where volume is nil in some cases.
What you expected to happen:
I did not expect to see the kublet log full of "invalid memory address or nil pointer dereference" errors.
How to reproduce it (as minimally and precisely as possible):
Sorry, I don't have a good way to reproduce this error.
Anything else do we need to know:
I'm running several pods, each with a GCE persistent disk mounted. When the pods finish, I run a different set of pods that mount the same persistent disks. I've seen various issues in past K8S versions where persistent disks don't get detached and reattached properly across K8S nodes, but so far with 1.5.1 it seems to have been working well up until now.
Is this a request for help?
No
What keywords did you search in Kubernetes issues before filing this one?
panic, findAndRemoveDeletedPods
Is this a BUG REPORT or FEATURE REQUEST?
BUG REPORT
Kubernetes version (use
kubectl version): 1.5.1Environment:
uname -a):Linux gke-xxx 4.4.21+ #1 SMP Thu Oct 27 02:50:36 PDT 2016 x86_64 Intel(R) Xeon(R) CPU @ 2.30GHz GenuineIntel GNU/LinuxWhat happened:
I created a pod and it went into the failed state. I couldn't find any info about the failure, so I logged into the GKE node and ran "journalctl -u kubelet" and saw:
When I tail the log, the
invalid memory address or nil pointer dereferenceerror messages continue indefinitely with multiple messages printed every second.Digging into pkg/kubelet/volumemanager/populator/desired_state_of_world_populator.go#L163 I see:
So I'm wondering if perhaps there is a bug where
volumeisnilin some cases.What you expected to happen:
I did not expect to see the kublet log full of "invalid memory address or nil pointer dereference" errors.
How to reproduce it (as minimally and precisely as possible):
Sorry, I don't have a good way to reproduce this error.
Anything else do we need to know:
I'm running several pods, each with a GCE persistent disk mounted. When the pods finish, I run a different set of pods that mount the same persistent disks. I've seen various issues in past K8S versions where persistent disks don't get detached and reattached properly across K8S nodes, but so far with 1.5.1 it seems to have been working well up until now.