Suite-k8s-1.18.K8sIdentity Identity expiration Expiration of CiliumIdentity
Initial triage suggests that this is a test pollution issue from Bookinfo tests, and is likely to affect other tests depending on test ordering.
https://jenkins.cilium.io/job/Cilium-PR-K8s-newest-kernel-4.9/318/testReport/junit/Suite-k8s-1/18/K8sIdentity_Identity_expiration_Expiration_of_CiliumIdentity/
test_results_Cilium-PR-K8s-newest-kernel-4.9_318_BDD-Test-PR.zip
Stacktrace
/home/jenkins/workspace/Cilium-PR-K8s-newest-kernel-4.9/k8s-1.18-gopath/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:387
Found 1 Cilium logs matching list of errors that must be investigated
/home/jenkins/workspace/Cilium-PR-K8s-newest-kernel-4.9/k8s-1.18-gopath/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:385
Standard Output
⚠️ Found a "JoinEP: " in logs
Number of "context deadline exceeded" in logs: 0
⚠️ Number of "level=error" in logs: 6
⚠️ Number of "level=warning" in logs: 50
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
Top 5 errors/warnings:
removing identity not added to the identity manager!
Error while rewriting endpoint BPF program
before filtering labels during regenerating restored endpoint
endpoint regeneration failed
Mutation detector is enabled, this will result in memory leakage.
Cilium pods: [cilium-lmkhb cilium-rpnbj]
Netpols loaded:
CiliumNetworkPolicies loaded:
Endpoint Policy Enforcement:
Pod Ingress Egress
hubble-cli-j5n24
hubble-cli-tw429
hubble-relay-58c8798f47-xhsgn
coredns-7964865f77-wxxxc
Cilium agent 'cilium-lmkhb': Status: Ok Health: Ok Nodes "" ContinerRuntime: Kubernetes: Ok KVstore: Ok Controllers: Total 26 Failed 0
Cilium agent 'cilium-rpnbj': Status: Ok Health: Ok Nodes "" ContinerRuntime: Kubernetes: Ok KVstore: Ok Controllers: Total 19 Failed 0
Standard Error
08:37:41 STEP: Running BeforeAll block for K8sIdentity
08:37:41 STEP: Ensuring the namespace kube-system exists
08:37:41 STEP: Deleting cep [details-v1-55cf6d74d-k98wn,reviews-v1-6948677f85-mnb8v,reviews-v2-77c4d85c6c-8kzqk] in namespace default
08:37:41 STEP: Waiting for 3 deletes to return (details-v1-55cf6d74d-k98wn,reviews-v1-6948677f85-mnb8v,reviews-v2-77c4d85c6c-8kzqk)
08:37:41 STEP: Unable to delete cep details-v1-55cf6d74d-k98wn with 'kubectl -n default delete cep details-v1-55cf6d74d-k98wn': Exitcode: 1
Stdout:
Stderr:
Error from server (NotFound): ciliumendpoints.cilium.io "details-v1-55cf6d74d-k98wn" not found
08:37:41 STEP: Installing Cilium
08:37:42 STEP: Waiting for Cilium to become ready
08:37:42 STEP: Cilium DaemonSet not ready yet: only 0 of 2 desired pods are ready
08:37:47 STEP: Cilium DaemonSet not ready yet: only 0 of 2 desired pods are ready
08:37:52 STEP: Cilium DaemonSet not ready yet: only 1 of 2 desired pods are ready
08:37:57 STEP: Cilium DaemonSet not ready yet: only 1 of 2 desired pods are ready
08:38:02 STEP: Number of ready Cilium pods: 2
08:38:02 STEP: Installing DNS Deployment
08:38:02 STEP: Restarting DNS Pods
08:38:09 STEP: Validating Cilium Installation
08:38:09 STEP: Performing Cilium controllers preflight check
08:38:09 STEP: Performing Cilium status preflight check
08:38:09 STEP: Performing Cilium health check
08:38:12 STEP: Performing Cilium service preflight check
08:38:12 STEP: Performing K8s service preflight check
08:38:12 STEP: Waiting for cilium-operator to be ready
08:38:12 STEP: Waiting for kube-dns to be ready
08:38:12 STEP: Running kube-dns preflight check
08:38:15 STEP: Performing K8s service preflight check
08:38:16 STEP: Making sure all endpoints are in ready state
08:38:17 STEP: Creating unused CiliumIdentity
08:38:17 STEP: Waiting for CiliumIdentity to be garbage collected
=== Test Finished at 2020-05-21T08:38:22Z====
08:38:22 STEP: Running JustAfterEach block for K8sIdentity
FAIL: Found 1 Cilium logs matching list of errors that must be investigated
===================== TEST FAILED =====================
08:38:22 STEP: Running AfterFailed block for K8sIdentity
cmd: kubectl get pods -o wide --all-namespaces
Exitcode: 0
Stdout:
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-system cilium-lmkhb 1/1 Running 0 41s 192.168.36.12 k8s2 <none> <none>
kube-system cilium-operator-6c758b54bf-68sbw 1/1 Running 0 41s 192.168.36.12 k8s2 <none> <none>
kube-system cilium-rpnbj 1/1 Running 0 41s 192.168.36.11 k8s1 <none> <none>
kube-system coredns-7964865f77-wxxxc 1/1 Running 0 21s 10.0.0.10 k8s2 <none> <none>
kube-system etcd-k8s1 1/1 Running 0 17m 192.168.36.11 k8s1 <none> <none>
kube-system hubble-cli-j5n24 0/1 CrashLoopBackOff 5 13m 10.0.1.131 k8s1 <none> <none>
kube-system hubble-cli-tw429 0/1 CrashLoopBackOff 5 13m 10.0.0.131 k8s2 <none> <none>
kube-system hubble-relay-58c8798f47-xhsgn 1/1 Running 0 13m 10.0.0.229 k8s2 <none> <none>
kube-system kube-apiserver-k8s1 1/1 Running 0 17m 192.168.36.11 k8s1 <none> <none>
kube-system kube-controller-manager-k8s1 1/1 Running 0 17m 192.168.36.11 k8s1 <none> <none>
kube-system kube-proxy-mbkzb 1/1 Running 0 17m 192.168.36.11 k8s1 <none> <none>
kube-system kube-proxy-szqxv 1/1 Running 0 16m 192.168.36.12 k8s2 <none> <none>
kube-system kube-scheduler-k8s1 1/1 Running 0 17m 192.168.36.11 k8s1 <none> <none>
kube-system log-gatherer-69z26 1/1 Running 0 15m 192.168.36.12 k8s2 <none> <none>
kube-system log-gatherer-wbfqv 1/1 Running 0 15m 192.168.36.11 k8s1 <none> <none>
kube-system registry-adder-9jr2j 1/1 Running 0 15m 192.168.36.11 k8s1 <none> <none>
kube-system registry-adder-ps2ql 1/1 Running 0 15m 192.168.36.12 k8s2 <none> <none>
Stderr:
Fetching command output from pods [cilium-lmkhb cilium-rpnbj]
cmd: kubectl exec -n kube-system cilium-lmkhb -- cilium endpoint list
Exitcode: 0
Stdout:
ENDPOINT POLICY (ingress) POLICY (egress) IDENTITY LABELS (source:key[=value]) IPv6 IPv4 STATUS
ENFORCEMENT ENFORCEMENT
2 Disabled Disabled 1 k8s:cilium.io/ci-node=k8s2 ready
reserved:host
177 Disabled Disabled 4 reserved:health fd00::98 10.0.0.197 ready
1017 Disabled Disabled 1797 k8s:io.cilium.k8s.policy.cluster=default fd00::b 10.0.0.10 ready
k8s:io.cilium.k8s.policy.serviceaccount=coredns
k8s:io.kubernetes.pod.namespace=kube-system
k8s:k8s-app=kube-dns
1623 Disabled Disabled 2044 k8s:io.cilium.k8s.policy.cluster=default fd00::7c 10.0.0.131 ready
k8s:io.cilium.k8s.policy.serviceaccount=default
k8s:io.kubernetes.pod.namespace=kube-system
k8s:k8s-app=hubble-cli
3562 Disabled Disabled 56769 k8s:io.cilium.k8s.policy.cluster=default fd00::82 10.0.0.229 ready
k8s:io.cilium.k8s.policy.serviceaccount=default
k8s:io.kubernetes.pod.namespace=kube-system
k8s:k8s-app=hubble-relay
Stderr:
cmd: kubectl exec -n kube-system cilium-rpnbj -- cilium endpoint list
Exitcode: 0
Stdout:
ENDPOINT POLICY (ingress) POLICY (egress) IDENTITY LABELS (source:key[=value]) IPv6 IPv4 STATUS
ENFORCEMENT ENFORCEMENT
269 Disabled Disabled 2044 k8s:io.cilium.k8s.policy.cluster=default fd00::172 10.0.1.131 ready
k8s:io.cilium.k8s.policy.serviceaccount=default
k8s:io.kubernetes.pod.namespace=kube-system
k8s:k8s-app=hubble-cli
1342 Disabled Disabled 1 k8s:cilium.io/ci-node=k8s1 ready
k8s:node-role.kubernetes.io/master
reserved:host
4048 Disabled Disabled 4 reserved:health fd00::16b 10.0.1.52 ready
Stderr:
===================== Exiting AfterFailed =====================
08:39:13 STEP: Running AfterEach for block EntireTestsuite
08:39:13 STEP: Running AfterAll block for K8sIdentity
Suite-k8s-1.18.K8sIdentity Identity expiration Expiration of CiliumIdentity
Initial triage suggests that this is a test pollution issue from Bookinfo tests, and is likely to affect other tests depending on test ordering.
https://jenkins.cilium.io/job/Cilium-PR-K8s-newest-kernel-4.9/318/testReport/junit/Suite-k8s-1/18/K8sIdentity_Identity_expiration_Expiration_of_CiliumIdentity/
test_results_Cilium-PR-K8s-newest-kernel-4.9_318_BDD-Test-PR.zip
Stacktrace
Standard Output
Standard Error