https://jenkins.cilium.io/job/cilium-ginkgo/job/cilium/job/master/5573/testReport/junit/Suite-k8s-1/12/K8sPolicyTest_GuestBook_Examples_checks_policy_example/
e60a837a_K8sPolicyTest_GuestBook_Examples_checks_policy_example.zip
Observed on master branch, v1.7 branch; coming up on 40 failures today. Seems like something in guestbook is either reliant on external infrastructure that is no longer available or otherwise configured with some kind of timebomb (expired cert, etc.).
Stacktrace
/home/jenkins/workspace/cilium-ginkgo_cilium_master/k8s-1.12-gopath/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:514
Cannot get web pods
Expected
<*errors.errorString | 0xc001b9ebc0>: {
s: "cannot retrieve pods: error: error executing jsonpath \"{range .items[*]}{@.metadata.name}{\\\"=\\\"}{@.spec.nodeName}{\\\"\\\\n\\\"}{end}\": Error executing template: not in range, nothing to end. Printing more information for debugging the template:\n\ttemplate was:\n\t\t{range .items[*]}{@.metadata.name}{\"=\"}{@.spec.nodeName}{\"\\n\"}{end}\n\tobject given to jsonpath engine was:\n\t\tmap[string]interface {}{\"apiVersion\":\"v1\", \"items\":[]interface {}{}, \"kind\":\"List\", \"metadata\":map[string]interface {}{\"resourceVersion\":\"\", \"selfLink\":\"\"}}\n\n\n",
}
to be nil
/home/jenkins/workspace/cilium-ginkgo_cilium_master/k8s-1.12-gopath/src/github.com/cilium/cilium/test/k8sT/Policies.go:1585
Standard Output
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Cilium pods: [cilium-ctcs5 cilium-mhwk6]
Netpols loaded: default::guestbook-policy-redis
CiliumNetworkPolicies loaded: default::guestbook-policy-web
Endpoint Policy Enforcement:
Pod Ingress Egress
frontend-c749f689b-m92c8
redis-master-t8r9v
redis-slave-9mltw
coredns-5544f57546-ts98r
grafana-559c9666fd-ngbwd
prometheus-7b5bd859df-k4m7c
Cilium agent 'cilium-ctcs5': Status: Ok Health: Ok Nodes "" ContinerRuntime: Kubernetes: Ok KVstore: Ok Controllers: Total 27 Failed 0
Cilium agent 'cilium-mhwk6': Status: Ok Health: Ok Nodes "" ContinerRuntime: Kubernetes: Ok KVstore: Ok Controllers: Total 26 Failed 0
Standard Error
20:35:07 STEP: Running BeforeEach block for EntireTestsuite K8sPolicyTest GuestBook Examples
20:35:08 STEP: WaitforPods(namespace="default", filter="-l zgroup=guestbook")
20:35:13 STEP: WaitforPods(namespace="default", filter="-l zgroup=guestbook") => <nil>
20:35:13 STEP: Apply policy to web
20:35:20 STEP: Apply policy to Redis
FAIL: Cannot get web pods
Expected
<*errors.errorString | 0xc001b9ebc0>: {
s: "cannot retrieve pods: error: error executing jsonpath \"{range .items[*]}{@.metadata.name}{\\\"=\\\"}{@.spec.nodeName}{\\\"\\\\n\\\"}{end}\": Error executing template: not in range, nothing to end. Printing more information for debugging the template:\n\ttemplate was:\n\t\t{range .items[*]}{@.metadata.name}{\"=\"}{@.spec.nodeName}{\"\\n\"}{end}\n\tobject given to jsonpath engine was:\n\t\tmap[string]interface {}{\"apiVersion\":\"v1\", \"items\":[]interface {}{}, \"kind\":\"List\", \"metadata\":map[string]interface {}{\"resourceVersion\":\"\", \"selfLink\":\"\"}}\n\n\n",
}
to be nil
=== Test Finished at 2020-08-27T20:35:21Z====
20:35:21 STEP: Running JustAfterEach block for EntireTestsuite K8sPolicyTest
===================== TEST FAILED =====================
20:35:22 STEP: Running AfterFailed block for EntireTestsuite K8sPolicyTest
cmd: kubectl get pods -o wide --all-namespaces
Exitcode: 0
Stdout:
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE
cilium-monitoring grafana-559c9666fd-ngbwd 0/1 Running 0 100m 10.0.0.34 k8s2 <none>
cilium-monitoring prometheus-7b5bd859df-k4m7c 1/1 Running 0 100m 10.0.0.172 k8s2 <none>
default frontend-c749f689b-m92c8 1/1 Running 0 17s 10.0.0.87 k8s2 <none>
default redis-master-t8r9v 1/1 Running 0 17s 10.0.1.163 k8s1 <none>
default redis-slave-9mltw 1/1 Running 0 17s 10.0.1.178 k8s1 <none>
kube-system cilium-ctcs5 1/1 Running 0 2m10s 192.168.36.12 k8s2 <none>
kube-system cilium-mhwk6 1/1 Running 0 2m10s 192.168.36.11 k8s1 <none>
kube-system cilium-operator-fccff88f7-n95vq 1/1 Running 0 2m10s 192.168.36.13 k8s3 <none>
kube-system coredns-5544f57546-ts98r 1/1 Running 0 30m 10.0.0.66 k8s2 <none>
kube-system etcd-k8s1 1/1 Running 0 107m 192.168.36.11 k8s1 <none>
kube-system kube-apiserver-k8s1 1/1 Running 0 107m 192.168.36.11 k8s1 <none>
kube-system kube-controller-manager-k8s1 1/1 Running 0 108m 192.168.36.11 k8s1 <none>
kube-system kube-scheduler-k8s1 1/1 Running 0 108m 192.168.36.11 k8s1 <none>
kube-system log-gatherer-bwgjt 1/1 Running 0 100m 192.168.36.13 k8s3 <none>
kube-system log-gatherer-fvpzp 1/1 Running 0 100m 192.168.36.11 k8s1 <none>
kube-system log-gatherer-l2kzh 1/1 Running 0 100m 192.168.36.12 k8s2 <none>
kube-system registry-adder-k7xpv 1/1 Running 0 106m 192.168.36.12 k8s2 <none>
kube-system registry-adder-vd8tp 1/1 Running 0 106m 192.168.36.11 k8s1 <none>
kube-system registry-adder-z8kzb 1/1 Running 0 106m 192.168.36.13 k8s3 <none>
Stderr:
Fetching command output from pods [cilium-ctcs5 cilium-mhwk6]
cmd: kubectl exec -n kube-system cilium-ctcs5 -- cilium service list
Exitcode: 0
Stdout:
ID Frontend Service Type Backend
1 10.108.48.190:3000 ClusterIP
2 10.109.108.136:9090 ClusterIP 1 => 10.0.0.172:9090
3 10.96.0.1:443 ClusterIP 1 => 192.168.36.11:6443
4 10.96.0.10:53 ClusterIP 1 => 10.0.0.66:53
95 10.104.42.229:6379 ClusterIP 1 => 10.0.1.163:6379
96 10.100.159.139:6379 ClusterIP 1 => 10.0.1.178:6379
Stderr:
cmd: kubectl exec -n kube-system cilium-ctcs5 -- cilium endpoint list
Exitcode: 0
Stdout:
ENDPOINT POLICY (ingress) POLICY (egress) IDENTITY LABELS (source:key[=value]) IPv6 IPv4 STATUS
ENFORCEMENT ENFORCEMENT
560 Disabled Disabled 51634 k8s:io.cilium.k8s.policy.cluster=default fd00::66 10.0.0.66 ready
k8s:io.cilium.k8s.policy.serviceaccount=coredns
k8s:io.kubernetes.pod.namespace=kube-system
k8s:k8s-app=kube-dns
1521 Enabled Disabled 17289 k8s:app=guestbook fd00::7e 10.0.0.87 ready
k8s:io.cilium.k8s.policy.cluster=default
k8s:io.cilium.k8s.policy.serviceaccount=default
k8s:io.kubernetes.pod.namespace=default
k8s:tier=frontend
2147 Disabled Disabled 4 reserved:health fd00::46 10.0.0.64 ready
3986 Disabled Disabled 1 k8s:cilium.io/ci-node=k8s2 ready
reserved:host
Stderr:
cmd: kubectl exec -n kube-system cilium-mhwk6 -- cilium service list
Exitcode: 0
Stdout:
ID Frontend Service Type Backend
1 10.96.0.1:443 ClusterIP 1 => 192.168.36.11:6443
2 10.96.0.10:53 ClusterIP 1 => 10.0.0.66:53
3 10.108.48.190:3000 ClusterIP
4 10.109.108.136:9090 ClusterIP 1 => 10.0.0.172:9090
86 192.168.36.11:40000 HostPort 1 => 10.0.1.79:8080
87 10.0.2.15:40000 HostPort 1 => 10.0.1.79:8080
88 0.0.0.0:40000 HostPort 1 => 10.0.1.79:8080
89 10.104.42.229:6379 ClusterIP 1 => 10.0.1.163:6379
90 10.100.159.139:6379 ClusterIP 1 => 10.0.1.178:6379
Stderr:
cmd: kubectl exec -n kube-system cilium-mhwk6 -- cilium endpoint list
Exitcode: 0
Stdout:
ENDPOINT POLICY (ingress) POLICY (egress) IDENTITY LABELS (source:key[=value]) IPv6 IPv4 STATUS
ENFORCEMENT ENFORCEMENT
220 Enabled Disabled 50389 k8s:io.cilium.k8s.policy.cluster=default fd00::1bf 10.0.1.163 ready
k8s:io.cilium.k8s.policy.serviceaccount=default
k8s:io.kubernetes.pod.namespace=default
k8s:k8s-app.guestbook=redis
k8s:role=master
k8s:zgroup=guestbook
539 Disabled Disabled 1 k8s:cilium.io/ci-node=k8s1 ready
k8s:node-role.kubernetes.io/master
reserved:host
862 Disabled Disabled 4 reserved:health fd00::1fb 10.0.1.182 ready
1255 Enabled Disabled 42266 k8s:io.cilium.k8s.policy.cluster=default fd00::18e 10.0.1.178 ready
k8s:io.cilium.k8s.policy.serviceaccount=default
k8s:io.kubernetes.pod.namespace=default
k8s:k8s-app.guestbook=redis
k8s:role=slave
k8s:zgroup=guestbook
Stderr:
===================== Exiting AfterFailed =====================
20:36:02 STEP: Running AfterEach for block EntireTestsuite K8sPolicyTest GuestBook Examples
FAIL: Redis deprecated policy cannot be deleted
Expected command: kubectl delete -f /home/jenkins/workspace/cilium-ginkgo_cilium_master/k8s-1.12-gopath/src/github.com/cilium/cilium/test/k8sT/manifests/guestbook-policy-redis-deprecated.json
To succeed, but it failed:
Exitcode: 1
Err: exit status 1
Stdout:
Stderr:
Error from server (NotFound): error when deleting "/home/jenkins/workspace/cilium-ginkgo_cilium_master/k8s-1.12-gopath/src/github.com/cilium/cilium/test/k8sT/manifests/guestbook-policy-redis-deprecated.json": networkpolicies.extensions "guestbook-redis-deprecated" not found
20:36:07 STEP: Running AfterEach for block EntireTestsuite K8sPolicyTest
20:36:07 STEP: Running AfterEach for block EntireTestsuite
https://jenkins.cilium.io/job/cilium-ginkgo/job/cilium/job/master/5573/testReport/junit/Suite-k8s-1/12/K8sPolicyTest_GuestBook_Examples_checks_policy_example/
e60a837a_K8sPolicyTest_GuestBook_Examples_checks_policy_example.zip
Observed on master branch, v1.7 branch; coming up on 40 failures today. Seems like something in guestbook is either reliant on external infrastructure that is no longer available or otherwise configured with some kind of timebomb (expired cert, etc.).
Stacktrace
Standard Output
Standard Error