Skip to content

CI: k8s-1.7.K8sValidatedPolicyTest KubernetesNetworkPolicy between server and client should enforce policy based on NamespaceSelector flake #3901

Closed
Closed
CI: k8s-1.7.K8sValidatedPolicyTest KubernetesNetworkPolicy between server and client should enforce policy based on NamespaceSelector flake#3901
Labels
kind/bug/CIThis is a bug in the testing code.needs/triageThis issue requires triaging to establish severity and next steps.
@joestringer

Description

@joestringer
joestringer
opened on Apr 25, 2018

Summary

In the KubernetesNetworkPolicy test, the pod did not successfully exit.

Stacktrace

/home/jenkins/workspace/Cilium-PR-Ginkgo-Tests-Validated/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:312
Pod did not finish as expected.
Expected an error to have occurred.  Got:
    <nil>: nil
/home/jenkins/workspace/Cilium-PR-Ginkgo-Tests-Validated/src/github.com/cilium/cilium/test/k8sT/Policies.go:861

Standard Output

�[1mSTEP�[0m: Creating the namespace that will be used for the pods
time="2018-04-25T19:31:10Z" level=debug msg=HTTPRequest data="{\"kind\":\"Namespace\",\"apiVersion\":\"v1\",\"metadata\":{\"name\":\"namespace-selector-test\",\"creationTimestamp\":null},\"spec\":{},\"status\":{}}\n" method=POST result="201 Created" testName=K8sPolicyTest to="https://192.168.36.11:6443/api/v1/namespaces"
�[1mSTEP�[0m: Creating a simple server that serves on port 80 and 81.
�[1mSTEP�[0m: Creating a server pod server in namespace namespace-selector-test
time="2018-04-25T19:31:10Z" level=debug msg=HTTPRequest data="{\"kind\":\"Pod\",\"apiVersion\":\"v1\",\"metadata\":{\"name\":\"server\",\"creationTimestamp\":null,\"labels\":{\"pod-name\":\"server\"}},\"spec\":{\"containers\":[{\"name\":\"server-container-80\",\"image\":\"gcr.io/kubernetes-e2e-test-images/porter-amd64:1.0\",\"ports\":[{\"name\":\"serve-80\",\"containerPort\":80}],\"env\":[{\"name\":\"SERVE_PORT_80\",\"value\":\"foo\"}],\"resources\":{},\"readinessProbe\":{\"httpGet\":{\"path\":\"/\",\"port\":80,\"scheme\":\"HTTP\"}}},{\"name\":\"server-container-81\",\"image\":\"gcr.io/kubernetes-e2e-test-images/porter-amd64:1.0\",\"ports\":[{\"name\":\"serve-81\",\"containerPort\":81}],\"env\":[{\"name\":\"SERVE_PORT_81\",\"value\":\"foo\"}],\"resources\":{},\"readinessProbe\":{\"httpGet\":{\"path\":\"/\",\"port\":81,\"scheme\":\"HTTP\"}}}],\"restartPolicy\":\"Never\"},\"status\":{}}\n" method=POST result="201 Created" testName=K8sPolicyTest to="https://192.168.36.11:6443/api/v1/namespaces/namespace-selector-test/pods"
�[1mSTEP�[0m: Creating a service svc-server for pod server in namespace namespace-selector-test
time="2018-04-25T19:31:10Z" level=debug msg=HTTPRequest data="{\"kind\":\"Service\",\"apiVersion\":\"v1\",\"metadata\":{\"name\":\"svc-server\",\"creationTimestamp\":null},\"spec\":{\"ports\":[{\"name\":\"server-80\",\"port\":80,\"targetPort\":80},{\"name\":\"server-81\",\"port\":81,\"targetPort\":81}],\"selector\":{\"pod-name\":\"server\"}},\"status\":{\"loadBalancer\":{}}}\n" method=POST result="201 Created" testName=K8sPolicyTest to="https://192.168.36.11:6443/api/v1/namespaces/namespace-selector-test/services"
�[1mSTEP�[0m: Waiting for pod ready
time="2018-04-25T19:31:11Z" level=debug msg=HTTPRequest data="<empty>" method=GET result="200 OK" testName=K8sPolicyTest to="https://192.168.36.11:6443/api/v1/namespaces/namespace-selector-test/pods/server"
time="2018-04-25T19:31:13Z" level=debug msg=HTTPRequest data="<empty>" method=GET result="200 OK" testName=K8sPolicyTest to="https://192.168.36.11:6443/api/v1/namespaces/namespace-selector-test/pods/server"
time="2018-04-25T19:31:13Z" level=debug msg=HTTPRequest data="<empty>" method=GET result="200 OK" testName=K8sPolicyTest to="https://192.168.36.11:6443/api/v1/namespaces/namespace-selector-test/pods/server"
time="2018-04-25T19:31:14Z" level=debug msg=HTTPRequest data="<empty>" method=GET result="200 OK" testName=K8sPolicyTest to="https://192.168.36.11:6443/api/v1/namespaces/namespace-selector-test/pods/server"
time="2018-04-25T19:31:15Z" level=debug msg=HTTPRequest data="<empty>" method=GET result="200 OK" testName=K8sPolicyTest to="https://192.168.36.11:6443/api/v1/namespaces/namespace-selector-test/pods/server"
time="2018-04-25T19:31:16Z" level=debug msg=HTTPRequest data="<empty>" method=GET result="200 OK" testName=K8sPolicyTest to="https://192.168.36.11:6443/api/v1/namespaces/namespace-selector-test/pods/server"
time="2018-04-25T19:31:17Z" level=debug msg=HTTPRequest data="<empty>" method=GET result="200 OK" testName=K8sPolicyTest to="https://192.168.36.11:6443/api/v1/namespaces/namespace-selector-test/pods/server"
time="2018-04-25T19:31:18Z" level=debug msg=HTTPRequest data="<empty>" method=GET result="200 OK" testName=K8sPolicyTest to="https://192.168.36.11:6443/api/v1/namespaces/namespace-selector-test/pods/server"
time="2018-04-25T19:31:19Z" level=debug msg=HTTPRequest data="<empty>" method=GET result="200 OK" testName=K8sPolicyTest to="https://192.168.36.11:6443/api/v1/namespaces/namespace-selector-test/pods/server"
time="2018-04-25T19:31:20Z" level=debug msg=HTTPRequest data="<empty>" method=GET result="200 OK" testName=K8sPolicyTest to="https://192.168.36.11:6443/api/v1/namespaces/namespace-selector-test/pods/server"
time="2018-04-25T19:31:21Z" level=debug msg=HTTPRequest data="<empty>" method=GET result="200 OK" testName=K8sPolicyTest to="https://192.168.36.11:6443/api/v1/namespaces/namespace-selector-test/pods/server"
�[1mSTEP�[0m: Testing pods can connect to both ports when no policy is present.
time="2018-04-25T19:31:21Z" level=debug msg=HTTPRequest data="{\"kind\":\"Pod\",\"apiVersion\":\"v1\",\"metadata\":{\"name\":\"client-can-connect-80\",\"creationTimestamp\":null,\"labels\":{\"pod-name\":\"client-can-connect-80\"}},\"spec\":{\"containers\":[{\"name\":\"client-can-connect-80-container\",\"image\":\"busybox\",\"args\":[\"/bin/sh\",\"-c\",\"for i in $(seq 1 5); do wget -T 8 svc-server.namespace-selector-test:80 -O - \\u0026\\u0026 exit 0 || sleep 1; done; exit 1\"],\"resources\":{}}],\"restartPolicy\":\"Never\"},\"status\":{}}\n" method=POST result="201 Created" testName=K8sPolicyTest to="https://192.168.36.11:6443/api/v1/namespaces/namespace-selector-test/pods"
time="2018-04-25T19:31:22Z" level=debug msg=HTTPRequest data="<empty>" method=GET result="200 OK" testName=K8sPolicyTest to="https://192.168.36.11:6443/api/v1/namespaces/namespace-selector-test/pods/client-can-connect-80"
time="2018-04-25T19:31:23Z" level=debug msg=HTTPRequest data="<empty>" method=GET result="200 OK" testName=K8sPolicyTest to="https://192.168.36.11:6443/api/v1/namespaces/namespace-selector-test/pods/client-can-connect-80"
time="2018-04-25T19:31:24Z" level=debug msg=HTTPRequest data="<empty>" method=GET result="200 OK" testName=K8sPolicyTest to="https://192.168.36.11:6443/api/v1/namespaces/namespace-selector-test/pods/client-can-connect-80"
time="2018-04-25T19:31:25Z" level=debug msg=HTTPRequest data="<empty>" method=GET result="200 OK" testName=K8sPolicyTest to="https://192.168.36.11:6443/api/v1/namespaces/namespace-selector-test/pods/client-can-connect-80"
time="2018-04-25T19:31:27Z" level=debug msg=HTTPRequest data="<empty>" method=GET result="200 OK" testName=K8sPolicyTest to="https://192.168.36.11:6443/api/v1/namespaces/namespace-selector-test/pods/client-can-connect-80"
time="2018-04-25T19:31:27Z" level=debug msg=HTTPRequest data="<empty>" method=GET result="200 OK" testName=K8sPolicyTest to="https://192.168.36.11:6443/api/v1/namespaces/namespace-selector-test/pods/client-can-connect-80"
time="2018-04-25T19:31:28Z" level=debug msg=HTTPRequest data="<empty>" method=GET result="200 OK" testName=K8sPolicyTest to="https://192.168.36.11:6443/api/v1/namespaces/namespace-selector-test/pods/client-can-connect-80"
time="2018-04-25T19:31:29Z" level=debug msg=HTTPRequest data="<empty>" method=GET result="200 OK" testName=K8sPolicyTest to="https://192.168.36.11:6443/api/v1/namespaces/namespace-selector-test/pods/client-can-connect-80"
time="2018-04-25T19:31:30Z" level=debug msg=HTTPRequest data="<empty>" method=GET result="200 OK" testName=K8sPolicyTest to="https://192.168.36.11:6443/api/v1/namespaces/namespace-selector-test/pods/client-can-connect-80"
time="2018-04-25T19:31:31Z" level=debug msg=HTTPRequest data="<empty>" method=GET result="200 OK" testName=K8sPolicyTest to="https://192.168.36.11:6443/api/v1/namespaces/namespace-selector-test/pods/client-can-connect-80"
time="2018-04-25T19:31:32Z" level=debug msg=HTTPRequest data="<empty>" method=GET result="200 OK" testName=K8sPolicyTest to="https://192.168.36.11:6443/api/v1/namespaces/namespace-selector-test/pods/client-can-connect-80"
�[1mSTEP�[0m: Cleaning up the pod client-can-connect-80
time="2018-04-25T19:31:33Z" level=debug msg=HTTPRequest data="<empty>" method=DELETE result="200 OK" testName=K8sPolicyTest to="https://192.168.36.11:6443/api/v1/namespaces/namespace-selector-test/pods/client-can-connect-80"
time="2018-04-25T19:31:33Z" level=debug msg=HTTPRequest data="{\"kind\":\"Pod\",\"apiVersion\":\"v1\",\"metadata\":{\"name\":\"client-can-connect-81\",\"creationTimestamp\":null,\"labels\":{\"pod-name\":\"client-can-connect-81\"}},\"spec\":{\"containers\":[{\"name\":\"client-can-connect-81-container\",\"image\":\"busybox\",\"args\":[\"/bin/sh\",\"-c\",\"for i in $(seq 1 5); do wget -T 8 svc-server.namespace-selector-test:81 -O - \\u0026\\u0026 exit 0 || sleep 1; done; exit 1\"],\"resources\":{}}],\"restartPolicy\":\"Never\"},\"status\":{}}\n" method=POST result="201 Created" testName=K8sPolicyTest to="https://192.168.36.11:6443/api/v1/namespaces/namespace-selector-test/pods"
time="2018-04-25T19:31:34Z" level=debug msg=HTTPRequest data="<empty>" method=GET result="200 OK" testName=K8sPolicyTest to="https://192.168.36.11:6443/api/v1/namespaces/namespace-selector-test/pods/client-can-connect-81"
time="2018-04-25T19:31:35Z" level=debug msg=HTTPRequest data="<empty>" method=GET result="200 OK" testName=K8sPolicyTest to="https://192.168.36.11:6443/api/v1/namespaces/namespace-selector-test/pods/client-can-connect-81"
time="2018-04-25T19:31:36Z" level=debug msg=HTTPRequest data="<empty>" method=GET result="200 OK" testName=K8sPolicyTest to="https://192.168.36.11:6443/api/v1/namespaces/namespace-selector-test/pods/client-can-connect-81"
time="2018-04-25T19:31:38Z" level=debug msg=HTTPRequest data="<empty>" method=GET result="200 OK" testName=K8sPolicyTest to="https://192.168.36.11:6443/api/v1/namespaces/namespace-selector-test/pods/client-can-connect-81"
time="2018-04-25T19:31:38Z" level=debug msg=HTTPRequest data="<empty>" method=GET result="200 OK" testName=K8sPolicyTest to="https://192.168.36.11:6443/api/v1/namespaces/namespace-selector-test/pods/client-can-connect-81"
time="2018-04-25T19:31:39Z" level=debug msg=HTTPRequest data="<empty>" method=GET result="200 OK" testName=K8sPolicyTest to="https://192.168.36.11:6443/api/v1/namespaces/namespace-selector-test/pods/client-can-connect-81"
time="2018-04-25T19:31:40Z" level=debug msg=HTTPRequest data="<empty>" method=GET result="200 OK" testName=K8sPolicyTest to="https://192.168.36.11:6443/api/v1/namespaces/namespace-selector-test/pods/client-can-connect-81"
time="2018-04-25T19:31:41Z" level=debug msg=HTTPRequest data="<empty>" method=GET result="200 OK" testName=K8sPolicyTest to="https://192.168.36.11:6443/api/v1/namespaces/namespace-selector-test/pods/client-can-connect-81"
time="2018-04-25T19:31:42Z" level=debug msg=HTTPRequest data="<empty>" method=GET result="200 OK" testName=K8sPolicyTest to="https://192.168.36.11:6443/api/v1/namespaces/namespace-selector-test/pods/client-can-connect-81"
time="2018-04-25T19:31:43Z" level=debug msg=HTTPRequest data="<empty>" method=GET result="200 OK" testName=K8sPolicyTest to="https://192.168.36.11:6443/api/v1/namespaces/namespace-selector-test/pods/client-can-connect-81"
time="2018-04-25T19:31:44Z" level=debug msg=HTTPRequest data="<empty>" method=GET result="200 OK" testName=K8sPolicyTest to="https://192.168.36.11:6443/api/v1/namespaces/namespace-selector-test/pods/client-can-connect-81"
time="2018-04-25T19:31:45Z" level=debug msg=HTTPRequest data="<empty>" method=GET result="200 OK" testName=K8sPolicyTest to="https://192.168.36.11:6443/api/v1/namespaces/namespace-selector-test/pods/client-can-connect-81"
time="2018-04-25T19:31:46Z" level=debug msg=HTTPRequest data="<empty>" method=GET result="200 OK" testName=K8sPolicyTest to="https://192.168.36.11:6443/api/v1/namespaces/namespace-selector-test/pods/client-can-connect-81"
�[1mSTEP�[0m: Cleaning up the pod client-can-connect-81
time="2018-04-25T19:31:46Z" level=debug msg=HTTPRequest data="<empty>" method=DELETE result="200 OK" testName=K8sPolicyTest to="https://192.168.36.11:6443/api/v1/namespaces/namespace-selector-test/pods/client-can-connect-81"
time="2018-04-25T19:31:46Z" level=debug msg="applying https://raw.githubusercontent.com/cilium/microscope/master/docs/microscope.yaml" testName=K8sPolicyTest
time="2018-04-25T19:31:46Z" level=debug msg="running command: kubectl apply -f  https://raw.githubusercontent.com/cilium/microscope/master/docs/microscope.yaml"
time="2018-04-25T19:31:52Z" level=debug msg="running command: kubectl -n kube-system get pods -l k8s-app=microscope -o json"
time="2018-04-25T19:31:52Z" level=debug msg=HTTPRequest data="{\"kind\":\"NetworkPolicy\",\"apiVersion\":\"networking.k8s.io/v1\",\"metadata\":{\"name\":\"allow-ns-b-via-namespace-selector\",\"creationTimestamp\":null},\"spec\":{\"podSelector\":{\"matchLabels\":{\"pod-name\":\"server\"}},\"ingress\":[{\"from\":[{\"namespaceSelector\":{\"matchLabels\":{\"ns-name\":\"namespace-selector-test\"}}}]}]}}\n" method=POST result="201 Created" testName=K8sPolicyTest to="https://192.168.36.11:6443/apis/networking.k8s.io/v1/namespaces/namespace-selector-test/networkpolicies"
time="2018-04-25T19:31:52Z" level=debug msg=HTTPRequest data="{\"kind\":\"Pod\",\"apiVersion\":\"v1\",\"metadata\":{\"name\":\"client-cannot-connect\",\"creationTimestamp\":null,\"labels\":{\"pod-name\":\"client-cannot-connect\"}},\"spec\":{\"containers\":[{\"name\":\"client-cannot-connect-container\",\"image\":\"busybox\",\"args\":[\"/bin/sh\",\"-c\",\"for i in $(seq 1 5); do wget -T 8 svc-server.namespace-selector-test:80 -O - \\u0026\\u0026 exit 0 || sleep 1; done; exit 1\"],\"resources\":{}}],\"restartPolicy\":\"Never\"},\"status\":{}}\n" method=POST result="201 Created" testName=K8sPolicyTest to="https://192.168.36.11:6443/api/v1/namespaces/namespace-selector-test/pods"
time="2018-04-25T19:31:53Z" level=debug msg=HTTPRequest data="<empty>" method=GET result="200 OK" testName=K8sPolicyTest to="https://192.168.36.11:6443/api/v1/namespaces/namespace-selector-test/pods/client-cannot-connect"
time="2018-04-25T19:31:54Z" level=debug msg=HTTPRequest data="<empty>" method=GET result="200 OK" testName=K8sPolicyTest to="https://192.168.36.11:6443/api/v1/namespaces/namespace-selector-test/pods/client-cannot-connect"
time="2018-04-25T19:31:55Z" level=debug msg=HTTPRequest data="<empty>" method=GET result="200 OK" testName=K8sPolicyTest to="https://192.168.36.11:6443/api/v1/namespaces/namespace-selector-test/pods/client-cannot-connect"
time="2018-04-25T19:31:56Z" level=debug msg=HTTPRequest data="<empty>" method=GET result="200 OK" testName=K8sPolicyTest to="https://192.168.36.11:6443/api/v1/namespaces/namespace-selector-test/pods/client-cannot-connect"
time="2018-04-25T19:31:57Z" level=debug msg=HTTPRequest data="<empty>" method=GET result="200 OK" testName=K8sPolicyTest to="https://192.168.36.11:6443/api/v1/namespaces/namespace-selector-test/pods/client-cannot-connect"
time="2018-04-25T19:31:58Z" level=debug msg=HTTPRequest data="<empty>" method=GET result="200 OK" testName=K8sPolicyTest to="https://192.168.36.11:6443/api/v1/namespaces/namespace-selector-test/pods/client-cannot-connect"
time="2018-04-25T19:31:59Z" level=debug msg=HTTPRequest data="<empty>" method=GET result="200 OK" testName=K8sPolicyTest to="https://192.168.36.11:6443/api/v1/namespaces/namespace-selector-test/pods/client-cannot-connect"
time="2018-04-25T19:32:00Z" level=debug msg=HTTPRequest data="<empty>" method=GET result="200 OK" testName=K8sPolicyTest to="https://192.168.36.11:6443/api/v1/namespaces/namespace-selector-test/pods/client-cannot-connect"
time="2018-04-25T19:32:01Z" level=debug msg=HTTPRequest data="<empty>" method=GET result="200 OK" testName=K8sPolicyTest to="https://192.168.36.11:6443/api/v1/namespaces/namespace-selector-test/pods/client-cannot-connect"
time="2018-04-25T19:32:02Z" level=debug msg=HTTPRequest data="<empty>" method=GET result="200 OK" testName=K8sPolicyTest to="https://192.168.36.11:6443/api/v1/namespaces/namespace-selector-test/pods/client-cannot-connect"
�[1mSTEP�[0m: Cleaning up the pod client-cannot-connect
time="2018-04-25T19:32:02Z" level=debug msg=HTTPRequest data="<empty>" method=DELETE result="200 OK" testName=K8sPolicyTest to="https://192.168.36.11:6443/api/v1/namespaces/namespace-selector-test/pods/client-cannot-connect"
time="2018-04-25T19:32:02Z" level=debug msg="running command: kubectl -n kube-system logs --timestamps=true -l k8s-app=cilium --since=53.758091936s --previous"
time="2018-04-25T19:32:02Z" level=debug msg="running command: kubectl -n kube-system logs --timestamps=true -l k8s-app=cilium --since=53.758091936s"
time="2018-04-25T19:32:03Z" level=debug msg="running command: kubectl get pods -o wide"
time="2018-04-25T19:32:03Z" level=debug msg="running command: kubectl -n kube-system logs cilium-0n96t"
time="2018-04-25T19:32:03Z" level=debug msg="running command: kubectl exec -n kube-system cilium-0n96t -- [cilium service list cilium endpoint list]"
time="2018-04-25T19:32:04Z" level=debug msg="running command: kubectl -n kube-system logs cilium-jmlq3"
time="2018-04-25T19:32:04Z" level=debug msg="running command: kubectl exec -n kube-system cilium-jmlq3 -- [cilium service list cilium endpoint list]"
time="2018-04-25T19:32:05Z" level=debug msg="running command: kubectl exec -n kube-system cilium-0n96t -- cilium endpoint list -o json"
time="2018-04-25T19:32:05Z" level=debug msg="running command: kubectl exec -n kube-system cilium-0n96t -- cilium service list -o json"
time="2018-04-25T19:32:06Z" level=debug msg="running command: kubectl exec -n kube-system cilium-0n96t -- cilium bpf tunnel list"
time="2018-04-25T19:32:06Z" level=debug msg="running command: kubectl exec -n kube-system cilium-0n96t -- cilium policy get"
time="2018-04-25T19:32:07Z" level=debug msg="running command: kubectl exec -n kube-system cilium-0n96t -- cilium config"
time="2018-04-25T19:32:07Z" level=debug msg="running command: kubectl exec -n kube-system cilium-0n96t -- cilium bpf ct list global"
time="2018-04-25T19:32:08Z" level=debug msg="running command: kubectl exec -n kube-system cilium-0n96t -- cilium status --all-controllers"
time="2018-04-25T19:32:08Z" level=debug msg="running command: kubectl exec -n kube-system cilium-0n96t -- cilium bpf lb list"
time="2018-04-25T19:32:09Z" level=debug msg="running command: kubectl exec -n kube-system cilium-0n96t -- cilium kvstore get cilium --recursive"
time="2018-04-25T19:32:10Z" level=debug msg="running command: kubectl exec -n kube-system cilium-0n96t -- cilium-bugtool"
time="2018-04-25T19:32:13Z" level=debug msg="running command: kubectl exec -n kube-system cilium-0n96t -- ls /tmp/"
time="2018-04-25T19:32:14Z" level=debug msg="running command: kubectl cp kube-system/cilium-0n96t:/tmp/cilium-bugtool-20180425-193210.663+0000-UTC-789251506.tar /tmp/cilium-bugtool-20180425-193210.663+0000-UTC-789251506.tar"
time="2018-04-25T19:32:14Z" level=debug msg="running command: cp /tmp/cilium-bugtool-20180425-193210.663+0000-UTC-789251506.tar /home/vagrant/go/src/github.com/cilium/cilium/test/test_results/17-/K8sValidatedPolicyTestKubernetesNetworkPolicybetweenserverandclientshouldenforcepolicybasedonNamespaceSelector/cilium-0n96t-cilium-bugtool-20180425-193210.663+0000-UTC-789251506.tar"
time="2018-04-25T19:32:14Z" level=debug msg="running command: kubectl exec -n kube-system cilium-jmlq3 -- cilium policy get"
time="2018-04-25T19:32:17Z" level=debug msg="running command: kubectl exec -n kube-system cilium-jmlq3 -- cilium kvstore get cilium --recursive"
time="2018-04-25T19:32:19Z" level=debug msg="running command: kubectl exec -n kube-system cilium-jmlq3 -- cilium service list -o json"
time="2018-04-25T19:32:22Z" level=debug msg="running command: kubectl exec -n kube-system cilium-jmlq3 -- cilium bpf ct list global"
time="2018-04-25T19:32:25Z" level=debug msg="running command: kubectl exec -n kube-system cilium-jmlq3 -- cilium bpf tunnel list"
time="2018-04-25T19:32:29Z" level=debug msg="running command: kubectl exec -n kube-system cilium-jmlq3 -- cilium endpoint list -o json"
time="2018-04-25T19:32:31Z" level=debug msg="running command: kubectl exec -n kube-system cilium-jmlq3 -- cilium config"
time="2018-04-25T19:32:34Z" level=debug msg="running command: kubectl exec -n kube-system cilium-jmlq3 -- cilium bpf lb list"
time="2018-04-25T19:32:36Z" level=debug msg="running command: kubectl exec -n kube-system cilium-jmlq3 -- cilium status --all-controllers"
time="2018-04-25T19:32:39Z" level=debug msg="running command: kubectl exec -n kube-system cilium-jmlq3 -- cilium-bugtool"
time="2018-04-25T19:33:11Z" level=debug msg="running command: kubectl exec -n kube-system cilium-jmlq3 -- ls /tmp/"
time="2018-04-25T19:33:12Z" level=debug msg="running command: kubectl cp kube-system/cilium-jmlq3:/tmp/cilium-bugtool-20180425-193240.203+0000-UTC-284662039.tar /tmp/cilium-bugtool-20180425-193240.203+0000-UTC-284662039.tar"
time="2018-04-25T19:33:13Z" level=debug msg="running command: cp /tmp/cilium-bugtool-20180425-193240.203+0000-UTC-284662039.tar /home/vagrant/go/src/github.com/cilium/cilium/test/test_results/17-/K8sValidatedPolicyTestKubernetesNetworkPolicybetweenserverandclientshouldenforcepolicybasedonNamespaceSelector/cilium-jmlq3-cilium-bugtool-20180425-193240.203+0000-UTC-284662039.tar"
time="2018-04-25T19:33:13Z" level=debug msg="running command: kubectl -n kube-system logs --timestamps --previous cilium-jmlq3"
time="2018-04-25T19:33:14Z" level=debug msg="running command: kubectl describe pods --all-namespaces"
time="2018-04-25T19:33:16Z" level=debug msg="running command: kubectl -n kube-system logs -l k8s-app='kube-dns' -c kubedns"
time="2018-04-25T19:33:16Z" level=debug msg="running command: kubectl -n kube-system logs -l k8s-app='kube-dns' -c dnsmasq"
time="2018-04-25T19:33:17Z" level=debug msg="running command: kubectl get pods --all-namespaces -o json"
time="2018-04-25T19:33:17Z" level=debug msg="running command: kubectl get cnp --all-namespaces -o json"
time="2018-04-25T19:33:17Z" level=debug msg="running command: kubectl -n kube-system logs -l k8s-app='kube-dns' -c sidecar"
time="2018-04-25T19:33:17Z" level=debug msg="running command: kubectl -n kube-system logs --timestamps cilium-0n96t"
time="2018-04-25T19:33:18Z" level=debug msg="running command: kubectl -n kube-system logs --timestamps --previous cilium-0n96t"
time="2018-04-25T19:33:18Z" level=debug msg="running command: kubectl get ds --all-namespaces -o json"
time="2018-04-25T19:33:18Z" level=debug msg="running command: kubectl get deployment --all-namespaces -o json"
time="2018-04-25T19:33:18Z" level=debug msg="running command: kubectl -n kube-system logs --timestamps cilium-jmlq3"
time="2018-04-25T19:33:19Z" level=debug msg="running command: kubectl get services --all-namespaces -o json"
time="2018-04-25T19:33:19Z" level=debug msg="running command: kubectl get replicationcontroller --all-namespaces -o json"
time="2018-04-25T19:33:19Z" level=debug msg="running command: kubectl -n kube-system logs --timestamps=true -l k8s-app=cilium | grep -qi -B 5 -A 5 deadlock"
time="2018-04-25T19:33:20Z" level=debug msg="running command: kubectl -n kube-system logs --timestamps=true --previous -l k8s-app=cilium | grep -qi -B 5 -A 5 deadlock"
�[1mSTEP�[0m: Cleaning up the policy namespace-selector-test/allow-ns-b-via-namespace-selector
time="2018-04-25T19:33:20Z" level=debug msg=HTTPRequest data="<empty>" method=DELETE result="200 OK" testName=K8sPolicyTest to="https://192.168.36.11:6443/apis/networking.k8s.io/v1/namespaces/namespace-selector-test/networkpolicies/allow-ns-b-via-namespace-selector"
�[1mSTEP�[0m: Cleaning up the server namespace-selector-test/server
time="2018-04-25T19:33:20Z" level=debug msg=HTTPRequest data="<empty>" method=DELETE result="200 OK" testName=K8sPolicyTest to="https://192.168.36.11:6443/api/v1/namespaces/namespace-selector-test/pods/server"
�[1mSTEP�[0m: Cleaning up the server's service namespace-selector-test/svc-server
time="2018-04-25T19:33:20Z" level=debug msg=HTTPRequest data="<empty>" method=DELETE result="200 OK" testName=K8sPolicyTest to="https://192.168.36.11:6443/api/v1/namespaces/namespace-selector-test/services/svc-server"
�[1mSTEP�[0m: Deleting the namespace that was used for the pods
time="2018-04-25T19:33:20Z" level=debug msg=HTTPRequest data="{\"kind\":\"DeleteOptions\",\"apiVersion\":\"v1\"}\n" method=DELETE result="200 OK" testName=K8sPolicyTest to="https://192.168.36.11:6443/api/v1/namespaces/namespace-selector-test"
time="2018-04-25T19:33:25Z" level=debug msg="running command: kubectl get pods --all-namespaces -o jsonpath='{.items[*].metadata.deletionTimestamp}'"
time="2018-04-25T19:33:25Z" level=info msg="List of pods terminating" Terminating pods=1 testName=K8sPolicyTest
time="2018-04-25T19:33:30Z" level=debug msg="running command: kubectl get pods --all-namespaces -o jsonpath='{.items[*].metadata.deletionTimestamp}'"
time="2018-04-25T19:33:30Z" level=info msg="List of pods terminating" Terminating pods=1 testName=K8sPolicyTest
time="2018-04-25T19:33:35Z" level=debug msg="running command: kubectl get pods --all-namespaces -o jsonpath='{.items[*].metadata.deletionTimestamp}'"

Potentially relevant bits from cilium-0n96t-logs.log:

2018-04-25T19:19:26.700971231Z level=warning msg="Ignoring error while deleting endpoint" endpointID=173 error="unable to delete element 173 from map /sys/fs/bpf/tc/globals/cilium_policy_173: []"
2018-04-25T19:19:32.701458108Z level=warning msg="Error while getting Kubernetes labels" error="pods \"frontend-07mv8\" not found" subsys=containerd-watcher
2018-04-25T19:19:44.71659572Z level=warning msg="Error while getting Kubernetes labels" error="pods \"frontend-07mv8\" not found" subsys=containerd-watcher
2018-04-25T19:19:57.721438724Z level=warning msg="Error while getting Kubernetes labels" error="pods \"frontend-07mv8\" not found" subsys=containerd-watcher
...
2018-04-25T19:24:25.211267837Z level=warning msg="Ignoring error while deleting endpoint" endpointID=33243 error="unable to delete element 33243 from map /sys/fs/bpf/tc/globals/cilium_policy_33243: []"
2018-04-25T19:24:25.587088301Z level=warning msg="Ignoring error while deleting endpoint" endpointID=6458 error="unable to delete element 6458 from map /sys/fs/bpf/tc/globals/cilium_policy_6458: []"
2018-04-25T19:24:28.454527864Z level=warning msg="Ignoring error while deleting endpoint" endpointID=64351 error="unable to delete element 64351 from map /sys/fs/bpf/tc/globals/cilium_policy_64351: []"
2018-04-25T19:24:31.609747524Z level=warning msg="Ignoring error while deleting endpoint" endpointID=51796 error="unable to delete element 51796 from map /sys/fs/bpf/tc/globals/cilium_policy_51796: []"
2018-04-25T19:24:32.5299511Z level=warning msg="Error while getting Kubernetes labels" error="pods \"app1-1207786732-1cf5f\" not found" subsys=containerd-watcher
2018-04-25T19:24:33.316158688Z level=warning msg="Error while getting Kubernetes labels" error="pods \"app1-1207786732-fzh8f\" not found" subsys=containerd-watcher

Logs

test_results_Cilium-PR-Ginkgo-Tests-Validated_2723.zip

Other relevant info

This may require a fix for #3900 to further investigate.

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/bug/CIThis is a bug in the testing code.needs/triageThis issue requires triaging to establish severity and next steps.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions