https://jenkins.cilium.io/job/Cilium-PR-Ginkgo-Tests-Validated/15184/testReport/junit/Suite-k8s-1/11/K8sServicesTest_Checks_service_across_nodes_Tests_NodePort_BPF_Tests_with_direct_routing/
Found in a PR build, but the PR was unrelated to nodeport or datapath changes.
Stacktrace
/home/jenkins/workspace/Cilium-PR-Ginkgo-Tests-Validated/k8s-1.11-gopath/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:420
Pod "testclient-kw9ct" can not connect to service "http://192.168.36.11:31365"
Expected command: kubectl exec -n default testclient-kw9ct -- curl --path-as-is -s -D /dev/stderr --fail --connect-timeout 5 --max-time 8 http://192.168.36.11:31365 -w "time-> DNS: '%{time_namelookup}(%{remote_ip})', Connect: '%{time_connect}',Transfer '%{time_starttransfer}', total '%{time_total}'"
To succeed, but it failed:
Exitcode: 28
Stdout:
time-> DNS: '0.000020()', Connect: '0.000000',Transfer '0.000000', total '5.001223'
Stderr:
command terminated with exit code 28
/home/jenkins/workspace/Cilium-PR-Ginkgo-Tests-Validated/k8s-1.11-gopath/src/github.com/cilium/cilium/test/k8sT/Services.go:235
Standard Output
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Cilium pods: [cilium-7fvsd cilium-9g7bf]
Netpols loaded:
CiliumNetworkPolicies loaded:
Endpoint Policy Enforcement:
Pod Ingress Egress
testds-sc4xb false false
coredns-687db6485c-vcr9k false false
testclient-fkhxr false false
testclient-kw9ct false false
testds-5jgmm false false
Cilium agent 'cilium-7fvsd': Status: Ok Health: Ok Nodes "" ContinerRuntime: Ok Kubernetes: Ok KVstore: Ok Controllers: Total 23 Failed 0
Cilium agent 'cilium-9g7bf': Status: Ok Health: Ok Nodes "" ContinerRuntime: Ok Kubernetes: Ok KVstore: Ok Controllers: Total 28 Failed 0
Standard Error
STEP: Installing Cilium
STEP: Installing DNS Deployment
STEP: Performing Cilium preflight check
STEP: Performing Cilium status preflight check
STEP: Performing Cilium controllers preflight check
STEP: Performing Cilium health check
STEP: Performing Cilium service preflight check
STEP: Performing K8s service preflight check
STEP: Waiting for cilium-operator to be ready
STEP: Waiting for kube-dns to be ready
STEP: Running kube-dns preflight check
STEP: Performing K8s service preflight check
STEP: Making ten HTTP requests from "testclient-fkhxr" to "http://10.109.184.104:10080"
STEP: Making ten HTTP requests from "testclient-kw9ct" to "http://10.109.184.104:10080"
STEP: Making 10 HTTP requests from k8s1 to "http://127.0.0.1:31365"
STEP: Making 10 HTTP requests from k8s1 to "http://192.168.36.11:31365"
STEP: Making 10 HTTP requests from k8s1 to "http://192.168.36.12:31365"
STEP: Making ten HTTP requests from "testclient-fkhxr" to "http://192.168.36.11:31365"
STEP: Making ten HTTP requests from "testclient-kw9ct" to "http://192.168.36.11:31365"
=== Test Finished at 2019-09-27T20:20:09Z====
===================== TEST FAILED =====================
cmd: kubectl get pods -o wide --all-namespaces
Exitcode: 0
Stdout:
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE
default testclient-fkhxr 1/1 Running 0 3m 10.10.1.237 k8s2 <none>
default testclient-kw9ct 1/1 Running 0 3m 10.10.0.110 k8s1 <none>
default testds-5jgmm 1/1 Running 0 3m 10.10.1.186 k8s2 <none>
default testds-sc4xb 1/1 Running 0 3m 10.10.0.158 k8s1 <none>
kube-system cilium-7fvsd 1/1 Running 0 1m 192.168.36.11 k8s1 <none>
kube-system cilium-9g7bf 1/1 Running 0 1m 192.168.36.12 k8s2 <none>
kube-system cilium-operator-786886584-8g7zt 1/1 Running 0 5m 192.168.36.12 k8s2 <none>
kube-system coredns-687db6485c-vcr9k 1/1 Running 0 17m 10.10.1.81 k8s2 <none>
kube-system etcd-k8s1 1/1 Running 0 16m 192.168.36.11 k8s1 <none>
kube-system kube-apiserver-k8s1 1/1 Running 0 16m 192.168.36.11 k8s1 <none>
kube-system kube-controller-manager-k8s1 1/1 Running 0 16m 192.168.36.11 k8s1 <none>
kube-system kube-proxy-hjknv 1/1 Running 0 17m 192.168.36.11 k8s1 <none>
kube-system kube-proxy-m6rsf 1/1 Running 0 6m 192.168.36.12 k8s2 <none>
kube-system kube-scheduler-k8s1 1/1 Running 0 16m 192.168.36.11 k8s1 <none>
kube-system log-gatherer-gtvnj 1/1 Running 0 5m 192.168.36.11 k8s1 <none>
kube-system log-gatherer-tkjvd 1/1 Running 0 5m 192.168.36.12 k8s2 <none>
Stderr:
Fetching command output from pods [cilium-7fvsd cilium-9g7bf]
cmd: kubectl exec -n kube-system cilium-7fvsd -- cilium service list
Exitcode: 0
Stdout:
ID Frontend Backend
1 10.96.0.1:443 1 => 192.168.36.11:6443
2 10.96.0.10:53 1 => 10.10.1.81:53
3 10.10.0.144:31365 1 => 10.10.1.186:80
2 => 10.10.0.158:80
4 10.105.14.20:80 1 => 10.10.1.186:80
2 => 10.10.0.158:80
5 10.109.184.104:10080 1 => 10.10.1.186:80
2 => 10.10.0.158:80
6 0.0.0.0:31365 1 => 10.10.1.186:80
2 => 10.10.0.158:80
7 192.168.36.11:31365 1 => 10.10.1.186:80
2 => 10.10.0.158:80
Stderr:
cmd: kubectl exec -n kube-system cilium-7fvsd -- cilium endpoint list
Exitcode: 0
Stdout:
ENDPOINT POLICY (ingress) POLICY (egress) IDENTITY LABELS (source:key[=value]) IPv6 IPv4 STATUS
ENFORCEMENT ENFORCEMENT
429 Disabled Disabled 38885 k8s:io.cilium.k8s.policy.cluster=default f00d::a0a:0:0:98e2 10.10.0.110 ready
k8s:io.cilium.k8s.policy.serviceaccount=default
k8s:io.kubernetes.pod.namespace=default
k8s:zgroup=testDSClient
660 Disabled Disabled 43962 k8s:io.cilium.k8s.policy.cluster=default f00d::a0a:0:0:1eb2 10.10.0.158 ready
k8s:io.cilium.k8s.policy.serviceaccount=default
k8s:io.kubernetes.pod.namespace=default
k8s:zgroup=testDS
1292 Disabled Disabled 4 reserved:health f00d::a0a:0:0:631c 10.10.0.58 ready
Stderr:
cmd: kubectl exec -n kube-system cilium-9g7bf -- cilium service list
Exitcode: 0
Stdout:
ID Frontend Backend
1 10.96.0.1:443 1 => 192.168.36.11:6443
2 10.96.0.10:53 1 => 10.10.1.81:53
3 0.0.0.0:31365 1 => 10.10.1.186:80
2 => 10.10.0.158:80
4 10.105.14.20:80 1 => 10.10.1.186:80
2 => 10.10.0.158:80
5 10.109.184.104:10080 1 => 10.10.1.186:80
2 => 10.10.0.158:80
6 192.168.36.12:31365 1 => 10.10.1.186:80
2 => 10.10.0.158:80
7 10.10.1.122:31365 1 => 10.10.1.186:80
2 => 10.10.0.158:80
Stderr:
cmd: kubectl exec -n kube-system cilium-9g7bf -- cilium endpoint list
Exitcode: 0
Stdout:
ENDPOINT POLICY (ingress) POLICY (egress) IDENTITY LABELS (source:key[=value]) IPv6 IPv4 STATUS
ENFORCEMENT ENFORCEMENT
762 Disabled Disabled 38885 k8s:io.cilium.k8s.policy.cluster=default f00d::a0a:100:0:fbff 10.10.1.237 ready
k8s:io.cilium.k8s.policy.serviceaccount=default
k8s:io.kubernetes.pod.namespace=default
k8s:zgroup=testDSClient
1092 Disabled Disabled 43962 k8s:io.cilium.k8s.policy.cluster=default f00d::a0a:100:0:5f39 10.10.1.186 ready
k8s:io.cilium.k8s.policy.serviceaccount=default
k8s:io.kubernetes.pod.namespace=default
k8s:zgroup=testDS
1727 Disabled Disabled 104 k8s:io.cilium.k8s.policy.cluster=default f00d::a0a:100:0:14d 10.10.1.81 ready
k8s:io.cilium.k8s.policy.serviceaccount=coredns
k8s:io.kubernetes.pod.namespace=kube-system
k8s:k8s-app=kube-dns
2751 Disabled Disabled 4 reserved:health f00d::a0a:100:0:a3c6 10.10.1.175 ready
Stderr:
===================== Exiting AfterFailed =====================
[[ATTACHMENT|f9f3a0c9_K8sServicesTest_Checks_service_across_nodes_Tests_NodePort_BPF_Tests_with_direct_routing.zip]]
STEP: Installing Cilium
STEP: Installing DNS Deployment
STEP: Performing Cilium preflight check
STEP: Performing Cilium status preflight check
STEP: Performing Cilium controllers preflight check
STEP: Performing Cilium health check
STEP: Performing Cilium service preflight check
STEP: Performing K8s service preflight check
STEP: Waiting for cilium-operator to be ready
STEP: Waiting for kube-dns to be ready
STEP: Running kube-dns preflight check
STEP: Performing K8s service preflight check
f9f3a0c9_K8sServicesTest_Checks_service_across_nodes_Tests_NodePort_BPF_Tests_with_direct_routing.zip
https://jenkins.cilium.io/job/Cilium-PR-Ginkgo-Tests-Validated/15184/testReport/junit/Suite-k8s-1/11/K8sServicesTest_Checks_service_across_nodes_Tests_NodePort_BPF_Tests_with_direct_routing/
Found in a PR build, but the PR was unrelated to nodeport or datapath changes.
Stacktrace
Standard Output
Standard Error
f9f3a0c9_K8sServicesTest_Checks_service_across_nodes_Tests_NodePort_BPF_Tests_with_direct_routing.zip