Skip to content

CI: K8sDatapathConfig Host firewall With native routing: Managed to reach #15575

Closed
Closed
CI: K8sDatapathConfig Host firewall With native routing: Managed to reach#15575
Labels
area/CIContinuous Integration testing issue or flakearea/host-firewallImpacts the host firewall or the host endpoint.ci/flakeThis is a known failure that occurs in the tree. Please investigate me!
@pchaigno

Description

@pchaigno
pchaigno
opened on Apr 6, 2021

https://jenkins.cilium.io/job/Cilium-PR-K8s-1.20-kernel-4.19/75/testReport/junit/Suite-k8s-1/20/K8sDatapathConfig_Host_firewall_With_native_routing/
8fab5f05_K8sDatapathConfig_Host_firewall_With_native_routing.zip

There are other ongoing flakes for this test, so please check for the Managed to reach part before assuming it's the same flake.

Stacktrace

/home/jenkins/workspace/Cilium-PR-K8s-1.20-kernel-4.19/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:518
Managed to reach 10.0.1.15:69 from testclient-host-8m2gz
Expected command: kubectl exec -n 202104061126k8sdatapathconfighostfirewallwithnativerouting testclient-host-8m2gz -- curl --path-as-is -s -D /dev/stderr --fail --connect-timeout 5 --max-time 20 tftp://10.0.1.15:69/hello -w "time-> DNS: '%{time_namelookup}(%{remote_ip})', Connect: '%{time_connect}',Transfer '%{time_starttransfer}', total '%{time_total}'"
To have failed, but it was successful:
Exitcode: 0 
Stdout:
 	 
	 Hostname: testserver-pgqr4
	 
	 Request Information:
	 	client_address=192.168.36.12
	 	client_port=33119
	 	real path=/hello
	 	request_scheme=tftp
	 
	 time-> DNS: '0.000015()', Connect: '0.000032',Transfer '0.000000', total '0.001360'
Stderr:
 	 

/home/jenkins/workspace/Cilium-PR-K8s-1.20-kernel-4.19/src/github.com/cilium/cilium/test/k8sT/DatapathConfiguration.go:683

Standard Output

Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 2
Number of "level=error" in logs: 0
⚠️  Number of "level=warning" in logs: 7
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
Top 3 errors/warnings:
Disabling HostServicesPeer feature.
Session affinity for host reachable services needs kernel 5.7.0 or newer to work properly when accessed from inside cluster: the same service endpoint will be selected from all network namespaces on the host.
Unable to update ipcache map entry on pod add
Cilium pods: [cilium-f7c5z cilium-mbpff]
Netpols loaded: 
CiliumNetworkPolicies loaded: 
Endpoint Policy Enforcement:
Pod                           Ingress   Egress
testserver-pgqr4                        
grafana-d69c97b9b-2s6kl                 
prometheus-655fb888d7-bkd2j             
coredns-867bf6789f-ckr7m                
testclient-4g9z2                        
testclient-td67l                        
testserver-gln6g                        
Cilium agent 'cilium-f7c5z': Status: Ok  Health: Ok Nodes "" ContinerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 43 Failed 0
Cilium agent 'cilium-mbpff': Status: Ok  Health: Ok Nodes "" ContinerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 29 Failed 0

Standard Error

Click here to see 
11:25:29 STEP: Installing Cilium
11:25:31 STEP: Waiting for Cilium to become ready
11:26:10 STEP: Validating if Kubernetes DNS is deployed
11:26:10 STEP: Checking if deployment is ready
11:26:10 STEP: Checking if kube-dns service is plumbed correctly
11:26:10 STEP: Checking if pods have identity
11:26:10 STEP: Checking if DNS can resolve
11:26:16 STEP: Kubernetes DNS is not ready: 5s timeout expired
11:26:16 STEP: Restarting Kubernetes DNS (-l k8s-app=kube-dns)
11:26:17 STEP: Checking service kube-system/kube-dns plumbing in cilium pod cilium-mbpff: unable to find service backend 10.0.1.118:53 in datapath of cilium pod cilium-mbpff
11:26:30 STEP: Waiting for Kubernetes DNS to become operational
11:26:30 STEP: Checking if deployment is ready
11:26:30 STEP: Checking if kube-dns service is plumbed correctly
11:26:30 STEP: Checking if pods have identity
11:26:30 STEP: Checking if DNS can resolve
11:26:31 STEP: Validating Cilium Installation
11:26:31 STEP: Performing Cilium controllers preflight check
11:26:31 STEP: Performing Cilium health check
11:26:31 STEP: Performing Cilium status preflight check
11:26:34 STEP: Performing Cilium service preflight check
11:26:34 STEP: Performing K8s service preflight check
11:26:35 STEP: Waiting for cilium-operator to be ready
11:26:35 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator")
11:26:35 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator") => <nil>
11:26:35 STEP: Making sure all endpoints are in ready state
11:26:36 STEP: Creating namespace 202104061126k8sdatapathconfighostfirewallwithnativerouting
11:26:36 STEP: Deploying demo_hostfw.yaml in namespace 202104061126k8sdatapathconfighostfirewallwithnativerouting
11:26:36 STEP: Waiting for 4m0s for 8 pods of deployment demo_hostfw.yaml to become ready
11:26:36 STEP: WaitforNPods(namespace="202104061126k8sdatapathconfighostfirewallwithnativerouting", filter="")
11:26:46 STEP: WaitforNPods(namespace="202104061126k8sdatapathconfighostfirewallwithnativerouting", filter="") => <nil>
11:26:46 STEP: Applying policies /home/jenkins/workspace/Cilium-PR-K8s-1.20-kernel-4.19/src/github.com/cilium/cilium/test/k8sT/manifests/host-policies.yaml
11:26:59 STEP: Checking host policies on ingress from local pod
11:26:59 STEP: WaitforPods(namespace="202104061126k8sdatapathconfighostfirewallwithnativerouting", filter="-l zgroup=testClient")
11:26:59 STEP: WaitforPods(namespace="202104061126k8sdatapathconfighostfirewallwithnativerouting", filter="-l zgroup=testClient") => <nil>
11:26:59 STEP: WaitforPods(namespace="202104061126k8sdatapathconfighostfirewallwithnativerouting", filter="-l zgroup=testServerHost")
11:26:59 STEP: WaitforPods(namespace="202104061126k8sdatapathconfighostfirewallwithnativerouting", filter="-l zgroup=testServerHost") => <nil>
11:27:06 STEP: Checking host policies on ingress from remote pod
11:27:06 STEP: WaitforPods(namespace="202104061126k8sdatapathconfighostfirewallwithnativerouting", filter="-l zgroup=testClient")
11:27:06 STEP: WaitforPods(namespace="202104061126k8sdatapathconfighostfirewallwithnativerouting", filter="-l zgroup=testClient") => <nil>
11:27:06 STEP: WaitforPods(namespace="202104061126k8sdatapathconfighostfirewallwithnativerouting", filter="-l zgroup=testServerHost")
11:27:06 STEP: WaitforPods(namespace="202104061126k8sdatapathconfighostfirewallwithnativerouting", filter="-l zgroup=testServerHost") => <nil>
11:27:12 STEP: Checking host policies on egress to local pod
11:27:12 STEP: WaitforPods(namespace="202104061126k8sdatapathconfighostfirewallwithnativerouting", filter="-l zgroup=testClientHost")
11:27:12 STEP: WaitforPods(namespace="202104061126k8sdatapathconfighostfirewallwithnativerouting", filter="-l zgroup=testClientHost") => <nil>
11:27:12 STEP: WaitforPods(namespace="202104061126k8sdatapathconfighostfirewallwithnativerouting", filter="-l zgroup=testServer")
11:27:12 STEP: WaitforPods(namespace="202104061126k8sdatapathconfighostfirewallwithnativerouting", filter="-l zgroup=testServer") => <nil>
11:27:18 STEP: Checking host policies on egress to remote pod
11:27:18 STEP: WaitforPods(namespace="202104061126k8sdatapathconfighostfirewallwithnativerouting", filter="-l zgroup=testClientHost")
11:27:18 STEP: WaitforPods(namespace="202104061126k8sdatapathconfighostfirewallwithnativerouting", filter="-l zgroup=testClientHost") => <nil>
11:27:18 STEP: WaitforPods(namespace="202104061126k8sdatapathconfighostfirewallwithnativerouting", filter="-l zgroup=testServer")
11:27:18 STEP: WaitforPods(namespace="202104061126k8sdatapathconfighostfirewallwithnativerouting", filter="-l zgroup=testServer") => <nil>
FAIL: Managed to reach 10.0.1.15:69 from testclient-host-8m2gz
Expected command: kubectl exec -n 202104061126k8sdatapathconfighostfirewallwithnativerouting testclient-host-8m2gz -- curl --path-as-is -s -D /dev/stderr --fail --connect-timeout 5 --max-time 20 tftp://10.0.1.15:69/hello -w "time-> DNS: '%{time_namelookup}(%{remote_ip})', Connect: '%{time_connect}',Transfer '%{time_starttransfer}', total '%{time_total}'"
To have failed, but it was successful:
Exitcode: 0 
Stdout:
 	 
	 Hostname: testserver-pgqr4
	 
	 Request Information:
	 	client_address=192.168.36.12
	 	client_port=33119
	 	real path=/hello
	 	request_scheme=tftp
	 
	 time-> DNS: '0.000015()', Connect: '0.000032',Transfer '0.000000', total '0.001360'
Stderr:
 	 

=== Test Finished at 2021-04-06T11:27:18Z====
11:27:18 STEP: Running JustAfterEach block for EntireTestsuite K8sDatapathConfig
===================== TEST FAILED =====================
11:27:18 STEP: Running AfterFailed block for EntireTestsuite K8sDatapathConfig
cmd: kubectl get pods -o wide --all-namespaces
Exitcode: 0 
Stdout:
 	 NAMESPACE                                                    NAME                               READY   STATUS    RESTARTS   AGE    IP              NODE   NOMINATED NODE   READINESS GATES
	 202104061126k8sdatapathconfighostfirewallwithnativerouting   testclient-4g9z2                   1/1     Running   0          45s    10.0.1.87       k8s1   <none>           <none>
	 202104061126k8sdatapathconfighostfirewallwithnativerouting   testclient-host-8m2gz              1/1     Running   0          45s    192.168.36.12   k8s2   <none>           <none>
	 202104061126k8sdatapathconfighostfirewallwithnativerouting   testclient-host-rtslw              1/1     Running   0          45s    192.168.36.11   k8s1   <none>           <none>
	 202104061126k8sdatapathconfighostfirewallwithnativerouting   testclient-td67l                   1/1     Running   0          45s    10.0.0.152      k8s2   <none>           <none>
	 202104061126k8sdatapathconfighostfirewallwithnativerouting   testserver-gln6g                   2/2     Running   0          45s    10.0.0.17       k8s2   <none>           <none>
	 202104061126k8sdatapathconfighostfirewallwithnativerouting   testserver-host-758rd              2/2     Running   0          45s    192.168.36.12   k8s2   <none>           <none>
	 202104061126k8sdatapathconfighostfirewallwithnativerouting   testserver-host-vm4hm              2/2     Running   0          45s    192.168.36.11   k8s1   <none>           <none>
	 202104061126k8sdatapathconfighostfirewallwithnativerouting   testserver-pgqr4                   2/2     Running   0          45s    10.0.1.15       k8s1   <none>           <none>
	 cilium-monitoring                                            grafana-d69c97b9b-2s6kl            1/1     Running   0          73m    10.0.0.23       k8s2   <none>           <none>
	 cilium-monitoring                                            prometheus-655fb888d7-bkd2j        1/1     Running   0          73m    10.0.0.242      k8s2   <none>           <none>
	 kube-system                                                  cilium-f7c5z                       1/1     Running   0          110s   192.168.36.12   k8s2   <none>           <none>
	 kube-system                                                  cilium-mbpff                       1/1     Running   0          110s   192.168.36.11   k8s1   <none>           <none>
	 kube-system                                                  cilium-operator-76c8b94696-9qcjs   1/1     Running   0          110s   192.168.36.12   k8s2   <none>           <none>
	 kube-system                                                  cilium-operator-76c8b94696-qldvc   1/1     Running   0          110s   192.168.36.11   k8s1   <none>           <none>
	 kube-system                                                  coredns-867bf6789f-ckr7m           1/1     Running   0          65s    10.0.0.18       k8s2   <none>           <none>
	 kube-system                                                  etcd-k8s1                          1/1     Running   0          76m    192.168.36.11   k8s1   <none>           <none>
	 kube-system                                                  kube-apiserver-k8s1                1/1     Running   0          76m    192.168.36.11   k8s1   <none>           <none>
	 kube-system                                                  kube-controller-manager-k8s1       1/1     Running   0          76m    192.168.36.11   k8s1   <none>           <none>
	 kube-system                                                  kube-scheduler-k8s1                1/1     Running   0          76m    192.168.36.11   k8s1   <none>           <none>
	 kube-system                                                  log-gatherer-9jsrs                 1/1     Running   0          74m    192.168.36.11   k8s1   <none>           <none>
	 kube-system                                                  log-gatherer-m9bgx                 1/1     Running   0          74m    192.168.36.12   k8s2   <none>           <none>
	 kube-system                                                  registry-adder-2n9dj               1/1     Running   0          74m    192.168.36.11   k8s1   <none>           <none>
	 kube-system                                                  registry-adder-sdfn9               1/1     Running   0          74m    192.168.36.12   k8s2   <none>           <none>
	 
Stderr:
 	 

Fetching command output from pods [cilium-f7c5z cilium-mbpff]
cmd: kubectl exec -n kube-system cilium-f7c5z -- cilium status
Exitcode: 0 
Stdout:
 	 KVStore:                Ok   Disabled
	 Kubernetes:             Ok   1.20 (v1.20.5) [linux/amd64]
	 Kubernetes APIs:        ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumEndpoint", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "core/v1::Namespace", "core/v1::Node", "core/v1::Pods", "core/v1::Service", "discovery/v1beta1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
	 KubeProxyReplacement:   Strict   [enp0s8 192.168.36.12 fd04::12 (Direct Routing), enp0s3 10.0.2.15 fd04::12]
	 Cilium:                 Ok   1.9.90 (v.1.9.90-r.2d6fdc4)
	 NodeMonitor:            Listening for events on 3 CPUs with 64x4096 of shared memory
	 Cilium health daemon:   Ok   
	 IPAM:                   IPv4: 7/255 allocated from 10.0.0.0/24, IPv6: 7/255 allocated from fd02::/120
	 BandwidthManager:       Disabled
	 Host Routing:           Legacy
	 Masquerading:           BPF   [enp0s8, enp0s3]   10.0.0.0/8 [IPv4: Enabled, IPv6: Enabled]
	 Controller Status:      43/43 healthy
	 Proxy Status:           OK, ip 10.0.0.45, 0 redirects active on ports 10000-20000
	 Hubble:                 Ok              Current/Max Flows: 1044/4095 (25.49%), Flows/s: 11.51   Metrics: Disabled
	 Cluster health:         2/2 reachable   (2021-04-06T11:26:33Z)
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-f7c5z -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                                                                  IPv6       IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                                                                       
	 36         Disabled           Disabled          9202       k8s:io.cilium.k8s.policy.cluster=default                                                     fd02::31   10.0.0.18    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=coredns                                                                              
	                                                            k8s:io.kubernetes.pod.namespace=kube-system                                                                                  
	                                                            k8s:k8s-app=kube-dns                                                                                                         
	 208        Disabled           Disabled          8064       k8s:io.cilium.k8s.policy.cluster=default                                                     fd02::a    10.0.0.17    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                                              
	                                                            k8s:io.kubernetes.pod.namespace=202104061126k8sdatapathconfighostfirewallwithnativerouting                                   
	                                                            k8s:zgroup=testServer                                                                                                        
	 817        Disabled           Disabled          10448      k8s:io.cilium.k8s.policy.cluster=default                                                     fd02::ea   10.0.0.152   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                                              
	                                                            k8s:io.kubernetes.pod.namespace=202104061126k8sdatapathconfighostfirewallwithnativerouting                                   
	                                                            k8s:zgroup=testClient                                                                                                        
	 1251       Disabled           Disabled          21320      k8s:app=grafana                                                                              fd02::73   10.0.0.23    ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                                     
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                                              
	                                                            k8s:io.kubernetes.pod.namespace=cilium-monitoring                                                                            
	 2302       Enabled            Enabled           1          k8s:cilium.io/ci-node=k8s2                                                                                           ready   
	                                                            reserved:host                                                                                                                
	 2318       Disabled           Disabled          4          reserved:health                                                                              fd02::1    10.0.0.177   ready   
	 4053       Disabled           Disabled          21889      k8s:app=prometheus                                                                           fd02::75   10.0.0.242   ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                                     
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=prometheus-k8s                                                                       
	                                                            k8s:io.kubernetes.pod.namespace=cilium-monitoring                                                                            
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-mbpff -- cilium status
Exitcode: 0 
Stdout:
 	 KVStore:                Ok   Disabled
	 Kubernetes:             Ok   1.20 (v1.20.5) [linux/amd64]
	 Kubernetes APIs:        ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumEndpoint", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "core/v1::Namespace", "core/v1::Node", "core/v1::Pods", "core/v1::Service", "discovery/v1beta1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
	 KubeProxyReplacement:   Strict   [enp0s8 192.168.36.11 fd04::11 (Direct Routing), enp0s3 10.0.2.15 fd04::11]
	 Cilium:                 Ok   1.9.90 (v.1.9.90-r.2d6fdc4)
	 NodeMonitor:            Listening for events on 3 CPUs with 64x4096 of shared memory
	 Cilium health daemon:   Ok   
	 IPAM:                   IPv4: 4/255 allocated from 10.0.1.0/24, IPv6: 4/255 allocated from fd02::100/120
	 BandwidthManager:       Disabled
	 Host Routing:           Legacy
	 Masquerading:           BPF   [enp0s8, enp0s3]   10.0.0.0/8 [IPv4: Enabled, IPv6: Enabled]
	 Controller Status:      29/29 healthy
	 Proxy Status:           OK, ip 10.0.1.35, 0 redirects active on ports 10000-20000
	 Hubble:                 Ok              Current/Max Flows: 1338/4095 (32.67%), Flows/s: 16.06   Metrics: Disabled
	 Cluster health:         2/2 reachable   (2021-04-06T11:27:15Z)
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-mbpff -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                                                                  IPv6        IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                                                                        
	 399        Disabled           Disabled          8064       k8s:io.cilium.k8s.policy.cluster=default                                                     fd02::15e   10.0.1.15    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                                               
	                                                            k8s:io.kubernetes.pod.namespace=202104061126k8sdatapathconfighostfirewallwithnativerouting                                    
	                                                            k8s:zgroup=testServer                                                                                                         
	 767        Disabled           Disabled          10448      k8s:io.cilium.k8s.policy.cluster=default                                                     fd02::138   10.0.1.87    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                                               
	                                                            k8s:io.kubernetes.pod.namespace=202104061126k8sdatapathconfighostfirewallwithnativerouting                                    
	                                                            k8s:zgroup=testClient                                                                                                         
	 987        Disabled           Disabled          4          reserved:health                                                                              fd02::1c7   10.0.1.222   ready   
	 2160       Enabled            Enabled           1          k8s:cilium.io/ci-node=k8s1                                                                                            ready   
	                                                            k8s:node-role.kubernetes.io/control-plane                                                                                     
	                                                            k8s:node-role.kubernetes.io/master                                                                                            
	                                                            reserved:host                                                                                                                 
	 
Stderr:
 	 

===================== Exiting AfterFailed =====================
11:27:43 STEP: Running AfterEach for block EntireTestsuite K8sDatapathConfig Host firewall
11:27:43 STEP: Running AfterEach for block EntireTestsuite K8sDatapathConfig
11:27:43 STEP: Deleting deployment demo_hostfw.yaml
11:27:43 STEP: Deleting namespace 202104061126k8sdatapathconfighostfirewallwithnativerouting
11:27:43 STEP: Deleting namespace 202104061126k8sdatapathconfighostfirewallwithnativerouting
11:27:58 STEP: Running AfterEach for block EntireTestsuite

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/CIContinuous Integration testing issue or flakearea/host-firewallImpacts the host firewall or the host endpoint.ci/flakeThis is a known failure that occurs in the tree. Please investigate me!

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions