CI: K8sFQDNTest Validate that FQDN policy continues to work after being updated

### Test Name
```test-name
K8sFQDNTest Validate that FQDN policy continues to work after being updated
```

### Failure Output
```failure-output
FAIL: Can't connect to to a valid target when it should work
```

### Stacktrace
<details><summary>Click to show.</summary>

```stack-trace
/home/jenkins/workspace/Cilium-PR-K8s-1.22-kernel-4.19/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:527
Can't connect to to a valid target when it should work
Expected command: kubectl exec -n default app2-58757b7dd5-pvvvw -- curl --path-as-is -s -D /dev/stderr --fail --connect-timeout 5 --max-time 20 --retry 5 http://vagrant-cache.ci.cilium.io -w "time-> DNS: '%{time_namelookup}(%{remote_ip})', Connect: '%{time_connect}',Transfer '%{time_starttransfer}', total '%{time_total}'" 
To succeed, but it failed:
Exitcode: 28 
Err: exit status 28
Stdout:
 	 time-> DNS: '0.000016()', Connect: '0.000000',Transfer '0.000000', total '5.000836'
Stderr:
 	 command terminated with exit code 28
	 

/home/jenkins/workspace/Cilium-PR-K8s-1.22-kernel-4.19/src/github.com/cilium/cilium/test/k8sT/fqdn.go:317
```
</details>

### Standard Output
<details><summary>Click to show.</summary>

```stack-output
Cilium pods: [cilium-9gp4w cilium-vpsn6]
Netpols loaded: 
CiliumNetworkPolicies loaded: default::fqdn-proxy-policy.yaml 
Endpoint Policy Enforcement:
Pod   Ingress   Egress
Cilium agent 'cilium-9gp4w': Status: Ok  Health: Ok Nodes "" ContinerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 37 Failed 0
Cilium agent 'cilium-vpsn6': Status: Ok  Health: Ok Nodes "" ContinerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 33 Failed 0


```
</details>

### Standard Error
<details><summary>Click to show.</summary>

```stack-error
03:10:39 STEP: Validating APP2 policy connectivity
03:10:44 STEP: Updating the policy to include an extra FQDN allow statement
03:10:45 STEP: Validating APP2 policy connectivity after policy change
FAIL: Can't connect to to a valid target when it should work
Expected command: kubectl exec -n default app2-58757b7dd5-pvvvw -- curl --path-as-is -s -D /dev/stderr --fail --connect-timeout 5 --max-time 20 --retry 5 http://vagrant-cache.ci.cilium.io -w "time-> DNS: '%{time_namelookup}(%{remote_ip})', Connect: '%{time_connect}',Transfer '%{time_starttransfer}', total '%{time_total}'" 
To succeed, but it failed:
Exitcode: 28 
Err: exit status 28
Stdout:
 	 time-> DNS: '0.000016()', Connect: '0.000000',Transfer '0.000000', total '5.000836'
Stderr:
 	 command terminated with exit code 28
	 

=== Test Finished at 2021-12-09T03:11:46Z====
===================== TEST FAILED =====================
03:11:46 STEP: Running AfterFailed block for EntireTestsuite K8sFQDNTest
cmd: kubectl get pods -o wide --all-namespaces
Exitcode: 0 
Stdout:
 	 NAMESPACE           NAME                               READY   STATUS    RESTARTS   AGE     IP              NODE   NOMINATED NODE   READINESS GATES
	 cilium-monitoring   grafana-5747bcc8f9-bg6w8           1/1     Running   0          2m34s   10.0.1.137      k8s2   <none>           <none>
	 cilium-monitoring   prometheus-655fb888d7-r5ssj        1/1     Running   0          2m34s   10.0.1.199      k8s2   <none>           <none>
	 default             app1-6bf9bf9bd5-gm5fv              2/2     Running   0          87s     10.0.0.246      k8s1   <none>           <none>
	 default             app1-6bf9bf9bd5-q7csp              2/2     Running   0          87s     10.0.0.46       k8s1   <none>           <none>
	 default             app2-58757b7dd5-pvvvw              1/1     Running   0          87s     10.0.0.24       k8s1   <none>           <none>
	 default             app3-5d69599cdd-rvrg6              1/1     Running   0          87s     10.0.0.161      k8s1   <none>           <none>
	 kube-system         cilium-9gp4w                       1/1     Running   0          2m31s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         cilium-operator-7ff47c6d9b-c8z7w   1/1     Running   0          2m31s   192.168.56.12   k8s2   <none>           <none>
	 kube-system         cilium-operator-7ff47c6d9b-nqkp7   1/1     Running   0          2m31s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         cilium-vpsn6                       1/1     Running   0          2m31s   192.168.56.12   k8s2   <none>           <none>
	 kube-system         coredns-69b675786c-bcmnk           1/1     Running   0          96s     10.0.1.98       k8s2   <none>           <none>
	 kube-system         etcd-k8s1                          1/1     Running   0          5m16s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-apiserver-k8s1                1/1     Running   0          5m16s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-controller-manager-k8s1       1/1     Running   0          5m16s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-8ptvj                   1/1     Running   0          3m26s   192.168.56.12   k8s2   <none>           <none>
	 kube-system         kube-proxy-fhlf2                   1/1     Running   0          4m59s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-scheduler-k8s1                1/1     Running   0          5m15s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-85hkq                 1/1     Running   0          2m37s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-qvg2z                 1/1     Running   0          2m37s   192.168.56.12   k8s2   <none>           <none>
	 kube-system         registry-adder-86vbk               1/1     Running   0          3m18s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         registry-adder-9s6md               1/1     Running   0          3m18s   192.168.56.12   k8s2   <none>           <none>
	 
Stderr:
 	 

Fetching command output from pods [cilium-9gp4w cilium-vpsn6]
cmd: kubectl exec -n kube-system cilium-9gp4w -c cilium-agent -- cilium service list
Exitcode: 0 
Stdout:
 	 ID   Frontend              Service Type   Backend                   
	 1    10.96.0.10:53         ClusterIP      1 => 10.0.1.98:53         
	 2    10.96.0.10:9153       ClusterIP      1 => 10.0.1.98:9153       
	 3    10.103.50.196:3000    ClusterIP      1 => 10.0.1.137:3000      
	 4    10.100.178.174:9090   ClusterIP      1 => 10.0.1.199:9090      
	 5    10.96.0.1:443         ClusterIP      1 => 192.168.56.11:6443   
	 6    10.101.117.150:80     ClusterIP      1 => 10.0.0.46:80         
	                                           2 => 10.0.0.246:80        
	 7    10.101.117.150:69     ClusterIP      1 => 10.0.0.46:69         
	                                           2 => 10.0.0.246:69        
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-9gp4w -c cilium-agent -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                                              IPv6       IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                                                   
	 129        Disabled           Disabled          30402      k8s:id=app1                                                              fd02::42   10.0.0.46    ready   
	                                                            k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default                                   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                 
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=app1-account                                                     
	                                                            k8s:io.kubernetes.pod.namespace=default                                                                  
	                                                            k8s:zgroup=testapp                                                                                       
	 323        Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s1                                                                       ready   
	                                                            k8s:node-role.kubernetes.io/control-plane                                                                
	                                                            k8s:node-role.kubernetes.io/master                                                                       
	                                                            k8s:node.kubernetes.io/exclude-from-external-load-balancers                                              
	                                                            reserved:host                                                                                            
	 441        Disabled           Disabled          30402      k8s:id=app1                                                              fd02::4f   10.0.0.246   ready   
	                                                            k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default                                   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                 
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=app1-account                                                     
	                                                            k8s:io.kubernetes.pod.namespace=default                                                                  
	                                                            k8s:zgroup=testapp                                                                                       
	 515        Disabled           Enabled           1102       k8s:id=app3                                                              fd02::a0   10.0.0.161   ready   
	                                                            k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default                                   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                 
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                          
	                                                            k8s:io.kubernetes.pod.namespace=default                                                                  
	                                                            k8s:zgroup=testapp                                                                                       
	 544        Disabled           Enabled           27512      k8s:appSecond=true                                                       fd02::1a   10.0.0.24    ready   
	                                                            k8s:id=app2                                                                                              
	                                                            k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default                                   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                 
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=app2-account                                                     
	                                                            k8s:io.kubernetes.pod.namespace=default                                                                  
	                                                            k8s:zgroup=testapp                                                                                       
	 1797       Disabled           Disabled          4          reserved:health                                                          fd02::52   10.0.0.120   ready   
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-vpsn6 -c cilium-agent -- cilium service list
Exitcode: 0 
Stdout:
 	 ID   Frontend              Service Type   Backend                   
	 1    10.96.0.1:443         ClusterIP      1 => 192.168.56.11:6443   
	 2    10.96.0.10:9153       ClusterIP      1 => 10.0.1.98:9153       
	 3    10.96.0.10:53         ClusterIP      1 => 10.0.1.98:53         
	 4    10.103.50.196:3000    ClusterIP      1 => 10.0.1.137:3000      
	 5    10.100.178.174:9090   ClusterIP      1 => 10.0.1.199:9090      
	 6    10.101.117.150:80     ClusterIP      1 => 10.0.0.46:80         
	                                           2 => 10.0.0.246:80        
	 7    10.101.117.150:69     ClusterIP      1 => 10.0.0.46:69         
	                                           2 => 10.0.0.246:69        
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-vpsn6 -c cilium-agent -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                                                        IPv6        IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                                                              
	 1          Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s2                                                                                  ready   
	                                                            reserved:host                                                                                                       
	 344        Disabled           Disabled          47438      k8s:app=grafana                                                                    fd02::139   10.0.1.137   ready   
	                                                            k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=cilium-monitoring                                    
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                            
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                                     
	                                                            k8s:io.kubernetes.pod.namespace=cilium-monitoring                                                                   
	 1247       Disabled           Disabled          4          reserved:health                                                                    fd02::14c   10.0.1.51    ready   
	 1344       Disabled           Disabled          47062      k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=kube-system         fd02::1f2   10.0.1.98    ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                            
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=coredns                                                                     
	                                                            k8s:io.kubernetes.pod.namespace=kube-system                                                                         
	                                                            k8s:k8s-app=kube-dns                                                                                                
	 3229       Disabled           Disabled          8077       k8s:app=prometheus                                                                 fd02::176   10.0.1.199   ready   
	                                                            k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=cilium-monitoring                                    
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                            
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=prometheus-k8s                                                              
	                                                            k8s:io.kubernetes.pod.namespace=cilium-monitoring                                                                   
	 
Stderr:
 	 

===================== Exiting AfterFailed =====================
03:11:59 STEP: Running AfterEach for block EntireTestsuite K8sFQDNTest
03:11:59 STEP: Running AfterEach for block EntireTestsuite

[[ATTACHMENT|ec62dcff_K8sFQDNTest_Validate_that_FQDN_policy_continues_to_work_after_being_updated.zip]]
03:12:02 STEP: Running AfterAll block for EntireTestsuite K8sFQDNTest
03:12:02 STEP: Removing Cilium installation using generated helm manifest


```
</details>

ZIP Links: 
<details><summary>Click to show.</summary>

 https://jenkins.cilium.io/job/Cilium-PR-K8s-1.22-kernel-4.19//92/artifact/7fcfe7ee_K8sUpdates_Tests_upgrade_and_downgrade_from_a_Cilium_stable_image_to_master.zip
  https://jenkins.cilium.io/job/Cilium-PR-K8s-1.22-kernel-4.19//92/artifact/ec62dcff_K8sFQDNTest_Validate_that_FQDN_policy_continues_to_work_after_being_updated.zip
  https://jenkins.cilium.io/job/Cilium-PR-K8s-1.22-kernel-4.19//92/artifact/test_results_Cilium-PR-K8s-1.22-kernel-4.19_92_BDD-Test-PR.zip
 
</details>

Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.22-kernel-4.19/92/

If this is a duplicate of an existing flake, comment 'Duplicate of #\<issue-number>' and close this issue.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CI: K8sFQDNTest Validate that FQDN policy continues to work after being updated #18218

Test Name

Failure Output

Stacktrace

Standard Output

Standard Error

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

CI: K8sFQDNTest Validate that FQDN policy continues to work after being updated #18218

Description

Test Name

Failure Output

Stacktrace

Standard Output

Standard Error

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions