Skip to content

Test GKE#11949

Closed
errordeveloper wants to merge 0 commit intomasterfrom
pr/errordeveloper/test-gke
Closed

Test GKE#11949
errordeveloper wants to merge 0 commit intomasterfrom
pr/errordeveloper/test-gke

Conversation

@errordeveloper
Copy link
Copy Markdown
Contributor

@errordeveloper errordeveloper commented Jun 8, 2020

Opening this just to run GKE tests on Jenkins a few times.

@maintainer-s-little-helper

This comment has been minimized.

Sign in to view
@errordeveloper errordeveloper added dont-merge/preview-only Only for preview or testing, don't merge it. release-note/ci This PR makes changes to the CI. and removed dont-merge/needs-release-note labels Jun 8, 2020
@errordeveloper
Copy link
Copy Markdown
Contributor Author

errordeveloper commented Jun 8, 2020

test-gke

@coveralls
Copy link
Copy Markdown

coveralls commented Jun 8, 2020

Coverage Status

Coverage decreased (-0.02%) to 36.983% when pulling 6d37528 on pr/errordeveloper/test-gke into 651a3a8 on master.

@errordeveloper
Copy link
Copy Markdown
Contributor Author

errordeveloper commented Jun 8, 2020 •
edited
Loading

First attempt:

11:26:26  • Failure [416.014 seconds]
11:26:26  K8sDatapathConfig
11:26:26  /home/jenkins/workspace/Cilium-PR-K8s-GKE/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:476
11:26:26    MonitorAggregation
11:26:26    /home/jenkins/workspace/Cilium-PR-K8s-GKE/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:476
11:26:26      Checks that monitor aggregation restricts notifications [It]
11:26:26      /home/jenkins/workspace/Cilium-PR-K8s-GKE/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:512
11:26:26  
11:26:26      Kubernetes DNS did not become ready in time
11:26:26  
11:26:26      /home/jenkins/workspace/Cilium-PR-K8s-GKE/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:632

@errordeveloper
Copy link
Copy Markdown
Contributor Author

errordeveloper commented Jun 8, 2020

test-gke

@errordeveloper
Copy link
Copy Markdown
Contributor Author

errordeveloper commented Jun 9, 2020

Second attempt:

18:57:32  • Failure [482.828 seconds]
18:57:32  K8sIdentity
18:57:32  /home/jenkins/workspace/Cilium-PR-K8s-GKE/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:476
18:57:32    Identity expiration
18:57:32    /home/jenkins/workspace/Cilium-PR-K8s-GKE/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:476
18:57:32      Expiration of CiliumIdentity [It]
18:57:32      /home/jenkins/workspace/Cilium-PR-K8s-GKE/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:512
18:57:32  
18:57:32      Kubernetes DNS did not become ready in time
18:57:32  
18:57:32      /home/jenkins/workspace/Cilium-PR-K8s-GKE/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:632

@errordeveloper
Copy link
Copy Markdown
Contributor Author

errordeveloper commented Jun 9, 2020

test-gke

@errordeveloper
Copy link
Copy Markdown
Contributor Author

errordeveloper commented Jun 9, 2020

Quite different results on the 3rd attempt.

There are a few cases with health-check failing:

12:06:12  • Failure in Spec Setup (BeforeEach) [534.388 seconds]
12:06:12  K8sHealthTest
12:06:12  /home/jenkins/workspace/Cilium-PR-K8s-GKE/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:476
12:06:12    checks cilium-health status between nodes [BeforeEach]
12:06:12    /home/jenkins/workspace/Cilium-PR-K8s-GKE/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:512
12:06:12  
12:06:12    cilium pre-flight checks failed
[2020-06-09T11:06:12.869Z]   Expected
[2020-06-09T11:06:12.869Z]       <*errors.errorString | 0xc00080e9b0>: {
[2020-06-09T11:06:12.869Z]           s: "Cilium validation failed: 4m0s timeout expired: Last polled error: connectivity health is failing: Cluster connectivity is unhealthy on 'cilium-c6wt8': Exitcode: 255 \nStdout:\n \t \nStderr:\n \t Error: Cannot get status/probe: Put \"http://%2Fvar%2Frun%2Fcilium%2Fhealth.sock/v1beta/status/probe\": context deadline exceeded\n\t \n\t command terminated with exit code 255\n\t \n",
[2020-06-09T11:06:12.869Z]       }
[2020-06-09T11:06:12.869Z]   to be nil

And a few cases where test workload don't appear to get deployed in default namespace, namely:

12:40:48  • Failure [659.081 seconds]
12:40:48  K8sUpdates
12:40:48  /home/jenkins/workspace/Cilium-PR-K8s-GKE/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:476
12:40:48    Tests upgrade and downgrade from a Cilium stable image to master [It]
12:40:48    /home/jenkins/workspace/Cilium-PR-K8s-GKE/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:512
12:40:48  
12:40:48    Test pods are not ready after timeout
[2020-06-09T11:40:48.733Z]   Expected
[2020-06-09T11:40:48.733Z]       <*errors.errorString | 0xc0004f1550>: {
[2020-06-09T11:40:48.733Z]           s: "timed out waiting for pods with filter -l zgroup=testapp to be ready: 5m0s timeout expired",
[2020-06-09T11:40:48.733Z]       }
[2020-06-09T11:40:48.733Z]   to be nil
12:40:48  
12:40:48    /home/jenkins/workspace/Cilium-PR-K8s-GKE/src/github.com/cilium/cilium/test/k8sT/Updates.go:353

and here is the evidence that those pods are simply missing:

12:33:34  11:33:34 STEP: WaitforPods(namespace="default", filter="-l zgroup=testapp")
12:38:47  11:38:34 STEP: WaitforPods(namespace="default", filter="-l zgroup=testapp") => timed out waiting for pods with filter -l zgroup=testapp to be ready: 5m0s timeout expired
12:38:47  11:38:34 STEP: cmd: kubectl describe pods -n default -l zgroup=testapp
12:38:47  Exitcode: 0 
12:38:47  Stdout:
12:38:47   	 
12:38:47  Stderr:
12:38:47   	 No resources found in default namespace.
12:38:47  	 
12:38:47  
12:38:47  FAIL: Test pods are not ready after timeout
12:38:47  Expected
12:38:47      <*errors.errorString | 0xc0004f1550>: {
12:38:47          s: "timed out waiting for pods with filter -l zgroup=testapp to be ready: 5m0s timeout expired",
12:38:47      }
12:38:47  to be nil

@errordeveloper
Copy link
Copy Markdown
Contributor Author

errordeveloper commented Jun 9, 2020

test-gke

@errordeveloper
Copy link
Copy Markdown
Contributor Author

errordeveloper commented Jun 9, 2020

The 4th run appears to have a the DNS failure, as well as health-check one:

15:22:51  • Failure in Spec Setup (BeforeEach) [388.702 seconds]
15:22:51  K8sIstioTest
15:22:51  /home/jenkins/workspace/Cilium-PR-K8s-GKE/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:476
15:22:51    Istio Bookinfo Demo [BeforeEach]
15:22:51    /home/jenkins/workspace/Cilium-PR-K8s-GKE/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:476
15:22:51      Tests bookinfo inter-service connectivity
15:22:51      /home/jenkins/workspace/Cilium-PR-K8s-GKE/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:512
15:22:51  
15:22:51      Kubernetes DNS did not become ready in time
15:22:51  
15:22:51      /home/jenkins/workspace/Cilium-PR-K8s-GKE/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:632

15:33:11  • Failure [620.013 seconds]
15:33:11  K8sDatapathConfig
15:33:11  /home/jenkins/workspace/Cilium-PR-K8s-GKE/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:476
15:33:11    MonitorAggregation
15:33:11    /home/jenkins/workspace/Cilium-PR-K8s-GKE/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:476
15:33:11      Checks that monitor aggregation restricts notifications [It]
15:33:11      /home/jenkins/workspace/Cilium-PR-K8s-GKE/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:512
15:33:11  
15:33:11      cilium pre-flight checks failed
[2020-06-09T14:33:11.443Z]     Expected
[2020-06-09T14:33:11.443Z]         <*errors.errorString | 0xc0004918b0>: {
[2020-06-09T14:33:11.443Z]             s: "Cilium validation failed: 4m0s timeout expired: Last polled error: connectivity health is failing: Cluster connectivity is unhealthy on 'cilium-c6bs5': Exitcode: 255 \nStdout:\n \t \nStderr:\n \t Error: Cannot get status/probe: Put \"http://%2Fvar%2Frun%2Fcilium%2Fhealth.sock/v1beta/status/probe\": context deadline exceeded\n\t \n\t command terminated with exit code 255\n\t \n",
[2020-06-09T14:33:11.443Z]         }
[2020-06-09T14:33:11.443Z]     to be nil
15:33:11

@errordeveloper errordeveloper changed the title Test Test GKE Jun 9, 2020
This was referenced Jun 12, 2020
Closed
Closed
@errordeveloper errordeveloper force-pushed the pr/errordeveloper/test-gke branch from 6d37528 to cf76e2b Compare June 19, 2020 10:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

dont-merge/preview-only Only for preview or testing, don't merge it. release-note/ci This PR makes changes to the CI.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@errordeveloper @coveralls