E2E test framework by YaoZengzeng · Pull Request #443 · kmesh-net/kmesh

YaoZengzeng · 2024-06-18T02:01:07Z

What type of PR is this?

/kind feature

What this PR does / why we need it:

E2E test framework to make the project more robust

Which issue(s) this PR fixes:
Fixes most part of #137

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

LiZhenCheng9527 · 2024-06-18T03:16:06Z

deploy/yaml/kmesh.yaml

                  path: istio-token
      containers:
        - name: kmesh
-          image: ghcr.io/kmesh-net/kmesh:latest


It's better to open a new folder for e2e's yaml

Good catch, I should find a way to configure HUB of the image

How do you install, if using helm we can customize the image

Now I apply yamls directly for convenience and I will use helm later.

LiZhenCheng9527 · 2024-06-18T03:26:42Z

test/e2e/run_test.sh

+# Exit immediately for non zero status
+set -e
+
+DEFAULT_KIND_IMAGE="kindest/node:v1.30.0@sha256:047357ac0cfea04663786a612ba1eaba9702bef25227a794b52890dd8bcd692e"


If you limit the kind/node image version, you will also have a limit on the kind version.
I have another option：Check for kind, then create cluster without specifying image.

Istio uses this to test on compatability with different k8s version.

What we need maybe add a mechanism to test comaptability with differen istio versions

LiZhenCheng9527 · 2024-06-18T03:29:33Z

test/e2e/run_test.sh

+    build_and_push_images
+fi
+
+go test -v -tags=integ ./test/e2e/...


Suggested change

go test -v -tags=integ ./test/e2e/...

go test -v -tags=integ $ROOT_DIR/test/e2e/...

ROOT_DIR=$(git rev-parse --show-toplevel)
Preventing test failures by not running in the kmesh root directory.

hzxuzhonghu

Looked just a small part

hzxuzhonghu · 2024-06-20T02:26:21Z

docs/proposal/e2e.md

+
+### Motivation
+
+E2E testing is a software approach that tests an application's workflow from start to finish, simulating read-user scenarios. The main purpose of E2E testing is to validate the system as a whole,ensuring that all the individual components and integrations work together seamlessly. It helps to identify any issues or defects that may arise from the interation between different components of the appliction, ensuing the application works as expected under normal operating conditions.


s/defects/detects

simulating read-user scenarios. what does this mean?

s/defects/detects

It is defetcs actually :)

simulating read-user scenarios. what does this mean?

updated, it is real user scenarios.

hzxuzhonghu · 2024-06-24T01:35:50Z

docs/proposal/e2e.md

+
+1. Use [namespace](https://github.com/istio/istio/blob/master/pkg/test/framework/components/namespace/namespace.go) package to create namespace for deploying test applications.
+
+2. Use [deployment](https://github.com/istio/istio/blob/master/pkg/test/framework/components/echo/deployment/builder.go) package to build test applications. `WithClusters()` can be used to specify the clusters where the test applications should be deployed. Each call to `WithConfig()` will generate a test application with corresponding configuration. Actually we create all test applications at the beginning. In each test case, we select some suitable applications for testing. And an application can be used as both a client and a server.


Should document if these apps do not satisfy new tests, how can developer deploy new apps

hzxuzhonghu · 2024-06-24T01:39:01Z

go.mod

 // Client-go does not handle different versions of mergo due to some breaking changes - use the matching version
 replace github.com/imdario/mergo => github.com/imdario/mergo v0.3.5

+replace istio.io/api => istio.io/api v1.22.0-alpha.1.0.20240620154034-5b788fec62d2


why replace instead of bumping directly

If we bump directly, it will use version istio.io/api v1.22.1, there will be some interface incompatible

I donot understand what do you mean

hzxuzhonghu · 2024-06-24T01:40:53Z

test/e2e/baseline_test.go

@@ -0,0 +1,155 @@
+//go:build integ


do we really need this building tag

It will be added automaticlly in my vscode

hzxuzhonghu · 2024-06-24T01:47:11Z

test/e2e/run_test.sh

+# Exit immediately for non zero status
+set -e
+
+DEFAULT_KIND_IMAGE="kindest/node:v1.30.0@sha256:047357ac0cfea04663786a612ba1eaba9702bef25227a794b52890dd8bcd692e"


Istio uses this to test on compatability with different k8s version.

What we need maybe add a mechanism to test comaptability with differen istio versions

hzxuzhonghu · 2024-06-24T01:52:41Z

test/e2e/run_test.sh

+    fi
+
+    # Create KinD cluster.
+    cat <<EOF | kind create cluster --name="${NAME}" -v4 --retain --image "${IMAGE}" --config=-


can we add a multi nodes cluster, since we have daemonset

hzxuzhonghu · 2024-06-24T01:59:29Z

test/e2e/main_test.go

+
+	builder := deployment.New(t).
+		WithClusters(t.Clusters()...).
+		/*


hzxuzhonghu · 2024-06-24T02:01:05Z

test/e2e/main_test.go

+	// AllWaypoint is a waypoint for all types
+	AllWaypoint echo.Instances
+	// WorkloadAddressedWaypoint is a workload only waypoint
+	WorkloadAddressedWaypoint echo.Instances
+	// ServiceAddressedWaypoint is a service only waypoint
+	ServiceAddressedWaypoint echo.Instances


I am not sure this is suitable for kmesh, at least the names and comments are confusing

removed, we don't need these fields now.

hzxuzhonghu · 2024-06-24T02:01:50Z

test/e2e/main_test.go

+	Captured                  = "captured"
+	Uncaptured                = "uncaptured"
+	WaypointImageAnnotation   = "sidecar.istio.io/proxyImage"
+	KmeshCustomWaypointImage  = "ghcr.io/kmesh-net/waypoint-x86:v0.3.0"


should support customize it

hzxuzhonghu · 2024-06-24T02:05:47Z

test/e2e/baseline_test.go

+		}
+		if src.Config().IsUncaptured() {
+			// TODO: fix this and remove this skip
+			t.Skip("https://github.com/istio/istio/issues/43238")


The test case is direclty copied from ambient integration test (https://github.com/istio/istio/blob/master/tests/integration/ambient/baseline_test.go#L628`).

Actually I think most the test cases there should be copied and used direcly. Because we need to fully match ambient's capabilities and be transparent to isito and users.

hzxuzhonghu · 2024-06-24T02:06:09Z

test/e2e/baseline_test.go

+		if opt.Scheme != scheme.HTTP {
+			return
+		}
+		if !dst.Config().HasServiceAddressedWaypointProxy() {


Can you remove this istio special concept

Signed-off-by: YaoZengzeng <yaozengzeng@huawei.com>

hzxuzhonghu

looks super good, i will try with it. And understand it deeper

test/e2e/run_test.sh

hzxuzhonghu · 2024-06-27T02:16:05Z

test/e2e/run_test.sh

+fi
+
+if [[ -z "${SKIP_SETUP:-}" ]]; then
+    setup_kind_cluster


If i have already installed kind cluster and istio, how can i reuse them

You can use flags to skip some steps, the specific usage method has been added to the proposal.

Signed-off-by: YaoZengzeng <yaozengzeng@huawei.com>

YaoZengzeng · 2024-06-27T09:17:53Z

@hzxuzhonghu PTAL

hzxuzhonghu

/lgtm

kmesh-bot · 2024-06-27T09:26:43Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: hzxuzhonghu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [hzxuzhonghu]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

kmesh-bot added do-not-merge/work-in-progress kind/feature labels Jun 18, 2024

kmesh-bot requested review from hzxuzhonghu and supercharge-xsy June 18, 2024 02:01

kmesh-bot added the size/XXL label Jun 18, 2024

LiZhenCheng9527 reviewed Jun 18, 2024

View reviewed changes

YaoZengzeng force-pushed the e2e branch 4 times, most recently from 1438e87 to 52777b8 Compare June 18, 2024 07:50

YaoZengzeng mentioned this pull request Jun 18, 2024

Bump istio #445

Merged

YaoZengzeng force-pushed the e2e branch 2 times, most recently from 66ff584 to 59a61a6 Compare June 19, 2024 01:52

kmesh-bot added size/XL and removed size/XXL labels Jun 19, 2024

YaoZengzeng force-pushed the e2e branch from 59a61a6 to 54f7afe Compare June 19, 2024 02:12

kmesh-bot added size/XXL and removed size/XL labels Jun 19, 2024

YaoZengzeng force-pushed the e2e branch 4 times, most recently from 1fb6e0a to 8a7ffc4 Compare June 21, 2024 09:22

hzxuzhonghu previously approved these changes Jun 22, 2024

View reviewed changes

kmesh-bot added the approved label Jun 22, 2024

hzxuzhonghu removed the approved label Jun 22, 2024

YaoZengzeng dismissed hzxuzhonghu’s stale review via 6cf790e June 22, 2024 03:17

YaoZengzeng force-pushed the e2e branch from 6cf790e to c8d2502 Compare June 22, 2024 03:19

YaoZengzeng changed the title ~~[WIP] E2E test framework~~ E2E test framework Jun 22, 2024

kmesh-bot removed the do-not-merge/work-in-progress label Jun 22, 2024

hzxuzhonghu reviewed Jun 24, 2024

View reviewed changes

YaoZengzeng added 14 commits June 26, 2024 08:55

add app

22d0ce4

Signed-off-by: YaoZengzeng <yaozengzeng@huawei.com>

deploy test case

c7a8ee2

Signed-off-by: YaoZengzeng <yaozengzeng@huawei.com>

enable deploying waypoint in kmesh

0610c2c

Signed-off-by: YaoZengzeng <yaozengzeng@huawei.com>

pass first e2e test

541612a

Signed-off-by: YaoZengzeng <yaozengzeng@huawei.com>

setup part of local registry

57720cf

Signed-off-by: YaoZengzeng <yaozengzeng@huawei.com>

only pull kmesh image from local registry

968c923

Signed-off-by: YaoZengzeng <yaozengzeng@huawei.com>

add github action workflow

33c6663

Signed-off-by: YaoZengzeng <yaozengzeng@huawei.com>

part of e2e proposal

dade647

Signed-off-by: YaoZengzeng <yaozengzeng@huawei.com>

part II of e2e proposal

3dcc628

Signed-off-by: YaoZengzeng <yaozengzeng@huawei.com>

use helm to install Kmesh

d69fb73

Signed-off-by: YaoZengzeng <yaozengzeng@huawei.com>

fix comments

18062ae

Signed-off-by: YaoZengzeng <yaozengzeng@huawei.com>

fix comments

508dfa4

Signed-off-by: YaoZengzeng <yaozengzeng@huawei.com>

fix comments

5d9141f

Signed-off-by: YaoZengzeng <yaozengzeng@huawei.com>

install istio and kmesh in script

ab1ca85

Signed-off-by: YaoZengzeng <yaozengzeng@huawei.com>

YaoZengzeng force-pushed the e2e branch from ff3ffb1 to ab1ca85 Compare June 26, 2024 08:55

hzxuzhonghu reviewed Jun 26, 2024

View reviewed changes

hzxuzhonghu reviewed Jun 27, 2024

View reviewed changes

test/e2e/run_test.sh Outdated Show resolved Hide resolved

hzxuzhonghu reviewed Jun 27, 2024

View reviewed changes

test/e2e/run_test.sh Outdated Show resolved Hide resolved

hzxuzhonghu reviewed Jun 27, 2024

View reviewed changes

YaoZengzeng added 2 commits June 27, 2024 07:00

fix comments

f3f75c0

Signed-off-by: YaoZengzeng <yaozengzeng@huawei.com>

config local registry when setup cluster

f011a72

Signed-off-by: YaoZengzeng <yaozengzeng@huawei.com>

YaoZengzeng force-pushed the e2e branch from f7ea7bc to f011a72 Compare June 27, 2024 08:11

hzxuzhonghu approved these changes Jun 27, 2024

View reviewed changes

kmesh-bot assigned hzxuzhonghu Jun 27, 2024

kmesh-bot added the lgtm label Jun 27, 2024

kmesh-bot added the approved label Jun 27, 2024

kmesh-bot merged commit ba03077 into kmesh-net:main Jun 27, 2024

YaoZengzeng mentioned this pull request Oct 31, 2025

add YaoZengzeng to the maintainer list #1517

Merged

	go test -v -tags=integ ./test/e2e/...
	go test -v -tags=integ $ROOT_DIR/test/e2e/...


		### Motivation

		E2E testing is a software approach that tests an application's workflow from start to finish, simulating read-user scenarios. The main purpose of E2E testing is to validate the system as a whole,ensuring that all the individual components and integrations work together seamlessly. It helps to identify any issues or defects that may arise from the interation between different components of the appliction, ensuing the application works as expected under normal operating conditions.


		1. Use [namespace](https://github.com/istio/istio/blob/master/pkg/test/framework/components/namespace/namespace.go) package to create namespace for deploying test applications.

		2. Use [deployment](https://github.com/istio/istio/blob/master/pkg/test/framework/components/echo/deployment/builder.go) package to build test applications. `WithClusters()` can be used to specify the clusters where the test applications should be deployed. Each call to `WithConfig()` will generate a test application with corresponding configuration. Actually we create all test applications at the beginning. In each test case, we select some suitable applications for testing. And an application can be used as both a client and a server.

Conversation

YaoZengzeng commented Jun 18, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LiZhenCheng9527 Jun 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hzxuzhonghu left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hzxuzhonghu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

YaoZengzeng commented Jun 27, 2024

Uh oh!

hzxuzhonghu left a comment

Choose a reason for hiding this comment

Uh oh!

kmesh-bot commented Jun 27, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

LiZhenCheng9527 Jun 18, 2024 •

edited

Loading