Fix: patch flaky tests from submit-upgrade-test-cloud-build by lacroixthomas · Pull Request #4236 · agones-dev/agones

lacroixthomas · 2025-07-31T22:00:13Z

What type of PR is this?
/kind bug

What this PR does / Why we need it:
Fix flaky test from submit-upgrade-test-cloud-build

Ensure the gameserver are shuting down before uninstall helmrelease

Which issue(s) this PR fixes:

Closes #4235

Special notes for your reviewer:

agones-bot · 2025-08-01T00:02:42Z

Build Succeeded 🥳

Build Id: 47cfb212-6fc5-41f8-b760-a1d0e3b00943

The following development artifacts have been built, and will exist for the next 30 days:

image: us-docker.pkg.dev/agones-images/ci/agones-controller:1.51.0-dev-de2296c
image: us-docker.pkg.dev/agones-images/ci/agones-extensions:1.51.0-dev-de2296c
image: us-docker.pkg.dev/agones-images/ci/agones-sdk:1.51.0-dev-de2296c-linux
image: us-docker.pkg.dev/agones-images/ci/agones-ping:1.51.0-dev-de2296c
image: us-docker.pkg.dev/agones-images/ci/agones-allocator:1.51.0-dev-de2296c
Linux C++ SDK (build): agonessdk-1.51.0-dev-de2296c-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.51.0-dev-de2296c.zip

A preview of the website (the last 30 builds are retained):

https://de2296c-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/googleforgames/agones.git pull/4236/head:pr_4236 && git checkout pr_4236
helm install agones ./install/helm/agones --namespace agones-system --set agones.image.registry=us-docker.pkg.dev/agones-images/ci --set agones.image.tag=1.51.0-dev-de2296c

lacroixthomas · 2025-08-01T00:06:30Z

/gcbrun

agones-bot · 2025-08-01T01:09:25Z

Build Succeeded 🥳

Build Id: 28043ba6-150c-4dec-a2e7-078d803f04eb

The following development artifacts have been built, and will exist for the next 30 days:

image: us-docker.pkg.dev/agones-images/ci/agones-controller:1.51.0-dev-de2296c
image: us-docker.pkg.dev/agones-images/ci/agones-extensions:1.51.0-dev-de2296c
image: us-docker.pkg.dev/agones-images/ci/agones-sdk:1.51.0-dev-de2296c-linux
image: us-docker.pkg.dev/agones-images/ci/agones-ping:1.51.0-dev-de2296c
image: us-docker.pkg.dev/agones-images/ci/agones-allocator:1.51.0-dev-de2296c
Linux C++ SDK (build): agonessdk-1.51.0-dev-de2296c-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.51.0-dev-de2296c.zip

A preview of the website (the last 30 builds are retained):

https://de2296c-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/googleforgames/agones.git pull/4236/head:pr_4236 && git checkout pr_4236
helm install agones ./install/helm/agones --namespace agones-system --set agones.image.registry=us-docker.pkg.dev/agones-images/ci --set agones.image.tag=1.51.0-dev-de2296c

agones-bot · 2025-08-01T05:34:13Z

Build Succeeded 🥳

Build Id: 1a50399c-1717-4d34-8458-6a7edf7dd5db

The following development artifacts have been built, and will exist for the next 30 days:

image: us-docker.pkg.dev/agones-images/ci/agones-controller:1.52.0-dev-d40fbb7
image: us-docker.pkg.dev/agones-images/ci/agones-extensions:1.52.0-dev-d40fbb7
image: us-docker.pkg.dev/agones-images/ci/agones-sdk:1.52.0-dev-d40fbb7-linux
image: us-docker.pkg.dev/agones-images/ci/agones-ping:1.52.0-dev-d40fbb7
image: us-docker.pkg.dev/agones-images/ci/agones-allocator:1.52.0-dev-d40fbb7
Linux C++ SDK (build): agonessdk-1.52.0-dev-d40fbb7-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.52.0-dev-d40fbb7.zip

A preview of the website (the last 30 builds are retained):

https://d40fbb7-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/googleforgames/agones.git pull/4236/head:pr_4236 && git checkout pr_4236
helm install agones ./install/helm/agones --namespace agones-system --set agones.image.registry=us-docker.pkg.dev/agones-images/ci --set agones.image.tag=1.52.0-dev-d40fbb7

markmandel · 2025-08-03T21:00:53Z

test/upgrade/main.go

 	cancel()
-	// TODO: Replace sleep with wait for the existing healthy Game Servers finish naturally by reaching their shutdown phase.
-	time.Sleep(30 * time.Second)
+


Whoops! That one fell through the cracks. Glad to the TODO was there to provide guideance.

LGTM, but would like @igooch to triple check as she was the one who originally wrote this.

agones-bot · 2025-08-04T16:10:14Z

Build Failed 😭

Build Id: bd56edee-9493-489d-b52a-df61519c557a

Status: FAILURE

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

lacroixthomas · 2025-08-04T16:11:44Z

Well, that's new.. https://console.cloud.google.com/cloud-build/builds/5096077b-a0db-473a-97d0-87320db99856;step=0?inv=1&invt=Ab4lyw&project=agones-images

ERROR: failed to solve: executor failed running [/bin/sh -c curl -L  ${HELM_URL} > /tmp/helm.tar.gz     && tar -zxvf /tmp/helm.tar.gz -C /tmp     && mv /tmp/linux-amd64/helm /usr/local/bin/helm     && chmod go+rx /usr/local/bin/helm     && rm /tmp/helm.tar.gz && rm -rf /tmp/linux-amd64]: exit code: 7
------
0.402 curl: (7) Failed to connect to get.helm.sh port 443 after 23 ms: Connection refused
0.386 
0.382                                  Dload  Upload   Total   Spent    Left  Speed
 % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
 > [ 6/11] RUN curl -L  https://get.helm.sh/helm-v3.5.0-linux-amd64.tar.gz > /tmp/helm.tar.gz     && tar -zxvf /tmp/helm.tar.gz -C /tmp     && mv /tmp/linux-amd64/helm /usr/local/bin/helm     && chmod go+rx /usr/local/bin/helm     && rm /tmp/helm.tar.gz && rm -rf /tmp/linux-amd64:
------
#10 ERROR: executor failed running [/bin/sh -c curl -L  ${HELM_URL} > /tmp/helm.tar.gz     && tar -zxvf /tmp/helm.tar.gz -C /tmp     && mv /tmp/linux-amd64/helm /usr/local/bin/helm     && chmod go+rx /usr/local/bin/helm     && rm /tmp/helm.tar.gz && rm -rf /tmp/linux-amd64]: exit code: 7
#10 0.402 curl: (7) Failed to connect to get.helm.sh port 443 after 23 ms: Connection refused

lacroixthomas · 2025-08-04T16:11:50Z

/gcbrun

agones-bot · 2025-08-04T16:25:13Z

Build Failed 😭

Build Id: f97e2dc3-136a-4588-8fe7-d8fea6e93977

Status: FAILURE

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

lacroixthomas · 2025-08-04T16:30:39Z

Seems that it's still flacky, it's not always the same error though

igooch · 2025-08-04T16:57:06Z

Well, that's new.. https://console.cloud.google.com/cloud-build/builds/5096077b-a0db-473a-97d0-87320db99856;step=0?inv=1&invt=Ab4lyw&project=agones-images

ERROR: failed to solve: executor failed running [/bin/sh -c curl -L  ${HELM_URL} > /tmp/helm.tar.gz     && tar -zxvf /tmp/helm.tar.gz -C /tmp     && mv /tmp/linux-amd64/helm /usr/local/bin/helm     && chmod go+rx /usr/local/bin/helm     && rm /tmp/helm.tar.gz && rm -rf /tmp/linux-amd64]: exit code: 7
------
0.402 curl: (7) Failed to connect to get.helm.sh port 443 after 23 ms: Connection refused
0.386 
0.382                                  Dload  Upload   Total   Spent    Left  Speed
 % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
 > [ 6/11] RUN curl -L  https://get.helm.sh/helm-v3.5.0-linux-amd64.tar.gz > /tmp/helm.tar.gz     && tar -zxvf /tmp/helm.tar.gz -C /tmp     && mv /tmp/linux-amd64/helm /usr/local/bin/helm     && chmod go+rx /usr/local/bin/helm     && rm /tmp/helm.tar.gz && rm -rf /tmp/linux-amd64:
------
#10 ERROR: executor failed running [/bin/sh -c curl -L  ${HELM_URL} > /tmp/helm.tar.gz     && tar -zxvf /tmp/helm.tar.gz -C /tmp     && mv /tmp/linux-amd64/helm /usr/local/bin/helm     && chmod go+rx /usr/local/bin/helm     && rm /tmp/helm.tar.gz && rm -rf /tmp/linux-amd64]: exit code: 7
#10 0.402 curl: (7) Failed to connect to get.helm.sh port 443 after 23 ms: Connection refused

Haven't seen this one before. It might be an issue with Helm version, or possibly with the way the variable is declared: https://github.com/search?q=repo%3Agoogleforgames%2Fagones%20HELM_URL&type=code. In any case these setups should all be the same, and they've diverged.

agones-bot · 2025-08-04T17:01:09Z

Build Failed 😭

Build Id: 12efc4ea-908a-4a60-918a-1737916f3fd2

Status: FAILURE

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

igooch · 2025-08-04T17:39:41Z

Build Failed 😭

Build Id: f97e2dc3-136a-4588-8fe7-d8fea6e93977

Status: FAILURE

Cloud Build view

Cloud Build log download

The most recent failure is Error from server (NotFound): gameservers.agones.dev "sdk-client-test-hqrr5" not found.

We have a log (times are UTC) for deleting a job that was still running:

INFO 2025-08-04T16:25:04.707185589Z Step #25 - "submit-upgrade-test-cloud-build": Deleting job from previous run of upgrade-test-runner on cluster standard-upgrade-test-cluster-1-31.

from https://github.com/googleforgames/agones/blob/231024b332b723b6b60be3ed9057ef42715e85b5/build/e2e_upgrade_test.sh#L94

But no log for Deleting game servers from previous run of upgrade-test-runner on cluster from https://github.com/googleforgames/agones/blob/231024b332b723b6b60be3ed9057ef42715e85b5/build/e2e_upgrade_test.sh#L104

Which is odd. There's also no Running command delete gs -l app=sdk-client-test coming from within the test itself, which would be expected if the test itself was trying to delete the game server.
https://github.com/googleforgames/agones/blob/231024b332b723b6b60be3ed9057ef42715e85b5/test/upgrade/main.go#L240

In general though it seems like the failed test runs are not cleaning up properly.

For this PR I'd suggest that we wait for all non-unheathly game servers, including those in Shutdown state to finish before continuing. So probably doing the inverse, i.e. wait until there are no game servers or all known game servers are "Unhealthy".

markmandel · 2025-08-04T18:04:08Z

0.402 curl: (7) Failed to connect to get.helm.sh port 443 after 23 ms: Connection refused

That looks like get.helm.sh was just not playing nice. So external dependency being flaky. Not an us problem.

markmandel · 2025-08-04T18:07:47Z

Deleting agones-system namespace from previous run of upgrade-test-runner on cluster standard-upgrade-test-cluster-1-31.
namespace "agones-system" deleted
Error from server (InternalError): an error on the server ("Internal Server Error: \"/api/v1/namespaces/agones-system\": the server is currently unable to handle the request") has prevented the request from succeeding (get namespaces agones-system)

Is what I saw in the last log -- is that the issue? there was some issue deleting the namespace?

markmandel · 2025-08-04T18:07:53Z

/gcbrun

agones-bot · 2025-08-04T19:05:39Z

Build Succeeded 🥳

Build Id: d09c5852-6f8a-4875-b839-04954609024f

The following development artifacts have been built, and will exist for the next 30 days:

image: us-docker.pkg.dev/agones-images/ci/agones-controller:1.52.0-dev-4d12197
image: us-docker.pkg.dev/agones-images/ci/agones-extensions:1.52.0-dev-4d12197
image: us-docker.pkg.dev/agones-images/ci/agones-sdk:1.52.0-dev-4d12197-linux
image: us-docker.pkg.dev/agones-images/ci/agones-ping:1.52.0-dev-4d12197
image: us-docker.pkg.dev/agones-images/ci/agones-allocator:1.52.0-dev-4d12197
image: us-docker.pkg.dev/agones-images/ci/agones-processor:1.52.0-dev-4d12197
Linux C++ SDK (build): agonessdk-1.52.0-dev-4d12197-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.52.0-dev-4d12197.zip

A preview of the website (the last 30 builds are retained):

https://4d12197-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/googleforgames/agones.git pull/4236/head:pr_4236 && git checkout pr_4236
helm install agones ./install/helm/agones --namespace agones-system --set agones.image.registry=us-docker.pkg.dev/agones-images/ci --set agones.image.tag=1.52.0-dev-4d12197

markmandel · 2025-08-05T16:08:06Z

test/upgrade/main.go

+		// Count Game Servers that are still in healthy states (not shutdown/terminated)
+		healthyCount := 0
+		for _, gs := range gameServers.Items {
+			state := gs.Status.State
+			// Consider Game Servers healthy if they're in Ready, Allocated, or Reserved states
+			if state == agonesv1.GameServerStateReady ||
+				state == agonesv1.GameServerStateAllocated ||
+				state == agonesv1.GameServerStateReserved {
+				healthyCount++
+			}
+		}
+
+		log.Printf("Waiting for %d healthy Game Servers to shutdown", healthyCount)
+
+		// Return true when no healthy Game Servers remain
+		return healthyCount == 0, nil


Suggested change

// Count Game Servers that are still in healthy states (not shutdown/terminated)

healthyCount := 0

for _, gs := range gameServers.Items {

state := gs.Status.State

// Consider Game Servers healthy if they're in Ready, Allocated, or Reserved states

if state == agonesv1.GameServerStateReady ||

state == agonesv1.GameServerStateAllocated ||

state == agonesv1.GameServerStateReserved {

healthyCount++

}

}

log.Printf("Waiting for %d healthy Game Servers to shutdown", healthyCount)

// Return true when no healthy Game Servers remain

return healthyCount == 0, nil

return len(gameServers) == 0, nil

Is what we're saying?

Out of curiosity - why do we wait for GameServers to be totally deleted?

Yea, not fully sure anymore
The call just after is cleanUpResources(), which already delete gs etc.: https://github.com/googleforgames/agones/blob/231024b332b723b6b60be3ed9057ef42715e85b5/test/upgrade/main.go#L564C1-L592C1

If we want we can do some check there to ensure everything is cleaned-up in between steps, to avoid the uninstall failing / timing out if there is still gs running ? 🤔

It's a guess, but the issue could be that:

We delete the GS using kubectl, they are in shutdown etc. Them we delete the helm release without waiting that it's done, which start but never finish and fails on delete namespace as there is still resources on it ?

The issue being that we dont wait between the delete gs and the unisntall of the helm release, we might set a finalizers and it never be deleted as the helm uninstall doesn't wait that the resources are cleaned up, this might be the explanation about why we want to wait that they are either non ready or deleted

Ill apply this changes on the clean resources fonction, to wait that the GS are cleaned up before uninstalling the helmrelease

Actually, we could just do a helm uninstall agones -n agones-system --wait --timeout which should wait that all resources are deleted, on need to "wait" for the game servers to be terminated manually

@igooch correct me if I'm wrong, but does that defeat the purpose of the test - which is to test in place upgrades.

But I've not dug that deep into what's going on in this test, so take what I say with a large grain of salt.

From what I understood it's only after the test that runs the in place upgrades (runConfigWalker) which is called just before the cleanup: here, at the really end, once the test is done

But I might have missed something or miss-understood some part 😄

agones-bot · 2025-08-09T01:24:33Z

Build Failed 😭

Build Id: e235f4bb-d7e1-4fec-8a1e-dcac0849ac2f

Status: FAILURE

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

agones-bot · 2025-08-09T02:26:04Z

Build Succeeded 🥳

Build Id: bdbba812-6297-47b1-870e-2be6367530de

The following development artifacts have been built, and will exist for the next 30 days:

image: us-docker.pkg.dev/agones-images/ci/agones-controller:1.52.0-dev-c9bdecc
image: us-docker.pkg.dev/agones-images/ci/agones-extensions:1.52.0-dev-c9bdecc
image: us-docker.pkg.dev/agones-images/ci/agones-sdk:1.52.0-dev-c9bdecc-linux
image: us-docker.pkg.dev/agones-images/ci/agones-ping:1.52.0-dev-c9bdecc
image: us-docker.pkg.dev/agones-images/ci/agones-allocator:1.52.0-dev-c9bdecc
image: us-docker.pkg.dev/agones-images/ci/agones-processor:1.52.0-dev-c9bdecc
Linux C++ SDK (build): agonessdk-1.52.0-dev-c9bdecc-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.52.0-dev-c9bdecc.zip

A preview of the website (the last 30 builds are retained):

https://c9bdecc-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/googleforgames/agones.git pull/4236/head:pr_4236 && git checkout pr_4236
helm install agones ./install/helm/agones --namespace agones-system --set agones.image.registry=us-docker.pkg.dev/agones-images/ci --set agones.image.tag=1.52.0-dev-c9bdecc

markmandel · 2025-08-09T23:41:00Z

Also possible I have no idea 😁

markmandel · 2025-08-09T23:41:29Z

/gcbrun

agones-bot · 2025-08-10T00:14:56Z

Build Failed 😭

Build Id: 7c27b5f6-077d-42f7-886c-90a5efdc99dd

Status: FAILURE

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

markmandel · 2025-08-10T00:33:06Z

Could also be there are multiple flakes in here.

lacroixthomas · 2025-08-10T13:43:16Z

This error might gonna be fixed by this PR : #4224

lacroixthomas · 2025-08-10T13:43:31Z

/gcbrun

agones-bot · 2025-08-10T14:40:03Z

Build Succeeded 🥳

Build Id: 6d155009-00dc-43b2-8499-de68fad42baf

The following development artifacts have been built, and will exist for the next 30 days:

image: us-docker.pkg.dev/agones-images/ci/agones-controller:1.52.0-dev-c9bdecc
image: us-docker.pkg.dev/agones-images/ci/agones-extensions:1.52.0-dev-c9bdecc
image: us-docker.pkg.dev/agones-images/ci/agones-sdk:1.52.0-dev-c9bdecc-linux
image: us-docker.pkg.dev/agones-images/ci/agones-ping:1.52.0-dev-c9bdecc
image: us-docker.pkg.dev/agones-images/ci/agones-allocator:1.52.0-dev-c9bdecc
image: us-docker.pkg.dev/agones-images/ci/agones-processor:1.52.0-dev-c9bdecc
Linux C++ SDK (build): agonessdk-1.52.0-dev-c9bdecc-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.52.0-dev-c9bdecc.zip

A preview of the website (the last 30 builds are retained):

https://c9bdecc-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/googleforgames/agones.git pull/4236/head:pr_4236 && git checkout pr_4236
helm install agones ./install/helm/agones --namespace agones-system --set agones.image.registry=us-docker.pkg.dev/agones-images/ci --set agones.image.tag=1.52.0-dev-c9bdecc

igooch

LGTM

github-actions bot added kind/bug These are bugs. size/S labels Jul 31, 2025

markmandel requested a review from igooch July 31, 2025 22:07

markmandel approved these changes Aug 3, 2025

View reviewed changes

igooch mentioned this pull request Aug 4, 2025

Use same Helm package manager for all tests #4237

Closed

markmandel reviewed Aug 5, 2025

View reviewed changes

Fix flacky test around helm uninstall

376240f

lacroixthomas force-pushed the bugfixes/fix-flacky-test-25 branch from 4d12197 to 376240f Compare August 9, 2025 00:50

Merge branch 'main' into bugfixes/fix-flacky-test-25

1f3206a

github-actions bot added the size/XS label Aug 9, 2025

lacroixthomas added 2 commits August 8, 2025 21:25

Fix: remove sleep for resources to be cleaned up

77c6e1a

Fix: add timeout on helm uninstall --wait

c9bdecc

igooch approved these changes Aug 12, 2025

View reviewed changes

igooch merged commit 9e74311 into agones-dev:main Aug 12, 2025
4 checks passed

lacroixthomas mentioned this pull request Aug 13, 2025

Approver request #4245

Closed

5 tasks

Conversation

lacroixthomas commented Jul 31, 2025

Uh oh!

agones-bot commented Aug 1, 2025

Uh oh!

lacroixthomas commented Aug 1, 2025

Uh oh!

agones-bot commented Aug 1, 2025

Uh oh!

agones-bot commented Aug 1, 2025

Uh oh!

markmandel Aug 3, 2025

Choose a reason for hiding this comment

Uh oh!

agones-bot commented Aug 4, 2025

Uh oh!

lacroixthomas commented Aug 4, 2025

Uh oh!

lacroixthomas commented Aug 4, 2025

Uh oh!

agones-bot commented Aug 4, 2025

Uh oh!

lacroixthomas commented Aug 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

igooch commented Aug 4, 2025

Uh oh!

agones-bot commented Aug 4, 2025

Uh oh!

igooch commented Aug 4, 2025

Uh oh!

markmandel commented Aug 4, 2025

Uh oh!

markmandel commented Aug 4, 2025

Uh oh!

markmandel commented Aug 4, 2025

Uh oh!

agones-bot commented Aug 4, 2025

Uh oh!

markmandel Aug 5, 2025

Choose a reason for hiding this comment

Uh oh!

lacroixthomas Aug 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lacroixthomas Aug 9, 2025

Choose a reason for hiding this comment

Uh oh!

lacroixthomas Aug 9, 2025

Choose a reason for hiding this comment

Uh oh!

lacroixthomas Aug 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

markmandel Aug 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lacroixthomas Aug 9, 2025

Choose a reason for hiding this comment

Uh oh!

agones-bot commented Aug 9, 2025

Uh oh!

agones-bot commented Aug 9, 2025

Uh oh!

markmandel commented Aug 9, 2025

Uh oh!

markmandel commented Aug 9, 2025

Uh oh!

agones-bot commented Aug 10, 2025

Uh oh!

markmandel commented Aug 10, 2025

Uh oh!

lacroixthomas commented Aug 10, 2025

Uh oh!

lacroixthomas commented Aug 10, 2025

Uh oh!

agones-bot commented Aug 10, 2025

lacroixthomas commented Aug 4, 2025 •

edited

Loading

lacroixthomas Aug 5, 2025 •

edited

Loading

lacroixthomas Aug 9, 2025 •

edited

Loading

markmandel Aug 9, 2025 •

edited

Loading