Controller for Pod in Succeeded state. by markmandel · Pull Request #4201 · agones-dev/agones

markmandel · 2025-06-06T07:43:08Z

What type of PR is this?

Uncomment only one /kind <> line, press enter to put that in a new line, and remove leading whitespace from that line:

/kind breaking

/kind bug

/kind cleanup
/kind documentation
/kind feature
/kind hotfix
/kind release

What this PR does / Why we need it:

When sidecar containers are enabled, if a Pod exit(0)s, then it will end up in a Succeeded state, and the backing GameServer will not get deleted.

This introduces a new controller to check for this Pod state and move the GameServer to a Shutdown state.

Also updates all records of simple-game-server to the latest version, since we need it to test that this fix works.

Which issue(s) this PR fixes:

Closes #4188

Special notes for your reviewer:

Won't pass until https://us-docker.pkg.dev/agones-images/examples/simple-game-server:0.38 is published.

agones-bot · 2025-06-06T07:49:47Z

Build Failed 😭

Build Id: 0d598dd6-57f6-4526-be3c-bf5b43bd7ce6

Status: FAILURE

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

0xaravindh · 2025-06-11T09:39:49Z

/gcbrun

agones-bot · 2025-06-11T09:47:14Z

Build Failed 😭

Build Id: 5926551c-2d54-4be0-bf50-f8a1af75f2e8

Status: FAILURE

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

markmandel · 2025-06-11T10:54:52Z

Gah, lint test failure! Rookie mistake! 🤣

agones-bot · 2025-06-11T12:06:11Z

Build Failed 😭

Build Id: c9e7a4f8-88f8-4606-b11c-938ccebc616b

Status: FAILURE

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

markmandel · 2025-06-12T05:12:35Z

Welp, something went definitely wrong here:

VERBOSE:     gameserver_test.go:326: 
VERBOSE:         	Error Trace:	/go/src/agones.dev/agones/test/e2e/gameserver_test.go:326
VERBOSE:         	Error:      	Received unexpected error:
VERBOSE:         	            	GameServer reached terminal state Shutdown
VERBOSE:         	            	agones.dev/agones/test/e2e/framework.(*Framework).WaitForGameServerState.func1
VERBOSE:         	            		/go/src/agones.dev/agones/test/e2e/framework/framework.go:279
VERBOSE:         	            	k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func2
VERBOSE:         	            		/go/src/agones.dev/agones/vendor/k8s.io/apimachinery/pkg/util/wait/loop.go:87
VERBOSE:         	            	k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext
VERBOSE:         	            		/go/src/agones.dev/agones/vendor/k8s.io/apimachinery/pkg/util/wait/loop.go:88
VERBOSE:         	            	k8s.io/apimachinery/pkg/util/wait.PollUntilContextTimeout
VERBOSE:         	            		/go/src/agones.dev/agones/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:48
VERBOSE:         	            	agones.dev/agones/test/e2e/framework.(*Framework).WaitForGameServerState
VERBOSE:         	            		/go/src/agones.dev/agones/test/e2e/framework/framework.go:259
VERBOSE:         	            	agones.dev/agones/test/e2e.TestGameServerUnhealthyAfterDeletingPod
VERBOSE:         	            		/go/src/agones.dev/agones/test/e2e/gameserver_test.go:325
VERBOSE:         	            	testing.tRunner
VERBOSE:         	            		/usr/local/go/src/testing/testing.go:1690
VERBOSE:         	            	runtime.goexit
VERBOSE:         	            		/usr/local/go/src/runtime/asm_amd64.s:1700
VERBOSE:         	            	waiting for GameServer 1749643344/game-servergxm9n to be Unhealthy
VERBOSE:         	            	agones.dev/agones/test/e2e/framework.(*Framework).WaitForGameServerState
VERBOSE:         	            		/go/src/agones.dev/agones/test/e2e/framework/framework.go:288
VERBOSE:         	            	agones.dev/agones/test/e2e.TestGameServerUnhealthyAfterDeletingPod
VERBOSE:         	            		/go/src/agones.dev/agones/test/e2e/gameserver_test.go:325
VERBOSE:         	            	testing.tRunner
VERBOSE:         	            		/usr/local/go/src/testing/testing.go:1690
VERBOSE:         	            	runtime.goexit
VERBOSE:         	            		/usr/local/go/src/runtime/asm_amd64.s:1700
VERBOSE:         	Test:       	TestGameServerUnhealthyAfterDeletingPod
VERBOSE: --- FAIL: TestGameServerUnhealthyAfterDeletingPod (9.76s)
VERBOSE: FAIL

When sidecar containers are enabled, if a Pod `exit(0)`s, then it will end up in a Succeeded state, and the backing GameServer will not get deleted. This introduces a new controller to check for this Pod state and move the `GameServer` to a `Shutdown` state. Also updates all records of simple-game-server to the latest version, since we need it to test that this fix works. Closes agones-dev#4188

markmandel · 2025-06-13T23:11:23Z

🤞🏻 I believe that should fix all those issues.

agones-bot · 2025-06-13T23:40:05Z

Build Failed 😭

Build Id: a62d63b0-fb18-4333-803a-9fff2fece65f

Status: FAILURE

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

markmandel · 2025-06-14T02:08:09Z

Gah! #4163 got me again.

Warning: autopilot-default-resources-mutator:Autopilot updated Job default/upgrade-test-runner: defaulted unspecified 'cpu' resource for containers [upgrade-test-controller] (see http://g.co/gke/autopilot-defaults).
job.batch/upgrade-test-runner created
pod/upgrade-test-runner-q6shk condition met
Wait for job upgrade-test-runner to complete or fail on cluster gke-autopilot-upgrade-test-cluster-1-32
SuccessCriteriaMetSuccessCriteriaMetSuccessCriteriaMetSuccessCriteriaMetSuccessCriteriaMet/tmp/tmp.9Mw2PmalbK/standard-upgrade-test-cluster-1-31.log: SuccessCriteriaMet
/tmp/tmp.9Mw2PmalbK/standard-upgrade-test-cluster-1-33.log: SuccessCriteriaMet
/tmp/tmp.9Mw2PmalbK/standard-upgrade-test-cluster-1-32.log: SuccessCriteriaMet
FailureTarget/tmp/tmp.9Mw2PmalbK/gke-autopilot-upgrade-test-cluster-1-31.log: FailureTarget

agones-bot · 2025-06-14T03:00:23Z

Build Failed 😭

Build Id: 23ea4f3f-2a99-4a6c-b248-32e540b65d56

Status: FAILURE

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

markmandel · 2025-06-14T06:37:57Z

Ooh, new bug.

Feature flags disabled.

time="2025-06-14 02:56:56.232" level=info msg="checking if GameServer exists" error="<nil>" gs=game-servertlcr2 state=Unhealthy test=TestGameServerPodCompletedAfterCleanExit
time="2025-06-14 02:56:59.231" level=info msg="checking if GameServer exists" error="<nil>" gs=game-servertlcr2 state=Unhealthy test=TestGameServerPodCompletedAfterCleanExit
time="2025-06-14 02:57:02.187" level=info msg="checking if GameServer exists" error="client rate limiter Wait returned an error: context deadline exceeded" gs=game-servertlcr2 state= test=TestGameServerPodCompletedAfterCleanExit
    gameserver_test.go:544: 
        	Error Trace:	/go/src/agones.dev/agones/test/e2e/gameserver_test.go:544
        	Error:      	Received unexpected error:
        	            	client rate limiter Wait returned an error: context deadline exceeded
        	Test:       	TestGameServerPodCompletedAfterCleanExit
--- FAIL: TestGameServerPodCompletedAfterCleanExit (183.19s)

agones-bot · 2025-06-14T07:11:59Z

Build Failed 😭

Build Id: fab54ff8-f77c-497c-9347-ec4805f480c1

Status: FAILURE

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

agones-bot · 2025-06-14T12:06:42Z

Build Failed 😭

Build Id: 1015e240-5015-44bb-84a8-f41488273b27

Status: FAILURE

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

* Ignore terminating pods. * Ignores Error or Unhealthy states. * Minor cleanup of failing e2e test, to help with error reporting.

markmandel · 2025-06-14T12:15:33Z

Blarg. More submit-upgrade-test-cloud-build flakiness. Unless I'm doing something wrong? 🤔

…rContainers

agones-bot · 2025-06-14T12:25:45Z

Build Failed 😭

Build Id: e58b3483-5854-40dc-bcfc-287047224b6e

Status: FAILURE

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

markmandel · 2025-06-16T00:04:07Z

Okay, seems like submit-upgrade-test-cloud-build is failing for a reason, since it's failed twice in a row -- @igooch , @0xaravindh , @peterzhongyi - do you have special abilities to tell me why, and what's the reason?

markmandel · 2025-06-16T07:53:39Z

eeyyy! it finally passed. Okay, was just bad flakiness.

We should really get that step de-flaked.

agones-bot · 2025-06-23T10:18:46Z

Build Succeeded 🥳

Build Id: afb9221a-7879-48af-9fb4-06b1c7feed0c

The following development artifacts have been built, and will exist for the next 30 days:

image: us-docker.pkg.dev/agones-images/ci/agones-controller:1.51.0-dev-7c1a019
image: us-docker.pkg.dev/agones-images/ci/agones-extensions:1.51.0-dev-7c1a019
image: us-docker.pkg.dev/agones-images/ci/agones-sdk:1.51.0-dev-7c1a019-linux
image: us-docker.pkg.dev/agones-images/ci/agones-ping:1.51.0-dev-7c1a019
image: us-docker.pkg.dev/agones-images/ci/agones-allocator:1.51.0-dev-7c1a019
Linux C++ SDK (build): agonessdk-1.51.0-dev-7c1a019-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.51.0-dev-7c1a019.zip

A preview of the website (the last 30 builds are retained):

https://7c1a019-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/googleforgames/agones.git pull/4201/head:pr_4201 && git checkout pr_4201
helm install agones ./install/helm/agones --namespace agones-system --set agones.image.registry=us-docker.pkg.dev/agones-images/ci --set agones.image.tag=1.51.0-dev-7c1a019

markmandel · 2025-06-26T17:08:16Z

bump bump @igooch , @peterzhongyi @gongmax please.

lacroixthomas · 2025-06-26T17:49:52Z

LGTM

igooch

LGTM

github-actions bot added the kind/bug These are bugs. label Jun 6, 2025

markmandel requested review from igooch and peterzhongyi June 6, 2025 07:43

github-actions bot added the size/M label Jun 6, 2025

markmandel force-pushed the example/crash-exist-code branch from aff1ca0 to 354b91d Compare June 11, 2025 11:32

github-actions bot added the size/L label Jun 11, 2025

markmandel added 2 commits June 14, 2025 08:00

Fix lint issues.

a3d1061

markmandel force-pushed the example/crash-exist-code branch from 354b91d to a14eb74 Compare June 13, 2025 23:11

markmandel force-pushed the example/crash-exist-code branch from a14eb74 to c36629c Compare June 14, 2025 02:08

markmandel force-pushed the example/crash-exist-code branch from b1d1c1a to 8e6055a Compare June 14, 2025 11:36

Fix bugs in SucceededController

c36629c

* Ignore terminating pods. * Ignores Error or Unhealthy states. * Minor cleanup of failing e2e test, to help with error reporting.

Disable TestGameServerPodCompletedAfterCleanExit during FeatureSideca…

6ec5418

…rContainers

markmandel force-pushed the example/crash-exist-code branch from 8e6055a to 6ec5418 Compare June 14, 2025 12:15

markmandel requested a review from gongmax June 22, 2025 22:14

Merge branch 'main' into example/crash-exist-code

7c1a019

igooch approved these changes Jun 27, 2025

View reviewed changes

igooch merged commit 6b6fb43 into agones-dev:main Jun 27, 2025
4 checks passed

markmandel deleted the example/crash-exist-code branch June 30, 2025 03:33

Conversation

markmandel commented Jun 6, 2025

Uh oh!

agones-bot commented Jun 6, 2025

Uh oh!

0xaravindh commented Jun 11, 2025

Uh oh!

agones-bot commented Jun 11, 2025

Uh oh!

markmandel commented Jun 11, 2025

Uh oh!

agones-bot commented Jun 11, 2025

Uh oh!

markmandel commented Jun 12, 2025

Uh oh!

markmandel commented Jun 13, 2025

Uh oh!

agones-bot commented Jun 13, 2025

Uh oh!

markmandel commented Jun 14, 2025

Uh oh!

agones-bot commented Jun 14, 2025

Uh oh!

markmandel commented Jun 14, 2025

Uh oh!

agones-bot commented Jun 14, 2025

Uh oh!

agones-bot commented Jun 14, 2025

Uh oh!

markmandel commented Jun 14, 2025

Uh oh!

agones-bot commented Jun 14, 2025

Uh oh!

markmandel commented Jun 16, 2025

Uh oh!

markmandel commented Jun 16, 2025

Uh oh!

agones-bot commented Jun 23, 2025

Uh oh!

markmandel commented Jun 26, 2025

Uh oh!

lacroixthomas commented Jun 26, 2025

Uh oh!

igooch left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants