Handle missing upgrade-test-runner pod to avoid log collection errors by 0xaravindh · Pull Request #4224 · agones-dev/agones

0xaravindh · 2025-07-23T05:57:24Z

What type of PR is this?

Uncomment only one /kind <> line, press enter to put that in a new line, and remove leading whitespace from that line:

/kind breaking
/kind bug

/kind cleanup

/kind documentation
/kind feature
/kind hotfix
/kind release

What this PR does / Why we need it:

Which issue(s) this PR fixes:

Closes #

Special notes for your reviewer:

Avoids errors when no pod is found for the job. Adds a check before running 'kubectl logs' to prevent running it with an empty pod name

agones-bot · 2025-07-23T06:27:17Z

Build Failed 😭

Build Id: 207e43f9-5b7d-4a10-a6ea-b1934df227d2

Status: FAILURE

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

agones-bot · 2025-07-23T14:22:56Z

Build Succeeded 🥳

Build Id: 758d4b29-b376-4fa1-8472-a9d6794b85ec

The following development artifacts have been built, and will exist for the next 30 days:

image: us-docker.pkg.dev/agones-images/ci/agones-controller:1.51.0-dev-e771a50
image: us-docker.pkg.dev/agones-images/ci/agones-extensions:1.51.0-dev-e771a50
image: us-docker.pkg.dev/agones-images/ci/agones-sdk:1.51.0-dev-e771a50-linux
image: us-docker.pkg.dev/agones-images/ci/agones-ping:1.51.0-dev-e771a50
image: us-docker.pkg.dev/agones-images/ci/agones-allocator:1.51.0-dev-e771a50
Linux C++ SDK (build): agonessdk-1.51.0-dev-e771a50-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.51.0-dev-e771a50.zip

A preview of the website (the last 30 builds are retained):

https://e771a50-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/googleforgames/agones.git pull/4224/head:pr_4224 && git checkout pr_4224
helm install agones ./install/helm/agones --namespace agones-system --set agones.image.registry=us-docker.pkg.dev/agones-images/ci --set agones.image.tag=1.51.0-dev-e771a50

link.sh

agones-bot · 2025-07-25T05:25:20Z

Build Failed 😭

Build Id: 18a243e3-68f8-4586-9820-71c445cf56c1

Status: FAILURE

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

0xaravindh · 2025-07-25T06:00:57Z

/gcbrun

agones-bot · 2025-07-25T06:29:14Z

Build Failed 😭

Build Id: d62b41aa-f2ab-4f91-87f0-58fc9b82b931

Status: FAILURE

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

lacroixthomas · 2025-07-25T16:27:31Z

From the error logs of the failed build, I can see that it the logs looks correct: {"lastProbeTime":"2025-07-25T06:23:12Z","lastTransitionTime":"2025-07-25T06:23:12Z","message":"Reached expected number of succeeded pods","reason":"CompletionsReached","status":"True","type":"SuccessCriteriaMet"}{"lastProbeTime":"2025-07-25T06:24:47Z","lastTransitionTime":"2025-07-25T06:24:47Z","message":"Reached expected number of succeeded pods","reason":"CompletionsReached","status":"True","type":"SuccessCriteriaMet"}{"lastProbeTime":"2025-07-25T06:27:03Z","lastTransitionTime":"2025-07-25T06:27:03Z","message":"Reached expected number of succeeded pods","reason":"CompletionsReached","status":"True","type":"SuccessCriteriaMet"}{"lastProbeTime":"2025-07-25T06:28:44Z","lastTransitionTime":"2025-07-25T06:28:44Z","message":"Reached expected number of succeeded pods","reason":"CompletionsReached","status":"True","type":"SuccessCriteriaMet"}

But I'm still pretty sure it's using the wrong kube context. It's the call of the print_failure_logs from this line: https://github.com/googleforgames/agones/blob/main/build/e2e_upgrade_test.sh#L191

        outputLog="${waitPids[$pid]}"
        echo "One of the upgrade tests pid $pid from cluster log $outputLog exited with a non-zero status ${status}."
        print_failure_logs "$(basename "$outputLog" .log)"

The function print_failure_logs do some calls with kubectl, but it doesn't have the right kube context, it only have the last kube context set (which currently seems to have valid logs / no errors). The last kube context set from this loop: https://github.com/googleforgames/agones/blob/main/build/e2e_upgrade_test.sh#L70-L84

It's maybe worth trying to run gcloud container clusters get-credentials "$testCluster" --region="$testClusterLocation" --project="$PROJECT_ID" before calling this print_failure_logs ? Or setting it in the function ?

What do you think about it ?

0xaravindh · 2025-07-25T17:10:14Z

Cloud Build view

@lacroixthomas this issue is not related to the upgrade test. We're encountering problems in the e2e test on gke-autopilot-e2e-test-cluster-1-33 due to node unavailability. . link

lacroixthomas · 2025-07-25T17:23:59Z

Cloud Build view

@lacroixthomas this issue is not related to the upgrade test. We're encountering problems in the e2e test on gke-autopilot-e2e-test-cluster-1-33 due to node unavailability. . link

Oh ok, but I was expecting to see some error logs regarding it though, maybe I was not looking at the right logs, will double check that

I don't have access to this link, seems to be internal to Google, it's asking SS0 from google

0xaravindh · 2025-07-28T04:35:38Z

/gcbrun

0xaravindh · 2025-07-29T09:46:04Z

/gcbrun

0xaravindh · 2025-07-30T04:33:32Z

/gcbrun

0xaravindh · 2025-07-30T14:14:22Z

/gcbrun

agones-bot · 2025-07-30T21:21:19Z

Build Failed 😭

Build Id: fb60720a-60a5-46f4-9c82-d4a9f179fe10

Status: FAILURE

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

agones-bot · 2025-07-30T21:21:20Z

Build Failed 😭

Build Id: 453e1389-ddff-4065-b723-e2f9ed2bb51c

Status: FAILURE

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

agones-bot · 2025-07-30T21:21:20Z

Build Failed 😭

Build Id: 4e22ef2b-ec81-4dc1-ad1b-fd64b0740b02

Status: FAILURE

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

agones-bot · 2025-07-30T21:21:20Z

Build Failed 😭

Build Id: 502f4f56-d54c-49fb-9d60-7a7de015598a

Status: FAILURE

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

agones-bot · 2025-07-30T21:21:20Z

Build Failed 😭

Build Id: a055be56-6a12-487f-af4e-1ac1da618fbd

Status: FAILURE

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

agones-bot · 2025-07-31T00:20:58Z

Build Failed 😭

Build Id: 23bb5739-3b26-4ce7-8b98-a99c92102552

Status: FAILURE

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

build/e2e_upgrade_test.sh

…ogs-ci

agones-bot · 2025-08-05T07:00:11Z

Build Succeeded 🥳

Build Id: 2f62dda8-8475-4424-8e10-9d63e5d47a32

The following development artifacts have been built, and will exist for the next 30 days:

image: us-docker.pkg.dev/agones-images/ci/agones-controller:1.52.0-dev-940962d
image: us-docker.pkg.dev/agones-images/ci/agones-extensions:1.52.0-dev-940962d
image: us-docker.pkg.dev/agones-images/ci/agones-sdk:1.52.0-dev-940962d-linux
image: us-docker.pkg.dev/agones-images/ci/agones-ping:1.52.0-dev-940962d
image: us-docker.pkg.dev/agones-images/ci/agones-allocator:1.52.0-dev-940962d
image: us-docker.pkg.dev/agones-images/ci/agones-processor:1.52.0-dev-940962d
Linux C++ SDK (build): agonessdk-1.52.0-dev-940962d-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.52.0-dev-940962d.zip

A preview of the website (the last 30 builds are retained):

https://940962d-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/googleforgames/agones.git pull/4224/head:pr_4224 && git checkout pr_4224
helm install agones ./install/helm/agones --namespace agones-system --set agones.image.registry=us-docker.pkg.dev/agones-images/ci --set agones.image.tag=1.52.0-dev-940962d

agones-bot · 2025-08-07T05:46:25Z

Build Failed 😭

Build Id: 65666d55-bc2a-4afd-b71e-446d82a1038f

Status: FAILURE

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

0xaravindh · 2025-08-07T06:21:45Z

Looks like the VirtualBox website (https://www.virtualbox.org/) is currently down. I’ll wait for a couple of hours and try again.

igooch

There's two more calls to print_failure_logs "$(basename "$outputLog" .log)" that will need the correct cluster region passed through. Or alternatively you could make an associative array that maps cluster name -> cluster location, and just pass in the cluster name to the print_failure_logs() function, and query the associative array for the region.

agones-bot · 2025-08-08T06:11:54Z

Build Succeeded 🥳

Build Id: 793cba50-b810-41b5-b1fe-c40d863ea711

The following development artifacts have been built, and will exist for the next 30 days:

image: us-docker.pkg.dev/agones-images/ci/agones-controller:1.52.0-dev-d860f15
image: us-docker.pkg.dev/agones-images/ci/agones-extensions:1.52.0-dev-d860f15
image: us-docker.pkg.dev/agones-images/ci/agones-sdk:1.52.0-dev-d860f15-linux
image: us-docker.pkg.dev/agones-images/ci/agones-ping:1.52.0-dev-d860f15
image: us-docker.pkg.dev/agones-images/ci/agones-allocator:1.52.0-dev-d860f15
image: us-docker.pkg.dev/agones-images/ci/agones-processor:1.52.0-dev-d860f15
Linux C++ SDK (build): agonessdk-1.52.0-dev-d860f15-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.52.0-dev-d860f15.zip

A preview of the website (the last 30 builds are retained):

https://d860f15-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/googleforgames/agones.git pull/4224/head:pr_4224 && git checkout pr_4224
helm install agones ./install/helm/agones --namespace agones-system --set agones.image.registry=us-docker.pkg.dev/agones-images/ci --set agones.image.tag=1.52.0-dev-d860f15

agones-bot · 2025-08-13T06:14:47Z

Build Failed 😭

Build Id: 65b7f682-adc6-496c-ad16-7e56fb533b73

Status: FAILURE

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

0xaravindh · 2025-08-13T07:08:32Z

/gcbrun

agones-bot · 2025-08-13T12:09:53Z

Build Failed 😭

Build Id: cb706c18-f5ec-48fc-b654-44ad4db2c81b

Status: TIMEOUT

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

igooch · 2025-08-14T16:22:17Z

/gcbrun

agones-bot · 2025-08-14T16:57:37Z

Build Failed 😭

Build Id: cdda7cc9-bad4-43e9-90f3-8d236f0a3ff5

Status: FAILURE

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

lacroixthomas · 2025-08-15T22:59:08Z

/gcbrun

agones-bot · 2025-08-15T23:55:32Z

Build Succeeded 🥳

Build Id: 77e500ac-0cde-4ee5-a857-c20b79afd9bf

The following development artifacts have been built, and will exist for the next 30 days:

image: us-docker.pkg.dev/agones-images/ci/agones-controller:1.52.0-dev-5634b55
image: us-docker.pkg.dev/agones-images/ci/agones-extensions:1.52.0-dev-5634b55
image: us-docker.pkg.dev/agones-images/ci/agones-sdk:1.52.0-dev-5634b55-linux
image: us-docker.pkg.dev/agones-images/ci/agones-ping:1.52.0-dev-5634b55
image: us-docker.pkg.dev/agones-images/ci/agones-allocator:1.52.0-dev-5634b55
image: us-docker.pkg.dev/agones-images/ci/agones-processor:1.52.0-dev-5634b55
Linux C++ SDK (build): agonessdk-1.52.0-dev-5634b55-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.52.0-dev-5634b55.zip

A preview of the website (the last 30 builds are retained):

https://5634b55-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/googleforgames/agones.git pull/4224/head:pr_4224 && git checkout pr_4224
helm install agones ./install/helm/agones --namespace agones-system --set agones.image.registry=us-docker.pkg.dev/agones-images/ci --set agones.image.tag=1.52.0-dev-5634b55

fix: skip log collection if upgrade-test-runner pod is missing

ca47e01

Avoids errors when no pod is found for the job. Adds a check before running 'kubectl logs' to prevent running it with an empty pod name

github-actions bot added kind/cleanup Refactoring code, fixing up documentation, etc size/S labels Jul 23, 2025

fixes

e771a50

0xaravindh requested review from gongmax and igooch July 23, 2025 14:34

gongmax reviewed Jul 24, 2025

View reviewed changes

link.sh Outdated Show resolved Hide resolved

fixes

dc58c60

Merge branch 'main' into logs-ci

5125945

Merge branch 'main' into logs-ci

df90ea9

igooch requested changes Aug 4, 2025

View reviewed changes

build/e2e_upgrade_test.sh Show resolved Hide resolved

Merge branch 'googleforgames:main' into logs-ci

49988c8

0xaravindh added 3 commits August 5, 2025 05:58

fixes

f9c7eed

Merge branch 'logs-ci' of https://github.com/0xaravindh/agones into l…

b8f8616

…ogs-ci

fixes

940962d

gongmax approved these changes Aug 6, 2025

View reviewed changes

gongmax requested a review from igooch August 6, 2025 16:03

Merge branch 'main' into logs-ci

64d5ea9

igooch requested changes Aug 7, 2025

View reviewed changes

Fix print_failure_logs calls by passing cluster region correctly

d860f15

lacroixthomas mentioned this pull request Aug 10, 2025

Fix: patch flaky tests from submit-upgrade-test-cloud-build #4236

Merged

Merge branch 'main' into logs-ci

5634b55

igooch approved these changes Aug 14, 2025

View reviewed changes

igooch enabled auto-merge (squash) August 14, 2025 16:21

lacroixthomas mentioned this pull request Aug 15, 2025

Feat: implement rust sdk (counter and list) #4247

Merged

igooch merged commit f0c08d3 into agones-dev:main Aug 15, 2025
4 checks passed

Conversation

0xaravindh commented Jul 23, 2025

Uh oh!

agones-bot commented Jul 23, 2025

Uh oh!

agones-bot commented Jul 23, 2025

Uh oh!

Uh oh!

agones-bot commented Jul 25, 2025

Uh oh!

0xaravindh commented Jul 25, 2025

Uh oh!

agones-bot commented Jul 25, 2025

Uh oh!

lacroixthomas commented Jul 25, 2025

Uh oh!

0xaravindh commented Jul 25, 2025

Uh oh!

lacroixthomas commented Jul 25, 2025

Uh oh!

0xaravindh commented Jul 28, 2025

Uh oh!

0xaravindh commented Jul 29, 2025

Uh oh!

0xaravindh commented Jul 30, 2025

Uh oh!

0xaravindh commented Jul 30, 2025

Uh oh!

agones-bot commented Jul 30, 2025

Uh oh!

agones-bot commented Jul 30, 2025

Uh oh!

agones-bot commented Jul 30, 2025

Uh oh!

agones-bot commented Jul 30, 2025

Uh oh!

agones-bot commented Jul 30, 2025

Uh oh!

agones-bot commented Jul 31, 2025

Uh oh!

Uh oh!

agones-bot commented Aug 5, 2025

Uh oh!

agones-bot commented Aug 7, 2025

Uh oh!

0xaravindh commented Aug 7, 2025

Uh oh!

igooch left a comment

Choose a reason for hiding this comment

Uh oh!

agones-bot commented Aug 8, 2025

Uh oh!

agones-bot commented Aug 13, 2025

Uh oh!

0xaravindh commented Aug 13, 2025

Uh oh!

agones-bot commented Aug 13, 2025

Uh oh!

igooch commented Aug 14, 2025

Uh oh!

agones-bot commented Aug 14, 2025

Uh oh!

lacroixthomas commented Aug 15, 2025

Uh oh!

agones-bot commented Aug 15, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants