What type of PR is this?
/kind bug

Which issue(s) this PR fixes:
Closes #4162

Fix GameServer Not Created on addMoreGameServers Failure

What this PR does / Why we need it
affected version <= 1.50.0
The core of my pull request is that I identified an issue where, if all GameServers are deleted and the addMoreGameServers function fails, no further GameServers are created. To resolve this, I added logic to reprocess the GameServerSet using c.workerqueue.EnqueueAfter.

Without this fix, since no GameServers exist, there are no further GameServer informer updates or resyncs triggered. As a result, the addMoreGameServers function is never called again, and no new GameServers are created.

POC

Imagine that the Agones Controller and Agones Extension are deployed on top of Kubernetes, and there is one GameServerSet (test-gss) with two GameServers (test-gss-1 and test-gss-2).

Delete the two GameServers (test-gss-1 and test-gss-2) while the Agones Extension is in the TLS certs changing state.

# Agones Extension Pod
http: TLS handshake error from ip:port: EOF
http: TLS handshake error from ip:port: EOF
http: TLS handshake error from ip:port: EOF
[...]

# my cli terminal
kubectl delete pod test-gss-1 test-gss-2 -n default

In a normal scenario, the addMoreGameServers function is triggered when a GameServer transitions to the Shutdown state or when its Pod is deleted.

but we can see this error message when invoke addMoreGameServers

{
  "error": "error creating gameserver for gameserverset test-gss: Internal error occurred: failed calling webhook \"mutations.agones.dev\": failed to call webhook: Post \"https://agones-controller-service.agones-system.svc:443/mutate?timeout=10s\": tls: failed to verify certificate: x509~~"
}

Due to the above error, the addMoreGameServers function is no longer invoked.

Solution

The solution is simple: if the addMoreGameServers function call fails, it should be reprocessed using c.workerqueue.EnqueueAfter.

	if numServersToAdd > 0 {
		if err := c.addMoreGameServers(ctx, gsSet, numServersToAdd); err != nil {
+			c.workerqueue.EnqueueAfter(gsSet, 1*time.Second) // retry!
			loggerForGameServerSet(c.baseLogger, gsSet).WithError(err).Warning("error adding game servers and will retry")
		}
	}

Limitation

A potential drawback is that retries may continue indefinitely until success.

If you have any suggestions for a better approach, I'd appreciate your comments.

Special notes for your reviewer:
N/A