Should we be doing this for both databases?

Frontend?

Added to frontend, too ✅

Would it be possible to ensure that the database is actually starting up and not unresponsive via pg_isready?

Basically in the startup condition we want:

• a small window where postgres isn't bound to the socket
• a larger window where postgres is actively not accepting connections ("server is starting" error from clients) but replaying the WAL

Would that help? Whether the startup probe fails because Postgres hasn't bound to the socket yet or it's rejecting connections, the probe would return a failure either way. K8s would continue probing until it uses up all of the failureThreshold retries.

The only way I can see any benefit from pg_isready is if it's use is split across probes:

• Startup probe (short timeout?): succeeds when Postgres has bound to the socket (but might not be accepting connections yet)
• Liveness probe (long timeout?): succeeds when Postgres is accepting connections

Even if we did that, the only potential benefit I see is in the case where Postgres is failing to bind to the socket due to some bad temporary state in the container that would get wiped upon restart. K8s would restart the container and Postgres would become ready after the shorter timeout rather than the longer timeout. That case doesn't seem very likely.

My idea as code!

What you have now:

for i = 0; i < 360 && !started(); i++ {
     sleep 10s
}

What pg_isready would allow:

for i = 0; i < 360 && !started() && starting(); i++ {
     sleep 10s
}

The difference is that the first one will continue to loop on more critical conditions besides the database starting up. If the database doesn't start at all it might take an hour to restart.

K8s would continue probing until it uses up all of the failureThreshold retries.

Ooh I see now. We can't say "hard exit" from the startup probe.

Won't the first one continue to loop on fewer conditions? The second one has the additional condition && starting(). 🤔

Ideally we'd be able to do the following but I'm not sure k8s probes are enough to accomplish it:

• if we're not bound or starting up (after like 10s) then exit HARD because postgres isn't doing anything
• if we're starting up keep polling
• if we're ready succeed the probe and go ready

Not sure we can distinguish the last two. Failures are failures (unless there's some exit code shenanigans we can do, like exit 256 sends a different signal to k8s).

I don't think probes can hard exit https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#define-startup-probes