@larkee Do we have any guidance on how long we should be expecting the backend to take making / restoring backups? The latest set of failures are showing 10 minute timeouts failing for backup samples: 3.6, 3.7, 3.8.

Ugh, at this point the system-3.8 job has run for 90 minutes without completing.

@larkee Do we have any guidance on how long we should be expecting the backend to take making / restoring backups? The latest set of failures are showing 10 minute timeouts failing for backup samples: 3.6, 3.7, 3.8.

Unfortunately, backup creation and database restores are run as low priority so they can often take a long time if higher priority operations come in. We use 10 minute timeouts for backup creation and 20 minutes for database restores. Unfortunately, these will occasionally fail as there is no guarantee they will complete within this time.

Ugh, at this point the system-3.8 job has run for 90 minutes without completing.

This is likely related to the above point. There are currently two restores and five backup creations which may account for this long execution time as currently the backup test do not specify a timeout when waiting for a result (e.g. operation.result() rather than operation.result(1200)). I will put up a PR which will add the timeout and reduce the number of backup created by one which should prevent it for running as long.