Test: Retry apt and snap downloads#1140
Conversation
There was a problem hiding this comment.
Pull request overview
This PR introduces a retry mechanism to handle transient failures when downloading packages from external sources during test execution. The changes add a retry() function that attempts operations up to 10 times with exponential backoff (5, 10, 15... 50 seconds between retries).
- Implements a new
retry()helper function with exponential backoff - Wraps all
apt-getupdate/install operations with retry logic - Wraps all
snap refreshandsnap installoperations (from external sources) with retry logic
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| test/includes/microcloud.sh | Adds the retry() function and wraps apt-get and snap install commands in the system setup with retry logic |
| test/suites/upgrade.sh | Wraps all snap refresh commands during upgrade tests with retry logic to handle download failures |
| test/suites/basic.sh | Wraps snap refresh command in the service mismatch test with retry logic |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
2af329f to
f5bca4e
Compare
f5bca4e to
1762f47
Compare
This allows running an arbitrary command ten times. Signed-off-by: Julian Pelizäus <julian.pelizaeus@canonical.com>
Signed-off-by: Julian Pelizäus <julian.pelizaeus@canonical.com>
Signed-off-by: Julian Pelizäus <julian.pelizaeus@canonical.com>
Signed-off-by: Julian Pelizäus <julian.pelizaeus@canonical.com>
As we are now running tests actively on both edge and candidate channels, we should rather not have this check to also keep the noise on the pipeline summary page low. Signed-off-by: Julian Pelizäus <julian.pelizaeus@canonical.com>
1762f47 to
6c48cd6
Compare
| retry lxc exec "${name}" -- snap install snapd | ||
| retry lxc exec "${name}" -- snap install yq |
There was a problem hiding this comment.
For another PR, yq is best installed with apt-get on 24.04 as it's faster than pulling the snap.
For now, I think just snap install yq will also bring snapd
There was a problem hiding this comment.
Ah yes, I remember you providing this hint some time ago, most likely in the LXD CI repo when I was adding checks for the backup configs which use YAML.
Will fix in a follow up PR.
| @@ -723,7 +723,7 @@ test_service_mismatch() { | |||
| reset_systems 1 3 1 | |||
|
|
|||
| lxc exec micro01 -- sh -c "echo 1 > /proc/sys/net/ipv6/conf/enp5s0/disable_ipv6" | |||
There was a problem hiding this comment.
Unrelated but the sh wrapper could be avoided:
| lxc exec micro01 -- sh -c "echo 1 > /proc/sys/net/ipv6/conf/enp5s0/disable_ipv6" | |
| lxc exec micro01 -- sysctl net.ipv6.conf.eth0.disable_ipv6=1 |
There was a problem hiding this comment.
Thanks, let me fix this in a follow up.
Just found two more snap installs in our custom `system-test` GH action. One of them just recently failed here during install: https://github.com/canonical/microcloud/actions/runs/20171402120/job/57908105443?pr=1132. To reuse the same `retry` func, I am sourcing the respective file. The PR also addresses the comments in #1140.
Try to better deal with stability issues in the infrastructure when performing apt/snap downloads.
Also added a commit to drop the
check_snap_channelsfunc to reduce noise in the pipeline summary page.