Skip to content

[kernel] block on cloud-init before beginning any setup#552

Merged
ArangoGutierrez merged 1 commit intomainfrom
block-on-cloud-init
Dec 13, 2025
Merged

[kernel] block on cloud-init before beginning any setup#552
ArangoGutierrez merged 1 commit intomainfrom
block-on-cloud-init

Conversation

@tariq1890
Copy link
Contributor

This change should fix the apt install linux-image-${KERNEL_VERSION} failures observed in our driver container's e2e pipelines:

  1. https://github.com/NVIDIA/gpu-driver-container/actions/runs/20184774381/job/57953216112
  2. https://github.com/NVIDIA/gpu-driver-container/actions/runs/19608858826/job/56152423167

My theory is that holodeck is attempting to perform instance setup a bit too early as I did see some differences in the output of apt-get update command when called from holodeck at different times. Below are the apt-get update outputs gathered from passing pipelines both of which are called from the holodeck run but at different times:

apt-get update output 1

+ sudo apt-get update -y
Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB]
Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease
Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB]
Get:4 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [2892 kB]
Get:5 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB]
Get:6 http://archive.ubuntu.com/ubuntu jammy/universe amd64 Packages [14.1 MB]
Get:7 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [416 kB]
Get:8 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [14.0 kB]
Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [4867 kB]
Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [913 kB]
Get:11 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [1007 kB]
Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [221 kB]
Get:13 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [22.3 kB]
Get:14 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [50.5 kB]
Get:15 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [10.2 kB]
Get:16 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [376 B]
Get:17 http://archive.ubuntu.com/ubuntu jammy/universe Translation-en [5652 kB]
Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 c-n-f Metadata [286 kB]
Get:19 http://archive.ubuntu.com/ubuntu jammy/multiverse amd64 Packages [217 kB]
Get:20 http://archive.ubuntu.com/ubuntu jammy/multiverse Translation-en [112 kB]
Get:21 http://archive.ubuntu.com/ubuntu jammy/multiverse amd64 c-n-f Metadata [8372 B]
Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [3157 kB]
Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [484 kB]
Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [19.0 kB]
Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [5044 kB]
Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [944 kB]
Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1245 kB]
Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [310 kB]
Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [30.0 kB]
Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [57.6 kB]
Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [13.2 kB]
Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [600 B]
Get:33 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [69.4 kB]
Get:34 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.5 kB]
Get:35 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [412 B]
Get:36 http://archive.ubuntu.com/ubuntu jammy-backports/restricted amd64 c-n-f Metadata [116 B]
Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [30.1 kB]
Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.6 kB]
Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B]
Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/multiverse amd64 c-n-f Metadata [116 B]
Fetched 42.6 MB in 8s (5514 kB/s)

apt-get output 2

+ sudo apt-get update
Get:1 http://us-west-1.ec2.archive.ubuntu.com/ubuntu jammy InRelease [270 kB]
Get:2 http://us-west-1.ec2.archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB]
Get:3 http://us-west-1.ec2.archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB]
Get:4 http://us-west-1.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 Packages [1395 kB]
Hit:5 http://security.ubuntu.com/ubuntu jammy-security InRelease
Get:6 http://us-west-1.ec2.archive.ubuntu.com/ubuntu jammy/main Translation-en [510 kB]
Get:7 http://us-west-1.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 c-n-f Metadata [30.3 kB]
Get:8 http://us-west-1.ec2.archive.ubuntu.com/ubuntu jammy/restricted amd64 Packages [129 kB]
Get:9 http://us-west-1.ec2.archive.ubuntu.com/ubuntu jammy/restricted Translation-en [18.6 kB]
Get:10 http://us-west-1.ec2.archive.ubuntu.com/ubuntu jammy/restricted amd64 c-n-f Metadata [488 B]
Get:11 http://us-west-1.ec2.archive.ubuntu.com/ubuntu jammy/universe amd64 Packages [14.1 MB]
Get:12 http://us-west-1.ec2.archive.ubuntu.com/ubuntu jammy/universe Translation-en [5652 kB]
Get:13 http://us-west-1.ec2.archive.ubuntu.com/ubuntu jammy/universe amd64 c-n-f Metadata [286 kB]
Get:14 http://us-west-1.ec2.archive.ubuntu.com/ubuntu jammy/multiverse amd64 Packages [217 kB]
Get:15 http://us-west-1.ec2.archive.ubuntu.com/ubuntu jammy/multiverse Translation-en [112 kB]
Get:16 http://us-west-1.ec2.archive.ubuntu.com/ubuntu jammy/multiverse amd64 c-n-f Metadata [8372 B]
Get:17 http://us-west-1.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [3157 kB]
Get:18 http://us-west-1.ec2.archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [484 kB]
Get:19 http://us-west-1.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [19.0 kB]
Get:20 http://us-west-1.ec2.archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [5044 kB]
Get:21 http://us-west-1.ec2.archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [944 kB]
Get:22 http://us-west-1.ec2.archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [640 B]
Get:23 http://us-west-1.ec2.archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1245 kB]
Get:24 http://us-west-1.ec2.archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [310 kB]
Get:25 http://us-west-1.ec2.archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [30.0 kB]
Get:26 http://us-west-1.ec2.archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [57.6 kB]
Get:27 http://us-west-1.ec2.archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [13.2 kB]
Get:28 http://us-west-1.ec2.archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [600 B]
Get:29 http://us-west-1.ec2.archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [69.4 kB]
Get:30 http://us-west-1.ec2.archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.5 kB]
Get:31 http://us-west-1.ec2.archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [412 B]
Get:32 http://us-west-1.ec2.archive.ubuntu.com/ubuntu jammy-backports/restricted amd64 c-n-f Metadata [116 B]
Get:33 http://us-west-1.ec2.archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [30.1 kB]
Get:34 http://us-west-1.ec2.archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.6 kB]
Get:35 http://us-west-1.ec2.archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B]
Get:36 http://us-west-1.ec2.archive.ubuntu.com/ubuntu jammy-backports/multiverse amd64 c-n-f Metadata [116 B]
Fetched 34.4 MB in 5s (7304 kB/s)

Comparing the above two outputs, one can see that the http://us-west-1.ec2.archive.ubuntu.com shows up as an apt source later. This leads to me believe that we may be running our apt calls too prematurely before cloud-init gets a chance to complete its setup.

Signed-off-by: Tariq Ibrahim <tibrahim@nvidia.com>
@coveralls
Copy link

coveralls commented Dec 13, 2025

Pull Request Test Coverage Report for Build 20185497932

Details

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage remained the same at 13.102%

Totals Coverage Status
Change from base Build 20030137740: 0.0%
Covered Lines: 272
Relevant Lines: 2076

💛 - Coveralls

@tariq1890
Copy link
Contributor Author

This change is inspired by this comment hashicorp/packer#2639 (comment)

@ArangoGutierrez ArangoGutierrez merged commit 690eb4d into main Dec 13, 2025
19 checks passed
@tariq1890 tariq1890 deleted the block-on-cloud-init branch December 13, 2025 07:36
ArangoGutierrez added a commit to ArangoGutierrez/holodeck that referenced this pull request Jan 8, 2026
- Add cloud-init status --wait to common functions so ALL templates
  wait for cloud-init to complete before provisioning
- Remove duplicate cloud-init wait from kernel.go (now in common)
- Increase container runtime startup timeouts to 120s for slow VMs
- Log progress every 15s instead of 10s during wait loops

This addresses race conditions where provisioning starts before
cloud-init finishes setting up apt sources and systemd services.

Signed-off-by: Carlos Eduardo Arango Gutierrez <eduardoa@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants