Description

Cannot SSH into an OpenStack VM created from a Flatcar 3227.2.0 image if the VM is provisioned with an ssh key (not with ignition user_data). Most likely the ssh key is not actually provisioned.

Impact

Our pipelines that publish Flatcar images to Glance include a basic test that it would be possible to ssh into a VM built from the image and that test fails now.

Environment and steps to reproduce

1. Set-up:
   a. Use a Flatcar image downloaded from https://stable.release.flatcar-linux.net/amd64-usr/current/flatcar_production_openstack_image.img.bz2
   b. Create a glance image from it
   c. Create a compute instance based on the image with a key name (e.g. openstack server create with --key-name xxx, but without --user-data)
2. Task: Try to ssh into the VM as user core
3. Error: Public-key authentication is skipped and you get a password prompt

Expected behavior

ssh with public-key authentication works

Additional information

I tried to provision the same public key via ignition user-data instead and then ssh worked. So it's probably not a problem with the type of key or host key algorithm. With previous Flatcar image releases provisioning the ssh key worked fine.

Command line used to create a VM (where security group ssh is one that allows incoming port 22)

openstack server create <name> --key-name <key-name> --image <image-id> --flavor <flavor-name> --network <network-name> --security-group default --security-group ssh

or with terraform, including the key pair creation:

resource "tls_private_key" "ssh" {
  algorithm   = "RSA"
  rsa_bits    = 4096
}

resource "openstack_compute_keypair_v2" "ssh" {
  name       = "image_test"
  public_key = tls_private_key.ssh.public_key_openssh
}

resource "openstack_compute_instance_v2" "server" {
  name        = "image_test"

  image_id    = "31115d3f-50d9-4290-9cea-d2c362560290"
  flavor_name = "m1c.tiny"
  key_pair    = openstack_compute_keypair_v2.ssh.name

  security_groups = [
    "default",
    openstack_compute_secgroup_v2.sg_ssh.name,
  ]

  network {
    name = openstack_networking_network_v2.network.name
  }

  depends_on = [
    openstack_networking_subnet_v2.subnet,
  ]
}

I tried to understand the difference of the current release to previous ones and my idea is that cloudinit is not run anymore.
Cloudinit used to query the metadata agent via http://169.254.169.254/2009-04-04/meta-data/public-keys/0/openssh-key and provide the returned key for the core user.

If I configure the VM via user-data I can see in the journal that enabling cloudinit is skipped ("enable-oem-cloudinit.service: Skipped due to 'exec-condition'.") and that is probably because there is already user-data. I couldn't read the journal of a VM that was not configured with user-data though, so I try to guess now, what happened.

In enable-oem-cloudinit.service the ExecConditions should match (.userConfigProvided is probably false if the VM was not configured with user-data). So oem-cloudinit.service will most likely be enabled. But in oem-cloudinit.service there is an ExecCondition that checks if the OEM_ID is one from the list (aws gcp rackspace-onmetal azure cloudsigma packet vmware digitalocean) and openstack is not in it. So cloudinit is not run. But we would need it to fetch the public key.

In older releases there was a default ignition file that created and enabled some oem-cloudinit.service that didn't have that ExecCondition, so cloudinit was run (maybe again only if user-data wasn't available).