Add AWS IMDSv2 mode support for ignition and afterburn #220
Description
Description
Booting Flatcar Stable (075585003325/Flatcar-stable-2605.6.0-hvm) (or even Alpha 075585003325/Flatcar-alpha-2661.0.0-hvm) in AWS on an instance with IMDSv2 turned on causes coreos-metadata (or its ignition friend) to exit, which causes #cloud-config to not execute, potentially locking the user out of the instance
Impact
Instances enter "Emergency Mode", which means no ssh access, nor SSM, and generally the instance is orphaned
Environment and steps to reproduce
-
Set-up: please see below
-
Task: booting
-
Action(s):
- boot instance
- observe one cannot connect to it
-
Error: [describe the error that was triggered]
[ 147.653951] ignition[5873]: Ignition v0.34.0-22-g032f620 ...snip... [ 147.844754] ignition[5873]: no config at "/usr/lib/ignition/user.ign" [ 147.851439] systemd[1]: Reached target Host and Network Name Lookups. [ 147.872665] ignition[5873]: GET http://169.254.169.254/2009-04-04/user-data: attempt #1 [ 147.885479] systemd[1]: Starting Ignition (disks)... [ 147.932668] ignition[5873]: GET result: Unauthorized [ 147.937613] systemd[1]: ignition-disks.service: Main process exited, code=exited, status=1/FAILURE [ 147.949266] ignition[5873]: failed to fetch config: failed to fetch resource [ 147.958620] systemd[1]: ignition-disks.service: Failed with result 'exit-code'. [ 148.008043] ignition[5873]: failed to acquire config: failed to fetch resource [ 148.017145] systemd[1]: Failed to start Ignition (disks). [ 148.022175] ignition[5873]: Ignition failed: failed to fetch resource
Expected behavior
The instance should honor the EC2 KeyName parameter, and/or the ssh_authorized_keys: [], and/or run the units: specified in #cloud-config
Additional information
The docs point to container-linux-config-transpiler whose repo is marched "archived," but the supported data by provider page cites the coreos-metadata repo (which is the process in journalctl on stable that fails, not ignition) but that repo redirects to coreos/afterburn which, helpfully, does seem to support IMDSv2
As an aside, I actually can't tell if this is my fault for attempting to use practically a standard or if it's because I haven't run my UserData through some kind of yaml-to-json compiler first. It's similarly confusing to have docs that say do not use #cloud-config but the reference CloudFormation stack still says #cloud-config
Reproduction Steps
I took the suggested stack, used AWS's "hello world VPC" as a substack just to boot up an instance with IMDSv2 to demonstrate what's going on; the interesting bits are:
aws cloudformation create-stack --template-body "$(cat flatcar-alpha-hvm.yaml)" --parameters ParameterKey=KeyPair,ParameterValue=$USER --stack-name flatcar-alpha-hvm
aws cloudformation wait stack-create-complete --stack-name flatcar-alpha-hvm
read -p 'instance-id? ' I_ID
while true; do
aws ec2 get-console-output --instance-id $I_ID | tee flatcar-alpha-hvm.console.log
if grep -q Output flatcar-alpha-hvm.console.log; then break; fi
echo '30s...'
sleep 30
donewhere flatcar-alpha-hvm.yaml is:
AWSTemplateFormatVersion: '2010-09-09'
Description: 'Flatcar Linux on EC2: https://docs.flatcar-linux.org/os/booting-on-ec2/'
Parameters:
InstanceType:
Description: EC2 HVM instance type (m3.medium, etc).
Type: String
Default: m3.medium
ConstraintDescription: Must be a valid EC2 HVM instance type.
KeyPair:
Description: The name of an EC2 Key Pair to allow SSH access to the instance.
but it doesn't matter, since this stack is to demonstrate the console error
# Type: AWS::EC2::KeyPair::KeyName
Type: String
Resources:
FlatcarServerLT:
# one must use an LT to set `MetadataOptions:`
Type: AWS::EC2::LaunchTemplate
Properties:
LaunchTemplateData:
MetadataOptions:
# THIS IS THE MAGIC SAUCE
HttpTokens: required
#/THIS IS THE MAGIC SAUCE
# --- everything below this line isn't important ---
ImageId: !FindInMap
- RegionMap
- Ref: 'AWS::Region'
- AMI
InstanceType: !Ref InstanceType
KeyName: !Ref KeyPair
UserData:
# this can be anything
Fn::Base64: |
#cloud-config
write_files:
- path: /root/boot0.sh
permissions: "0755"
content: |
#! /usr/bin/env bash
echo hello world
VpcStack:
DependsOn:
# put this in flight early to avoid a CFN race condition
- FlatcarServerLT
Type: AWS::CloudFormation::Stack
Properties:
TemplateURL: https://s3.amazonaws.com/cloudformation-templates-us-east-1/VPC_With_PublicIPs_And_DNS.template
Parameters:
KeyName: !Ref KeyPair
# Outputs: [ PublicSubnet, VPCId ]
FlatcarSecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: Flatcar Linux SecurityGroup
SecurityGroupIngress:
- IpProtocol: tcp
FromPort: 22
ToPort: 22
CidrIp: 0.0.0.0/0
VpcId: !GetAtt [VpcStack, Outputs.VPCId]
FlatcarServer:
Type: AWS::EC2::Instance
Properties:
LaunchTemplate:
LaunchTemplateId: !Ref FlatcarServerLT
Version: !GetAtt [FlatcarServerLT, LatestVersionNumber]
SecurityGroupIds:
- Ref: FlatcarSecurityGroup
SubnetId: !GetAtt [VpcStack, Outputs.PublicSubnet]
Mappings:
RegionMap:
eu-central-1:
AMI: ami-0d748198043c0c255
ap-northeast-1:
AMI: ami-0ed3e6682b9c4d495
ap-northeast-2:
AMI: ami-066c23653a0c45465
ca-central-1:
AMI: ami-0308d875f0b87b531
ap-south-1:
AMI: ami-0cd30a2e85af98cca
sa-east-1:
AMI: ami-04db15e4f88a7504c
ap-southeast-2:
AMI: ami-0d83f60257e42437b
ap-southeast-1:
AMI: ami-0174e8bc61f1d1907
us-east-1:
AMI: ami-0c7fa97342e18ae64
us-east-2:
AMI: ami-0c5acb9322e623e4e
us-west-2:
AMI: ami-0544e9163376d21a2
us-west-1:
AMI: ami-0d8cea3887521a7e3
eu-west-1:
AMI: ami-0e23ec5ac0146a7a9
eu-west-2:
AMI: ami-03bb370cfd70f297d
eu-west-3:
AMI: ami-049de7754895b3950
eu-north-1:
AMI: ami-0c85acc9d84626d05
ap-east-1:
AMI: ami-0d03469bdf7d0a3d3
me-south-1:
AMI: ami-00301d028f3249639