SSM Agent fails to start on latest alpha #1307
Description
Description
I'm testing out the latest Alpha (3815.0.0) on EC2 and I see that the amazon-ssm-agent service is throwing an error and is not able to start. Basically:
amazon-ssm-agent[8929]: 2023-12-28 15:29:32 WARN [ssm-agent-worker] failed to read runtime config 'identity_config.json': open /var/lib/amazon/ssm/runtimeconfig/identity_config.json: no such file or directory
Impact
The impact is not high since rolling back to the latest Stable AMI (3602.2.3) does not produce the error, but I wanted to make this visible somewhere.
Environment and steps to reproduce
- Set-up: Arm64 EC2 instance running Alpha (3815.0.0) and a few docker containers via systemd. All EC2 instance roles are configured correctly to allow the SSM agent to connect.
- Task: After launching the test instance, I tried to execute some SSM documents and noticed that no managed instance targets were found (because the SSM agent wasn't connected).
- Action(s): In my EC2 launch-template, I simply swapped Stable AMI with Alpha AMI and re-launched the instance.
- Error: I see multiple errors related to
identity_config.json. See below for some examples. The systemd service does not start and enters a failed state after some time.
amazon-ssm-agent[8929]: 2023/12/28 15:29:32 Found config file at /etc/amazon/ssm/amazon-ssm-agent.json.
amazon-ssm-agent[8929]: Applying config override from /etc/amazon/ssm/amazon-ssm-agent.json.
amazon-ssm-agent[8929]: 2023/12/28 15:29:32 processing appconfig overrides
amazon-ssm-agent[8929]: 2023-12-28 15:29:32 WARN [ssm-agent-worker] failed to read runtime config 'identity_config.json': open /var/lib/amazon/ssm/runtimeconfig/identity_config.json: no such file or directory
amazon-ssm-agent[8929]: 2023-12-28 15:29:32 INFO [ssm-agent-worker] Checking if agent identity type CustomIdentity can be assumed
amazon-ssm-agent[8929]: 2023-12-28 15:29:32 WARN [ssm-agent-worker] failed to read runtime config 'identity_config.json': open /var/lib/amazon/ssm/runtimeconfig/identity_config.json: no such file or directory
amazon-ssm-agent[8929]: 2023-12-28 15:29:32 ERROR [ssm-agent-worker] Agent failed to assume any identity
amazon-ssm-agent[8929]: 2023-12-28 15:29:32 ERROR [ssm-agent-worker] failed to find identity, retrying: failed to find agent identity
amazon-ssm-agent[8929]: 2023-12-28 15:29:32 INFO [ssm-agent-worker] Checking if agent identity type OnPrem can be assumed
amazon-ssm-agent[8929]: 2023-12-28 15:29:32 WARN [ssm-agent-worker] failed to read runtime config 'identity_config.json': open /var/lib/amazon/ssm/runtimeconfig/identity_config.json: no such file or directory
amazon-ssm-agent[8929]: 2023-12-28 15:29:32 INFO [ssm-agent-worker] Checking if agent identity type EC2 can be assumed
amazon-ssm-agent[8929]: 2023-12-28 15:29:32 WARN [ssm-agent-worker] failed to read runtime config 'identity_config.json': open /var/lib/amazon/ssm/runtimeconfig/identity_config.json: no such file or directory
amazon-ssm-agent[8929]: 2023-12-28 15:29:32 INFO [ssm-agent-worker] Checking if agent identity type CustomIdentity can be assumed
amazon-ssm-agent[8929]: 2023-12-28 15:29:32 WARN [ssm-agent-worker] failed to read runtime config 'identity_config.json': open /var/lib/amazon/ssm/runtimeconfig/identity_config.json: no such file or directory
amazon-ssm-agent[8929]: 2023-12-28 15:29:32 ERROR [ssm-agent-worker] Agent failed to assume any identity
amazon-ssm-agent[8929]: 2023-12-28 15:29:32 ERROR [ssm-agent-worker] error occurred when starting ssm-agent-worker: failed to find agent identity
systemd[1]: amazon-ssm-agent.service: Deactivated successfully.
Expected behavior
The amazon-ssm-agent service should start normally and the agent should connect.
Additional information
The only change I made to my configuration was the Flatcar version (AMI) for my region. As I mentioned, with the latest Stable version, everything works perfectly. I'm not sure what additional information to provide, but I am available to provide more if needed.