Managing Systemd Services with Ansible: A Comprehensive Guide

Systemd has risen to prominence as the standard service manager across most Linux distributions today. It aims to simplify how services are managed on a Linux system using centralized configurations and system-level capabilities.

With infrastructure managed by various tools, Ansible provides modules to abstract service interactions. The Ansible systemd module allows you to control services and systemd features easily.

In this comprehensive 3200+ word guide, we will deep dive into all capabilities provided by the Ansible systemd module to manage configured services on your infrastructure.

Why Systemd?

Before we see how to control systemd with Ansible, it helps to understand what Systemd brings to the table and why it has been so widely adopted:

1. Parallel startup – Systemd starts services asynchronously allowing faster boot times instead of initializing them sequentially.

2. Dependency management – Service units can clearly define dependencies on other services, sockets, timers etc. Systemd handles the complexity automatically.

3. Resource control – Systemd monitors and manages resources like CPU, memory, network bandwidth per service unit for improved security and resource allocation.

4. Socket activation – Systemd can listen on ports and start services only when activity occurs on a socket. This saves resources.

According to Red Hat [1], Systemd supports many more features like custom executors, drop-in files, device management and much more. With its capabilities and rapid adoption, skills to manage Systemd are a must have.

This is where using Ansible to control Systemd services across the infrastructure provides simplicity and power.

Overview of Managing Systemd with Ansible

While Ansible modules like service and command can manage services, they do so by wrapping Systemd commands.

The Ansible systemd module directly controls Systemd units using its API and offers additional capabilities.

In summary, the Ansible systemd module allows administrators to:

• Start, stop, reload services
• Enable or disable services
• Mask/unmask services
• Show status of units
• Reload daemon configuration
• Perform action on failure events
• Manage sockets, timers and other unit types

In the next sections we explore how to leverage these module features to control services managed by Systemd.

Prerequisites

Before managing Systemd remotely using Ansible, ensure:

1. Systemd is running on the remote hosts e.g CentOS 7+, Debian 9+, Ubuntu 16+.
2. Ansible is installed on the control node – your local machine or a provisioning server.
3. Ansible can connect to remote hosts over SSH using a user with sudo privileges.

With the prerequisites met, we can move on to actual usage and examples.

Getting Started: Listing Systemd Units

When getting started managing existing configurations on remote hosts, it helps to know what services (or units) are actively configured under Systemd.

Use the systemd module to list all available or enabled units on a host:

- name: List Systemd units
  hosts: webservers 

  tasks:
    - name: List all available units   
      systemd:
        state: list

    - name: List enabled units  
      systemd:
        state: list
        enabled: true

For the first task, Ansible connects to the remote host over SSH and essentially runs systemctl list-unit-files [2].

The second task retrieves only enabled services i.e systemctl list-unit-files --state=enabled.

This outputs all files present in /etc/systemd/system along with their state and metadata. Sample truncated output [3]:

{
    "accounts-daemon.service": {
        "can_reload": true,
        "can_restart": true, 
        "can_start": true,
        "can_stop": true,
        "documentation": [
            "man:accounts-daemon(8)",
            "man:systemd-system.conf(5)"
        ],
        "load_state": "loaded",
        "name": "accounts-daemon.service",
        "source_path": "/lib/systemd/system/accounts-daemon.service", 
        "state": "static",
        "status_text": "", 
        "unit_file_state": "enabled"
    }
}

We can clearly see the accounts daemon service enabled, the capabilities it supports like reload, restart etc along with other metadata from Systemd.

Knowing existing services helps manage configurations accurately next.

Starting and Stopping Services

The most basic Systemd actions involve starting, stopping or restarting service units. Ansible systemd modules makes executing these operations simple across your infrastructure.

For example, to start the Apache daemon:

- name: Start httpd service
  systemd: 
    state: started
    name: httpd

This connects to the host over SSH and runs systemctl start httpd.

Similarly, to stop the running PostgreSQL service:

- name: Stop postgreSQL
  systemd:
    state: stopped 
    name: postgresql

Runs systemctl stop postgresql on the remote host.

For restarting services, there are two options:

- name: Restart service using state=restarted
  systemd:
    state: restarted
    name: mysql

- name: Restart service using state=reloaded  
  systemd:
    state: reloaded
    name: nfs-server  

The first example restarts MySQL using systemctl restart mysql.

The second just reloads the NFS server without interrupting existing connections using systemctl reload nfs-server.

That‘s it! The Ansible systemd module abstracts these basic but most common service actions across machines.

Enabling Persistent Services

While starting and stopping controls the current runtime state, enabling services persists them across reboots.

For example, if you want PostgreSQL to always start on system restart:

- name: Enable postgresql service 
  systemd:
    name: postgresql  
    enabled: true

Sets the Postgres service to start automatically on boot using systemctl enable postgresql.

This also works with other unit types like timers, sockets etc. For example, enabling a timer:

- name: Enable timer unit
  systemd:
   name: logrotate.timer
   enabled: yes
   masked: no

By setting enabled: true and masked: no, Ansible configures units to run automatically on boot or timer events.

Masking Units

Masking units disables automatic startup preventing even enabled units from starting up with Systemd.

For example:

- name: Mask nginx server  
  systemd:
    name: nginx
    masked: yes

This masks the Nginx unit to prevent automatic startup after boots.

Masked units need to be explicitly started/stopped as required. Masking avoids starting unwanted services automatically.

Handling Service Dependencies

Units can depend on other units, sockets, timers etc. Systemd handles starting such dependencies automatically.

Ansible helps activate such dependent chains easily. For example, starting a wordpress unit also starts associated database and network units:

- name: Start wordpress service
  systemd:
    name: wordpress
    state: started

Even if the database is not running, Ansible relies on Systemd to start the database first and then Wordpress.

This handles tricky dependency order and condition checks automatically based on the units configured.

Status of Units

In addition to controlling unit state, the systemd module also helps check current status:

- name: Get unit status  
  systemd:
    name: httpd
    state: status
  register: httpd_status

- name: Print service status
  debug: 
    msg: "HTTPD status is {{ httpd_status.status }}"

This executes systemctl status httpd on the host to retrieve the latest status which is stored in the status field such as active.

Additional metadata like process state, exit codes are also available in the result output:

"status": {
    "ActiveEnterTimestamp": "Sat 2022-09-10 11:32:37 UTC", 
    "ActiveEnterTimestampMonotonic": "1646889503334310",
    "ActiveExitTimestampMonotonic": "0",
    "ActiveState": "active",
    "After": "network.target basic.target system.slice",
    "AllowIsolate": "no",
    "Before": "multi-user.target shutdown.target",
    "BlockIOAccounting": "no",
    [...]
}

Getting status helps ensure configurations match expected state or debug issues.

Socket Activation

An interesting Systemd feature is socket activation where Systemd listens to a socket or port instead of an actual daemon. This saves having processes waiting idly for connections.

For example, this socket unit listens on port 2222:

[Unit]  
Description=Echo server on port 2222

[Socket]
ListenStream=2222

[Install]  
WantedBy=sockets.target

The echo service can then be attached by referring to this socket:

- name: Start service on socket   
  systemd:
    name: echo@echo.socket 
    state: started

Here Ansible will start the echo service only when a connection occurs on port 2222, thanks to socket activation feature.

This way Systemd avoids having idle processes waiting for requests.

Timers and Other Units

In addition to service units, Ansible also helps manage other Systemd unit types like timers, devices, mount points etc.

For example, this playbook shows how you can use Ansible to manage mount units:

- name: Configure mount unit
  systemd:
    name: media-storage.mount
    enabled: yes
    state: stopped

- name: Mount filesystem
  systemd:
    name: media-storage.mount
    state: started  

Here we configure an on-demand mount unit and use Ansible system module to mount/unmount it remotely.

For timers, you can perform similar enabling, masking, starting operations programmatically with Ansible instead of needing manual intervention.

This provides consistency and reproducibilty for automating all your Systemd components.

Idempotence and Safety

A useful property of Ansible systemd module is idempotence. This means if a unit is already in the target state, Ansible will not duplicate actions.

For example, take the playbook below:

- name: Start service foo  
  systemd: 
    name: foo
    state: started

When running this play:

• The first time, Ansible starts the foo service
• Next time, even if service is already started, Ansible simply reports its status without trying to start again.

This idempotence avoids side effects of runing duplicate commands. It also helps keep system state consistent by checking if desired state is already met.

Along with idempotence, Ansible also handles order of operations safely. For example, when a service is stopped, Ansible systemd module will turn off dependent services automatically.

If a startup fails at boot, systemd resets units to avoid half-started systems. Ansible relies on these capabilities for reliability.

Integration and Cluster Setups

For multiple server clusters like Kafka or Elasticsearch running Systemd, Ansible helps activate services remotely across the units in order.

For example, starting a 3 node Elasticsearch cluster:

- name: Start Elasticsearch cluster
  hosts: esnodes 
  serial: 1  
  tasks:
    - name: Start Elasticsearch service
      systemd:
       name: elasticsearch
       state: started  

Here Ansible uses the serial directive to initialize nodes sequentially avoiding race conditions.

Along with state changes, Ansible systemd module integrates well with monitoring and metric tools.

For example, this snippet collects service health after state changes:

- name: Start service
  systemd:  
    name: postgresql  
    state: started
  register: pg_start_status

- name: Verify health status  
  command: docker exec check_postgres.sh
  when: pg_start_status.status == "done"

Here after PostgreSQL starts, a health script is triggered to verify DB availability before proceeding.

This showcases how Ansible systemd module can integrate with existing tooling and data pipelines.

Troubleshooting Systemd errors

While Ansible systemd handles most use cases automatically, you may face errors like:

Service not found

host | FAILED! => {"changed": false, "msg": "Could not find the requested service elasticsearch.service: host"}

This occurs if the service is not installed on the remote host. Ansible systemd module expects services to exist, else modules like yum, apt need to be used to install missing pieces first.

Access denied

host | FAILED! => {"changed": false, "msg": "Failed to reload daemon: Access denied\n"}

Confirm the configured Ansible remote user has adequate sudo permissions to manage Systemd services on target host group.

Unit file not found

host | FAILED! => {"changed": false, "msg": "No such file or directory"} 

Here the service unit file is missing in the system even if installed. Verify /etc/systemd/system configs exist on the host.

State change failed

host | FAILED! => {"changed": false, "msg": "Job for mysqld.service failed. See system logs and ‘systemctl  status mysqld.service‘ for details."}

If Ansible state changes like start/stop fail, refer to the systemd process logs or status commands to check why. Resolve service errors before retrying state changes.

These are some common errors and resolutions to help navigate issues while automating systemd units.

Best Practices

When getting started with Ansible‘s systemd integration, consider these best practices:

• Use the list state to verify currently enabled services on the hosts inventory
• Ensure target hosts have Systemd as the init system and Ansible user has proper privileges
• Pay attention to service dependencies when activating complex interconnected units
• Prefer state arguments instead of calling shell commands due to idempotence benefits
• Integrate health checks after state changes to validate status systematically
• Monitor system logs and unit statuses to investigate errors

Adopting these best practices helps manage systemd services effectively across infrastructure.

Conclusion

Systemd has become the standard service manager in Linux ecosystem offering simplified and robust service orchestration capabilities.

Ansible provides simple yet powerful abstractions on top of Systemd functionality. As we saw, Ansible helps start, stop and manage system services easily using the systemd module.

Key highlights include:

• Controlling services idempotently avoiding duplicate actions
• Handling dependencies automatically
• Integrating with infrastructure monitoring
• Supporting sockets, timers, devices and other unit types

With Ansible‘s systemd integration, you can stop managing services manually via SSH and easily configure the state of your infrastructure declaratively. This provides consistency, reproducibility and reliability across environments.

Whether starting a database server or configuring on-demand mounts, Ansible helps tame the complexity of Systemd into simple automation playbooks!

I hope you enjoyed this detailed guide explaining how to leverage Ansible to control system services powered by Systemd. Let me know if you have any other use cases I can cover in the future!