Running Scripts and Commands During Debian Boot with Systemd

Booting a modern Debian system is far more complex under the hood than simply "turning on" the machine. Myriads of background services, socket activations, device initialization, filesystem mounting and more occur in a robust and controlled manner.

Systemd has become the new standard as the init system that handles the userspace boot process and system management across most Linux distributions. Its modular architecture based on unit files offers unmatched flexibility to customize service startups.

In this comprehensive expert guide, we will cover:

• Diving deep into the systemd architecture
• Mapping the Debian Linux boot procedure
• Creating and customizing systemd service units
• Integrating scripts into different boot targets
• Following best practices for robust script design
• Troubleshooting common errors and issues

So let‘s understand how you can directly shape system behavior during boot with custom scripts using systemd.

An Expert Look Under the Hood of Systemd

At its core, systemd provides a system and service manager for Linux operating systems. It is the first process to start at boot even before the kernel begins initializing services.

systemd ──> kernel ──> services

Thus, systemd handles starting the userspace components and applications after the core OS components are loaded in memory.

The key goals of systemd architecture include:

• Fast boot times with parallel service activations
• On-demand activation of daemon processes
• Configuration via unit files instead of shell scripts
• Transactional dependency ordering of units
• Robust process tracking and integration
• Detailed logging for diagnostics

There are three core systemd component types:

systemd (PID 1): The first process to start at boot, acting as the parent for rest of boot.

Init Process ID: 1
Service: systemd

systemctl: Primary command line tool to manage systemd, services, view logs, etc.

systemd units: Configuration files that define what/how to start services.

Let‘s understand these components within the entire boot process flow.

Mapping the Debian Boot Procedure and Systemd Initialization

Booting a Debian system is an intricate sequence spanning firmware, bootloaders, kernel initialization and ultimately systemd orchestrating user space startups.

Here is an overview of the key phases:

1. BIOS and Bootloader Startup

The system firmware (BIOS/UEFI) runs mandatory Power On Self Tests (POST) to initialize central hardware like CPU, memory, storage, etc.

Once the tests pass, the first-stage bootloader stored in disk MBR (Master Boot Record) gets loaded into memory and executed. In Debian, this is usually GRUB.

GRUB then displays the OS selection menu and invokes the second-stage bootloader once Debian is chosen. This scans available disks to load the Linux kernel image and init RAM disk.

2. Kernel and Initramfs Initialization

The uncompressed Linux kernel initializes virtual memory, CPU exceptions, detects available hardware, mounts the initramfs transient root filesystem, and triggers udev device creation.

The temporary initramfs unpacked into memory has just enough drivers and tools to identify storage devices with the real root filesystem.

The initramfs passes control to the kernel init process once the disks are unlocked to mount the actual root filesystem path.

3. systemd Service Manager Initialization

One of the final initramfs steps is to prepare all information to execute the first user space process systemd (PID 1).

The systemd process starts running in the new root filesystem with all hardware information discovered so far. Its first responsibility is triggering udev to create device nodes under /dev to match detected system hardware.

systemd then activates various pre-defined target units based on configuration it finds under /etc/systemd/system/*.target. Starting the default.target subsequently launches all dependent services via their unit files.

This ultimately leads the boot process reaching the final graphical graphical.target from where the Linux login screen appears.

As we can see, systemd is pivotal for coordinating all the core user space activities after the Linux kernel phases. Now let‘s see how we can tap into systemd to alter service startups.

An Anatomy of Systemd Service Unit Files

systemd configuration relies heavily on unit files that dictate how a service process should be started, stopped or managed.

Unit files contain information spread across multiple sections like:

[Unit]
Description = Simple Demo Service
After = network.target

[Service]
ExecStart = /opt/scripts/start.sh
Restart = always

[Install] 
WantedBy = multi-user.target 

Let‘s dissect the purpose of each unit file section:

[Unit] Section

The [Unit] section defines metadata and ordering dependencies:

• Description: A short textual description about this unit
• After: Which target/service should this service start after
• Before: Which target/service should start before this unit
• Wants: A weaker dependency that won‘t fail if unavailable

[Service] Section

The [Service] section specifies the actual process execution:

• Type: Service process type – simple, forking, oneshot etc
• ExecStart: Commands to start this service process
• ExecStop: Commands to stop the service process
• Restart: Restart policy if the service exits or crashes

[Install] Section

The [Install] section rarely used, it defines aliases and multi-user dependencies:

• WantedBy: Targets like multi-user.target where this unit will be enabled

With this foundation on systemd, let‘s move onto creating our own units with boot scripts.

Creating Systemd Service Units for Scripts

Let‘s create a systemd unit file so we can control custom script execution during Debian startup.

We will running a shell script /opt/scripts/start.sh – this can initialize databases, launch memory caching or do any other task.

Follow these steps to define the new system service:

1. Create Unit File

Launch a text editor like vim/nano and create a new service file called myscript.service:

sudo nano /etc/systemd/system/myscript.service

2. Add Unit Metadata

Let‘s name our service and make it run after networking starts:

[Unit]
Description = My Startup Script
After = network.target

3. Specify Service Details

We will run our script as root user given the privileges needed:

[Service]
Type = simple
User = root
ExecStart = /opt/scripts/start.sh

4. Set Install Dependencies

Finally, enable the script for all multi-user/graphical boot targets:

[Install]
WantedBy = multi-user.target graphical.target  

This covers creating a basic systemd unit file to control any custom script or tool via predictable boot events.

Loading New Systemd Units

For systemd to detect our new myscript.service unit, we have to notify it explicitly:

sudo systemctl daemon-reload

This scans and loads all unit files under /etc/systemd/system. Now we can start/enable our script service.

Executing Scripts Under Different Boot Targets

One key benefit of systemd is intergrating scripts within modular boot targets and stages. This provides extensive control for the Linux experts over the exact sequence.

For example, instead of just starting with generic multi-user target, we can tap into much earlier boot events like:

Early Boot:

• cryptsetup.target – After unlocking LUKS encrypted disks
• remote-fs.target – Once remote network filesystems are mounted

Core Services

• network.target – After setting up all networking
• syslog.target – For log aggregation daemons
• rpcbind.target – After enabling RPC port mapping

User Session

• graphical.target – Right before launching the GUI login prompt
• suspend.target – On system wakeup after sleep/hibernate

This offers incredible flexibility over traditional SysV init approach that relied on a linear numbered script order.

Ordering Dependencies Between Targets

While integrating scripts spanning early device init all the way until the graphical session, we need to carefully specify ordering depedencies with After=, Before=, Wants=.

For example, here is a common requirement – we want myscript.service to initialize the application after networking starts AND also before graphical login.

This can be achieved by:

After = network.target nss-lookup.target  
Before = graphical.target
WantedBy = graphical.target

The nobootwait option is useful to avoid delaying subsequent targets when integrating into early boot sequence.

Best Practices for Robust Script Design

When controlling entire system behavior via scripts, adopting defensive design practices is vital given the risks like:

• Runtime failures, exits leading to failed boots
• Resource leaks causing performance drops
• Downtime if maintenance needs reboot
• Conflicting portfolio of tools over time

Key guidelines include:

• Modular scripts that do ONE thing with well defined inputs/outputs
• Extensive logging and runtime validation checks
• Graceful handling for crashes, signals, early exits
• Generating system notifications on status changes
• Following configuration management using Ansible/Puppet
• Runtime security with limited Linux capabilities/namespaces
• Using Docker containers for full isolation (if possible)
• Stateless logic without dependencies on external state

Adhering to these scripting best practices prevents messy and fragile boot orchestration logic over time.

Troubleshooting Tricky Systemd Services Issues

Despite best efforts, issues can still arise with system services – some common Linux boot problems include:

1. Service Deadlocks During Startup

This occurs when a script waits on another service or resource leading to circular blocking. Always design systemd units to start in parallel without strict ordering.

2. Hitting Systemd Timeout Limits
Scripts that take long to initialize (due to network calls etc.) may exceed the 90 second default startup timeout. Consider increasing TimeoutStartSec=300 in the unit file.

3. Resource Constraint Errors
Some tools may fail to start in low memory or disk situations. You can configure systemd to retry or even reboot systems automatically before retrying service activation.

4. Logging and Debugging Issues
Misconfigurations or runtime crashes within scripts can become hard to troubleshoot. Follow logging best practices and enable debug mode for verbose systemd logs.

Armed with the knowledge of targets analysis, ordering dependencies and process tracking – these Linux boot issues can be diagnosed reliably.

Conclusion: Architecting Linux Boot with Systemd

Through this detailed guide, we covered the integral role systemd plays within modern Debian and Linux systems for user space initialization.

The systemd service manager introduces an elegant set of abstractions like targets, units and ordering dependencies. This addresses complexities in orchestrating hundreds of processes starting in parallel during the boot procedure.

Constructing units for our custom scripts and integrating them alongside the multitude of services gives immense fine-grained control over boot flow. Doing this robustly does require an expert eye on interdependencies, resource usage and failure handling.

I hope you‘ve found this helpful in unlocking the power of architecting Linux startup sequences using systemd. Let me know if you have any other questions!