homelab (wip)

Hardware	OS	Tools	Secrets

• This repo contains the source of my personal homelab that runs a Raspberry Pi 4 Model B 8GB locally on my network.

• Containers run using podman and systemd as Quadlets and are deployed using ansible.

• Secrets are encrypted using sops and a age asymmetric key pair.

• TLS termination and reverse proxying is done using traefik.

1. Use the private key of the root.pub public ssh key to generate a installable fedora iot image.
2. Then use the ansible bootstrap playbook to setup some bare minimums, like users and permissions.
3. Use the configure playbook to setup system components like the firewall.
4. Configure your specific cert provider inside traefik.yml.
5. Finally use the containers playbook to deploy all containers.

├── 📁 .github                      # CI/CD workflows and actions
├── 📁 ansible                      # Application deployments
│   ├── 📁 inventory
│   ├── 📁 roles
│   └──    ansible.cfg
├── 📁 configs                      # Configuration files for containers
│   ├── 📁 adguard
│   ├── 📁 traefik
│   └── 📁 other apps
├── 📁 quadlets                     # Systemd unit file templates for pods and containers
├── 📁 scripts                      # Builds and helper scripts
├── 📁 secrets                      # SOPS-encrypted secrets
├── 📁 users                        # Public ssh keys for users
├── .sops.yaml                      # SOPS configuration
└── README.md

systemd unit files are used to define podman pods and containers. This allows them to really nicely integrate into the linux system (like starting on boot, restart on failure, dependency resolution, etc.). It is like a integrated kubernetes-lite.

Containers to not run rootless, but use userns=auto to run processes in a random user namespace. This prevents container processes to be able to attack each other, as they all run in different user namespaces. If a attacker manages to break out of one container, they still cannot access files or processes of other containers or the host, as they are now a random user without privileges. See also containers/podman#20845 and containers/podman#13728.

Additionally, newer linux kernels support idmapped mounts, which allow the system to remap the ownership of files on the fly for a specific mount point, without actually changing the file ownership on the physical disk. This is used to map config and data files into containers with the correct ownership. Special attention is given to containers that do not run the root user inside. Here the mapping has to be matched to the target uid:gid inside the container (see n8n as an example).

Ansible is used to deploy and manage the homelab.

The bootstrap playbook is used to setup bare minimums on a fresh fedora iot installation, like creating the admin user and setting up ssh access.

The configure playbook sets up basic port forwarding (mainly 443 and 53).

The containers playbook deploys all containers defined in the quadlets folder + their respective configuration files from the configs folder + any necessary secrets from the secrets folder. Secrets are first decrypted using the age private key before being copied to the server. This means that you have to enter your age private key each time when running the playbook. The playbooks is not (yet) idempotent, but already tries to only restart containers if something changed. If anything goes wrong you can always restart services manually using systemtl restart <service>.

Each service is exposed under its own subdomain, e.g. adguard.example.com.

Traefik is configured via traefik.yml. Basic rules for tls, entrypoints and the docker provider life there. Inside of the container quadlets you only have to define traefik.enable=true for traefik to pick up the service and route it correctly. Otherwise it will be skipped. Using the defaultRule option from traefik, the subdomain is either defined by the container name or an explicit custom subdomain=xyz label (see vaultwarden quadlet for an example).

Traefik automatically provisions and renews TLS certificates using Let's Encrypt. It uses a DNS challenge to issue a wildcard certificate (*.example.com) that all services then use.

This must be adapted to your specific environment. Define your DNS provider and environment args required inside traefik.yml.

• root.pub: Used when installing linux
• jh.pub: The jh user that has sudo permissions without password

Secrets are encrypted using sops and an age key that is stored securely, e.g. in bitwarden, as a secure note.

How to edit secrets (using bitwarden):

1. Unlock the vault by reading the master password from stdin or a file and then setting the BW_SESSION environment variable:

    export BW_SESSION=$(bw unlock --raw)

2. Instruct SOPS to retrieve the age private key by executing the following command and taking its output:

    export SOPS_AGE_KEY_CMD="zsh -c \"bw list items --search 'homelab secrets age de-/encryption key' | jq -r '.[0].fields[] | select(.name == \\\"private key\\\") | .value'\""

3. Edit the secrets file using sops:

    sops edit files/secrets/.env.adguard

TODO!

• /usr/lib/systemd/system-generators/podman-system-generator --dryrun
• systemd-analyze --generators=true verify X.service - Check for errors during systemd unit generation
• journalctl
  • -r - Show all messages in the journal sorted by recently.
  • -f - Follow logs as they are coming in.
  • -u - Select a service unit to filter for.
  • -n X - Show the last X lines of the journal.