The Aero has a design problem in the display clutch (see 2.1.5.7 Screen hinge problem (the darn “clutch”) in the FAQ) which makes very hard to find one without a broken display bezel.

Over the span of a few years I bought two units on eBay. The first one had the broken bezel. I bought a second unit, without the cracks but it had a display that turned on and off intermittently. The plan was to make one out of two or three units.

Luckily, in addition to the FAQ, there are different “cult” sites like https://remember.the-aero.org/ where one can find all kind of information.

Opening the device requires patience and being careful.

• My previous computer to this one, a rare taiwanese CAF 386SX which I was looking for years and finally found and acquired.
• An Atari 130XE, which I disassembled and applied the process to get rid of the yellow color, but never assembled back.
• My Timex Sinclair rescued from my parents (in perfect condition). Need to figure the TV/display part.

Description from its website:

Tree-sitter is a parser generator tool and an incremental parsing library. It can build a concrete syntax tree for a source file and efficiently update the syntax tree as the source file is edited.

Why is this important? This comment summarizes it well:

Incremental parsing of incorrect code is one of those things that is literally impossible in the general case, but tree-sitter has found a lot of good ways to do it that are not just possible for a large fraction of reality, but also performant. It’s hard to understate how impressive a piece of engineering this is.

I see this technology having an impact on IDEs, editors, linters and other tools similar to what the LLVM project did years ago for the compiler and interpreter ecosystem, and what Language Server Protocol did for IDEs during the last years.

For example, the Emacs editor adopted LSP by including eglot by default.

Originally, you could replace the limited regexp-based syntax highlighting in Emacs with the emacs-tree-sitter modes. This is no longer necessary, as from version 29+, Tree-sitter support is part of Emacs by default.

Description from its website:

WireGuard® is an extremely simple yet fast and modern VPN that utilizes state-of-the-art cryptography.

Wireguard’s simplicity:

• implemented in ~5000 lines of code, when most VPN solutions range from tens of thousands to hundreds of thousands
• works at the interface level, which means you can treat it like any other interface
• the state is hidden from the user, so things like roaming just work
• It is incorporated in most open-source operating systems. There is a Windows native version and a multi-platform userspace version written in Go.
  • NetworkManager, which most people use to manage networks in desktop Linux, has native support for it, and GNOME even displays the toggle for Wireguard connections.
• Android/iOS app
• Most commercial VPN providers support it, including the only one that is worth your time.

Some higher-level solutions have been built on top of WireGuard. The most impressive is Tailscale, which brings magical usability to access private networks spread across the world.

And last but not least, Fritzboxes, one of the most popular consumer routers in Germany, supports Wireguard natively, since December 2022, which means I now can access my home LAN very easily and from almost any device.

I believe many of the complicated app architectures today are either too early or just unnecessary.

SQLite is a local database engine that operates on a single file per database. It is the most deployed database in the world, and it is likely running in your pocket inside your phone on many different apps.

Many businesses could start in a single machine using a SQLite database.

Litestream is a project from Ben Johnson that replicates sqlite3 databases to make sqlite globally distributed. The replication part was extracted into LiteFS, while Litestream kept the disaster recovery replication.

With this model, LiteFS uses FUSE (Filesystem in userspace) as a pass-through filesystem to intercept writes to the database to detect transaction boundaries and replicate those in the replica nodes.

Litestream allows replicating databases by continuously copying write-ahead log pages to cloud storage.

An alternative implementation of transaction replication using SQLite built-in VFS is also planned.

The project was since then acquired by Fly.io, which specializes in deploying apps close to the users.

Both projects give SQLite superpowers and allow for resilient and performant applications while keeping the setup and architecture lean and simple.

During the Twitter exodus to Mastodon, I saw people dealing with the complexity and resource requirements of operating Mastodon for a single user. My Fediverse instance is not Mastodon, but gotosocial. Uses 128M ram, a 140M SQLite database, and runs on a 5€ micro VM. The database is replicated to an sftp share with Litestream.

Nix is a tool for producing reproducible builds and deployments. It takes a different approach to package management using a declarative and functional build description.

When you build something with Nix, it ends in its own directory in the Nix store e.g. /nix/store/hxxrbmr2zh6ph90qi8b4n2m53yvan3fr-curl-7.85.0/ and as long as the inputs do not change, the location, which is content-addressed, will not change either. They will also depend on the exact versions they were built against.

This allows you the installation of multiple versions in parallel, and the current system profile itself is a collection of symbolic links to the right binaries, which means you can roll back very easily.

While Nix can be used on Linux and macOS, there is a full Linux distribution built on this model.

While it can also be used for CI, building container images, etc., I use Nix in two ways:

• Declare project dependencies

  If I have e.g. a folder with some Ansible roles I use to configure my home gadgets, I can make that project independent from where I am running it by just having a top shell.nix declaring dependencies. Then a simple .envrc file with the line use_nix and direnv setup in my shell.

  As soon as I cd into the directory, Ansible is installed and appears in the path. I cd out and it disappears. The nix store is cached, so the second time is very fast (until you nix store gc).

  You can use this to have reproducible developer environments.

  Nix Flakes is a new format to package Nix-based projects in a more discoverable, composable, consistent and reproducible way.

  With Flakes, you could even pin your environment to a specific revision of the package descriptions.

• Manage packages, including my own

  Some packages I need all the time: Emacs, Chromium, tarsnap, etc. I use Nix for that, and keep my distribution just for the base system.

  nix profile install nixpkgs#tarsnap and the package is now always available. I also have packages that are not free to distribute, so I can keep the recipe to build it in git, or just override a few compile options from another package. It is just flexible.

The language is a functional DSL that takes some curve to learn, just like the built-in functions. I am not sure if this will be someday the future of deployments, but for me as been agreat addition to those two use cases..

StableDiffusion is an AI model which allows to:

• transform text prompt into images
• transform images plus a text prompt into new images
• edit images by selecting an area and a prompt

Also impressive are the creations where StableDiffusion is used to change a single video frame, and another model is used to extrapolate the change to the rest of the frames, resulting in full video editing.

The Dreambooth model allows to finetune StableDiffusion for specific subjects. This is what the Lensa app does when generating many avatars from your selfies.

I believe this will have a huge impact on creative industries (design, gaming), and will make their software understand the semantics of the image, just like IDEs have been doing for years offering syntax-aware refactorings.

I’d like to mention ChatGPT together with Copilot, but I haven’t tried Copilot yet.

These technologies are already proving to be very useful in the context of programming.

Leaving out the controversial topic of training proprietary models on GPL code for another occasion, I am impressed how good ChatGPT is to port code from one dimension to another, eg. rewriting using a different language, library, etc. I think it will become very useful for porting, refactoring and updating software.

For example, I was very pleased with ChatGPT being able to take some Linux commands, and generating me a set of Ansible tasks to replicate the configuration

Single-page applications (SPA) are with us for longer than I can remember, but the feeling something is not right in that model continues to live with me.

The architecture duplication on the server and client-side (controllers, views, stores), dividing teams through json messages in two worlds speaking different languages seems broken. The instability of the Javascript eco-system just makes things worse.

I can’t however, picture how to solve the challenges SPAs aim to solve when it comes to highlyy interactive applications.

Phoenix is a web framework for Elixir, a language running on the Erlang VM. His creator has a Rails background, so he took off from where Rails left and brought innovation to the space in the form of Phoenix LiveView, a technique that allows for highly interactive applications without abandoning the server side paradigm.

Other toolkits have appeared which allow to start server side and add interactivity in a structured way without abandoning the server side paradigm. One is HotWire from Basecamp, which includes Turbo and other libraries, and htmx, which works by just annotating HTML.

Something I always disliked about virtualization was the use of images. It added a whole layer of complexity.

virtio-fs is a filesystem that allows sharing the host filesystem with the guest. Unlike virtio-9p (the one used by Windows Subsystem for Linux), it has local semantics.

qemu has support for it, so you can boot a root filesystem.

One tool that takes advantage of virtio-fs is krunvm. It allows to run container images as micro virtual machines. The machines implement a few simple virtio devices enough to run an embedded kernel in libkrun.

krunvm takes virtio-fs to the next level, basically making it invisible, allowing you to mount any host folder into the virtual machine the same way that you do it with container images.

Follow the work Sergio Lopez is doing in this space.

These are my picks. What are yours?

After evaluating several mail providers including Tutanota, ProtonMail, Mailfence, Soverin, Runbox, I decided for https://mailbox.org:

• Provides IMAP and can be used with generic existing open-source clients
• Company has a focus on privacy
• Attractive price
• Located in Germany
• Based on Open-Xchange, which is Open-Source
• Provides a Calendar feature based on open standards (CalDAV)
• Provides encryption in several forms
• Company values: privacy, eco-friendly, and work-life balance for their employees align with my own

Mailbox.org is not perfect. 2FA is bolted-on like many other parts of the application. I don’t think you can beat Google when it comes to security, but the risk of getting your account locked at Google and no escalation path or human to talk to makes all the technicallities irrelevant.

To migrate, I planned to use mbsync, which I already use to download my work email in my mu4e/Emacs setup. The idea is to create two channels, one for GMail, one for the new provider, download the whole GMail archive (forcing pull in a sync), and then force a push on the new provider.

Downloading all my mail with mbsync did not work. GMail has download limits for IMAP. The next thing to try was Google Takeout, a service that allows you to dowload your Google data. This gave me an mbox with all my GMail messages. mbsync only works with Maildir, so I tried to upload the mbox messages with Thunderbird, but did not get far. At the end, I used mb2md to convert the mbox to Maildir format, and then used mbsync to upload the messages to the new provider. This worked.

In order to prevent lock-in in the future, I used a custom domain. My go-to registrar is Namecheap and I have no complaints. I went with Gandi, as they are based in France and I read good things about them.

To be able to migrate at my own pace, I setup a forward and delete filter rule in Gmail. I had hundred of accounts using my email address as username. Thankfully, my password manager knows about those and I changed the ones I use more often. Every time I get a newsletter or notification, I take the chance to unsubscribe, and check the To: field and update my profile, or delete that account.

I replaced the GMail mobile application with FairMail. The build is not free as in beer ($), but it is Open-Source (GPL). Paying to get a working binary and some support is worth it.

I switched to DuckDuckGo a long time ago already. I tried Quant (based in France) but for some reason it takes seconds to connect. I can’t believe they fail in this obvious detail.

My usual workflow has been to download photos locally and upload to Google Photos. The lack of a good sync mechanism resulted in glitches over time. Some albums were present locally and some existed only in Google Photos.

I used both gohotos-sync and Google Takeout to get two full copies of albums and the photo stream. gphotos-sync has a useful flag --compare-folder which hels comparing albums in Google Photos with the local version of those, creating symlinks for local and remote missing files.

I then used exiftool to sort pictures further. If you don’t know this tool, I highly recommned you learn it.

I selected Hetzner Storage Share, an affordable Nextcloud based service hosted in Germany. Nextcloud is intended to replace Google Drive, which means it allows to share files via eg. public links. You can, however, install many applications in it, including a very simple photo gallery.

The feature I will miss the most is to be able to do AI based search on my photos. I search often by keywords and concepts.

I setup the Linux and mobile clients. The Linux client syncs a part of my Pictures folder that is ready and organized. I configured Instant-Upload on my phone which auto-uploads photos I take with the camera. The upload is unidirectional, but as they land on a folder I have configured to be synced with my computer, they reach my laptop to be further organized. I can delete the camera files without risk of losing what has been uploaded.

I still depend on Drive for sharing files with my band. I relegated Drive to its own Firefox Container, this way I am not permanently logged into the Google Account as I browse the Web, but do not need to log-in again to use Drive.

For personal notes, I use org-mode on a synced folder. I sync the folder to my Nextcloud instance. Orgzly provides a TODO widget and access the files via WebDAV. Orgro gives you a more sophisticated viewer.

I do share a shopping list with my family in Keep and I haven’t yet solved that problem. I have thought about a Keep-like view for Orgzly -it is open-source-, by transforming each headline into a card.

I track my runs in Fit. My ideal solution would be to store tracks directly as a file in a NextCloud folder. An alternative is to store them in a internal database and do an Export from time to time.

Google Takeout allows you to export tracks in TCX format, with summaries as CSV files. I ended with 300+ TCX files.

I evaluated many apps that required no Cloud service. RunnerUp, ForRunners and FitoTrack are also Open-Source, where Sportractive is not.

FitoTrack and Sportractive where the most promissing ones. In both apps I could not import more than one file at a time so I contacted the authors asking for tips how to import my data. Sportractive author mentioned this was not possible. FitoTrack author found this a simple addition, implemented it and pointed me to the next release. Due to a glitch, took longer to show up in the Play Store, but I built the app from source and started experimenting with this feature.

To convert the TCX files to GPX I used gpsbabel. FitoTrack has trouble with Fit multiple laps/tracks. The pack option in gpsbabel merges them.

for fn in ../*.tcx; do gpsbabel -i gtrnctr -f "$fn" -x track,pack -o gpx -F $(basename $fn .tcx).gpx; done

I had now 300+ files with names like 2018-04-15T00_44_22+02_00_PT38M17.962S_Running.gpx, no description and no metadata specifying it was “Running”.

I hacked this script which finds the starting point, does reverse geolocation to find the place name, cleans it up and then renames the file. It also sets the description to something like “Run in Madrid, Spain”.

import os
import time
import unidecode

import gpxpy
import gpxpy.gpx
from geopy.geocoders import Nominatim

geolocator = Nominatim(user_agent="JustATestScript")

for filename in os.listdir("."):
    if not filename.endswith(".gpx"):
        continue

    print("Current: {}".format(filename))
    gpx_file = open(filename, "r")
    gpx = gpxpy.parse(gpx_file)

    # get first point
    point = None
    try:
        point = gpx.tracks[0].segments[0].points[0]
    except Exception:
        print(" `-> No point 0")
        continue

    location = geolocator.reverse(
        (point.latitude, point.longitude),
        language="en",
        addressdetails=True,
    )
    country = unidecode.unidecode(location.raw["address"]["country"])
    # City is not so easy. Fallback until we get something
    city = None
    for place in ["city", "village", "suburb", "town"]:
        if place not in location.raw["address"]:
            continue
        import re
        city = re.sub(r".+/\s+", "", location.raw["address"][place])
        city = unidecode.unidecode(city)
        break
    if not city:
        raise Exception("No place in address: {}".format(location.raw))
    newname = "{}-Running-{}_{}.gpx".format(
        point.time.strftime("%Y-%m-%d_T%H_%m"),
        city.replace(" ", "_"),
        country.replace(" ", "_"),
    )
    print(" `-> new name: {}".format(newname))
    gpx.tracks[0].name = "Run in {}, {}".format(city, country)
    gpx.tracks[0].description = None
    # FIXME: does not serialize. Fix with xmlstarlet
    gpx.tracks[0].type = "running"
    with open(filename, "w") as out:
        out.write(gpx.to_xml())
    try:
        os.rename(filename, newname)
    except Exception as e:
        print(location.raw)
        raise e
    # do not call the API too fast
    time.sleep(1)

The result was:

2018-07-29_T08_07-Munich_Germany.gpx
2019-08-10_T14_08-Barcelone_Spain.gpx
2020-08-22_T07_08-Warsaw_Poland.gpx
2020-07-11_T15_07-Nuremberg_Germany.gpx
2020-08-20_T06_08-Valencia_Spain.gpx
2018-06-03_T10_06-Stuttgart_Germany.gpx
...

(city names and dates are not the real ones)

Setting the sport type in the metadata did not get serialized back, so I fix it with xmlstarlet:

xmlstarlet ed --inplace -N x="http://www.topografix.com/GPX/1/0" -s /x:gpx/x:trk -t elem -n type -v "running" *.gpx

Then, mass import into FitoTrack and I got all my activities with nice descriptions and the right “Running” icon.

My new mail setup is working for some weeks already without problems. I miss some Photos features, but that’s it. I was not expecting Fit to take that much effort.

In general, I am happy with the results. I regained control of my data and I got to use more open-source.

We start by creating k3s/init.sls. For the k3s state I defined a minimal pillar defining the server and the shared token:

k3s:
  token: xxxxxxxxx
  server: rpi03

The first part of the k3s state ensures cgroups are configured correctly and disables swap:

k3s.boot.cmdline:
  file.managed:
    - name: /boot/cmdline.txt
    - contents: |
	cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory

k3s.disable.swap:
  cmd.run:
    - name: swapoff -a
    - onlyif:  swapon --noheadings --show=name,type | grep .

As the goal was to avoid using the SD card, the next state makes sure /var/lib/rancher/k3s is a mount. I have to admit I wasted quite some time getting right the state for the storage mount. Using mount.mounted did not work because it is buggy and took different btrfs subvolume mounts from the same device as the same mount.

k3s.volume.mount:
  mount.mounted:
    - name: /var/lib/rancher/k3s
    - device: /dev/sda1
    - mkmnt: True
    - fstype: btrfs
    - persist: False
    - opts: "subvol=/@k3s"

I resorted then to write my own state. I discovered the awesome findmnt command, and my workaround looked like:

k3s.volume.mount:
  cmd.run:
    - name: mount -t btrfs -o subvol=/@{{ grains['id'] }}-data /dev/sda1 /data
    - unless: findmnt --mountpoint /data --noheadings | grep '/dev/sda1[/@k3s]'
    - require:
	- file: k3s.volume.mntpoint

This turned later to be a pain, as the k3s installer started k3s without caring much if this volume was mounted or not. Then I remembered: systemd does exactly that. It manages mount and dependencies. This simplified the mount state to:

k3s.volume.mount:
  file.managed:
    - name: /etc/systemd/system/var-lib-rancher-k3s.mount
    - contents : |
	[Unit]

	[Install]
	RequiredBy=k3s
	RequiredBy=k3s-agent

	[Mount]
	What=/dev/sda1
	Where=/var/lib/rancher/k3s
	Options=subvol=/@k3s
	Type=btrfs
  cmd.run:
    - name: systemctl daemon-reload
    - onchanges:
	- file: k3s.volume.mount
  service.running:
    - name: var-lib-rancher-k3s.mount

The k3s state works as follows: it runs the installation script in server or agent mode depending if the pillar k3s:server entry matches with the node where the state is applied.

{%- set k3s_server = salt['pillar.get']('k3s:server') -%}
{%- if grains['id'] == k3s_server %}
{%- set k3s_role = 'server' -%}
{%- set k3s_suffix = "" -%}
{%- else %}
{%- set k3s_role = 'agent' -%}
{%- set k3s_suffix = '-agent' -%}
{%- endif %}

k3s.{{ k3s_role }}.install:
  cmd.run:
    - name: curl -sfL https://get.k3s.io | sh -s -
    - env:
	- INSTALL_K3S_TYPE: {{ k3s_role }}
{%- if k3s_role == 'agent' %}
	- K3S_URL: "https://{{ k3s_server }}:6443"
{%- endif %}
	- INSTALL_K3S_SKIP_ENABLE: "true"
	- INSTALL_K3S_SKIP_START: "true"
	- K3S_TOKEN: {{ salt['pillar.get']('k3s:token', {}) }}
    - unless:
	# Run install on these failed conditions
	# No binary
	- ls /usr/local/bin/k3s
	# Token changed/missing
	- grep '{{ salt['pillar.get']('k3s:token', {}) }}' /etc/systemd/system/k3s{{ k3s_suffix }}.service.env
	# Changed/missing server
{%- if k3s_role == 'agent' %}
	- grep 'K3S_URL=https://{{ k3s_server }}:6443' /etc/systemd/system/k3s{{ k3s_suffix }}.service.env
{%- endif %}
    - require:
	- service: k3s.volume.mount
	- service: k3s.kubelet.volume.mount

k3s.{{ k3s_role }}.running:
  service.running:
    - name: k3s{{ k3s_suffix }}
    - enable: True
    - require:
      - cmd: k3s.{{ k3s_role }}.install

The next step is to move workloads like homeassistant into this setup.

k3s allows to automatically deploy manifests located in /var/lib/rancher/server/manifests. We can deploy eg. mosquitto like the following:

homeassistant.mosquitto:
  file.managed:
    - name: /var/lib/rancher/k3s/server/manifests/mosquitto.yml
    - source: salt://homeassistant/files/mosquitto.yml
    - require:
      - k3s.volume.mount

With mosquito.yml being:

---
apiVersion: v1
kind: Namespace
metadata:
  name: homeassistant
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mosquitto
  namespace: homeassistant
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mosquitto
  template:
    metadata:
      labels:
	app: mosquitto
    spec:
      containers:
	- name: mosquitto
	  image: docker.io/library/eclipse-mosquitto
	  resources:
	    requests:
	      memory: "64Mi"
	      cpu: "100m"
	    limits:
	      memory: "128Mi"
	      cpu: "500m"
	  ports:
	  - containerPort: 1883
	  imagePullPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
  name: mosquitto
  namespace: homeassistant
spec:
  ports:
  - name: mqtt
    port: 1883
    targetPort: 1883
    protocol: TCP
  selector:
    app: mosquitto

Homeassistant is no different, except that we use a ConfigMap resource to store the configuration and define an Ingress resource to access it from the LAN:

---
apiVersion: v1
kind: Namespace
metadata:
  name: homeassistant
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: homeassistant-config
  namespace: homeassistant
data:
  configuration.yaml: |
    homeassistant:
      auth_providers:
	- type: homeassistant
	- type: trusted_networks
	  trusted_networks:
	    - 192.168.178.0/24
	    - 10.0.0.0/8
	    - fd00::/8
	  allow_bypass_login: true
      name: Home
      latitude: xx.xxxx
      longitude: xx.xxxx
      elevation: xxx
      unit_system: metric
      time_zone: Europe/Berlin
    frontend:
    config:
    http:
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: homeassistant
  namespace: homeassistant
spec:
  replicas: 1
  selector:
    matchLabels:
      app: homeassistant
  template:
    metadata:
      labels:
	app: homeassistant
    spec:
      containers:
	- name: homeassistant
	  image: homeassistant/raspberrypi3-64-homeassistant:stable
	  volumeMounts:
	    - name: config-volume-configuration
	      mountPath: /config/configuration.yaml
	      subPath: configuration.yaml
	  livenessProbe:
	    httpGet:
	      scheme: HTTP
	      path: /
	      port: 8123
	    initialDelaySeconds: 30
	    timeoutSeconds: 30
	  resources:
	    requests:
	      memory: "512Mi"
	      cpu: "100m"
	    limits:
	      memory: "1024Mi"
	      cpu: "500m"
	  ports:
	    - containerPort: 8123
	      protocol: TCP
	  imagePullPolicy: Always
      volumes:
	- name: config-volume-configuration
	  configMap:
	    name: homeassistant-config
	    items:
	    - key: configuration.yaml
	      path: configuration.yaml
---
apiVersion: v1
kind: Service
metadata:
  name: homeassistant
  namespace: homeassistant
spec:
  selector:
    app: homeassistant
  ports:
    - port: 8123
      targetPort: 8123
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: homeassistant
  namespace: homeassistant
  annotations:
    kubernetes.io/ingress.class: traefik
    traefik.frontend.rule.type: PathPrefixStrip
spec:
  rules:
  - host: homeassistant.int.mydomain.com
    http:
      paths:
      - path: /
	backend:
	  serviceName: homeassistant
	  servicePort: 8123

Setting up Ingress was the most time consuming part. It took me a while to figure out how it was supposed to work, and customizing the Treafik Helm chart is not intuitive to me. While homeassistant was more straightforward as it is a simple HTTP behind SSL proxy service, the Kubernetes dashboard is already deployed with SSL inside the cluster. I am still figuring out how ingress.kubernetes.io/protocol: https, traefik.ingress.kubernetes.io/pass-tls-cert: "true" (oh, don’t forget the quotes!) or insecureSkipVerify work toghether and what is the best way to expose it to the LAN.

In a future post, I will describe the dashboards setup, and other improvements.

Jekyll just worked. I can’t complain about it. However, I feel too tied to the Github Pages environment. You had to use a Jekyll version that was close to a year behind and live with the plugins that the environment supported, and nothing more.

If I was to setup my own CI workflow to overcome this limitation, why keep using Jekyll?. Hugo started to feel faster, easier to deploy locally and mostly compatible.

I have been using Emacs for more than 10 years. 3 years ago, I switched mail clients from Thunderbird to mu4e on top of Emacs. Then I discovered org-mode as a plain-text personal organization system and gradually started to live more time inside emacs. Microsoft did a so good job with Visual Studio Code that for a moment I thought I would not resist. However, Microsoft created an ecosystem by making the interaction with programming languages a standard, via the Language Server Protocol, and emacs-lsp made my programming experience with emacs just better.

I knew that org-mode was quite good at exporting. After I saw a couple of websites generated from org, I started to toy with the idea of using org too. It could also be a good chance to learn Emacs Lisp for real. So I started learning about org and websites.

As I did not know where to start, I started by reading a lot of solutions by other people, documentation, posts, etc.

Most of the structure of the final solution, its ideas, conventions, configuration and some snippets were taken from the following projects:

• Toon Claes’s blog
• Example org-mode website using GitLab Pages by Rasmus
• https://github.com/bastibe/org-static-blog
• The theme is a port of the original Lagom theme I was using with Jekyll

While locally you can test by running emacs via the Makefile, I wanted a new way to run the generation on every git push:

• I started with Travis, but I could not find container jobs anymore. When I relalized that Ubuntu had an older Emacs version I just lost interest.
• Gitlab was easy to get working. Not only because is very simple and elegant, but some of the examples I took inspiration from where already using it, along the Emacs Alpine container image. However, I did not want to have everything in Github, except my site
• Then I realized Github Actions was in beta, but I was not yet in. Until:

You’re in 🥰

— Nat Friedman (@natfriedman) August 26, 2019

Not only I have a website powered by the tool I use daily, but it is also packed with awesome features. For example, the diagrams in this post are inlined as PlantUML code in the org file and exported via Org Babel.

It also gave me project to learn Emacs Lisp. I do plan to add some minor features, like blog tags or categories and perhaps commenting. The learning will also benefit personalizing my editor and mail client.

The source of this site is available on Github.

So with this you can use your configuration management to manage your state against a defined state and on top of that we give you the tooling to inspect and rollback configuration changes.

We will continue adding the missing pieces to give the administrators full overview and control over their running systems.

You can find our current work in this github repository. We plan of course to send it upstream once the design and implementation settles down.

This means there’s usually a “central” server running Ansible commands, although there’s nothing particularly special about what server Ansible is installed on. Ansible is “agentless” - there’s no central agent(s) running. We can even run Ansible from any server; I often run Tasks from my laptop.

The salt package is made of various components, among others:

• salt: the framework, libraries, modules, etc.
• salt-minion: the minion daemon, runs on the managed hosts.
• salt-master: the master daemon, runs on the management server.
• salt-ssh: a tool to manage servers over ssh.

If you want to run Salt like Ansible, you only need to install salt-ssh in your machine (the machine where you want to run tasks from).

You don’t need anything else than Python on the hosts you will manage.

Well, there are a couple of other packages required

ssh $HOST zypper -n install python-pyOpenSSL python-xml

Salt is available out of the box on openSUSE Leap and Tumbleweed so if you are using them just type:

zypper in salt-ssh

For other platforms, please refer to the install section of the Salt documentation.

Ansible has a default inventory file used to define which servers it will be managing. After installation, there’s an example one you can reference at /etc/ansible/hosts.

The equivalent file in salt-ssh is /etc/salt/roster.

That’s good enough for now. If needed, we can define ranges of hosts, multiple groups, reusable variables, and use other fancy setups, including creating a dynamic inventory.

Salt can also provide the roster with custom modules. Funnily enough, ansible is one of them.

As I am using a self-contained setup, I create ~/Project/etc/salt/roster:

node1:
  host: node1.example.com
node2:
  host: node2.example.com

Ansible will assume you have SSH access available to your servers, usually based on SSH-Key. Because Ansible uses SSH, the server it’s on needs to be able to SSH into the inventory servers. It will attempt to connect as the current user it is being run as. If I’m running Ansible as user vagrant, it will attempt to connect as user vagrant on the other servers.

salt-ssh is not very different here. Either you already have access to the server, otherwise it will optionally ask you for the password and deploy the generated key-pair etc/salt/pki/master/ssh/salt-ssh.rsa.pub to the host so that you have access to it in the future.

So, in the Ansible tutorial, you did:

$ ansible all -m ping
127.0.0.1 | success >> {
    "changed": false,
    "ping": "pong"
}

The equivalent in salt-ssh would be:

salt-ssh '*' test.ping
node1:
    True
node2:
    True

Just like the Ansible tutorial covers, salt-ssh also has options to change the user, output, roster, etc. Refer to man salt-ssh for details.

Ansible uses “modules” to accomplish most of its Tasks. Modules can do things like install software, copy files, use templates and much more.

If we didn’t have modules, we’d be left running arbitrary shell commands like this:

ansible all -s -m shell -a 'apt-get install nginx'

However this isn’t particularly powerful. While it’s handy to be able to run these commands on all of our servers at once, we still only accomplish what any bash script might do.

If we used a more appropriate module instead, we can run commands with an assurance of the result. Ansible modules ensure indempotence - we can run the same Tasks over and over without affecting the final result.

For installing software on Debian/Ubuntu servers, the “apt” module will run the same command, but ensure idempotence.

ansible all -s -m apt -a 'pkg=nginx state=installed update_cache=true'
127.0.0.1 | success >> {
    "changed": false
}

The equivalent in Salt is also called “modules”. There are two types of modules: Execution modules and State modules. Execution modules are imperative actions (think of install!). State modules are used to build idempotent declarative state (think of installed).

There are two execution modules worth to mention:

• The cmd module, which you can use to run shell commands when you want to accomplish something that is not provided by a built-in execution module. Taking the example above:

salt-ssh '*' cmd.run 'apt-get install nginx'

• The state module, which is the execution module that allows to apply state modules and more complex composition of states, known as sls files.

salt-ssh '*' pkg.install nginx

You don’t need to use the apt module, as it implements the virtual pkg module. So you can use the same module on every platform.

On Salt you would normally use the non-idempotent execution modules from the command line and use the idempotent state module in sls files (equivalent to Ansible’s playbooks).

If you still want to apply state data like ansible does it:

salt-ssh '*' state.high '{"nginx": {"pkg": ["installed"]}}'

Playbooks can run multiple Tasks and provide some more advanced functionality that we would miss out on using ad-hoc commands. Let’s move the above Task into a playbook.

The equivalent in Salt is found in states.

Create srv/salt/nginx/init.sls:

nginx:
  pkg.installed

To apply this state, you can create a top.sls and place it in srv/salt:

base:
  `*`:
    - nginx

This means, all hosts should get that state. You can do very advanced targetting of minions. When you write a top, you are defining what it will be the highstate of a host.

So when you run:

salt-ssh '*' state.apply

You are applying the highstate on all hosts, but the highstate of each host is different for each one of them. With the salt-ssh command you are defining which hosts are getting their configuration applied. Which configuration is applied is defined by the top.sls file.

You can as well apply a specific state, even if that state does not form part of the host highstate:

salt-ssh '*' state.apply nginx

Or as we showed above, you can use state.high to apply arbitrary state data.

Salt has a similar concept called events and reactors which allow you to define a fully reactive infrastructure.

For the example given here, a simple state watch argument will suffice:

nginx:
  pkg.installed: []
  service.running:
    - watch: pkg: nginx

Note:

The full syntax is:

someid:
  pkg.installed:
    name: foo

But if name is missing, someid is used, so you can write:

#+BEGIN_SRC yaml foo: pkg.installed #+END_END

Looking at the given Ansible example:

{% raw %}
---
- hosts: local
  vars:
   - docroot: /var/www/serversforhackers.com/public
  tasks:
   - name: Add Nginx Repository
     apt_repository: repo='ppa:nginx/stable' state=present
     register: ppastable

   - name: Install Nginx
     apt: pkg=nginx state=installed update_cache=true
     when: ppastable|success
     register: nginxinstalled
     notify:
      - Start Nginx

   - name: Create Web Root
     when: nginxinstalled|success
     file: dest={{ docroot }} mode=775 state=directory owner=www-data group=www-data
     notify:
      - Reload Nginx

  handlers:
   - name: Start Nginx
     service: name=nginx state=started

    - name: Reload Nginx
      service: name=nginx state=reloaded
{% endraw %}

You can see that Ansible has a way to specify variables. Salt has the concept of pillar which allows you to define data and then make that data visible to hosts using a top.sls matching just like with the states. Pillar data is data defined on the “server” (there is a equivalent grains for data defined in the client).

Edit srv/pillar/paths.sls:

{% raw %}
docroot: /var/www/serversforhackers.com/public
{% endraw %}

Edit srv/pillar/top.sls and define who will see this pillar (in this case, all hosts):

base:
  '*':
    - paths

Then you can see which data every host sees:

salt-ssh '*' pillar.items
node1:
    ----------
    docroot:
        /var/www/serversforhackers.com/public
node2:
    ----------
    docroot:
        /var/www/serversforhackers.com/public

With this you can make sensitive information visible on the hosts that need it. Now that the data is available, you can use it in your sls files, you can add to

{% raw %}
nginx package:
  pkg.installed

nginx service:
  service.running:
    - watch: pkg: 'nginx package'

nginx directory:
  file.directory:
    - name: {{ pillar['docroot'] }}

{% endraw %}

Which can be abbreviated as:

{% raw %}
nginx:
  pkg.installed: []
  service.running:
    - watch: pkg: nginx

{{ pillar['docroot'] }}:
  file.directory
{% endraw %}

Roles are good for organizing multiple, related Tasks and encapsulating data needed to accomplish those Tasks. For example, installing Nginx may involve adding a package repository, installing the package and setting up configuration. We’ve seen installation in action in a Playbook, but once we start configuring our installations, the Playbooks tend to get a little more busy.

There is no 1:1 concept in Salt as it already organizes the data around a different set of ideas (eg: gains, pillars), but for the utility of the specific Ansible tutorial, lets look at a few examples.

These are equivalent to grains, and you can see what grains you have available by calling:

salt-ssh '*' grains.items

You can use them from Jinja2 as grains:

{% raw %}
{% if grains['os_family'] == 'RedHat' %}
...
{% endif %}
{% endraw %}

If you need a custom grain definition, you can write your own and distribute them from the server.

The equivalent in Salt would be to use the Pillar. If you need encryption support you have various options:

• Use a external pillar which fetches the data from a vault service
• Use the renderer system and add the gpg renderer to the chain. (Disclaimer: I haven’t tried this myself).

You will need a pillar:

admin_password: $6$lpQ1DqjZQ25gq9YW$mHZAmGhFpPVVv0JCYUFaDovu8u5EqvQi.Ih
deploy_password: $6$edOqVumZrYW9$d5zj1Ok/G80DrnckixhkQDpXl0fACDfNx2EHnC
common_public_key: ssh-rsa ALongSSHPublicKeyHere

And then refer to it from the user state:

{% raw %}
admin:
  user.present:
    - password: {{ pillar['admin_password'] }}
    - shell: /bin/bash

sshkeys:
  ssh_auth.present:
    - user: admin
    - name: {{ pillar['common_public_key'] }}
{% endraw %}

In order to refresh the pillar data, you can use:

salt-ssh '*' saltutil.refresh_pillar

So, this is how you use Salt in a way similar to Ansible. The best part of this is that you can start learning about Salt without having to deploy a Salt master/minion infrastructure.

The master/minion infrastructure brings a whole new set of possibilities. The reason we chose Salt is because here is where it starts, and not where it ends.

• Chris Fidao for the original Ansible tutorial.
• Konstantin Baikov for corrections and suggestions.

In mid-2014 I wrote about SUSE Manager 2.1. It was an important release for us because at that point we became very active upstream, to the point that on a typical day, a considerable chunk of the open pull requests came from SUSE, including a refreshed mobile-enabled user interface.

But the world is changing, and management offerings are taking new innovative directions, especially around Configuration Management and Compliance. This trend was in several dimensions quite radical for our product, based on the Spacewalk project, which has been managing Linux systems with the same paradigm since 2008.

Deploying a management solution is a considerable investment for the customer: operation, training, processes, etc. Telling a customer you will be completely replacing their deployed solution because new trends have appeared means you are throwing all their efforts in the trash.

On the other hand, it is very easy to find excuses to rewrite software from scratch and there are well-known industry voices giving advice on the topic.

So are we up to the challenge?. Can we evolve SUSE Manager iteratively within this new world without endangering our customer investments?

After we decided that we would bring this new world into the existing SUSE Manager, we clearly didn’t want to write a configuration management engine ourselves: there were multiple opensource projects that were up to the task. We also understood that customers could be already invested into one of them, and that is fine: we were looking for the one that could be tightly integrated into SUSE Manager and had a design that matched our vision.

And here is where things got even less sweet: they got salty!

We ended up choosing Salt. The reasons are endless: From its vibrant community to the “aha!” moment you get when understanding its internals and seeing the vision behind the architecture. It is a complete toolkit to build datacenter automation, wrapped in usable tools with sane defaults that work out-of-the-box.

SaltStack platform or Salt is a Python-based open source configuration management software and remote execution engine. Supporting the “Infrastructure as Code” approach to deployment and cloud management, it competes primarily with Puppet, Chef, and Ansible. (Source: Wikipedia]

SaltStack software orchestrates the build and ongoing management of any modern infrastructure. SaltStack is also the most scalable and flexible configuration management software for event-driven automation of CloudOps, ITOps and DevOps. (Source: SaltStack)

We did not get the same impression from the other tools we evaluated. It was clear that Salt fit into the SUSE Manager present and future.

The fact that Salt implements configuration management (states) on top of the execution engine (state.apply), and that the executions modules are first class, we could offer almost all features we already had in SUSE Manager and naturally extend the Configuration Management part. On top of that, Salt is written in Python, and the current Spacewalk client stack and partly the server stack are also Python components.

Things only got better from there. During all this time we got really crazy about Salt. Our sysadmins are using it as well, and a community was formed internally that went beyond our product. Not only with sysadmins and engineers, but even our Product Manager writes beacons.

Working on a new release means the opportunity to refresh the platforms and technologies you use, and to look for better alternatives for some of them.

• We keep rebasing and picking up enhancements from Spacewalk upstream.
• A mature codebase does not mean you should not get rid of code. E.g,: here is a pull request from the team to remove 30k lines of code that did not make much sense nowadays.

With the Salt and Compliance work there was going to be new code written, and that is an opportunity for choosing the right platforms and frameworks.

• From SLES-11-SP3 to SLES-12-SP1
• From Tomcat 6.x to Tomcat 8.x
• From Java 7 to Java 8
• We started to use Spark for server-side Java code.
• We started to use React on the client side.

The first attempt was done as part of Hackweek 11. A protoype known as Saltwalk was born.

This protoype (a simple python reactor) helped figuring out what the bulk of the work would be, the non-trivial parts and what decisions we needed to take to move forward.

The basic architecture of a reactor that handles Salt events and interacts with Spacewalk was in place. What we needed now was a way for Spacewalk to interact with Salt.

The Remote Commands page in the Salt section gives you a web based version of salt '*' cmd.run. You can preview which minions will be affected with the target before sending the commands and then see the results in real time:

Making Salt an important part of SUSE Manager does not end there. In our industry being an Enterprise distributor of open-source software only works if you are part of it.

• SUSE already has around 600 commits from 5+ developers in Salt upstream
• The SUSE Manager team is hiring so that we can do more work upstream and help shape Salt’s future
• SUSE will be gold sponsor at Saltconf 2016

As you can see, SUSE Manager with Salt is a powerful duo which allows you to continue managing your infrastructure with the same tool, be 100% backward compatible and start taking advantages of declarative configuration management and the orchestration capabilities of Salt, while keeping everything you have already deployed untouched.

We are very excited about the possibilities here, which will be guided by feedback from our customers and synergies we have with Salt and other SUSE technologies.

Configuration Management is only one of the features that will arrive SUSE Manager 3. Expect to hear from the powerful Subscription Matching and Topology Awareness in future posts.