frkl - consolidating complexity

A (practical) ssh primer

Mon, 26 Aug 2019 00:00:00 +0200

This post is the first of a series of articles about basic underlying principles that are useful to know for the types of devops-y and sys-admin-y type tasks freckles usually helps with. For each topic, I’ll write one article that tries to explain the general principle(s) involved, and a second one outlining how this is used in the context of freckles. Feel free to skip that later part if you don’t intend to use freckles.

If in doubt, I’ll trade ‘technically correct’-ness for ease of explanation. Treat those articles as a rough overview to get you started, and consider diving deeper using other resources once you feel like you have a basic understanding of what is going on.

This post (and the following ones) assume you have some understanding of how to use the command-line to do certain tasks (changing folders, move files, execute applications, …). I am pondering whether it’s worth to write a primer on command-line usage too, ping me if that is of interest to you.

So, what is this ssh you are talking about and why should I care?

As long as you do your work on only your own computer, you won’t have to worry about ssh at all. But once you want to host some software, be it Wordpress, Nextcloud, or any of the other awesome packages that require an application to run on a remote server constantly – so they can serve user requests via the web at any time – you need some way to connect to the machine that hosts that software package. To be able to install, update and generally maintain that service.

This is where ssh comes in. ssh is a sort of “Remote Desktop” for your command-line. It runs as a service on the remote machine, and, depending on how it is configured, lets you login with a username and password (or private key – more on that later). Once logged in, you can operate on the remote machine as you would do locally.

I won’t go into detail on how to setup and configure an ssh service itself, most cloud providers have ssh setup and running in a basic way on virtual machines (VMs) you create, and we can just use that for our purposes here.

Authentication

Authentication is the most important thing we need from ssh: it is supposed to only let only known users onto the remote machine. If you hear of security breaches in the media, often times attackers gain access to such servers via a mis-configured ssh service.

Username/password authentication

The most basic way to login via ssh is via a username/password combination. This is not the most secure way to use ssh, as usually a password is not long enough to be considered really secure. But in some cases it is necessary, most often for the initial configuration of a server, which, as last step, often involves actually disabling ssh password authentication.

Depending on the cloud provider you choose, you will have gotten a username and password combination along with the other connection details of your ssh server. For the purpose of this tutorial, I’ll use the Hetzner cloud to create virtual machines (mostly because I find their interface easy enough to use, and they are cheap, so it’s a good fit when you want to try things out). If you use any of the others (Linode, DigitalOcean, AWS, Google, Azure,…) you’ll encounter a similar interface and workflow, but it’ll differ enough that you’ll probably have to look up other tutorials on the web for some details.

So, let’s get started. Let’s sign up for an account and go the Hetzner cloud management interface. There we’ll create a new project (I’ll call it ‘Tutorial’) and click the “Add server” button.

The details you chose on the following screen are not really important, I’ll use a Debian image and the cheapest machine type (cx11), and give the machine the name ‘tutorial’. All other fields are either default or blank.

Sidenote: The price of €2.49 is per month, if the server only runs for part of a month, you’ll only have to pay the hourly rate (€0.004/hour).

Once we ‘buy’ our server, we wait until it is provisioned by Hetzner. This usually takes 10 to 20 seconds. Hetzner will send you an email with the connection details.

We need the IPv4 address, the username, and the password to be able to connect via ssh:

ssh <username>@<ipv4>

The IPv4 address is necessary for the underlying technologies to route our commands to the right server. In my case, the IP address I received was ‘116.203.23.175’, and the username ‘root’ (which is the main ‘admin’ user on Linux systems).

So, here’s what I do on my terminal:

$ ssh root@116.203.23.175
The authenticity of host '116.203.23.175 (116.203.23.175)' can't be established.
ECDSA key fingerprint is SHA256:RpD/BudrVmHy0mri4aFK493JSmIul0pRRrHJ2Rz9Log.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '116.203.23.175' (ECDSA) to the list of known hosts.
root@116.203.23.175's password: xxx

The ‘authenticity’ message will be displayed on your first connection attempt, and can be used to verify that you are connecting to the server you intend to connect to (and there is no man-in-the-middle attack happening). How to do that is an advanced topic, and in practice not many people do that when creating new VMs like we just did.

Once I entered the password from the email I received, I’m logged in. The Hetzner VMs are configured in a way that when logging in the first time, the user has to set a new password for the ‘root’ user. Which is a good practice, because passwords sent via email can not be considered secure. Here’s how that looked for me:

You are required to change your password immediately (root enforced)
Linux tutorial 4.9.0-9-amd64 #1 SMP Debian 4.9.168-1+deb9u2 (2019-05-13) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Sat Jun 22 12:18:36 2019 from 89.158.123.195
Changing password for root.
(current) UNIX password: xxx
Enter new UNIX password: yyy
Retype new UNIX password: yyy

And that’s basically all there is to it. Except, as I’ve mentioned, that using username/password authentication is not really the best/safest way to use ssh. So, let’s look at:

Private key authentication

You can think of a ‘private key’ (also called ssh key in this context) as a really long (and therefor really secure) password, one that you don’t want to type in every time (and you couldn’t really remember it anyway). That is why it is stored in a file on your local machine (usually at $HOME/.ssh/id_rsa – $HOME is a variable that points to the root folder for your user, usually the full path is: /home/<username>/.ssh/id_rsa).

For security reasons, most of the times that really long password is encrypted with another, shorter password (one that you actually can, and should, remember). This encryption protects against users on the same computer just copying the ‘password/private key-file’ and using it to impersonate you. But it is also an inconvenience, because it means that everytime you want to use it, you have to enter your (short) password, to decrypt the private key.

Now, how do we create the private key? There’s a tool for that on almost all Linux (and Mac OS X) machines: ssh-keygen. It supports different options, but until you know better, I’d recommend just using one of the recommended parameters that is compatible with most systems and which will result in a reasonable secure setup:

$ ssh-keygen -t rsa -b 4096

Generating public/private rsa key pair.
Enter file in which to save the key (/home/markus/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/markus/.ssh/id_rsa.
Your public key has been saved in /home/markus/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:rzDEUqBmzKqYkhjullfnhFRXYdJeP4nP6Ml2zOlpS1c markus@t440p
The key's randomart image is:
+---[RSA 4096]----+
|    .    .o+.    |
| o . . . .o. .   |
|  *   o . . . o .|
| +   +     . . + |
|o   o + S     + E|
|=+   = o .   . o.|
|B.. . *   . o +.o|
|oo .   + .   =.*o|
|...     .   . ++.|
+----[SHA256]-----+

Sidenote: if you are ok with a very small risk of incompatibilites, it is recommended to use the ‘Ed25519’ algorithm instead of ‘rsa’ (-t ed25519 instead of -t rsa). Check the internet (like here) for more information. Using ‘rsa’ with a keysize of 4096 is a good enough choice though.

Once that is done, the ‘ssh-keygen’ command will have created two

$HOME/.ssh/id_rsa
$HOME/.ssh/id_rsa.pub

The first one is our private key, encrypted with the password we provided. This file is important, and should be kept secure. Never send your private key file over a network, or let anybody access it.

The second one is a file that contains a sort of unique ‘fingerprint’ of our private key. This file does not need to be kept secure. In fact, often you’ll be asked to provide the content of this file (e.g. on GitHub or similar services).

Why do I need this ‘fingerprint’? Good question, and if you understand the answer you understand the basic principle of why ssh private key authentication is done the way it is done.

I won’t go into detail about how exactly this works under the hood (not that I actually know…), but (thankfully) that is not necessary to use any of this. The main thing you need to know is that for the process of getting a sorta ‘fresh’ fingerprint you need the private key (you can’t just copy the fingerprint itself), and every such fingerprint is, for all practical purposes, unique amongst all of the fingerprints of all ssh keys in existence.

To be able to assert that without a doubt is very useful, because now, if you are a remote server with only knowledge of the fingerprint (which is usually called ‘public key’), you can ask the owner of the private key to give you a fingerprint (in a secure way), compare that with what you have in your records, and if there is a match you can be certain whoever asked you to let them in is in possession of the private key that belongs to the fingerprint.

The alternative would have been for you have the content of the private key in your records. And you compare that to the private key somebody who wants to login presents. This would have been bad for several reasons though, the worst being that the private key is now stored on the remote server, and anybody who can access that (via vulnerabilities, or maybe just because they are an admin user on that server) can now impersonate you on other services where you use the same key. Which would mean we would have to use a different private key for every server we want to connect to. Because we only use the fingerprint, and it is not possible to ‘fake’ the fingerprint-taking process without the private key (this is the crux, really), we can distribute our fingerprint to as many services and servers as we want, without having to worry about anybody we give it to will be able to impersonate us.

Now, let’s use our key!

First, let’s add our public ssh key to the VM we created earlier, manually. Then we’ll create a new VM, using ssh key authentication from the start (the recommended way).

To add our newly created key, we need to get the content of the public key (‘fingerprint’):

$ echo $HOME/.ssh/id_rsa.pub
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC8TbDtJ4P3aF8eM0rAIrSrxea53p3qisN2jbvzW1rKGbaSJXKFCprVa05gFGUBeBZq2coHhIEOX3j+h+oNsWVWPdfY/mLt1a5uvQ+skY1j7Roc74xjNw0HzW71sTesodGXdXOACU4ins9FHUCkDdDW62sSARRTMaBksLvWMvzhR3dRiA3KgUTy8R2ObJITEqh4Ztbkfwzi+ntOOLJOJ+zediTsYlsT5ZPDlRcJamU6fV8XqinmuppEtBtuM8qd3cn42Oh+00AxyebgY0wyE9EJ9jr5S+cHICRoOxQGWagGBKhh8JAcFx3RwbSMY1+rk5lDIXavW+w6HA5WAntNZzLL markus@frkl.io

To add that key to the ‘root’ user account of our VM, we need to login (using the password method), and then copy and paste our public key into the file $HOME/.ssh/authorized_keys on the remote machine. Alternatively we could use the ‘ssh-copy-id’ command, which will read thee public ssh key, login to the remote server, and copy the public-key-content into the appropriate place:

ssh-copy-id root@116.203.23.175

/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@116.203.23.175 password:

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh root@116.203.23.175'"
and check to make sure that only the key(s) you wanted were added.

After that we can try to login again, and this time around we should not be asked for the root password of the server, but for the password of our ssh key:

$ ssh root@116.203.23.175
root@116.203.23.175's password: <root_password>
Linux tutorial 4.9.0-9-amd64 #1 SMP Debian 4.9.168-1+deb9u2 (2019-05-13) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Sat Jun 22 14:17:19 2019 from 89.158.123.195

root@tutorial:~# echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC8TbDtJ4P3aF8eM0rAIrSrxea53p3qisN2jbvzW1rKGbaSJXKFCprVa05gFGUBeBZq2coHhIEOX3j+h+oNsWVWPdfY/mLt1a5uvQ+skY1j7Roc74xjNw0HzW71sTesodGXdXOACU4ins9FHUCkDdDW62sSARRTMaBksLvWMvzhR3dRiA3KgUTy8R2ObJITEqh4Ztbkfwzi+ntOOLJOJ+zediTsYlsT5ZPDlRcJamU6fV8XqinmuppEtBtuM8qd3cn42Oh+00AxyebgY0wyE9EJ9jr5S+cHICRoOxQGWagGBKhh8JAcFx3RwbSMY1+rk5lDIXavW+w6HA5WAntNZzLL markus@frkl.io' >> $HOME/.ssh/authorized_keys

root@tutorial:~# exit
logout
Connection to 116.203.23.175 closed.

vagrant@debian-testing:~$ ssh root@116.203.23.175
Enter passphrase for key '/home/vagrant/.ssh/id_rsa':
Linux tutorial 4.9.0-9-amd64 #1 SMP Debian 4.9.168-1+deb9u2 (2019-05-13) x86_64
...
...
...

The one thing to note here is that the authorized_keys file (always in $HOME/.ssh/authorized_keys) is the servers record for which private key ‘fingerprints’ to allow access for the particular user that tries to login. Every line in it denotes one public key (‘fingerprint’).

We can let Hetzner do the same thing for us, by registering our public ssh key with them. Go to the Hetzner cloud ‘Access’ page of our project (left side, key symbol), and click the ‘Add ssh key’ button.

Copy and paste your public key (the line that starts with ‘ssh-rsa….’) into the approriate text field, and give your key a name (‘tutorial-key’).

Now, the next time you create a machine, the puchase page lets you select from your registered ssh keys. Select the one you want to be included, and after the server is up you should be able to use it to login the same way we did when we added the key to ‘authorized_keys’ manually.

Password protected private keys, or not?

As I’ve mentioned above, usually we protect our private key with a password, but sometimes we don’t. When should we do what? Well, rule of thumb: use a password.

Once you start managing servers, it’s getting a bit more difficult though. The issues start once we do ‘automation’. ‘Automation’ is everything we want our computers to do automatically, without us having to do anything, except for an initial manual setup of the automation process (and even that we sometimes automate – but that is a different topic).

‘Automation’ means we should not have to intervene at all. One example would be a task that gets done every day at the same time (often via ‘cron’ jobs). Or a setup task you want to be kicked off every time you commit to a git repository.

If that task involves your private ssh key in some way (for example: build the package, login to a server via ssh, then install the package), you somehow have to ‘unlock’ your private key for every one of those tasks. You could, of course, just store the password to the private key on the same machine as your private key. But that would defeat the purpose, and would be pretty much as secure as just using no password on your private key at all. Because, if an attacker can read you (encrypted) private key, they also have access to read your password file…

There are ways to protect your private key in scenarios like this, but almost none of them are flawless, and none of them are straight-forward. You having to manually enter a password is really the best in terms of security. And the worst in terms of convenience. This is why there are certain situations where people decide to not secure the private key with a password, and accept the security implications that come with this decision.

There is not much advice I can give on a primer like this, I’d just recommend that whenever you get stuck trying to automate something, and passwords are involved, try to search the internet for people with a similar problems and technologies, and check whether there seems to be a sort of consensus (well, as close an approximation to ‘consensus’ there exists on the internet, anyway) ‘best-practice’ for your type of situation.

The `ssh-agent`

On your local machine, there is one remedy for the inconvenience of having to enter the password to your ssh key every time you want to use it: ssh-agent.

It comes with a minor impact on security, but my feeling is that most people accept that for the huge boost in convenience it brings.

ssh-agent is a little daemon (a program that runs in the background) which can store passwords for ssh keys in a secure way. That means, you have to enter the password once, the first time you use a ssh key in a session, and after that the ssh-agent will intercept any access requests for the ssh key in question, and will provide the decrypted key to whatever program requested it.

Most people (and Linux distributions) start ssh-agent automatically in some way when the user logs in. It is hard to give general advice on how to use it because of the different ways this is done. So I’ll only outline how it works in principle, below.

If you type ssh-agent in your terminal, you’ll see something like:

$ ssh-agent
SSH_AUTH_SOCK=/tmp/ssh-eq76YPuBvNU8/agent.12962; export SSH_AUTH_SOCK;
SSH_AGENT_PID=12963; export SSH_AGENT_PID;
echo Agent pid 12963;

This does 2 things:

start the ssh-daemon (type ps axu|grep ssh-agent to confirm)
print out details that are needed to connect to the running ssh-agent daemon

If you start the agent several times, you’ll notice that the printed details are always slightly different. The reason for this is that you can have several ssh-agents processes running at the same time, and each of them can store different (or the same) ssh keys. In most cases we only want one of those agents running though.

Now, what are those SSH_AUTH_SOCK and SSH_AGENT_PID variables? Those need to be present in the environment of a process that wants to use the ssh-agent so the process knows how to connect to the agent. Reading the content of the SSH_AUTH_SOCK variable is how that is done. In our case, an application would read that value, and would use the /tmp/ssh-eq76YPuBvNU8/agent.12962 path (which is not a ‘normal’ file, but a unix socket) to ‘talk’ to our ssh-agent.

So, what the output of our ssh-agent command is telling us: ‘those are my connection details, please put them in your environment if you want to use me’ (it’can’t do it itself for … reasons). So, we can either do this now, manually (by copying and pasting the two relevant lines of the output):

$ SSH_AUTH_SOCK=/tmp/ssh-eq76YPuBvNU8/agent.12962; export SSH_AUTH_SOCK;
$ SSH_AGENT_PID=12963; export SSH_AGENT_PID;

Or, we use the eval command to do it for us (which is what the ssh-agent output is designed for), by wrapping the eval around our ssh-agent command:

$ eval $(ssh-agent)

Basically, this executes ssh-agent, then it executes the output of the ssh-agent execution. We can confirm this worked by checking our processes for the ‘ssh-agent’ process, and the contents of our SSH_AUTH_SOCK and SSH_AGENT_PID environment variables:

$ ps axu|grep ssh-agent
markus   24648  0.0  0.0   5164   336 ?        Ss   13:59   0:00 ssh-agent
markus   25117  0.0  0.0   6604   728 pts/3    S+   13:59   0:00
grep ssh-agent

$ echo $SSH_AUTH_SOCK
/tmp/ssh-i1u4UBNrqPQZ/agent.24647

$ echo $SSH_AGENT_PID
24648

Right. Now we have ‘ssh-agent’ running. What’s next?

We need to register the ssh keys we intend to use with it! In most cases, there exists only the default key: $HOME/.ssh/id_rsa

Because this is the default, we don’t need to provide a path to the key, we only need to call:

$ ssh-add
Enter passphrase for /home/markus/.ssh/id_rsa:
Identity added: /home/markus/.ssh/id_rsa (/home/markus/.ssh/id_rsa)

Once that is done, every ssh connection that uses that key will use the stored key in ssh-agent, and we’ll never have to use provide the ssh key password manually again.

'Dockerfiles' for LXD

Sat, 10 Aug 2019 00:00:00 +0200

tldr

freckles can be used as a generic, composable ‘Dockerfile’ replacement for LXD. To try it out, on a system that has freckles and LXD installed and working, put the following content into a file 'example-lxd.frecklet‘:

- nginx-service
- nginx-vhost-from-folder:
    document_root: /var/www/html
    default_server: true
    use_https: false
- file-with-content:
    owner: www-data
    path: /var/www/html/index.html
    content: |
      <h1><i>freckles</i> says "Hello World", from {{:: from ::}}!</h1>

This will act as our ‘Dockerfile’ and describes how to install and configure Nginx to serve a static webpage. To create an LXD image from it, execute:

$ frecklecute lxd-image-from-frecklet-file \
     --install-packer \
     --source-image images:debian/10 \
     --image-name freckles-image \
     --frecklet-path $(pwd)/example-lxd.frecklet \
     --frecklet-vars '{"from": "my first image"}'

Once that has finished, create a container from the new ‘freckles-image’ image:

$ lxc launch freckles-image freckles-test

To test whether the Nginx service that runs in the container works, issue:

$ curl http://<container_ip>
<h1><i>freckles</i> says "Hello World", from my first image!</h1>

Long-ish preamble

I think Dockerfiles played a fairly big part in Dockers success and adoption early on. They are simple to read/understand, flexible enough to be useful beyond basic tasks, and composable (if you squint your eyes a bit and look at ‘inheriting from another Docker image’ as composition).

Of course, there are other factors for Dockers success, being at the right time at the right place (a.k.a. luck) among them, but Dockerfiles are certainly important. Docker containers are usually considered to be ‘application containers’, and part of a microservice-type architecture. In practices, that is (and probably never was) 100% true, there are a lot of people trying to force Docker into something else, for example coming up with ‘init-systems’ for Docker, trying to run webapps incl. Database and Memcache and whatever else in a single container. Think of that what you will, it’s still a testament for Dockers success, and the flexibility of Dockerfiles, because they actually allow for all this to be built.

Now, if we just accept the premise there are cases where its advantageous to have multiple services on the same host without the overhead of a virtual network connecting all of them (and we could certainly argue when that makes and doesn’t make sense), we’ll sooner or later stumble upon LXD (I’ll ignore the more low-level LXC, but I acknowledge that sometimes it’d be a better fit).

LXD is a so-called ‘system container manager’ (as opposed to ‘application container’), which means it ‘contains’ whole operating systems, including a ‘normal’ init system. An LXD container behaves like a physical or virtual machine, except for some permission-related things, but without much overhead (like ‘real’ VMs, for example).

LXD does not have an equivalent to a Dockerfile though (fairly or unfairly ignoring ‘cloud-init’ in this instance – this is a topic for a whole other blog post). One reason for this is that it’s not really necessary. Since an LXD container behaves like a normal physical or virtual machine, we can use the same tooling to install applications, and generally get a container into the state we want it to be. And for the case where sysadmin/devops-professionals do the state-making, that is fine, they know what to do, and how to do it in an automated way. Using tools like Ansible, Puppet. Or maybe Packer, Terraform, etc.

The one advantage a ‘Dockerfile for LXD’ would have in that situation is to bring easy reproducability, re-usability, and just general ‘orderlyness’ to LXD for people who don’t have to deal with those things usually. In the same way Dockerfiles brought those things along with Docker. All you need is a single text file, and you’ll be able to replicate and costumize a Docker image. I think we can’t overestimate the effect of a technology that allows people who know what they are doing (those who create Dockerfiles) to share what they are doing (Dockerfiles) with people who wouldn’t have known how to do that particular thing (at least without spending a few hours on Stackoverflow) or can’t be bothered, and who now can just re-use that particular thing.

A thing which, by the way, now only has to be created once, and which then can be continuously improved upon by a community of people (who ideally know what they are doing). It’s like using a specialized library in any programming language, it allows you to focus on your core problem as an application developer, without too much distraction on those tangential details you’d have to worry about otherwise. It’s really not that deep of a problem, everything considered. But it’s wide-spread, and prevalent to the point that we often don’t even see and recognize it in our daily (working) life, which means we are not taking advantage of the already existing solutions for such a problem.

This is, incidentally, exactly what freckles tries to solve too, just in a more generic way than a Dockerfile does. Allow for re-usable recipes to bring computational environments into a certain state, technolgy- and stack-independent.

Which, finally, brings me to the topic of this post: using frecklets as sort of re-usable ‘Dockerfile-replacements’. In the context of LXD. Of course, since they are re-usable, you can use the same frecklet you use in LXD for Docker too (with some caveats), or, to provision a remote server (as I’ll show at the end of this post).

Using freckles with LXD

There are two separate areas where we can choose to use freckles in combination with LXD:

describing the content/state of the images
building the actual LXD images

In addition, we can use freckles to install and configure LXD:

Optional: using freckles to install LXD

Note: the following does not setup internet connectivity for containers who are connected to the newly created bridge ‘lxbr0’. This has to be done manually (for now), e.g. by doing iptables -t nat -A POSTROUTING -s <bridge_subnet>.0/24 -o eth0 -j MASQUERADE. This will be done automatically sometime in the future, hopefully.

For this, we can use the lxd-service frecklet:

$ frecklecute lxd-service --user markus

Under the hood, this uses the juju4.lxd Ansible role to install LXD on a Ubuntu or RedHat-based Linux system. Other distributions might work, but are not tested. Also, note the --user markus command-line argument. This lets you specify one or several users who’ll have permissions to use the LXD service on this host. After running this, you’ll have to either log-out of your current session and log-in again, or execute newgrp lxd to take advantage of this permission.

It is possible to adjust some parameters related to your LXD setup with this frecklet (like for example network addresses or storage settings), but this is out of scope in this context.

Describing the LXD-image contents

We’ll be writing a relatively simple frecklet, not different at all to anything we’d write for a different target type (e.g. a remote server). For more details on how to all this works, and how you can write your own, please check the freckles getting started guide and the other frecklet-related documentation.

For our example, lets install the Nginx webserver and configure it to host a single, static html site (which we also create dynamically).

As this is a relatively ‘normal’ thing to do, there are already pre-made frecklets that can help us implement the required steps, which are:

install the Nginx packages and enable/start the service (nginx-service)
create a configuration for our vhost (nginx-vhost-from-folder)
create our html file (file-with-content)

Here’s how that looks as a frecklet:

- nginx-service
- nginx-vhost-from-folder:
    document_root: /var/www/html
    default_server: true
    use_https: false
- file-with-content:
    owner: www-data
    path: /var/www/html/index.html
    content: |
      <h1><i>freckles</i> says "Hello World", from {{:: from ::}}!</h1>

Note the {{:: from ::}} template string in our frecklet code: this lets us customize the generated html for every LXD image (if we chose to do so).

To test, lets launch an LXC ocntainer and provision it with our new frecklet. We could do this with a simple lxc launch ... command, but while we’re at it let’s use freckles for this too. In an interactive session that doesn’t make much sense, since it’s also just one command, and comes with a bit of overhead. But it’s a good opportunity to show how one would do that if launching an LXD container was part of a provisioning pipeline. So, here goes:

$ frecklecute lxd-container-running --image-name 'debian/10' \
     --image-server https://images.linuxcontainers.org \
     --name 'freckles-test' \
     --register-addresses freckles_container_ip

╭╼ starting run
│  ├╼ running frecklet: lxd-container-running (on: localhost)
│  │  ├╼ starting Ansible run
│  │  │  ├╼ create/launch container 'freckles-test'
│  │  │  │  ╰╼ ok
│  │  │  ╰╼ ok
│  │  ╰╼ ok
│  ╰╼ ok
╰╼ ok

Result:

    freckles_container_ip:
      eth0:
      - 10.10.10.42

In this example, we provided the optional --register-addresses command-line argument. That prompts freckles to register the (network) details of our newly created container into a variable, which makes it easier for us to connect to it later. Alternatively, you can get the ip address with a simple lxc list.

Now that our container is running, we can provision it using our frecklet from above. We’ll save the content string as example-lxd.frecklet, then issue:

$ frecklecute --target lxd::freckles-test example-lxd.frecklet --from 'a test run'

The crucial thing is the --target lxd::<container_name> part. It tells freckles to not connect via ssh, but directly via ‘lxd’.

This will take a minute or two, as there is some bootstrapping to be done in addition to installing Nginx. Once the process is finished, you should be able to point your browser to the IP address displayed in the run before this, and see the message “freckles says “Hello World”, from a test run!”

Sweet, now we have everything we need to make that into an LXD container image…

Building an LXD image

While it’s possible to create an LXD image from scratch using freckles, for example by capturing the steps from this tutorial into a frecklet, we won’t do that here. We are going to be cheating a bit, and use Packer instead, since that offers the same functionality (and more), with less hassle.

As we already have our example-lxd.frecklet file, all we need to do is provide it’s full path in the following command:

$ frecklecute container-image-from-frecklet-file \
     --install-packer \
     --image-type lxd \
     --source-image images:debian/10 \
     --image-name freckles-image \
     --frecklet-path $(pwd)/example-lxd.frecklet \
     --frecklet-vars '{"from": "my first image"}'

Let’s look at that command in detail:

container-image-from-frecklet-file: the name of the frecklet we want to execute (we could have also used the wrapper frecklet lxd-image-from-frecklet-file
--install-packer: we make sure that the Packer executable is available on our system (using the packer-installed frecklet under the hood)
--image-type lxd: we want an LXD image (we could have also said: docker, but since there is some systemd/init stuff in our frecklet, indirectly, that won’t work in this case)
--source-image images:debian/10: the source image to use (notice the ‘images:‘-prefix, this is needed in this case to use the correct LXD image index
--image-name: the name of our new image
--frecklet-path $(pwd)/example-lxd.frecklet (absolute!) path to the frecklet that contains our provisioning instructions
--frecklet-vars '{"..."}': additional variables (as a JASON string) which are needed by our provisioning frecklet. The alternative to using this is hard-coding everything in the frecklet file

So, once this command finished, you should be able to see the new image via:

$ lxc image list
+----------------+--------------+--------+-----------------------------------------------+--------+-----------+-------------------------------+
|     ALIAS      | FINGERPRINT  | PUBLIC |                  DESCRIPTION                  |  ARCH  |   SIZE    |          UPLOAD DATE          |
+----------------+--------------+--------+-----------------------------------------------+--------+-----------+-------------------------------+
| freckles-image | f9600c979eba | no     | built with freckles.                          | x86_64 | 194.72MB  | Aug 10, 2019 at 10:37am (UTC) |
...
...

And we can use it like so:

$ frecklecute lxd-container-running --image-name 'freckles-image' \
      --name 'freckles-example' --register-addresses freckles_container_ip

╭╼ starting run
│  ├╼ running frecklet: lxd-container-running (on: localhost)
│  │  ├╼ starting Ansible run
│  │  │  ├╼ create/launch container 'freckles-example'
│  │  │  │  ╰╼ ok
│  │  │  ╰╼ ok
│  │  ╰╼ ok
│  ╰╼ ok
╰╼ ok

Result:

    freckles_container_ip:
      eth0:
      - 10.10.10.100

Now point your browser to the container ip (10.10.10.100 in my case), and you should see a message like: ‘freckles says “Hello World”, from my first image!’.

Re-using our frecklet

The main reason I wrote freckles was to have a generic, flexible and re-usable way to describe desired states of computational environments. We already used our frecklet to provision a running (empty) LXD container, and to create an LXD container image from it. If our frecklet didn’t contain init-system-specific instructions (setting up and restarting a systemd service unit), we could also use it to build a Docker image. This won’t work here, but lets use it instead to setup a remote server (like an EC2 instance, or a Digital Ocean droplet):

Requirements:

(empty) Ubuntu/Debian remote machine with public IP
root or sudo access to that machine

We don’t need to make any changes to our frecklet, all we need to do is change the --target in our command:

$ frecklecute -t root@159.69.201.220 example-lxd.frecklet --from "a remote machine"

╭╼ starting run
│  ├╼ running frecklet: /home/markus/projects/frkl-dev/frkl.io/example-lxd.frecklet (on: 159.69.201.220)
│  │  ├╼ starting Ansible run
│  │  │  ├╼ updating apt cache
│  │  │  │  ╰╼ ok
│  │  │  ├╼ ensure rsync, ca-certificates and unzip packages are installed
│  │  │  │  ╰╼ ok
│  │  │  ├╼ creating freckles share folder
│  │  │  │  ╰╼ ok
│  │  │  ├╼ creating box basics marker file
│  │  │  │  ╰╼ ok
│  │  │  ├╼ recording python interpreter metadata
│  │  │  │  ╰╼ ok
│  │  │  ├╼ recording box metadata for later runs
│  │  │  │  ╰╼ ok
│  │  │  ├╼ Ensure nginx is installed.
│  │  │  │  ╰╼ ok
│  │  │  ├╼ Copy nginx configuration in place.
│  │  │  ├╼ reload nginx
│  │  │  │  ╰╼ ok
│  │  │  ├╼ write content to file: /etc/nginx/sites-enabled/localhost.http.conf
│  │  │  │  ╰╼ ok
│  │  │  ├╼ ensure user 'www-data' exists
│  │  │  │  ╰╼ ok
│  │  │  ├╼ write content to file: /var/www/html/index.html
│  │  │  │  ╰╼ ok
│  │  │  ├╼ geerlingguy.nginx : reload nginx
│  │  │  │  ╰╼ ok
│  │  │  ╰╼ ok
│  │  ╰╼ ok
│  ╰╼ ok
╰╼ ok

$ curl http://159.69.201.220
<h1><i>freckles</i> says "Hello World", from a remote machine!</h1>%

On licensing freckles

Mon, 17 Jun 2019 18:00:00 +0200

Ok, I guess I have to write this. Why is freckles licensed using the Parity Public license, a license that most people haven’t heard of (yet), that is not approved by FSF/OSI, and that is also very very copy-left, compared to the usual suspects?

There are a few reasons, and it took me quite a while to arrive at the conclusions I eventually arrived at. I’m pretty confident anything I say here has been said before, and most likely much more eloquent and exhaustive. Probably by Kyle Mitchell, whose writings here and here I can’t recommend enough.

Sidenote: Parity allows you to run software licensed with it in combination with other open-source software, as much as you want. If you want to use it in combination with non-open software, you’ll have to purchase an alternative license for the code in question. Read the license text, it’s very short!

Oh, one thing, before I start: I usually avoid calling freckles ‘open-source’. Not because I think it is not ‘open-source software’, but because the OSI claims to own the term, and any license that is not approved by it should not be called ‘open-source’. A few (loud-ish) people on the usual opinion-haunts agree. So, that’s why I just don’t call freckles open source software: because I’m not interested in that particular argument. In this blog post I will do so though. Because in this context here it’s just easier, and to me, software licensed using the Parity Public license counts as open-source software, and is not too far away from what Richard Stallman wanted, back when. In a lot of ways it’s probably closer than to what the OSI stands for today. But, as I said: not having that argument. This is really about my (personal) decision to use a very strong copy-left license.

Oh, and another thing: the main reason I wrote freckles was because I very strongly felt that something like it should exists, and I could not understand that it didn’t yet. Because it seems really obvious to me, to a point where I can’t even explain why. I’ve actually waited a few years before I (reluctantly) started writing it. It’s not the most interesting thing to write, really. Anyway: the parts of me that have nothing to do with my programming-related obsessions obviously realize that the fact that nobody ever bothered writing it, in addition to the blank looks I usually get when I tell people about it, indicate that this is probably really nothing nobody ever wants to use. Which makes it a bit embarrassing when I talk about actually charging people money for it. For the purpose of this article, just assume I’m talking about something people would be at least somewhat inclined to pay for, ok?

With all that being said:

Reasons!!!

Idealism

I would like to live in a world where all software is open-source. Boring people will say that is not realistic, but boring people would say that, wouldn’t they? I reluctantly agree that this will not transpire in the next short to very long future, but hey, that doesn’t mean that we who think that this should be the case shouldn’t do their small parts. Parity gives open source software a leg up, because it can be used without any problems from other open-source software, but needs to be purchased when people want to use it with code they don’t want to share.

Money

This probably sounds like a humble-brag, but I’m not really good at half-assing things. I don’t see that as a strength all of the time, but sometimes it is. Anyway, I wasn’t really happy with my first ‘proof-of-concept’ for freckles, even if it kind of worked. I really wanted to properly understand the thing I was trying to fix, and make it as simple (not easy) as I could. This took me about 3 or 4 re-writes. Now I’m happy (enough) in this regard. But I also spent a ship-load of time on it. And I wouldn’t have done it if I didn’t think there was at least a very small chance that it could pay off in some other way than me being satisfied about having understood something. Motivations, and how the prospect of reward driving society, and all that.

If I had chosen any of the ‘normal’ open source licenses (even AGPL), there wouldn’t be much of an incentive for people to pay for freckles. The triggers that enforce sharing your code just don’t apply. I could have, of course, chose one of the other open-source business models, not dual-licensing:

Open-source business models, and what I don’t like about them

Donations

This is not the relationship I want to have with my users. I don’t think any party in such a relationship should feel ‘thankful’ for something the other party did. I don’t want to feel thankful for users having donated, even though they didn’t need to, and users shouldn’t need to feel thankful because I created something and gave it out for free. I like the sort of relationships that result from market-driven interactions much better: I take a risk, and spend time and money creating something I think can be of value for you. You wait until it is clear that thing is of value for you, and you pay for that value (plus a bit extra for the risk I took). My free decision to take that risk, your free decision to pay for a product after you compared it to other options. Nobody has to be thankful, and we can all be adults about it. No (perceived, or real) power imbalance.

Open core

I don’t like the overhead this creates, in terms of having to split up code more or less arbitrarily. Driven by business decisions, not what would make sense technically. A code-base likt that is much more cumbersome to deal with, especially in regards to having to setup everything twice (CI pipelines, release channels, …). But more importantly, it’s going to be really difficult to align my incentives with my users interests.

Also, ‘idealism’: everybody can use everything, if you make you own software freely available, you might have to have a slight advantage over your proprietary competitor by not having to pay.

Software/platform/whatever as a service

This would have been possible for freckles, and probably would have been a bit easier to implement too (but harder to run/maintain in the long run). But it would have taken away a lot of value from the user. The way it is now, freckles is super-flexible, and can be used in a lot of circumstances, without much hassle at all: embed the binary in your project, use the Python library and Python code, write a small REST-wrapper around it and call it remotely, host the binary yourself, use my hosted version, etc. If it was a hosted service, users would have to trust me with their secrets, and data in general. I might still decide to provide such a hosted service at some point, but it’d be in addition to all the other options users can pick from.

Also, I really dislike how there are so many startups creating SaaSes for things that should just be libraries, if you look at it from a technical perspective. I see how MRR is nice, but really, this is getting pretty silly.

Corporate sponsorship

I could live with that, I guess. Depending on the company, and terms. Still, I like the idea of a software-cornershop with a handful (enough to sustain) users better.

Tit for tat

I think it’s only fair: if you don’t want to give me your software, why should I let you use mine? There are not a lot of good moral arguments to be made for why the freedom of people who don’t want to protect their users freedom should be protected. There is an argument to be made by open-source developers who want to protect their users freedom (even if those users don’t want to protect their users freedom in turn). If that’s you, fine, I’ll grant you that. But I might want you to show me what and how much you created and invested, first.

For now, that’s all I can think of. Will add to this over time.

Data-centric environment management

Mon, 22 Jan 2018 18:00:00 +0200

Note: This was written for a previous version of freckles. The code/examples in here won’t work anymore.

tl;dr: Imagine you get a piece of data, some folder or zip file from a friend. It doesn’t matter what it is. Some source code in Python or Java or Cobol. Their dissertation in LaTEX. A backup of a Wordpress instance. A blender project. Research data. Anything, really.

Imagine there is an application you can just point to that folder, and run it, without having to tell that application what that data is. Imagine that run automatically sets up your laptop so that you can instantly work with or use that data. Without you having to do anything else. No installing of applications by hand, no having to hunt for the right versions of those applications, no configuration. No compiling.

That is what this is about.

I hate repetition. If you are working in I.T., chances are you hate repetitive work also. Luckily, a lot of I.T. work is devoted to cutting down on repetitive work. Because, after all, that’s what computers are good at: give them the same problem several times, and they’ll work on it the same way all those several times. Without complaining, mostly. Even if they have to do it repetitively a million-billion times and then a million-billion times again.

Besides any potential personal aversion against repetitive work one might have there is another reason humans shouldn’t be doing repetitive work if a computer can do it: the chance of making a mistake grows the more often you do a repetitive thing. And you might end up with a different end-result some of the time. Computers either do it always wrong, or if we’re lucky, always right. Provided, of course, somebody wrote tests for all possible and impossible edge cases. Which, naturally, we all do, every time.

So, long story short, there is one (non-obvious, but very generic) repetitive thing that, over the years, annoyed me more than other repetitive things: setting up an environment on a computer (virtual, physical, whatever) in order for this computer to deal with a set of files/data that is of a type the computer isn’t prepared to deal with yet.

For example:

if I have Python source files, I need to make sure I have (the right version of) Python installed, a virtualenv created, and dependencies installed via pip
if I have markdown files representing content for this here blog, I need to setup and configure a webserver (say, nginx), maybe install PHP and probably also some (the right) PHP libraries, then I have to download grav and put it into the right folder so nginx can find it
if I have backup data of a service that needs migration to a new machine (virtual or not) I have to re-setup and configure the service, and restore the backup data in some way or other

It appears to me that, if I know what type of data I deal with, and if I have a set of (ideally best) practices for that type of data, a computer would be able to do that sort of setting up for me. Right? … RIGHT???

Not only that, if the computer would know what kind of platform/distribution/version of distribution it runs (which it always does since, hey, it’s the one running it…), and if it had instructions that outline what to do differently an each of those platforms, it could always, automatically, prepare a host environment that is hospitable to the kind of data in question, and there would be no manual intervention necessary. AT bloody ALL.

What would be necessary is somebody preparing those sort of recipes, best practices and platform-dependent instructions for all the potential types of data we come across. In a way that the computer can understand. But, the good thing is, we could do that in a collaborative and evolutionary fashion, starting off with a simple use-case and best practice, and build on top of that to support more options, features, and platforms in the future. We’d have one place to improve a recipe for a given use-case or type of data, and that recipe would go through the normal stages of software development until it can be considered stable and comprehensive enough.

So, what’s left is the glue, an application that runs on the computer, is pointed at the data we are interested in, parses that data (and potentially existing augmenting metadata), chooses the right recipes for that type of data and platform it runs on, and executes those recipes in the way the data/metadata demands.

There are two classes of existing applications that do parts of what I’m describing: configuration management engines like Ansible, SaltStack, Puppet, etc., and build systems like make, maven, Rake and so on. But those are either focused on a bigger infrastructure and network environment, only understand a certain type of data (Java project, Ruby project, …), or are very low-level and don’t have the building blocks to manipulate the state of a machine in an efficient way. There might be other tools, but if there are, I don’t know about them.

Now, all of this led me to work on freckelize. freckelize is part of a project called freckles which is designed as a layer on top of Ansible, and is an experiment to find ways to re-use all those existing tasty Ansible modules and roles for things besides ‘traditional’ configuration management.

In this blog post I’m not going into too much detail about how freckelize works, what features – aside from the basic ones – it has, what the security implications of using a tool like that are. And anything else that might distract from getting across the basic idea behind it. I’ll write about all that in a follow-up post.

So, to illustrate that basic idea I’ll use the very simple example of hosting a static webpage, where the data we work with is a single html file. I also wrote a more in-depth blog post about this exact usage scenario, so if you are interested in those details, check it out here later.

NOTE: the ‘static-website’ recipe I’m using below is currently only tested on Debian Stretch. Help me improve it and add more supported platforms?

The example dataset – a single html file, plus an optional metadata file named .freckle – can be found here: https://github.com/freckles-io/example-static-website.

Let’s put those two files in a folder called example-static-website. The index.html file looks like this:

<!DOCTYPE html>
<html>
<body>
<h1>Now, what is all this?</h1>
<p>No idea at all, mate.</p>
</body>
</html>

And this is .freckle file, which contains additional metadata:

- static-website:
    static_website_domain: 127.0.0.1   # ip address or domain name used by this server
    static_website_port: 80            # port the webserver listens to

This latter .freckle file is optional, but useful to adjust some of freckelize’s behavior. It uses yaml syntax, and, at it’s root level, contains a list of types of data to be considered, including potential variables per type. In this case it contains two variables, which both are set to default values, which means that this file doesn’t affect behavior just yet.

So, this is what you have to do (assuming freckles is already installed) to install a webserver (nginx in this case) and configure it to host our website:

freckelize static-website -f example-static-website

Done. Simple, he? Check if it’s working by visiting: http://127.0.0.1

Since this folder already contains a .freckle file that includes the ‘static-website’ type, we could have just omitted the ‘static-website’ command:

freckelize -f example-static-website

In the case that we don’t have that folder on our local machine but only on Github, we can let freckelize also clone it for us, before doing it’s thing:

freckelize -f https://github.com/freckles-io/example-static-website.git -t /var/lib/freckles

This will check out the repository as a sub-folder of /var/lib/freckles (which is a nice place to collect those sort of folders). Then it’ll do exactly what it did before, when using the local folder.

There are more scenarios freckelize supports, like for example pointing it to a remote tarball of the data. Refer to the documentation for details. In the future, anyway. Need to re-write parts of that documentation to bring it up-to-date. Soooooorry.

As an example this is not really impressive, I’m sure, as this is something that would not take a lot of time to do by hand. Just a sudo apt-get install nginx, and some configuration editing somewhere in /etc/nginx/.

To illustrate how easy it is to accomplish more complex tasks: let’s say we want to host that website on a VPS somewhere, via https and a (valid) Let’s encrypt certificate. This is supported by the static-website (source) recipe (‘adapter’ in freckelize-speak). We need to provide a bit more information to freckelize though, as it wouldn’t know the domain name to use, and the email address the folks over at “Let’s encrypt” require. Also, we need to configure DNS so that the domain name we use points to the VPS IP address. This last thing has to be done manually though, and since it depends a lot on the providers that are used I won’t write about how to accomplish it.

Let’s edit the .freckle file:

- freckle:
    owner: www-data
    group: www-data

- static-website:
    static_website_domain: example.frkl.io
    static_website_port: 80
    lets_encrypt_email: makkus@frkl.io

We add a generic section (always called ‘freckle’) about who should own the data (which will also determine the user nginx runs under). We leave the port as ‘80’, the adapter will automatically create a vhost configuration to forward all traffic to the default https port (443) if configured to use https. The adapter is written in a way that, if it encounters the lets_encrypt_email variable with a string other than ‘none’, it’ll use that value as email address and request a https certificate for the domain specified from the “Let’s encrypt” service. In addition, it’ll setup a cron job that makes sure that certificate will be re-newed before it expires.

So, that was that. I hope all that made at least a tiny bit of sense to anybody other than myself…

There’s a lot more to be said. For example, how that is different from ‘normal’ configuration management, why you would choose one over the other, how one could compliment the other, how this could be used with technologies like LXC, Docker, Vagrant. What the disadvantages of using this kind of thing are. And what else could be done with it that I haven’t even hinted at yet.

My plan is to write about all that and more in the future. So, please, do check back here every now and then.

Also, get in touch if you have questions, or suggestions. Either via email, gitter, or a Github issue.

Writing declarative command-line scripts, using Ansible modules and roles

Sat, 04 Nov 2017 13:00:00 +0200

Note: This was written for a previous version of freckles. The code/examples in here won’t work anymore.

Ever wanted to just quickly run a few Ansible tasks or apply a role from Ansible Galaxy, without having to manually setup Ansible, or create an inventory, and/or download role(s) from Ansible Galaxy, etc…? Or have you ever wished you could write a re-usable command-line script using Ansible modules and roles? Or, maybe you haven’t thought about it before, but now that I mention it…

If, this here blog post is for you.

We’re going to be using the frecklecute command-line interface for this, which is part of the freckles package. To learn more about freckles itself: I’ve written about what it is and what you can do with it here.

tldr; show me the goods

Here’s one such example script (a ‘frecklecutable’) which sets up a user account (if it doesn’t exist yet) and makes sure the user is part of the wheel group (which will also be created if necessary). The script also sets up ‘passwordless’ sudo permissions for that wheel group.

Right. This is the script (let’s name it setup_sudo_user):

doc:
  short_help: "setup new sudo user"
  help: "Sets up a new user with (passwordless) sudo privileges.\n\nInstalls the 'sudo' package if necessary, and creates a group 'wheel' which will be allowed passwordless sudo-access."

args:
  user_name:
     help: "the name of the user"
     is_var: false
     required: yes
  password:
     help: "the user password hash (generate with 'mkpasswd -m sha-512')"
     is_var: false
     required: yes

tasks:
  - group:
      name: wheel
      state: present
  - package:
      name: sudo
      state: present
  - lineinfile:
      dest: /etc/sudoers
      state: present
      regexp: "^%wheel"
      line: "%wheel ALL=(ALL) NOPASSWD: ALL"
      validate: "/usr/sbin/visudo -cf %s"
  - user:
      name: "{{:: user_name ::}}"
      password: "{{:: password ::}}"
      update_password: always
      groups: wheel
      append: yes

As you can see, we use the yaml format to describe what we want to achive. To use this script, assuming we already bootstrapped freckles, all we need to do is (as the ‘root’ user, in this case):

# frecklecute setup_sudo_user --help
Usage: frecklecute setup_sudo_user [OPTIONS]

  Setups a new user with (passwordless) sudo privileges.

  Installs the 'sudo' package if necessary, and creates a group 'wheel'
  which will be allowed passwordless sudo-access.

Options:
  --password TEXT   the user password hash (generate with 'mkpasswd -m
                    sha-512')  [required]
  --user_name TEXT  the name of the user  [required]
  --help            Show this message and exit.

  For more information about frecklecute and the freckles project, please
  visit: https://github.com/makkus/freckles

# mkpasswd -m sha-512 hello_password
$6$h5OgOaSfa$rwIgBF1Ds/YKx9200agirpmdjG/8D5ThsM3AG9ozvlwci3DzZrBcqRA6LbOQMRAStQop0MWlDes5atB/E7BR6.

# frecklecute setup_sudo_user --user_name fancy_new_user --password '$6$h5OgOaSfa$rwIgBF1Ds/YKx9200agirpmdjG/8D5ThsM3AG9ozvlwci3DzZrBcqRA6LbOQMRAStQop0MWlDes5atB/E7BR6'

* starting tasks (on 'localhost')...
 * starting custom tasks:
     * group... ok (changed)
     * package... ok (no change)
     * lineinfile... ok (changed)
     * user... ok (changed)
   => ok (changed)

That was easy, right?

Why should I use that? When should I use that

This is not something that couldn’t be done with a bash script, or just plain Ansible. If you already have setup Ansible, it probably makes sense to just use that. If not, then this is an easy way to create scripts to manage local machines, still taking advantage of the power of Ansible, and the hundreds or thousands of existing modules and roles.

Compared to a bash script, I think this is quite well suited in cases where you want to manage state on a machine. Not so much (rather, not at all) when you have ‘actual’ work to do, like for example parsing a huge chunk of text files.

A declarative script like the example above is much easier to read (I think, anyway), and in most cases the script you are writing will be idempotent, which you might or might not appreciate. It is certainly handy not having to check states or files or installed applications and handle those cases differently depending on the result of the check, because this is already done in every one of your building blocks. I’m not saying it’s not possible to write idempotent bash scripts, but I’d argue it’s quite a bit more work than just re-using all those readymade Ansible modules and roles.

Plus, you know. All those readymade Ansible modules and roles. There is one for almost everything you can imagine…

I see, I see. How does that work then?

Let’s go through this script, and see how that works. A frecklecutable supports 5 key-names in the root of the document: doc, defaults, args, vars and tasks. Only the last one, tasks is required for a valid frecklecutable. To learn more about the keys not covered here, visit the frecklecutable documentation.

`doc`

The values under the doc key help make a nice commandline application out of the frecklecutable. As you can see in the example above, frecklecute setup_sudo_user --help prints out a nice help message with the text we assign to the help key, along usage hints for any potential arguments to the script.

`args`

This key gathers all user specify-able command-line arguments of the frecklecutable. Under the hood this uses the Click python package, so you can use most of the ‘click’-supported options for such an option or argument. More details can be found in the freckles documentation, but here’s a quick example of the kind of customization that’s possible:

become_root:
  help: whether to become root for this task or not
  arg_name: become
  required: false
  is_flag: true
  default: false
  is_var: false

This creates a variable with the name ‘become_root’, which is of type boolean (because it’s specified as a flag), which is not required and defaults to ‘false’ if not specified by the user (by providing the --become option). It also contains a short help text to tell the user what it means to set it.

The only ‘non-Click’ key in this example is the is_var one. This tells frecklecute to not add the resulting value to every task in the tasks list, but use it for templating purposes. More details on this can be found here. In our example, both user_name and password are used as templating variables.

`tasks`

This is the main section of a frecklecutable, and where things get done. This key contains a list of tasks to execute. Those tasks can be either Ansible modules, or roles. In contrast to Ansible playbooks, modules and roles are treated the same within a frecklecutable.

There are two ways to define a task item in the task list, a verbose, ‘exploded’ way which has yet to be documented and written about, and a concise, short way which I’ll describe here. The ‘exploded’ way is basically a dictionary with explicit metadata, and it can be necessary when using one of the more uncommon features or special cases. Both can be used interchangeably, as internally a short description is converted into the more explicit, verbose way.

Anyway, to add a task to the list, the most important thing is it’s name. A name can either be an alias you define (which we’ll ignore for the purpose of this blog post, but can be read about here), the name of a module, or the name of a role from Ansible Galaxy. frecklecute will know which is which, because roles always contain a ‘ . ’ in their name, whereas modules never do.

Let’s have a look at the example from above:

tasks:
  - group:
      name: wheel
      state: present
  - package:
      name: sudo
      state: present
  - lineinfile:
      dest: /etc/sudoers
      state: present
      regexp: "^%wheel"
      line: "%wheel ALL=(ALL) NOPASSWD: ALL"
      validate: "/usr/sbin/visudo -cf %s"
  - user:
      name: "{{:: user_name ::}}"
      password: "{{:: password ::}}"
      update_password: always
      groups: wheel
      append: yes

Those are all Ansible modules. The tasks are all specified as ‘single-key’ dictionaries, with the single key being the name of the module (or role), and the value of the key being a dictionary using any of the supported keys of a module: group, package, lineinfile, and user. This is a slightly unusual way to use the yaml-syntax, but I found it to be the easiest to read (and write, for that matter) – something that was fairly important to me.

As frecklecute supports basic templating (using Jinja2, similar to Ansible itself), any values under tasks can contain templating markers ({{:: and ::}}, we don’t use the ‘normal’ Jinja2 markers, because we want to be able to ‘forward’ those to Ansible and use vars like {{ ansible_env.USER }}).

So, how would we use Ansible roles with this then? In case you’ve never heard of Ansible roles: those are basically collections of tasks, which may or may not support different platforms to do one particular thing. For example, setting up the nginx webserver, or ldap authentication. Similar to what we do in our example here, there are also roles that can do some basic security- and user-management.

Let’s have a look at the geerlingguy.security role from here. This can do the ‘sudo’ setup, as well as some other basic stuff that is considered ‘good practice’, like setting up fail2ban and unattended security updates. The following is how you’d use it to setup a passwordless sudo user with it. You’d create a playbook that contains:

- hosts: localhost
  vars_files:
    - vars/main.yml
  roles:
    - geerlingguy.security

And put this in any of the places where you add variables:

security_sudoers_passwordless:
  - johndoe

This role supports other basic security tasks, like for example controlling ssh server behaviour, like allowing or disallowing password authentication. Do do that, as well as prevent root ssh logins, you could add:

security_ssh_password_authentication: "no"
security_ssh_permit_root_login: "no"

Translating this into a (minimal) *frecklecutable would look like so:

args:
  user_name:
     help: the name of the user
     is_var: false
     required: yes
  password:
     help: the user password hash (generate with 'mkpasswd -m sha-512')
     is_var: false
     required: yes
tasks:
  - user:
      name: "{{:: user_name ::}}"
      password: "{{:: password ::}}"
      update_password: always
  - geerlingguy.security:
      security_sudoers_passwordless:
        - "{{:: user_name ::}}"
      security_ssh_password_authentication: "no"
      security_ssh_permit_root_login: "no"

We still need to setup the user manually, as the role only adds the username to the /etc/sudoers file. This frecklecutable will create the user, download the geerlingguy.security role from Ansible Galaxy, then execute the role using the variables we provided:

$ frecklecute /frecklets/frecklecutables/setup_sudo_user_role --user_name fancy_new_user --password test

Downloading external roles...
  - downloading role 'security', owned by geerlingguy
  - downloading role from https://github.com/geerlingguy/ansible-role-security/archive/1.5.0.tar.gz
  - extracting geerlingguy.security to /home/vagrant/.cache/ansible-roles/geerlingguy.security
  - geerlingguy.security (1.5.0) was installed successfully

* starting tasks (on 'localhost')...
 * starting custom tasks:
     * USER... ok (changed)
   => ok (changed)
 * applying role geerlingguy.security'......
   - Include OS-specific variables. => ok (no change)
   - Install fail2ban. => ok (changed)
   - Ensure fail2ban is running and enabled on boot. => ok (no change)
   - Update SSH configuration to be more secure. =>
       - {u'regexp': u'^PasswordAuthentication', u'line': u'PasswordAuthentication no'} => ok (no change)
       - {u'regexp': u'^PermitRootLogin', u'line': u'PermitRootLogin no'} => ok (changed)
       - {u'regexp': u'^Port', u'line': u'Port 22'} => ok (changed)
       - {u'regexp': u'^UseDNS', u'line': u'UseDNS no'} => ok (no change)
       - {u'regexp': u'^PermitEmptyPasswords', u'line': u'PermitEmptyPasswords no'} => ok (changed)
       - {u'regexp': u'^ChallengeResponseAuthentication', u'line': u'ChallengeResponseAuthentication no'} => ok (no change)
       - {u'regexp': u'^GSSAPIAuthentication', u'line': u'GSSAPIAuthentication no'} => ok (changed)
       - {u'regexp': u'^X11Forwarding', u'line': u'X11Forwarding no'} => ok (changed)
   - Add configured user accounts to passwordless sudoers. =>
       - fancy_new_user => ok (changed)
   - Install unattended upgrades package. => ok (changed)
   - Copy unattended-upgrades configuration files in place. =>
       - 10periodic => ok (changed)
       - 50unattended-upgrades => ok (changed)
   => ok (changed)

Being able to do all this, combined with the multitude of available roles on Ansible Galaxy can be quite powerful, and potentially save quite a bit of time in the day of a typical developer or systems/devops person.

Conventions

One of the main goals for frecklecute was to have an easy-to-read scripting language, using Ansible modules and roles as building blocks. To achieve that, there are some conventions that need to be understood, at least once your scripts get more complex and use the more advanced features.

Those conventions obviously need to be documented. Unfortunately this is still work-in-progress, and not finished yet. The most important of those relates to executing tasks with ‘root’ or ‘sudo’ permissions, which is something I’ve skipped over so far because we just used the ‘root’ user to execute the examples.

In ‘real life’ we wouldn’t do that, but use a ‘normal’ user account which has ‘sudo’ permissions already. Without those we wouldn’t be able to install packages using the system manager, or add new users and groups.

Ansible deals with this by providing a become keyword, which, if set to true means that Ansible will execute the task (or role) in question using ‘root’ permissions. This goes along with the --ask-become-pass command-line flag in the ansible-playbook application.

frecklecute does support that flag as well, and you can also tell it to execute some tasks using elevated permissions. If that is necessary, one can either use the ‘exploded’ form of task description I was referring to earlier, or, in the case of the short-form you specify the task name all uppercase. In our example, this would look like:

tasks:
  - USER:
      name: "{{:: user_name ::}}"
      password: "{{:: password ::}}"
      update_password: always
  - GEERLINGGUY.SECURITY:
      security_sudoers_passwordless:
        - "{{:: user_name ::}}"
      security_ssh_password_authentication: "no"
      security_ssh_permit_root_login: "no"

Obviously, this won’t work for roles that have uppercase characters in their name. In those cases we’d have to use the ‘exploded’ form of configuration. Fortunately, this doesn’t happen very often.

Extras

As a member of the freckles family, there are a few extra features frecklecute has that mesh quite nicely with being able to quickly write those state-altering scripts:

transparent bootstrap

freckles can use inaugurate for bootstrap, which means all you need to have installed to execute a frecklecutable is either curl or wget:

$ curl https://freckles.io | bash -s -- frecklecute setup_sudo_user --user_name fancy_new_user --password <xxxxx>

remote script locations

You can keep a repository of your frecklecutables somewhere online (e.g. your Github account), and let frecklecute execute it from that remote place:

$ frecklecute gh:makkus/freckles/examples/setup_sudo_user --user_name fancy_new_user --password <xxxxxx>

self-hosted execution context

This is related to the previous point – and the details are a topic for another post – but in addition to executing remote frecklecutables you can also self-host all the roles you need for a frecklecuteable run. That means that, except for the freckles package itself (which you get from pypi, all dependencies for a run can be hosted somewhere remotely by yourself, and as such re-used from any type of newly setup machine that has internet access). So, executing a frecklecutable which is hosted somewhere on your Github account (alongside the Ansible roles it uses) on a newly installed machine would look something like:

$ curl https://freckles.io | bash -s -- frecklecute -r gh:makkus/frecklets setup_sudo_user --user_name fancy_new_user --password <xxxxxxxxxxxx>

using a frecklecutable directly

Since frecklecute acts like a kind of interpreter for a frecklecutable, you can use every frecklecutable directly, like you do with bash scripts for example. You need to add a shebang line to the beginning of the yaml file:

#! /usr/bin/env frecklecute

tasks:
  ...
  ...

Then make it executable and put it into your path:

$ chmod +x setup_sudo_user

$ mv setup_sudo_user /usr/local/bin    # assuming /usr/loca/bin is in your $PATH

$ setup_sudo_user --help

Usage: frecklecute /home/vagrant/.local/bin/setup_sudo_user
           [OPTIONS]

  Setups a new user with (passwordless) sudo privileges.

  Installs the 'sudo' package if necessary, and creates a group 'wheel'
  which will be allowed passwordless sudo-access.

Options:
  --password TEXT   the user password hash (generate with 'mkpasswd -m
                    sha-512')  [required]
  --user_name TEXT  the name of the user  [required]
  --help            Show this message and exit.

  For more information about frecklecute and the freckles project, please
  visit: https://github.com/makkus/freckles

That is all. For now.

How to manage my dotfiles with 'freckles'

Tue, 24 Oct 2017 15:00:00 +0200

Note: This was written for a previous version of freckles. The code/examples in here won’t work anymore.

Welcome to the last part of this three-part series about managing dotfiles with freckles, this is where it becomes interesting. The first part explained what dotfiles are, why one might want to manage them, and how to do that by hand. The second one explained how you can do the same thing using freckles, which is a configuration management tool I built on top of Ansible. This last installment will describe how I structured my own dotfiles, and how I use freckles to get them onto new environments.

[TOC] Maybe I should start off with a few key-points I want to be able to do with my dotfiles-setup:

I want to be able to easily setup a new machine that doesn’t have anything installed yet, with all my dotfiles and the applications I commonly use
I want to be able to draw my dotfiles from multiple sources, not just one git repository
I want to be able to choose which of my dotfiles are relevant for a certain environment (laptop, VM, container, remote server, etc.)
I want my dotfiles to contain additional metadata describing other (relevant) details about the environment they live in (like additional packages to install, ensure certain folders exist, etc.)
I want to be able to use alternative package managers, like for example ‘nix’ to install some packages
I want to be able to do all this on different platforms (at least Debian- and RedHat-based distros as well as Mac OS X, using mostly the same configuration, if possible)

My dotfiles

Here’s where I keep them: https://github.com/makkus/dotfiles

There’s a few things I do differently than what I’ve seen from most other people’s dotfiles:

Usage ‘profiles’

I sort my dotfiles into different folders by ‘usage profile’:

terminal: for terminal/console applications
x: for X/gui applications
sec: misc applications where config files contain sorta semi-private information

Within the ‘terminal’ and ‘x’ profiles, I have ‘sub-profiles’:

minimal: applications used for a minimal environment, which I want to have available everywhere I do any sort of non-trivial work on (.bashrc, git config, ‘zile’ - a small emacs-like editor I often use for small tasks)
extra: applications I use often-ish, but only I’m on my own machines (laptops, private server)
dev: applications I use when I do development
i3-desktop: (only in ‘x’) my desktop environment setup, which I use on my 2 laptops, and sometimes in a VM

I like to be able to mix-and-match those. E.g. if I setup a new laptop or workstation, I’ll want all of those. If I login to a remote server I expect to work more than a minimal amount of time on, I want to use the ‘terminal’ profile, but only the ‘minimal’ and ‘extra’ (and, depending, maybe the ‘dev’) sub-profiles. If I prepare a Docker container, I only want the ‘minimal’ sub-profile of ‘terminal’ (which I’ll delete once the container is finished to save space). On a new (graphical) VM I might only want the ‘minimal’ sub-profile of both ‘terminal’ and ‘x’.

Applications

I don’t want to have to install the applications my configurations refer to manually. If there is a configuration file for git, I want git to be available on that machine.

Additional applications

Even though it’s important to install the applications I have configurations for, most applications I use day-to-day either don’t need configuration, or I’m happy with the defaults so I never bother to configure anything in the first place.

So, depending on the usage profile of an environment, I need additonal packages to be installed. htop would be one example I use almost everywhere. Or tree, which is often not installed by default.

Alternative package managers

Setting up my workstation would be easy if all I’d install are system packages. Alas, for me, that’s not really a realistic scenario. Often I need more recent versions of a package than is available on Debian or Ubuntu, or it’s not available at all. One needs to add extra repositories, PPAs, whatnot. Some packages are not even packaged at all but need to be installed from scratch.

Those are the additional sources I get my applications from:

nix

For most of the user-facing applications I use, I prefer to use versions installed from the nix packages collection, using the nix package manager. It’d go to far to explain why I prefer to do this, but the two main reasons are that I (mostly) get the latest (or at least reasonable fresh) versions of a package and don’t have to wait for my distribution of choice to catch up, and that I get the exact same version and build of an application on different platforms. It’s also possible to install nix on systems where you don’t have root access, but that’s a bit more involved, and I haven’t had to use that for a while now.

conda

When working on Python projects, I prefer to use conda. Again, the reasons for this choice are not really the topic of this blog post, so I’ll just say that I like how it’s possible to use most ‘system’ packages that are (still) required as dependencies for certain Python packages (e.g. cryptography, pycrypt), without having to use the system package manager to install them. Everything is neatly contained in a confa environment.

git

Some applications or plugins need to be installed via git. One example would be spacemacs, which is installed by checking out it’s source code into the $HOME/.emacs.d directory. Or, zplug, a plugin manager for zsh that also is installed by checking out it’s source into the home directory.

pip

I also use a few (mostly command-line) applications directly from pypi, installed in a special (conda) virtualenv, symlinked into my $PATH.

‘stray’ packages

Some applications I use (like for example the AirVPN client ‘eddie‘, or Vagrant) only offer deb or rpm packages, but no repository or similar to install the package from. I don’t really like having to download and install those packages manually every time that becomes necessary.

Additional repositories

A lot of applications nowadays come with their own package repository (e.g. PPA in Ubuntu) which contains the application and it’s dependencies as sytem-packages. In some cases this is the only way to get a package on a machine, without having to compile from source.

others

There are more sources I get packages from. For example, I use some Vagrant plugins, so Vagrant itself can be considered a plugin manager.

Also, even though I don’t use it myself, but people get packages via npm, ruby gems, cabal, etc. Ideally, I’d like to be able to easily install packages from any of those sources as well (including ‘transparent’ installation of the package manager involved).

‘application-less’ config files

With ‘application-less’ I mean config (or similar) files that need to be ‘stowed’ (put into place), but don’t need any packages installed. One example would be additional fonts I use in my desktop environment, which need to sit in the $HOME/.fonts folder. I have those in an extra git repository because I don’t want to have to checkout the added ‘bulk’ of too many fonts everywhere I go and where they are not needed (e.g. terminal environments).

Encrypted config files

I like to encrypt some of my semi-private config files before uploading them to github. One example would be email configuration, where I don’t want the world to see my account details. I’d never upload any configuration file containing a password, even encrypted, but some files I consider private enough so I prefer the world not to see them, but not important enough to worry too much if that would happen by accident.

I found git-crypt to be an adequate solution for this use-case. It uses my gpg key to encrypt certain files before upload to the git repository, and you can do a git-crypt unlock on first checkout (provided your gpg-key is in place).

Additional provisioning tasks

Checking out dotfiles, sym-linking them into the home directory, and installing applications is fun, but usually there are always additional tasks to be done before one can use a newly provisioned machine. Those are often not too many, so they can be done by hand. But you’ll have to remember what exactly you did the last time you setup your machine, or you have to document it. I haven’t been too good at either of those tasks in the past. Anyway, for my environment, those tasks include:

creating certain folders some applications expect but don’t create themselves
copy my i3 X xsession desktop file to /usr/share/xsessions so the login manager picks it up
copy the touchscreen configuration for my external monitor to /etc/X11/xorg.conf.d
import my public gpg key stub (I’m using a Yubikey to hold the actual key)
set the trust of my gpg key to ‘ultimate’

Ideally, I only want to execute the tasks that are relevant to the particular profile I selected for the environment I’m on. For example, most of the time I don’t need to import my gpg key when I setup a VM.

Automating all this

If I only wanted to do all this every few months, to setup a new, or re-setup an old workstation, I wouldn’t mind doing it all manually. I really wanted to use my dotfiles in most of the environments I do serious work on, and at some stage I decided I wanted to be able to use a framework that’d be able to setup working environments in an automated and flexible fashion. Hence freckles.

Bootstrapping (with `inaugurate`)

One of the main drivers behind writing freckles was me getting annoyed always having to setup requirements and dependencies before being able to run my bootstrap script which then would checkout my dotfiles, and install applications. I really wanted to do all this with only one command, and I wanted that command to work on all the machines and platforms I work on. Basically, what I needed was a bootstrap script for my bootstrap script. This is what inaugurate is. I’ve written about this and how it works in probably every blog post and readme about freckles I’ve written so far, but just for good measure, in case you haven’t read any of those (yet):

inaugurate can install an application and it’s requirements as well as execute it in one go. The only requirement for the machine it is run on is either wget or curl. It can be executed with or without sudo permissions, and depending which one it is it uses different ways to install the application (either using system packages, or conda). This is done by directly executing the bash script wget or curl downloads. If that concerns you, read this, this, and this.

The first time you need an ‘inauguratable’ application, you prepend the command you want to execute with:
curl https://freckles.io | bash -s -- [command incl. arguments]
After this, and after you either logged out and logged in again, or you sourced your .profile (source $HOME/.profile) to get freckles into your $PATH, you can use the inaugurated application directly. So, this is the one command I have to execute to get a new workstation setup with all my config files and applications, and extra bits and pieces:

curl https://freckles.io | bash -s -- freckelize dotfiles -f gh:makkus/dotfiles

Check out the previous post in this series to understand what that does.

Data-centric environment management using `freckelize`

This is a topic that deserves it’s own blog post. Until that is written: I think it makes sense, if at all possible, to keep metadata describing environment requirements with the data that is supposed to live in that environment. For the case of dotfiles, this means I think dotfile repositories should ideally contain, apart from the dotfiles itself, metadata about the applications that are required by the dotfiles, as well as other requirements (e.g. folders that need to exists, package managers that are used to install the required applications, details about how the config files itself should be setup, etc.).

The freckles project provides three command line interfaces. The one that is interesting in this case is called freckelize, and it’s written specifically to support and encourage data-driven environment management.

In the following I’ll describe how to use freckelize` to manage dotfiles and support the requirements I listed above.

Usage ‘profiles’

freckelize works on folders containing structured data. A certain, pre-defined data structure is called a ‘profile’. For each such supported profile, freckelize provides a so-called ‘adapter’, which connects the structured data with an environment by executing tasks to prepare the environment to host that type of data. The adapter we are interested in is (not surprisingly) called dotfiles.

I call a folder that contains structured data – where the type of data is supported by freckelize – a ‘freckle’. That’s a bit silly, I know, but I found it makes it easier to talk about all this.

Interlude 1: a ‘freckle’ folder, without any metadata

By default, if no extra metadata is added to a folder, freckelize considers the git repository it checks out to be the freckle. So, if I use this example dotfile repository with freckelize, the command would look like:

freckelize dotfiles -f gh:makkus/dotfiles-test-simple-2

freckelize checks out the repository to $HOME/freckles/dotfiles-test-simple-2, and as neither this folder nor any of it’s sub-folders contains a freckelize metadata file (which would be named .freckle) it will use the root of that local folder itself as the freckle.

Interlude 2: one (or several) ‘freckle’ folders, with metadata

If freckelize finds one or several files called .freckle within the checked out git repository, it assumes those are the one(s) it is supposed to operate on, and it’ll not use the root of the git repository except if that also contains a .freckle file. This is done so that in the majority of cases freckelize does the right thing by default: if no .freckle file exists, the user doesn’t even have to be aware of any conventions. If the user wants to do something out of the ordinary, they have to learn about the more advanced features of freckelize anyway.

So, if a folder contains one or more files named .freckle, freckelize will operate on all of those, using the parent folder of the .freckle file as the freckle folder. Those .freckle files can be empty, or contain additional metadata which freckelize will forward to the appropriate adapter.

My dotfiles ‘freckle’ folders

As I want to be able to use my dotfiles in several different usage scenarios, I have split up my dotfiles into 8 parts (as I’ve described above) by placing .freckle files in the roots of the ‘profile’ folders:

$ cd ~/dotfiles

$ find|grep /.freckle$

./sec/.freckle
./terminal/dev/.freckle
./terminal/extra/.freckle
./terminal/minimal/.freckle
./x/dev/.freckle
./x/extra/.freckle
./x/i3-desktop/.freckle
./x/minimal/.freckle

freckelize let’s you limit which freckle folders it uses with the --include and --exclude flags. Both of those can be used multiple times, and both of them look at the provided string, and in- or exclude freckle folders which full, absolute paths end with the provided string. So, for example, if I use freckelize dotfiles --include terminal/minimal -f gh:makkus/dotfiles, it’ll use one freckle folder:

dotfiles/terminal/minimal

If I use --include minimal instead, I’ll have two matches:

dotfiles/terminal/minimal
dotfiles/x/minimal

Or, if I specify --include terminal/minimal --include terminal/extra, I’ll get:

dotfiles/terminal/minimal
dotfiles/terminal/extra

And so on, you get the idea. This lets me mix and match among all my usage profiles, and gives me fine-grained control which applications and configurations I want on a certain machine.

One thing I should probably mention here is related to the usage of stow: stow often complains when you link into the same directory from different source ‘base’ paths. You can get it to work well in such cases by adding an (empty) file called .stow in the base of a ‘source’ folder. Read more about this here.

Applications

How to install applications that are related to the dotfiles we use is the topic of the previous blog post in this series, so just head over there for details.

Additional applications and alternative package managers

This is a huge can of worms, and although it sort of works now within freckles for my needs, there is still quite a bit of work to be done to make the implementation cleaner, and to support more than the few package managers I implemented support for so far. If you want to help out and contribute support for a package manager (nodejs anyone?), let me know and I’ll explain the (few) things that need to be put in place.

As I’ve mentioned, I use nix for most of my user-space applications. Not for applications in my ‘minimal’ profile though, because I don’t want the additional overhead of having to install nix for a few terminal appplications.

frecklecute can (in most cases) figure out which package managers it needs to install automatically, so there is no need to provide a directive to instruct it to do so.

As we’ve learned, frecklecute takes notice of files named .freckle. It uses them as marker files, to figure out which folders to process. But it also uses them to read additional user-provided metadata that is related to the ‘freckle’ folder and the adapter being used. To illustrate, this is how my .freckle file for the terminal/extra usage profile looks like:

- dotfiles:
    pkg_mgr: auto
    packages:
      - direnv
      - gawk
      - pypy
      - pypy-dev
      - python-dev
      - python3-dev

- dotfiles:
    pkg_mgr: nix
    packages:
      - di
      - emacs
      - fasd-unstable
      - git-crypt
      - htop
      - imagemagick
      - mu
      - pandoc
      - password-store
      - silver-searcher
      - ranger
      - ripgrep
      - trash-cli

This instructs frecklecute to use the default package manager on a system (‘auto’) to install the first set of packages (the one starting with ‘direnv’), and nix to install the other set. How to write those configuration files is a topic for another blog post, but it should be easy enough to see how it works in this case. When kicked off, frecklecute will parse all the freckle folder it encounters, then it’ll make a list of all packages to install and which package managers to use. If it comes across a package manager that isn’t installed on the machine yet, it’ll install it before attempting to install anything.

Another example is the .freckle file in my terminal/minimal profile:

- dotfiles:
    pkg_mgr: auto
    packages:
      - tree
      - zplug:
         pkg_mgr: git
         dest: ~/.zplug
         repo: https://github.com/zplug/zplug

This combines two package managers under the same dotfiles key. By default, all packages will inherit the package manager that is specified ‘adapter-wide’. But that can be overwritten for every package. In this case, we use ‘git’ to install zplug, a zsh plugin manaager (according to it’s install instructions). Internally, frecklecute uses the Ansible git module, so we can use all options that are listed in this modules help page.

If we want to use a non-default package manager for one of the folder-based applications installs, this can be done by adding a metadata file called .package.freckle in the relevant dotfile folder, like I did in the one that holds my tmux config:

pkg_mgr: nix

‘application-less’ config files

In order to be able to have the configuration files that don’t relate directly to an applications (e.g. .fonts) in the same repository as all your others, but not use them as package names when looking for applications, you can tell frecklecute to omit a folder. To do that, all you need to do is create an (empty) marker file called .no_stow.freckle in the folder you don’t want to be used as application name. Like I did for my dotfiles/x/minimal/xkb folder which contains keyboard layouts:

 $ tree -a xkb
xkb
├── keymap
│   ├── filco
│   └── storm
├── .no_install.freckle
├── .no_stow.freckle
└── symbols
    ├── filco
    └── storm

On a side-note: notice the .no_install.freckle marker file. This does a similar thing than the .no_stow.freckle one, it prevents frecklecute from installing the (folder-name-based) application for that particular folder.

Encrypted config files

There’s not all that much to tell about this, except that I put most of my semi-private files in the sec usage profile and added their names to the .gitattributes file of the repository.

Using this setup is a bit of a nightmare, because I can’t use any of the files (particularly the gpg.conf one) before ‘unlocking’ the repository. Which is kinda hard to do automatically as it needs a key passphrase, or my Yubikey pressed. Since I haven’t really figured out how to best handle this, I still do this part manually. And I don’t ‘stow’ the `dotfiles/sec/gnupg folder automatically. I do, however, automatically import my gpg key and give it ‘ultimate’ trust. For how that is done, read the next section:

Additional provisioning tasks

Because there are a lot of things that could potentially be required to setup a certain environment, it’s impossible to add ‘non-generic’ functionality to the ‘dotfiles’ adapter to satisfy a large enough percentage of such requirements to make sense.

The ‘ansible-tasks’ adapter to the rescue! This adapter is written for those cases that need generic task execution. This is a quite powerful feature, but also a dangerous one, as this can execute arbitrary commands. Be careful when you use this, and don’t use the ‘ansible-tasks’ adapter without checking first what it does in every case.

The way this adapter works is that it looks for a file called .tasks.freckle in the root of the freckle folder. If it finds it, and if it can parse it as yaml, it’ll execute all the tasks that it contains. The tasks itself are described using the ‘tasks’-part of the Ansible playbook format. It supports all the available official Ansible modules as building blocks. A simple example is a list of a (single) task to ensure a few directories are present:

- name: 'creating folders in home dir'
  file:
    path: "{{ item }}"
    state: directory
  with_items:
    - "~/.backups/zile"
    - "~/.emacs.d/cache/layouts"

Or, a bit more involved, here’s how I import my gpg when initializing a new machine:

- name: ensure gnupg config dir has right permissions
  file:
    path: "~/.gnupg"
    mode: 0700
    state: directory

- name: importing gpg key stub
  command: "gpg --import {{ freckle_path }}/gnupg/.gnupg/gpg_public.key"
  args:
    creates: "~/.gnupg/trustdb.gpg"
  become: no

- name: setting gpg key trust to 'ultimate'
  shell: 'echo -e "trust\n5\ny\n" | gpg --command-fd 0 --edit-key m@ilmark.us'
  become: no

This task list can re-use a few existing variables, freckle_path being among them, and probably the most important one.

Technical sidenote: In contrast to freckelize adapters which can execute both Ansible modules and Ansible roles, this task list can only contain Ansible module-tasks (for now, anyway).

tldr;

So, to sum it all up:

If you want to use freckelize to manage your dotfiles, including installation of applications and execution of additional provisioning tasks, you have to:

prepare one or more dotfile repositories
split up your dotfiles into ‘usage profiles’ if you want to
prepare (an optional) .freckle metadata file that contains additional applications which don’t need/have configurations
prepare (an optional) .tasks.freckle for every dotfile repository or usage profile that contains additional tasks to be executed
log into the environment you want to add your dotfiles to
execute curl https://freckles.io | bash -s -- freckelize -f <dotfile_repo_url> dotfiles ansible-tasks
get a coffee, as this might take a while, depending on how many applications there are to install

And that’s it for dotfiles and freckles.

How to manage your dotfiles with 'freckles'

Tue, 24 Oct 2017 14:00:00 +0200

Note: This was written for a previous version of freckles. The code/examples in here won’t work anymore.

This is part two of a three-part series about managing dotfiles with freckles. The first part explained what dotfiles are, why one might want to manage them, and how to do that by hand. This one here will show you how to do the same thing using freckles, and the last one will show how to use freckles with a more involved setup (mine).

If you haven’t – or just can’t be bothered to – read my introductory post about freckles: freckles is a configuration management tool to help you setup your working environment with as little fuss and configuration as necessary.

freckles comes with three different command-line applications, all with slightly different goals. freckles itself, frecklecute, and, for our purpose the most relevant:

freckelize

freckelize’s main goal is to support a data-centric configuration management approach, in your working environment. It supports plugins, so-called ‘adapters’, which help prepare an environment for certain types of data. One such type of data is a dotfiles folder laid out in the way described below (the relevant adapter is named ‘dotfiles’, unsurprisingly).

checking out your dotfiles repo, and ‘stowing’ your config

To recap, in the previous post I described a simple folder structure for configuration files which can easily be used with stow:

dotfiles
├── bash
│   ├── .bashrc
│   └── .profile
├── git
│   └── .gitconfig
├── i3
│   ├── .config
│   │   ├── i3
│   │   │   └── config
├── zile
│   └── .zile
└── zsh
    ├── .zprofile
    ├── .zshenv
    └── .zshrc

If you created your dotfiles repository similar to this, and uploaded it to github, you can already use freckles to initialize a new machine:

$ curl https://freckles.io | bash -s -- freckelize dotfiles -f gh:<your github username>/<your dotfiles repo name> --no-install

This is designed to be as easy to remember as such a curl/bash command can possibly be, and to do as much as makes sense:

it bootstraps the freckles python package from pypi including it’s dependencies, then runs freckelize (which comes with it)
it might or might not ask for your ‘sudo’ password, which the underlying Ansible application might need to install dependencies (like for example git)
it expands the gh:<xxx>/<xxx> url to a proper github one
it uses git to check out your dotfiles repository (to $HOME/freckles/<repo_name>)
it stows all the configuration files in the dotfile repositories sub-folders

How this bootstrap and freckelize work in detail is explained in the freckles documentation.

installing your applications

You might have noticed the --no-install flag at the end of the above command. This tells freckelize (more exactly, freckelizes ‘dotfiles’ adapter) to not execute the ‘install’ step, which it would do by default.

Most of the time, if you have have configuration for an app, you want that app to be installed. And most of the time that application’s package name is the same as the one you’ll have used as sub-folder name in your dotfiles repository (e.g. i3, zile, …). So, why not use that folder name as the metadata to make sure the application a set of configuration files is associated with is installed? This is what the freckelize dotfile adapter does by default. So let’s run the above command again, without --no-install, and without the curl part since freckles is already installed:

$ source ~/.profile     # in case you haven't logged out and logged in again, to pick up the PATH freckles is installed in
$ freckelize dotfiles -f gh:<your github username>/<your dotfiles repo name>

This will not only stow all your configuration files, but also install packages named after sub-folders in your dotfiles repository, using the system package manager.

installing (additional) applications that don’t have configurations

Now, most of the time you’ll want additional applications to be installed, ones that don’t have or need configuration files, or where you are just happy with the default config.

This is easy to do as well with freckelize, but we need to create an extra metadata file to be able to tell freckelize which applications to install. freckelize can do more than just install and manage dotfiles, which is a topic for other blog posts, but it has one file it always looks up: .freckle in the root of the repository you point it to. So, let’s add a few ‘non-configuration’ applications (using yaml syntax):

dotfiles:
  packages:
    - htop
    - tree

As we are using the freckelize dotfiles adapter, we’ll need to put details about it’s execution under the dotfiles key, otherwise it wouldn’t be picked up. The dotfiles adapter understands keys other than packages, which I’ll say more about below, and in the next blog post in this series. Or you can of course check out the dotfiles adapter documentation.

I’ve prepared an example repository that contains configuration for the fish shell, as well as the zile editor, and which uses the above .freckle file, here: https://github.com/makkus/dotfiles-test-simple. To apply that dotfile repository to your machine would look like:

$ freckelize dotfiles -f gh:makkus/dotfiles-test-simple

# using repo(s):

 - gh:makkus/dotfiles-test-simple
     -> remote: 'https://github.com/makkus/dotfiles-test-simple.git'
     -> local: '/home/vagrant/freckles/dotfiles-test-simple.git'

# starting ansible run...

* starting tasks (on 'localhost')...
 * starting to process freckle(s)...
   - checking out freckle(s) =>
       - https://github.com/makkus/dotfiles-test-simple.git => ok (changed)
   - starting adapter 'dotfiles' => ok (no change)
   - starting dotfile adapter execution => ok (no change)
   - install dotfile folders packages =>
       - zile (using: apt) => ok (changed)
       - fish (using: apt) => ok (changed)
   - install dotfiles packages-list packages =>
       - htop (using: apt) => ok (changed)
       - tree (using: apt) => ok (changed)
   - installing stow =>
       - stow (using: apt) => ok (changed)
   - stowing folders =>
       - zile => ok (changed)
       - fish => ok (changed)
   => ok (changed)

To check the created symlinks we can:

$ ls -lah ~
total 57M
drwxr-xr-x 8 vagrant vagrant 4.0K Oct 24 09:58 .
drwxr-xr-x 3 root    root    4.0K Jun 19 23:27 ..
drwx------ 3 vagrant vagrant 4.0K Oct 24 09:56 .ansible
-rw-r--r-- 1 vagrant vagrant  220 Jun 19 23:27 .bash_logout
-rw-r--r-- 1 vagrant vagrant 3.5K Jun 19 23:27 .bashrc
drwx------ 3 vagrant vagrant 4.0K Oct 24 09:55 .cache
lrwxrwxrwx 1 vagrant vagrant   42 Oct 24 09:58 .config -> freckles/dotfiles-test-simple/fish/.config
drwxr-xr-x 3 vagrant vagrant 4.0K Oct 24 09:58 freckles
drwxr-xr-x 5 vagrant root    4.0K Oct 24 09:57 .local
-rw-r--r-- 1 vagrant vagrant  809 Oct 24 09:56 .profile
drwx------ 2 vagrant vagrant 4.0K Oct 24 09:54 .ssh
-rw------- 1 vagrant vagrant   60 Oct 24 09:57 .Xauthority
lrwxrwxrwx 1 vagrant vagrant   40 Oct 24 09:58 .zile -> freckles/dotfiles-test-simple/zile/.zile

As I’ve mentioned before, freckelize can actually do a bit more, and it can be configured much more fine-grained, for example to install packages using other package-managers (git, conda, nix, …). You can specify different package names for an application. You can also run a set of additional tasks that create folders, import gpg keys, or do whatever else you might need done. I’ll write about all this in the last post of this series, using my own setup as an example.

Managing dotfiles

Tue, 24 Oct 2017 13:00:00 +0200

This is part one of a three-part series about managing dotfiles with freckles. This one covers the basics about what dotfiles are, why managing them might or might not make sense, and how to do it manually. The second one shows how to use a tool I wrote – freckles – to do the same thing with (arguably) less effort. The last post will cover how I manage my own dotfiles using freckles, as I’ve got a fairly involved setup which shows how to use all the more advanced options of freckles.

‘dotfiles’, and why you might want to manage them

‘dotfiles’ are configuration (text-) files, named so because on UNIX systems they usually start with a “ . “. If you are mostly using Windows, and/or GUI applications, this all might not really apply to you, as configurations are often stored in a non-text format, in a registry, or in other really weird places. If you use Linux or Mac OS X (or more recently, the Ubunty subsystem on Windows), and the command-line, you’ll have probably come across them though.

dotfiles usually live directly under your home directory or, quite often nowadays, in a folder called .config in your home directory. Sometimes applications have a single configuration file, sometimes they have a whole sub-folder of them, and sometimes they give you the choice, depending on how far you want to go customizing the apps behaviour.

As well as there not being a really standardized location where dotfiles can be found, there’s also no ‘one’ format for them. Some of them are ini-style key/value files, some of them are yaml, or json, or – god forbid – xml. Some of them are even python code or bash scripts. All depends on the application they belong to.

The more you use certain applications, the more you’ll find yourself working on it’s configuration to better support your way of working, or your use-cases. That also means that you spend a non-trivial amount of time on customizing your workstation experience, time you probably don’t want to loose if you loose your laptop, or even just the filesystem where the configuration is stored. Which is why a lot of people store their dotfiles in a version control system. Mostly git, since that is where most of their other work lives as well. This also gives you the advantage of tracking the history of the changes you made to them, so if something goes wrong, looking at the git history might give you some clue as to what changed.

different ways of managing your dotfiles

There are already several interesting tools out there to manage dotfiles, for example dotbot, homesick, rcm, or yadm. You can even use saltstack (which is seriously cool and which I’d probably have used if anyone had told me about it before I was almost finished writing freckles).

So, after you get an idea how to use freckles to manage dotfiles, you might want to check out some of those and see whether maybe they are a better fit for your setup. Workstation setups and configurations are very personal and individual things, which is probably why there hasn’t emerged a commonly used, ‘best-practice’ kind of tool or structure that almost everybody uses.

In addition to those alternative helper tools, there are usually two strategies (as far as I can tell) to use git to store your dotfiles:

In my opinion, both approaches have their advantages, so, again, I’d recommend reading up on both of them. For my workflow, as much as I like the idea, the ‘bare git repo’ approach is too inflexible to be of enough value, which is why I choose to use ‘stow’. Which is also the method freckles uses. I’ll cover the details of those more advanced use-cases in the third installment of this series, but if you have a fairly straight-forward setup, using a bare git repository might very well be the better solution.

dotfiles folder structure

As the ‘stow’ method symbolically links your configuration files into where they need to be, you have a certain degree of flexibility as to how you organize your dotfiles folder. For most use-cases, it’ll look something like this though:

dotfiles
├── bash
│   ├── .bashrc
│   └── .profile
├── git
│   └── .gitconfig
├── i3
│   ├── .config
│   │   ├── i3
│   │   │   └── config
├── zile
│   └── .zile
└── zsh
    ├── .zprofile
    ├── .zshenv
    └── .zshrc

This is probably the approach that gets recommended the most. It’s a neat and tidy structure as each application gets it’s own sub-folder. Within that sub-folder we re-create the folder structure the application expectes, and as it would look like from the root of the home directory. In this example, most configuration files live directly under $HOME (.bashrc, .zile, etc.), except for the i3 one, which has it’s configuration file (named config) in the folder $HOME/.config/i3.

In a setup like this, this is how you’d execute stow:

$ stow -d <dotfiles_dir> -t $HOME -S *

This will link the contents of each of the sub-directories into $HOME, using the relative paths to them we created earlier (if necessary):

$ ls -lah ~/.bashrc
lrwxrwxrwx 1 markus markus 38 Okt 22 16:38 /home/markus/.bashrc -> dotfiles/bash/.bashrc

stow is quite smart about how it does the linking, and how it treats existing (sub-)directories, check out its manual for more information.

setting up a newly installed machine

In order to manually configure a new machine using the configuration you created and uploaded to github, you’ll have to do something like this (let’s assume we are using an Ubuntu box):

$ sudo apt install git
...
$ git clone https://github.com/makkkus/dotfiles.git ~/dotfiles
...
$ sudo apt install stow
...
$ stow -d <dotfiles_dir> -t $HOME -S *
...
$ sudo apt install i3 zile zsh <all other applications you use>
...

You could probably do the app installs in one command, and git is most likely already installed. So, this doesn’t look too bad and could be scripted fairly easily, especially if you don’t have a lot of applications to install, and you always use the same operating system or distribution (as different OS’s or distributions have sometimes different package names for the same application).

So, as long as your setup is as simple as my example here, I’d recommend stop reading here, and just do it like this (or, as mentioned above, use a bare git repo instead).

But, if you have done that, and at some stage you find (as I did) that this eventually, with a growing amount of configuration folders and files and applications, gets a bit un-organized; or you’d like to just record new applications to install into a text file which gets read the next time you re-install a new machine; or you use more than one platform, and the difference between those always screws with your script, or makes it more complex than you like it to be; or you only want to use a sub-set of your application on certain targets; or you just think (like I do), that having to enter more than one line or write a script for something so trivial is annoying; – if any of those things apply to you, check out the next two installments of this series here and here.

So, I made this ..thing

Wed, 18 Oct 2017 14:00:00 +0200

Note: This was written for a previous version of freckles. The code/examples in here won’t work anymore.

It was one of those ‘scratch-your-own-itch’-situations, I suppose. Only, I really didn’t want to scratch that particular itch myself.

For one, because I knew exactly how long it’d take me to write this if I wanted to do it properly. And I really thought it is something so obvious that sooner or later somebody will build (something like) it, and all I have to do is go on Reddit, moan a bit about how it’s bloated, and (obviously) written in the wrong language. But I’ll use it anyway, you can thank me later.

It looks like I was wrong though, and it’s either not obvious at all, or other people were playing the same waiting game as me. Or, of course, it is just a really stupid idea. Not yet counting that one out either. Whatever the reason, I waited for a few years, and nobody scratched my itch. So, eventually I gave in and wrote that …thing. Named the thing ‘freckles’. Uploaded it to Github: https://github.com/makkus/freckles

Although it’s not quite finished yet, I consider it usable now.

What am I talking about? What is this thing?? Glad you asked! I’m not 100% happy with that description, but basically:

The thing: (quick and easy) ‘local’ configuration management

I’m not going to write about what exactly made me create freckles (might do that in a later blog post if anybody is interested) and how it works in detail (there’s https://docs.freckles.io for that). I should, however, probably explain what configuration management is, for those of you who don’t know.

Yes, what is this configuration management you speak of?

If you are familiar with Ansible, puppet, chef, or salt, you know about configuration management, and why (for the most part) it is a good idea. If not: in short, configuration management gives you a way to describe a machine/server and the services and applications it runs. Either in code, or a configuration format like json or yaml. Then it takes that configuration and applies it to a machine, removing the need for you to setup the machine manually, as well as guaranteeing that the machine is always setup the same way, even (or especially) after a re-install. As a bonus, often times you can use the configuration itself as a sort of always-up-to-date documentation on how your infrastructure is set up. Much more organized and parse-able than, say, a bash script.

Because of the overhead that comes with configuration management frameworks, using them is usually restricted to situations where the infrastructure to be controlled is deemed to cross a certain threshold of… let’s call it ‘importance’. While for production services or other business-relevant systems this threshold is often crossed even for single servers or services, this is not usually the case for the physical (or virtual) machines we developers (or somesuch) use when going about whatever we go about. There are exceptions of course, but spending the time to learn about, and then setting up a system like that is not always worth it.

Cue, ‘freckles’

freckles tries to change that effort/value equation by making it easier –and faster – to practice configuration management, at least in local development environments. I do think there’s a lot of developers time to be saved, to be used on actual development, rather than all the annoying stuff around it (like, for example, setting up and configuring webservers). Plus, a bit of good practice never hurt anybody, right?

freckles comes with (so far) three command-line interfaces (freckles itself, frecklecute and freckelize) which all do slightly different things. They all are designed to primarlily do cnfiguration management on single boxes. Physical or virtual ones, local or remote. Where you have ‘root’ access, or you don’t. Whichever distribution of Linux, or Mac OS X.

Basically any type of box you want to get into a certain state. Unlike other configuration management systems, freckles doesn’t need any infrastructure around it because everything runs on the box that needs the state change. That means, in turn, there is the overhead (mainly time and hard-disk space) of having to get it onto every one of those machines, instead of having a single ‘controller’ which distributes configuration changes across the network. So, it’s not a good fit for medium- or large-sized infrastructures.

Internally, freckles uses Ansible to apply state changes to the machine it is working on. Ansible is a comparably light-weight and easy to use configuration management framework, and it comes with hundreds of so-called modules and roles, which are easy-to use, pre-made building blocks to change state on a machine. And which can all be used with freckles.

There are a few areas where freckles does things differently than Ansible (and also other configuration management systems), or focuses on slightly different things:

(Transparent) bootstrap

Although (or because) you have to get freckles on every machine you want to get configured, I made it quite easy to do so, using the inaugurate bootstrap script (which is a spin-off of the freckles project itself). Running bash scripts directly from the internet is a slightly controversial topic, and I don’t really want to discuss this here. I’ve written about what that means, why it’s not the real problem, and how to make it work in a secure way for you in a few other places though: here, here, and here.

Basically, the first time you execute one of freckles command-line interfaces, you do it via curl (or wget), like:

curl https://freckles.io | bash -s -- freckles <freckles_arguments>

This will install freckles in your home directory (more details here), and also executes it for it’s first run. It’ll add itself to your PATH in $HOME/.profile, so once you either sourced that (source ~/.profile), or logged out of your current session and logged in again you can use it directly (using the same command you’d have used after the ... | bash -s -- in the example above).

Depending on the situation you might only need to execute freckles (or it’s companion interfaces freckelize or frecklecute) once. If that’s the case, you can delete the folder it was installed into straight away after execution, by setting an environment variable (e.g. to save some space in a Docker container):

curl https://freckles.io | SELF_DESTRUCT=true bash -s -- freckles <freckles_arguments>

One-line execution

Related to bootstrapping, freckles tries to let you execute a configuration run for a machine in one command. Before that works, you might have to prepare roles and/or execution scripts, and host those somewhere online, or you can re-use pre-made, shared ones (in which case you’ll have to be mindful that this could be a security issue). But once that is done, you can apply the same configuration on different machines, platforms, distribution versions, with said one-line execution, always using the same line. Provided, of course, you made sure to support all those different platforms etc in your roles. Configure once, run everywhere, sorta.

For example, in order to get all my dotfiles (‘dotfiles’ are configuration files) installed on a new machine, as well as all applications installed that are referenced in them, all I have to execute is:

curl https://freckles.io | bash -s -- freckelize dotfiles -f gh:makkus/dotfiles

# or, if freckles is alrady installed

freckelize dotfiles -f gh:makkus/dotfiles

(on a sidenote: gh:makkus/dotfiles is an abbreviation that freckles automatically expands to https://github.com/makkus/dotfiles.git)

I’ll talk a bit more about freckelize and dotfiles below.

Another example would be to setup a machine that runs a webserver to host/redirect readthedocs.io documentation via your own domain and https, as explained in the readthedocs documentation (weird example, I know – but I had to do that recently for obvious reasons):

curl https://freckles.io | bash -s -- frecklecute gh:makkus/freckles/examples/readthedocsforwarding.yml

# or, if freckles is alrady installed, and the script is available locally

frecklecute /home/markus/frecklecutables/readthedocsforwarding.yml

For the content of the readthedocsforwarding.yml file, read the next part:

One-file configuration

One of the slight annoyances I feel when using Ansible to run a few tasks and/or roles on my local machine is that I always have to touch quite a few files. There’s the inventory, then vars files, and the playbook file itself. And you might or might not have to download roles you use manually. For simple cases, I always wanted to be able to describe everything in just one file, kinda like a Dockerfile, but (much) more powerful. Or like a bash script, but more readable, and less time-consuming to create.

frecklecute, which comes as part of freckles can do just that. As you’ve seen in the above example, it can use either local or remote (yaml) files (which I call frecklecutables) to execute some tasks. Details about how this works can be found here, but I’ll show you the content of the readthedocsforwarding.yml file to give you an idea:

tasks:

  - install: fail2ban

  - thefinn93.letsencrypt:
      letsencrypt_webroot_path: /var/www/html
      letsencrypt_email: makkus@frkl.io
      letsencrypt_cert_domains:
        - docs.freckles.io
      letsencrypt_renewal_command_args: '--renew-hook "systemctl restart nginx"'

  - geerlingguy.nginx:
      nginx_remove_default_vhost: true
      nginx_vhosts:
       - listen: "80"
         server_name: "docs.freckles.io"
         return: "301 https://docs.freckles.io$request_uri"
       - listen: "443 ssl http2"
         server_name: "docs.freckles.io"
         state: "present"
         extra_parameters: |
            location / {
               proxy_pass https://freckles.readthedocs.io:443;
               proxy_set_header Host $http_host;
               proxy_set_header X-Forwarded-Proto https;
               proxy_set_header X-Real-IP $remote_addr;
               proxy_set_header X-Scheme $scheme;
               proxy_set_header X-RTD-SLUG freckles;
               proxy_connect_timeout 10s;
               proxy_read_timeout 20s;
            }
            ssl_certificate      /etc/letsencrypt/live/docs.freckles.io/fullchain.pem;
            ssl_certificate_key  /etc/letsencrypt/live/docs.freckles.io/privkey.pem;
            ssl_protocols        TLSv1.1 TLSv1.2;
            ssl_ciphers          HIGH:!aNULL:!MD5;

If you have used Ansible before, this should look familiar to you. This script first downloads all the external roles it needs, after that it installs the fail2ban package (because that’s always a good idea for a server), then uses the thefinn93.letsencrypt ansible role to retrieve a certificate from the let’s encrypt CA (which will only work if executed on a machine matching the specified hostname, obviously). Once that it done it’ll create a cron job to always re-new that certificate in time, and install the nginx webserver (using this role) including the site configuration that sets up the forwarding to readthedocs.io.

All that can obviously also be done with Ansible itself. And it should probably be, in case of a larger infrastructure and, you know, ‘production’. But for a single server, and prototyping or development frecklecute might be an adequate solution. One thing to mention is that freckles does not support all features of Ansible directly (like for example the when directive). This is partly deliberate to keep those frecklecutables simple and readable. And partly due to time constraints on my part. Not having those features is not really a problem though, because I recommend to always write an Ansible role once the task to be executed becomes non-trivial. Then include that role in the frecklecutable as a task item. Much cleaner that way.

A nice thing about all this is that you can use frecklecute and this frecklecutable in either a Docker container build process, a Vagrant box (well, probably not that file since it needs an outside internet connection and a proper hostname), or a VPS on whichever VPS provider you use.

Data-centric environment management

While working on freckles I realized that in a lot of cases the metadata that is required to setup a working environment is already present in the structure or content of the data or code that is supposed to be used in that working environment (or can be very easily added to that environment in the form of a metadata file).

This is often obvious for programming projects, where build tools expect for example a file called setup.py (similar for other programming languages). But it can be used in a lot more cases. For example, the dotfiles I mentioned earlier: if you structure them in a way that the configuration files for an application are in a folder that is named after the package name of the application itself, then you can use that information to install the application while at the same time putting the configuration files in the locations they need to be (e.g. via symlinks).

Above I’ve shown you how I initialize a new machine with my dotfiles:

freckelize dotfiles -f gh:makkus/dotfiles

The dotfiles part is referencing a so-called freckelize adapter, which – in the dotfiles case – is shipped with freckles. Again, I’ll not go into detail how exactly this works (go here and here if you are interested). But, in short, such an adapter expects data of a certain shape, and executes steps to prepare a host machine to be able to host that particular data profile. In this example it checks out my configuration files and links them to all the right places and installs all applications I usually work with. On any (physical or virtual) machine I happen to need them.

Another example is this very webpage you are reading at the moment. In order to setup a development environment for it on my workstation, I execute:

freckelize vagrant-dev -f gh:makkus/freckles_website

This will checkout the source code of this site, setup Vagrant if not already installed, as well as other potential requirements which might be specified in that repository (e.g. Virtualbox, Vagrant plugins).

In the Vagrantfile of this project I again use freckelize to bootstrap an environemt that contains nginx, php, and the grav cms, in the Vagrant box to be created:

wget -O - https://freckles.io | sudo bash -s -- freckelize -r gh:makkus/frecklets grav -f /vagrant/ --port 8280 --nginx-user vagrant

In order for all that to work I had to prepare an Ansible role and a freckelize adapter to setup a machine to be able to host a grav webpage, and add them, as well as roles to setup nginx and php, to a git repository. Once that is done, I can re-use those for every grav website I’ll create in the future (and so can you, if you decide I and the other role creators involved are trustworthy enough). I’ll probably write another blog-post about how this works in details later.

As is the case with the readthedocs example above, that setup can easily be used in a lot of different situations or technologies (Docker, Vagrant, LXC, physical host…).

Scripting

In addition to all this, freckles can also be used to quickly write commandline scripts that use frecklecute as their sort of ‘interpreter’. Again, not going into any detail here, instead, check out this link, and this link

A quick example script to create a folder using the ‘file’ Ansible module, and user-input for the folder to create would be:

#! /usr/bin/env frecklecute
doc:
  help: create a folder
args:
  path:
    help: the folder path
    default: ~/cool_folder
tasks:
  - file:
      state: directory

Saved in a file called ‘folder-create’, chmod’ed to be executable, and either put in your PATH or executed directly would look like:

$ create-folder --help
Usage: frecklecute ./create-folder [OPTIONS]

  create a folder

Options:
  --path TEXT  the folder path
  --help       Show this message and exit.

  For more information about frecklecute and the freckles project, please
  visit: https://github.com/makkus/freckles

$ create-folder --path ~/now-that-is-a-folder-created-in-an-interesting-way

* starting tasks (on 'localhost')...
 * starting custom tasks:
     * file... ok (changed)
   => ok (changed)

Or upload it to github and execute it like so:

$ frecklecute gh:makkus/freckles/examples/create-folder --path ~/a-folder-created-from-a-remote-frecklecutable

* starting tasks (on 'localhost')...
 * starting custom tasks:
     * file... ok (changed)
   => ok (changed)

Obviously, this all is a tad overkill just to create a folder. But as I’ve mentioned before, this can be used with all the Ansible modules and roles available. Galaxy is the limit…

And, if you really want to go all out, you can even combine this with the online bootstraping of freckles which means neither freckles nor your frecklecutable need to be on a machine to be able to run it. Only curl or wget. And you don’t need any root permissions either, as long as the Ansible roles or modules you use don’t require it. I think that’s quite cool. And probably pretty dangerous too, in the hand of fools. Good thing there are hardly any fools in this world!

Direct Ansible role or module execution

And, for extra credit, and those of us who only want to quickly execute an Ansible role, on a machine without anything useful installed (again, except for curl or wget):

$ curl https://freckles.io | bash -s -- frecklecute ansible-task --become --task-name mongrelion.docker

Not going to explain what that does, as it should be obvious by now (hint: mongrelion.docker is an Ansible role that installs … well). You get the idea…

Use-cases

Here is a random list of other use cases freckles can be used for:

after installing a new (physical or virtual) machine, quickly install and configure it with the applications you commonly use, by letting freckelize download and process a remote (or local) dotfile/configuration repository (adapter documentation: here)
setup the source code of the projects you are working on, including their dependencies, on your machine (e.g. Python projects, or (generic) projects that use Vagrant)
quickly write a short script to install/update some of your own (non-packaged) applications (e.g. freckles uses frecklecute to update itself)
ensure you and your team-mates use the same setup of a certain development environment
quickly execute a ‘one-off’ Ansible task or role (e.g. to install and configure Docker, or nginx, etc.), without having to install Ansible itself manually (more info: here)
write easy to read and understand deployment scripts, which can also be used for documentation or education purposes (e.g. in a blog post), and which can be used in combination with inaugurate to create ‘no-requirement’ bootstrap scripts
create scriptlets that are easy to share and execute, to init new (development or other) projects from templates
…

Basically, most things you can imagine which change the state of your machine/filesystem from a relatively ‘useless’, to a more ‘useful’ state. The definition of ‘useless’ and ‘useful’ is up to you, of course.

Where to, from here?

Not sure, really.

The main selling point for freckles are the many Ansible roles and modules it can use (as well as, of course, Ansible itself). One idea I have is to create a repository of ‘curated’ roles, which I want to call the ‘Ark’ (Ansible Role somewordstartingwith-K-maybeKiosk), and which will contain only one role per thing to do or install, and one role only. Ansible Galaxy is great, but I find it a bit hard at times to find the best role for what I try to do, and the platform I try to do it on. I think, in a lot of cases it’d be better to work together on commonly agreed upon “main” role for a problem, and improve it, than to create a new role that is targeted only on a certain platform, or version of software. There is a lot of that happening already, just not in a very structured way. As far as I know, anyway.

Roles in that repository would have a maintainer, would have tests, would work for as many platforms as possible, and would be continually improved upon. Haven’t really had time to think this through, and create a list of requirements. Ideally, I wouldn’t have to do that on my own though, and other people who also see value in this and are keen to collaborate on it would join in.

As for freckles itself, I still have quite a few ideas about features and improvements, but I reckon I’ll wait and see whether and how much uptake it sees in the next few weeks/months. Then decide whether to spend more time on it and for example give it proper unit-testing, logging and more documentation. Or whether to write the thing I actually wanted to write when my annoyance of something like freckles not existing became stronger than my reluctance to write it myself. Or whether to actually start earning money again. Bah, stupid money…

About

Tue, 27 Jun 2017 17:39:21 -0700

frkl is my small software and consulting enterprise.

It’s an experiment. I’m trying to figure out whether it’s possible to work on the (slightly weird) software I want to exist in the world and also earn a living.

frkl - consolidating complexity

A (practical) ssh primer

So, what is this ssh you are talking about and why should I care?

Authentication

Username/password authentication

Private key authentication

Password protected private keys, or not?

The ssh-agent

'Dockerfiles' for LXD

tldr

Long-ish preamble

Using freckles with LXD

Optional: using freckles to install LXD

Describing the LXD-image contents

Building an LXD image

Re-using our frecklet

On licensing freckles

Reasons!!!

Idealism

Money

Open-source business models, and what I don’t like about them

Donations

Open core

Software/platform/whatever as a service

Corporate sponsorship

Tit for tat

Data-centric environment management

Writing declarative command-line scripts, using Ansible modules and roles

tldr; show me the goods

Why should I use that? When should I use that

I see, I see. How does that work then?

doc

args

tasks

Conventions

Extras

transparent bootstrap

remote script locations

self-hosted execution context

using a frecklecutable directly

How to manage *my* dotfiles with 'freckles'

My dotfiles

Usage ‘profiles’

Applications

Additional applications

Alternative package managers

nix

conda

git

pip

‘stray’ packages

Additional repositories

others

‘application-less’ config files

Encrypted config files

Additional provisioning tasks

Automating all this

Bootstrapping (with inaugurate)

Data-centric environment management using freckelize

Usage ‘profiles’

Interlude 1: a ‘freckle’ folder, without any metadata

Interlude 2: one (or several) ‘freckle’ folders, with metadata

My dotfiles ‘freckle’ folders

Applications

Additional applications and alternative package managers

‘application-less’ config files

Encrypted config files

Additional provisioning tasks

tldr;

How to manage your dotfiles with 'freckles'

freckelize

checking out your dotfiles repo, and ‘stowing’ your config

installing your applications

installing (additional) applications that don’t have configurations

Managing dotfiles

‘dotfiles’, and why you might want to manage them

different ways of managing your dotfiles

dotfiles folder structure

setting up a newly installed machine

So, I made this ..thing

The `ssh-agent`

`doc`

`args`

`tasks`

How to manage my dotfiles with 'freckles'

Bootstrapping (with `inaugurate`)

Data-centric environment management using `freckelize`