The Grumpy Troll: The Grumpy Troll

(Go)Hugo Generated Text File

Phil Pennock — Fri, 23 Feb 2024 21:30:00 -0500

I migrated this blog to Hugo in 2013. At the time, I wrote features for Hugo to support the migration. For a while, I knew every feature of the software. Those days are long past. Much of what I knew, I have forgotten.

I almost entirely just have post content, setup in a pattern which predates the themes and so forth. I have some static files, of course. But when it came to setting up a plain-text file outside of the blog area, but with the content being generated with Go text/template, I found it surprisingly hard to find clear explicit guidance on how to do this.

It’s comparatively simple, once you know how. And I belatedly realised that this makes it an appropriate topic for the blog. How meta!

You might use this for:

robots.txt
humans.txt
security.txt

and anything else which has to live at explicit paths.

Except I don’t use humans.txt; and security.txt needs to be PGP-signed so is static data where it exists; and … hrm, there must be something other than robots.txt, right? Anyway, moving swiftly on.

Motivation: `robots.txt`

The robots.txt file is perhaps the oldest fixed-format fixed-URL “standard” site pages of the World Wide Web. It far predates /.well-known/. There are multiple parsers for it, as every crawler, of every search engine, can implement their own rules. And the search engines change their rules over time.

In particular, if someone wants to crawl what you’ve written, and you have a way to tell them not to, then “little mistakes” or “oh we changed that” updates which result in them being able to claim with a straight face that you didn’t tell them properly? That’s a situation to be avoided.

Thus for robots.txt it pays to be proactively defensive and use the simplest possible served format. No assumptions of batching rules. No fanciness.

But the moment you start writing the same text over and over again, typos will creep in. I mean, my spieling is alwais purfect so it must be the auto-corruption of the virtual keyboard, right?

Hugo Textfile Setup

I only noticed when writing this blog-post, but Hugo actually has a pre-registered output format explicitly for robots.txt. But only for that one specific text file. I want this setup for any text file I might choose to generate. So I’m sticking with what I configured because I can use it with more files in future.

We need one configuration change to be able to serve arbitrary text files, and then two files inside the tree for each file served. For robots.txt and auto-generation, I also added a “data file” to read as a data source.

I use a YAML configuration file, so in my blog’s git repository’s root directory I have config.yaml. (edit/2025-01: Newer versions of Hugo encourage a migration to hugo.yaml instead).

In that, I have multiple outputFormats but only one of interest here:

contentdir: "content"
layoutdir: "layouts"
publishdir: "public"

outputFormats:
  txt:
    mediaType: text/plain
    isPlainText: true

Those first three directories are (or used to be?) standard, and will be covered below.

This outputFormats entry says: when the page metadata sets the outputs list to contain txt, we want to serve it as text/plain, and we want the Hugo engine to use the Golang text/template parser instead of the html/template parser.

Then to generate the target file, we need a stub markdown page and the logic in the layout page.

Because contentdir: is "content", my stub markdown page is: content/robots.md

---
title: Robots
url: "robots.txt"
type: "page"
layout: "robots"
outputs:
 - txt
---

I don’t think the title really matters here, but it’s standard and available to templates.
The url: says “where to put this inside the public/ directory (the publishdir: directory), or when using hugo serve it is more directly just the URL
The types are used for grouping pages together, eg blog or post and is usually the first directory inside content/; since we’re not inside a directory, I’m just manually assigning something generic and "page" is in fact what the default should be outside a directory. So this is making an implicit value explicit. You could skip it.
The layout: is important and is what ties into the next file, below.
The outputs: list says which output renderers to use.
- If we had multiple here, they would all need to exist, but since we define one URL which is not a directory, only one will be used.
- If url: did not have an extension (or ended in a /) then the URL would create a directory and files with basename index would be created, each with an extension corresponding to the outputs.
There is no “content” after the front-matter. This stub is pure front-matter.

Now we can write the template!

The stub says that the page robots.txt should use the robots layout. The config.yaml says layoutdir: "layouts". We are not doing anything special with additional rendering views on the same content (layouts) so we use the layout named _default. The output is txt so we use that as our extension.

So we create: layouts/_default/robots.txt
{{ layoutdir }}/_default/{{ chosen_layout_name }}.{{ output_extension}}

And in that file we can write Golang text/template directives to construct a repetitive file with the most unambiguous rendering we can come up with.

My `robots.txt` template and data

I want highly structured input. I’m already using YAML, as penance for my sins. Hugo lets you put YAML, JSON, TOML, or XML (for those with even more sins than I) in files in the data/ directory. The loaded and parsed content of these files can then be accessed in templates via the .Site.Data dictionary.

After reading some documents such as https://www.eff.org/deeplinks/2023/12/no-robotstxt-how-ask-chatgpt-and-google-bard-not-use-your-website-training and https://github.com/samber/the-great-gpt-firewall I had decided to go ahead and explicitly deny permission to AI systems from using my blog in future.

I really truly want to make sure that those companies training AI by ignoring copyright have no excuses, thus the simplified robots.txt.

So I created a file data/ai_bots.yaml :

- ua: GPTBot
  description: "OpenAI's web crawler: GPT3.5, GPT4, ChatGPT"
- ua: ChatGPT-User
  description: "ChatGPT plugins"
- ua: Google-Extended
  description: "Google's web crawler: Bard, VertexAI, Gemini"
- ua: anthropic-ai
  description: "Claude"
- ua: CCBot
  description: "Common Crawl"

Note that the description is for me and is not currently used, but conceivably could be in other uses of this file. Also, I’m torn on ChatGPT-User being included, since that’s for AI agents reaching out and performing specific actions.

But ultimately this is a blog and I have copyright over the content, it is not an API which an AI could reasonably be interacting with, only plagiarising from.

Then I created layouts/_default/robots.txt:

# robots.txt
{{ range $bot := $.Site.Data.ai_bots }}
User-agent: {{ $bot.ua }}
Disallow: /
{{- end }}

User-agent: *
Allow: /

The resulting generated page is not much longer.

# robots.txt

User-agent: GPTBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: anthropic-ai
Disallow: /
User-agent: CCBot
Disallow: /

User-agent: *
Allow: /

But that’s not the point. I now have structure, consistency, and confidence that if I make a typo it will be repeated everywhere so that I can’t help but notice it, instead of quietly sabotaging subsets of my efforts to exclude AI bots.

Configuration Objects

Phil Pennock — Mon, 22 Jan 2024 01:45:00 -0500

I’m going to pick holes in an approach which I think is actively harmful to our industry and the maintainability of the systems we write and manage. The problem is one of approach, and nothing is unfixable.

The way is which YAML is used and abused is symptomatic of flaws in how we approach systems design and configuration. This is not the fault of YAML, as whichever language we use would suffer the same fate.

It is reasonable to not want to tie the configuration of a system to one language.

It is reasonable to not want to force the use of interpreters for various engines deep into your stack; to not want to make it impossible to add new flexibility later unless either some language maintainers agree with you, or you want to take on maintaining a language interpreter yourself.

When you choose to use a structured data format as the only input, if the format is human editable then the danger is people try layering as little as possible onto that format at first, and instead of getting a well designed data generator, you accrete generations of hacks.

Kubernetes

For instance, it’s reasonable for Kubernetes to pick something like YAML as the object interchange, but only if you demonstrate from the beginning that YAML is only an object representation layer; one with schemas available, and other good things; and demonstrate this with examples of how to do things otherwise.

Most Kubernetes admins should touch a tiny bit of YAML for initial bootstrap. Thereafter, they should know it as “that format the state snapshots and backups are written in”.

But because a human editable representation was used, with the supplied tools not geared for taking other formats, people stick with YAML. We see anchors and references, and then Helm becoming a Go text/template wrapper with distribution mechanisms, and incomprehensible messes as supplied Helm Charts either try to be all things to all people with a bazillion configuration options, or say “learn to write patch objects, then you can change anything, we merge those in” and suddenly most admins can’t tune their charts.

If Protobufs had been used to represent the Kubernetes API objects, it would have been less approachable but it would have been much more obvious that the configuration objects should be merely output artifacts from the configuration management for a set of clusters, not the native language for humans to edit.

I hope that in the future we look back on people using YAML with text templating layers on top of it with the same cautious respect with which we look at people writing raw Sendmail configs and bemoaning the use of macros by newcomers who aren’t real mail administrators until they do. I have a suspicion that attempts to nudge the Kubernetes culture away from YAML will meet just that sort of reaction.

And this is not specific to Kubernetes. I see it with many things using YAML. While YAML might have its problems (or NO problems), it’s broadly a decent human-editable way to represent some structured data.

OCI Image References

People write references to OCI images and write :latest, and then they get told off because you don’t know what a given machine will see at any point in time. It becomes a shibboleth, to tell people that :latest is wrong, rather than fixing the problem, and define the in-group of service admins who “understand why you should never use :latest”.

The problem is that the configuration describing the overall desired state is being used as the same configuration for telling the machines what to run now when they see the configuration.

Loosely, the human using :latest probably really does want a given deploy to grab the latest, but to have it be consistent across the deploy, and subject to being able to switch to a pinned version if needed, and perhaps :latest being interpreted as “the latest to pass our malware scanners, not the latest upstream”.

The build step for deployment configurations might be written in the same configuration language as the “desired state when we do deploys” configurations, but it is semantically different. There needs to be a translation layer.

Taking a desired state should resolve OCI image tags to checksums and lock down specific versions, without the human needing to care. They should know that they can see what it was for a given release/deploy but not be forced to manually maintain the lower layer configuration.

Summary

Configuration to describe what we want is a higher level of abstraction than configuration to be interpreted by individual machines during a deploy. Using the exact same files and models for both is an anti-pattern.

The runtime deploy configuration objects should have a schema which doesn’t allow for the human mutable labels which drift, perhaps mid-deploy. The human edited objects should have LSP integrations letting you see how a label might be resolved now, and why, or switching to how it was resolved, for a given past deploy, referencing a history of the generated configurations.

We should probably be using JSON or Protobufs or CBOR or “something” for the machine configurations. Just about anything will do, except ASN.1 or XML (I have my prejudices, you have yours). We don’t want comments there, or encouraging editing except in emergencies.

YAML might be a fair simple representation as a starting-point for human editing, but as long as there are solid schemas for the backend target, it should not be forced.

Tailscale and Docker Remote

Phil Pennock — Sun, 14 Jan 2024 22:45:00 -0500

I recently wanted a way for my team to run throw-away containers on a remote box, for development and testing. I did not want Kubernetes or a lot of overhead, I just wanted the ability for people to throw containers on the box and run them. The only access control needed was “is in the list of people allowed to talk to the container-runner service”: if you can talk to it, you can run containers. In host-networking mode, privileged, whatever.

There are all sorts of fleet orchestration systems for running containers across many nodes, but really I was after nothing more than “docker, but remote”.

Given a choice of Docker, how to secure the communications to the Docker API? This is not about “the apps run by Docker”, as some of those might be public, some not. Each app will have its own stance, although we do want good defaults and support for different models. My concern was how to talk securely to Docker itself.

At home I do this using my personal Certificate Authority, but we’ve been strenuously resisting needing to run our own CA at work. Let’s Encrypt manages almost everything we need, and CertManager inside K8S manages ad-hoc CAs with a limited audience as needed.

And then I remembered Tailscale and Andreas Fuchs’s tsnsrv front-end proxy: connect a machine to Tailscale, run a proxy service, and use Tailscale as the access control. There are even HTTP headers added to the proxied request, identifying the origin. Roughly. Exceptions apply with multiple authentication mechanisms allowed, etc; the ID presented is not absolutely trusted. Real access control should be done with ACLs in the Tailnet, and I do that, but it’s a nice extra.

The proxy appears on your Tailnet as a new “machine”, with its own hostname.

So, how hard could it be?

Turns out, not very hard. There was a fair bit of Ansible to write for our dependencies, and a lot of debugging to get Docker working, but conceptually it’s simple enough. For a few days, it was more complicated, until I realised that I could remove TLS from the picture. The original deployment was a single-evening task.

This blog-post will include an example systemd service file, but none of the rest of the Ansible we use.

No TLS??

Tailscale is built around WireGuard as the network encryption layer. WireGuard provides both encryption (ChaCha20) and integrity protection (Poly1305) between nodes on the network.

Tailscale tracks your identity, according to various identity providers, and lets you build ACLs inside Tailscale.

So TLS provides no real additional benefit here. It’s vaguely nice to have as a doubled layer and to reduce possible questioning from auditors.

Against that, the Docker CLI tool will only use TLS to talk to a remote service if configured with a TLS client key and cert. It doesn’t need to be verifiable by the peer! At first I just reused the existing key/cert from my home CA, and that was sufficient. A snake-oil cert (self-signed) would work just fine, we only need to to convince Docker to speak HTTPS instead of HTTP.

Setting up an additional snake-oil cert as a pre-requisite is a bunch of needless ceremony and an obstacle to easy use. So, just … switch the proxy to not require HTTPS and Docker is happy! We don’t need to worry about web-browsers which refuse “plaintext” here, we just need to know that the traffic on the wire is protected.

Appearance to an end user

To use this, someone runs this setup step:

tailnet_name='whatever-isyours'
H="tcp://docker-foo.${tailnet_name:?}.ts.net:2375"
docker context create work-foo \
  --description='foo CloudNameHere ArchHere' \
  --docker="host=$H"

and thereafter they can use --context work-foo for the Docker CLI. Or, more likely, export DOCKER_CONTEXT=work-foo and set that up via a direnv(1) hook to be set automatically as needed.

For us, using a Linode in Chicago, that was a hostname of docker-us-ord and a context of work-us-ord with a description of 'us-ord Linode amd64'

You can find your Tailnet name at https://login.tailscale.com/admin/dns

If you do need to use TLS, then it becomes:

CP='/path/to/keycert_without_suffix'

tailnet_name='whatever-isyours'
H="tcp://docker-foo.${tailnet_name:?}.ts.net:2376"
docker context create work-foo \
  --description='foo CloudNameHere ArchHere' \
  --docker="host=$H,cert=$CP.crt,key=$CP.key"

The Deployment

As a pre-requisite, you need a Tailscale account and some users. Note that they do have a generous free tier and you can probably get started without paying anything.

https://tailscale.com/

The setup once written

To add to a new host, you’d just add some variables to a host definition:

vars:
  want_svc_docker: true
  docker_tailsvc_name: "docker-us-ord"
  # Those first two are all that were really needed,
  # the next two ... see below.
  tailsvc_verbose_logging: true
  # docker_use_tsnsrv: true

and then run Ansible … three times. Alas. Why? Because I failed to use pre-authentication keys. Don’t be me. Read https://tailscale.com/kb/1085/auth-keys and work this into your Ansible secrets management. We will do this before we use this with a second host. At least, I hope we will.

You can even set the key to be pre-tagged, which will be wonderful for easing ACL setup for a given deployment role.

So we’ve got a clean-up task if we decide we’re going to do more with this approach, which I hope we will. So far, the additional services have been run inside Docker using $TS_AUTHKEY and I haven’t touched the Ansible rules.

The Multiple Ansible Invocations Without Pre-Auth Keys

The first time, you run without the docker_use_tsnsrv enabled.

You then run, as root on the host:

# tailsvc is a Unix user, chosen to match systemd setup below
tailscale up --operator=tailsvc

That will give you a URL to visit. Sign in with the authentication provider you use, and accept the machine into the Tailnet.

Then use https://tailscale.com/kb/1028/key-expiry to disable key expiry.

Then you uncomment docker_use_tsnsrv and run Ansible. This is the step where you install the tsnsrv proxy service. It will run, but with rather verbose logs. This is needed because (at time of writing) the URL needed to authenticate the proxy to the Tailnet is only printed in the verbose logs. But you do the same authentication flow once more, and disable key expiry again.

This time you should also apply a Tag to the pseudo-machine, for use in Tailscale ACLs.

Then set tailsvc_verbose_logging to false and deploy once more to clean up.

This could shrink to two deploys, if the issue with the authentication URL needing verbose logs is sorted out, and to a single deploy with pre-auth keys!

The systemd service file template

The following variables were needed:

tailscale_magicdns_domain: this is probably global within your deployment scope
tailsvc_name: this needs to be unique for each service using this setup
tailsvc_plaintext: a bool to choose whether or not to live without the TLS pain
tailsvc_verbose_logging: because I didn’t use pre-auth keys and so needed to be able to deploy with access to the logs to get the server auth URL.
tailscale_proxy_username, tailscale_proxy_groupname, tailscale_proxy_homedir: for a Unix user to run the proxy, so we can avoid running things as root.

{% set logparam = '-tsnetVerbose=true' if tailsvc_verbose_logging else '' -%}
{% if tailsvc_plaintext | bool -%}
{% set port = '2375' -%}
{% set listenspec %}-plaintext=true -listenAddr=:{{ port }}{% endset -%}
{% else -%}
{% set port = '2376' -%}
{% set listenspec %}-listenAddr=:{{ port }}{% endset -%}
{% endif -%}
{% set svcurl %}http://{{ tailsvc_name }}.{{ tailscale_magicdns_domain }}:{{ port }}{% endset -%}

[Unit]
Description=Tailscale Proxy for Docker
After=network-online.target docker.service

[Service]
Type=simple
ExecStart=/usr/local/bin/tsnsrv -name {{ tailsvc_name }} {{ logparam }} {{ listenspec }} -upstreamUnixAddr /run/docker.sock {{ svcurl }}
User={{ tailscale_proxy_username }}
Group={{ tailscale_proxy_groupname }}

# A bunch of config is written to .config/tsnet-tsnsrv/ in the homedir (/home/tailsvc)
# We do need access to the Docker control socket.
ProtectHome=tmpfs
BindPaths={{ tailscale_proxy_homedir }}  /run/docker.sock

NoNewPrivileges=true
MemoryDenyWriteExecute=true
LockPersonality=true
PrivateDevices=true
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true
RestrictSUIDSGID=true

ProtectSystem=strict
PrivateTmp=true

[Install]
Alias=tailsvc-docker.service
WantedBy=multi-user.target

{# vim: set filetype=systemd.jinja : #}

The ACLs

While tailsvc is able to present identity in HTTP headers, Docker isn’t going to use those and there’s no ACL management available there.

But visit https://login.tailscale.com/admin/acls/file and, if you’re ready to carve up your Tailnet, then you might define a group:myteam and then in the acls[] list include an item:

{
  "action": "accept",
  "src":    ["group:myteam"],
  "dst":    ["tag:docker:*"],
}

The Icing on the Cake

If you have firewall rules on the host, then you might reference the IP address ranges used by Tailscale to bypass most protections:

100.64.0.0/10: RFC 6598 Shared Address Space
fd7a:115c:a1e0::/96: random allocation in RFC 4193 Unique Local IPv6 Unicast Addresses

tailscale_tailnet_cidrs:
 - "100.64.0.0/10"
 - "fd7a:115c:a1e0::/96"

And then you might allow ports 80 and 443 through from the world, but no other ports, and suddenly folks can run things in Docker and expose on other ports and those run services will only be reachable for people connected to your Tailnet!

Now, some might use that to skip authentication again and end up with a soft chewy inside for your VPN “internal network”, but it’s on you and your policies to keep that from happening.

When running tsnet using services, those are inherently Tailscale-only; but for developing a new public service which isn’t ready to be public quite yet, having a toybox deployment host for containers where folks can run quick experiments without a lot of overhead, this visibility constraint is ideal.

Feedback

ACLs include ports, use daemon.json

Pete Keen notes that, depending upon access requirements, a simpler solution which might be available is to configure Docker to listen on the tailnet IP in its daemon.json file and then use ACL tags to only grant access to the 2375/2376 ports to my team.

This is great and has me wondering if I should have made different choices about how deployments can offer service to the company. :)

This won’t work for my deployment, because our model is that ports 80 and 443 are open to the world (and apps which want access control have to provide it) and every other port is open to the Tailnet, so simple apps can just provide external service on a port, and anyone in the company, on our Tailnet, can reach the app.

The problem is that the Tailscale ACL language only offers an accept action

Tailscale access rules are “default deny”, so the only possible action is accept, to allow traffic that would otherwise be denied.

If there were a Deny action available, then we could switch to that and just trust that the ACLs are well-maintained. We would use something like:

// Not supported by Tailscale at this time
[
  {
    "action": "accept",
    "src":    ["group:myteam"],
    "dst":    ["tag:docker:22", "tag:docker:2375", "tag:docker:2376"]
  },
  {
    "action": "deny",
    "src":    ["autogroup:member"],
    "dst":    ["tag:docker:22", "tag:docker:2375", "tag:docker:2376"]
  },
  {
    "action": "accept",
    "src":    ["autogroup:member"],
    "dst":    ["tag:docker:*"]
  }
]

and then have Ansible pick up the Tailscale IPs for the docker daemon.json file, and manage that file via Ansible.

If we had “the 80/443 service is for internal usage only and everyone outside the team has to go through that”, then we could still use this, and just grant "autogroup:member" access to ["tag:docker:80","tag:docker:443"] and all would be well.

What we could do is this, since Pete pointed out that ACL rules can match port ranges too:

[
  {
    "action": "accept",
    "src":    ["group:myteam"],
    "dst":    ["tag:docker:22", "tag:docker:2375", "tag:docker:2376"]
  },
  {
    "action": "accept",
    "src":    ["autogroup:member"],
    "dst":    ["tag:docker-companyopen:1-21", "tag:docker-companyopen:23-2374", "tag:docker-companyopen:2377-65535"]
  }
]

but by that point things are looking fragile enough that the “use tsnsrv as a proxy to present as a different machine” looks much nicer to maintain.

So yes, I might revisit, but I think I like the model I ended up with, of 80/443 being public-to-Internet and everything else internal.

Small Mailserver Best Current Practices

Phil Pennock — Fri, 24 Jul 2020 16:45:00 -0400

(This post was originally written as a reply on the mailop mailing-list, but a friend asked me to turn it into a blog post. I’ve edited it, mostly adding more links to elsewhere, but there are some additions here.)

Context: someone with a mail-server hosted in a German facility with a poor reputation for handling abuse reports was asking for help on sending email to their Gmail-using friends; they had SPF and didn’t see the point of DKIM; they had TLS setup for their mail-server, using a certificate from CACert.

There’s a PDF from Google from 2006 which is still worth reading: https://research.google.com/pubs/archive/45.pdf, entitled “Sender Reputation in a Large Webmail Service” by Bradley Taylor. Anyone running a mail-server today needs to read that document.

If you don’t send much email, then the only IP-based reputation which Google can assess you on is the reputation of your address-block, so being in a “troublesome” hosting provider will score heavily against you. At that point, if not moving away, you need to try to balance out that negative score with enough positives that any of the large providers using reputation scoring will accept the mail.

Working forward-and-reverse paired DNS is even more important for IPv6 than for IPv4; for better or worse, some of the large providers have decided that exemptions in old standards for old behavior should not apply when folks deploy standards which are far newer. So you absolutely need an MX record, you must not just rely upon address-records (A and AAAA).

With a poor IP-based reputation, you need to see if you can score a better domain-based reputation. This is where DKIM comes into play: once you can provably link a message to really be from a given domain, then even if you don’t send much mail you can benefit from stuff like “not on day-old-bread domain-lists”. But having DKIM and then a DMARC record does help (and I’m no fan of DMARC).

For the mail-server’s TLS: for that to count in your favor instead of being a wash, I strongly suspect that it needs to be a certificate which senders can verify. For those people scoring up for “better TLS”, those senders using DANE will be happy with a TLSA record in DNSSEC for your CACert anchor. But the large webmail providers are Resistant to having to deploy DNSSEC verification, so instead have pushed out an alternative called MTA-STS. With MTA-STS, you’re tied into “whichever subset of CAs all the large senders you care about will trust”, and then using that CA for the certificates both for the MTA-STS web-server and for your mail-server. Note that you don’t need to implement the client logic for MTA-STS (and I think it’s antithetical to an open federated platform) but do need to just publish the static information for those senders who do use it. At that point, CACert is not going to cut it. You’d need to try Let’s Encrypt instead.

The ongoing natural tendency from larger providers is to favor supporting what the majority of their users want the majority of the time. With so many people using larger providers, they naturally tilt towards stuff which works with the larger senders, and requiring more hoops. Those additional hoops create more work for smaller providers and self-hosters doing thing manually.

We need better automation tools around all of this. The below will make it clearer why.

So, here is my current understanding of the best current practices here, in reality not IETF idealism. This includes making mandatory stuff which some folks insist must be optional, because realistically to send to some large providers it’s not optional. This list includes features to make you compatible with ongoing trends in the EU (particularly Germany) to strongly disfavor allowing clear-text SMTP.

This is broad best practices, targeting more than just Gmail. Eg, DANE/DNSSEC will be ignored by the large webmail providers, but will matter with some other providers. [clarification added later]

This assumes that you are not a large sender who should also be setting up feedback loops, learning how to “warm” IPs, considering BIMI, postmaster tooling domain verification, etc.

Deliverability fixes

reverse DNS with matching forward DNS; the name used should not pattern-match anything generic and ideally would include a DNS label of mail or mx or the like in it.
MX record, always.
Accurate SPF;
- ideally not too broad; pay attention to SPF’s query limits.
- avoid -all at the end because, with the sole exception of “this domain never sends email” records, the larger operators have metrics to show that using -all tends to be a sign of over-enthusiasm rather than reality, so it will slightly count against you;
- remember to have an SPF record for your HELO hostname, because when you send a “bounce” rejection, this is the thing which will be looked up (since there’s no domain in <>).
DKIM set up, with thought towards the selector namespace.
- RSA2048 key is effectively a hard-requirement
  - DNS TXT records consist of one or more DNS strings, each of which is limited to 255 ASCII characters. For a key of this size, you will end up needing two DNS strings in the zonefile.
- Ed25519 keys are not yet widely supported, but by now are not likely to actively break and make things worse for you, if you dual-sign. This needs to be a different selector.
- Note that for various good reasons you should design this to be something you routinely rotate.
  - Some folks use yearly, some monthly
  - I rotate every three months.
DMARC record; see RFC 7489
- But for domains which humans send from don’t use p=quarantine or p=reject;
- Do consider setting up a receiver for reports, just so that you can see how much of a privacy breach DMARC reporting is when you send to mailing-lists which don’t re-sign. :-/
TLS certificate from a CA in the main trust anchor bundles;
- Just use Let’s Encrypt.
MTA-STS web-server with HTTPS certificate from the same CA, and the relevant MTA-STS txt file in place; add the DNS record when it’s up and happy. See RFC 8461.
- See https://esmtp.email/tools/mta-sts/ for a testing validation tool (with thanks to Luis Muñoz for the pointer).
For the independent mail providers using the stuff broadly supported in open source MTAs, you should look at DNSSEC, because the patterns here are less susceptible to rent-seeking pressures:
- DNSSEC-signed zone for your own domain
  - Use whichever signing algorithm CloudFlare are currently using: this should be both current for cryptography and widely enough supported that if it’s not supported by someone’s resolver, then they have bigger problems than just your domain.
- DNSSEC validating resolver for you to look up records of others (consider Unbound or Knot Resolver)
- DANE records for your own domain (TLSA records in DNS)
  - See RFC 7672 for the SMTP details.
  - See RFC 6698 for the base spec with RFC 7218 for some common acronyms which make talking about it easier.
  - See RFC 7671 for the updates and operational guidance.
  - There are other RFCs, for SRV records and for OpenPGP, etc.
- Tell your mail-server to obey DANE stuff, so that if there’s a TLSA record in DNSSEC-verified DNS then the mail-server can disable fallback to cleartext for delivery to MX (and ideally also then verify the TLS connection has a cert chain which is anchored in one of the TLSA records)
- https://dnsviz.net is your friend
_smtp._tls record so you can get reports of TLS failures sending to you
Seeing if you can get your IP onto one of the open DNS-based allow-lists (also called “whitelists” but some folks are moving away from that term), such as https://www.dnswl.org/ or Spamhaus’s SWL.
Periodically check if you appear in any DNS-based deny-lists.
Make sure you’re not sending from “ISP residential address-space”; if need be route your mail outbound via a host in better address-space (and update SPF etc to match)
Don’t do sender call-out verification to SMTP servers which aren’t yours.
For your own sanity, do make sure you set up fail2ban, or something like it, scanning your mail-server logs, because SMTP AUTH online cracking is widespread. If they ever get in, your deliverability will be negatively impacted by their spam campaign through your mail-server.

Convenience Stuff

Outside of “Phil’s BCP” above, additional non-deliverability but convenience options include:

DNS SRV records for submission(s)/imap(s)/pop3(s)/sieve, even if just to say with «0 0 0 .» that it’s not supported.
If your communications base includes people using OpenPGP with email, then set up WKD to publish OpenPGP keys for your domain too.
- This is just a fixed schema for laying out keys for HTTPS retrieval.
- See the WKD draft for details.
- I wrote https://github.com/PennockTech/openpgpkey-control as a management framework for an organization; the other/standalone-update-website script is designed to be embeddable into an existing site-building workflow without anything else from the repository.
- The GnuPG project has tooling available which manages the WKD layout as an email-integrated workflow, for people to update their own keys.
If your communications base includes people using S/MIME then set up SMIMEA records in your DNSSEC-signed DNS.
- They look a lot like TLSA records; both are trust anchors in DNS.
- See RFC 8162 for details.
The moment you start specifying “must be TLS-secured” it’s worth adding CAA records into DNS, so that Certificate Authorities which are broadly trusted will refuse to issue for your domain unless you list them.
- See RFC 8659 for details
- The values checked by each Certificate Authority as indicating they have permission are required to be listed in their Certification Practice Statement, as part of the CA/Browser forum’s Baseline Requirements. If it’s missing, then browsers are not supposed to be trusting that CA.
- For domain-validation CAs such as Let’s Encrypt, consider adding account information to those records to tie it to your specific account. See RFC 8657 for details.
- Beware that at time of writing, Let’s Encrypt only honors the accounturi restriction in their staging environment, not their production setup; this will likely change.
- Remember that DNS zonefiles support comments. You’ll want them here!
https://wiki.libravatar.org/api/ for Gravatar-style images in some MUAs (with caching for privacy reasons); this is a fixed schema HTTPS layout and a DNS record. I think almost nobody uses this but I don’t have usage figures. If you have a Claws user, perhaps?

A lot of the DNS items above can be found in a sample DNS zonefile contributed to the DNSControl project for their regression test suite, after my production zone broke their converters. DNS hosting platforms which try to over-simplify will make some of the above hard.

I’ve written this as a reply so it’s the stuff I can remember / spot now, so I might be missing something big or “too obvious to me”. Other tech exists.

Addenda

A second reason to run a local DNS resolver is that if you are checking any DNS-based access lists then you should not do so via public resolvers. All mail-server clusters or stand-alone systems really should have a dedicated DNS resolver for their use. Since you need the resolver to exist anyway, you might as well make it a validating resolver and get DNSSEC verification too, per the above. [2020-08-19]

Shell Locality

Phil Pennock — Thu, 16 Jul 2020 11:45:00 -0400

When a shell function declares a variable to be local and then unsets it, does the name return to being global in scope or is it still “known” to be local, even though unset (via some kind of tombstone mechanism, perhaps)?

Let’s test it. Spoiler: the results vary.

Note that POSIX does not provide local, this is a shell extension.

The Test

Here is our test as a pasteable one-liner:

i=0; s() { echo "$i: x: $x"; i=$((i+1));}; x=outside; s; foo() { s; local x; s;x=inside;s;unset x;s;unset x; s; }; foo; s

Breaking that down:

i=0                # loop-counter for our "show" function so we can see the steps
s() {              # a simple show function, using POSIX $((...)) for arithmetic
  echo "$i: x: $x"
  i=$((i+1))
}
x=outside    # our test variable is x and we give it a maskable value
s            # 0 -- prior state
foo() {
  s          # 1 -- inside function, before local
  local x    # from this point on, if `local` exists, x is non-global
  s          # 2 -- does the act of making the variable local clear its value?
  x=inside   # our "inside" value
  s          # 3 -- confirming value is set and variables work
  unset x    # reset x ... but what is left behind?
  s          # 4 -- should see an empty string
  unset x    # unknown: does this unset the global?
  s          # 5 -- final display inside the function
}
foo      # we need to call the function, since only zsh has anonymous functions
s            # 6 -- outside the function, show the global after double unset

The Results

Zsh

0: x: outside
1: x: outside
2: x:
3: x: inside
4: x:
5: x:
6: x: outside

The local keyword resets the value of the variable (counter 2).
The second unset did not unset the global (counter 6).

Bash (5)

0: x: outside
1: x: outside
2: x:
3: x: inside
4: x:
5: x:
6: x: outside

[edited to add Bash options:]

Bash behaves the same as zsh, by default. There is an option localvar_inherit to change this:

shopt -s localvar_inherit
foo
0: x: outside
1: x: outside
2: x: outside
3: x: inside
4: x:
5: x:
6: x: outside

There is also an option localvar_unset which impacts a more complicated form than we’re addressing here.

Dash

0: x: outside
1: x: outside
2: x: outside
3: x: inside
4: x:
5: x:
6: x: outside

The local keyword in dash does NOT reset the visible value, but does make future modifications only visible within local scope; this matches Bash with localvar_inherit.
The second unset did not unset the global.

Ksh

This test was of the shell from apk add -U loksh inside an Alpine 3.12 Docker container, which installed version 6.7.1-r0 of loksh, described as “A Linux port of OpenBSD’s ksh”.

0: x: outside
1: x: outside
2: x:
3: x: inside
4: x: outside
5: x:
6: x:

The local keyword in ksh is immediately resetting the visible value.
The second unset inside the function did unset the global variable!

FreeBSD sh

0: x: outside
1: x: outside
2: x:
3: x: inside
4: x:
5: x:
6: x: outside

This is the same as zsh and bash’s default.

Conclusion

Writing portable shell is hard. Duh.

Slack Tax

Phil Pennock — Thu, 12 Dec 2019 19:20:00 -0500

Chambers of commerce should be urgently talking with the FTC to try to stave off an imminent forced tax on their members, by Slack.

NB: I have no vested interest in any company mentioned here, unless my retirement index-tracker fund has done so, in which case I’m probably hurting myself by writing this.

Slack as a company makes Slack, the product. It’s for team communications and basic use is free but there are paid tiers which bill per active user. They’re scrupulously fair about tracking as users become inactive and issuing billing credits so that you don’t pay if people aren’t using it.

The Slack product is good. Some years back when it was new, I mistakenly objected to our company trying Slack to solve “get everyone using a team chat” because we’d tried two other products and only the engineers would actually use them, so I believed the problems were cultural and shuffling deck chairs by trying the latest trendy chat tool was avoiding addressing the real problem. I was persuaded to give it one more try. I was wrong: the ease of use and fun quirkiness of Slack made it enticing enough that the entire company got on board. For a while it solved our problems (right up until dropping the gateways for IRC & XMPP killed the accessibility solutions our blind employee was using).

If a company wants to use Slack, that’s their call to make. Competition is good and freedom to choose matters. There are competitors (eg Mattermost; or Microsoft Teams) which creates an open marketplace.

The problem comes when suddenly a company has no choice and in order to be viable they are forced to buy a particular product, licensed per user. Much like MS Office was effectively mandatory and changing document formats kept locking out competitors, Slack is making that transition with a new paid feature.

Barrons extracted the critical part from a recent Slack blog post:

“I’ve been making network-based software for a living for the last 20 years,” Butterfield wrote. “Over that time I’ve worked on a lot of great projects directly, and have seen many more from the inside, as an investor, board member or just as a friend. I have never been more excited about a product or its product market fit than I am about shared channels.”

Shared Channels let you link together Slack teams. Within a company, great. Between companies … it’s attractive. But you can only link together Slack teams. It is not an open protocol and competitors are not welcome. There’s no fundamental reason why you can’t bridge to other products but that’s not compatible with this Silicon Valley investment.

I’ve seen Shared Channels in beta used by a company to set up links with customers and vendors. Important business links get Slack links. There might be only two or three people routinely in such a channel, enough to avoid a single person dependency, but there’s an implicit expectation from both sides that when there’s a problem, other needed people can be brought in quickly to help solve it.

Shared Channels are undeniably useful, although there are cultural mores which need to be established to prevent abuse by larger companies treating employees of their business partners as their own, to delegate work to. Any company about to set up a direct synchronous communication system open potentially to “all their employees” and “all employees of all important business partners” needs to take seriously the problems that can arise from that. Synchronous communication is disruptive and it takes solid ground rules to keep productivity from being damaged by just a few people who don’t follow cultural expectations. When not all the people are your own employees, that will get harder.

The core problem though is that with Shared Channels there is effectively no choice. Slack has a critical mass of customers and Shared Channels turns that into a network effect, locking out competition. Who will pay for two team communication products, or try to maintain one paid and one free, when they can “just” use one? You can’t sanely just use Slack for a few liaison people because much of the value is the ability to pull in other people to solve problems in a hurry.

So with Shared Channels being a paid tier feature which excludes other companies’ team chat, and the expectation being that all employees are on the same team chat, suddenly we don’t have a good product winning in an open marketplace. Instead, we have the current market leader setting up a major barrier to competition and locking themselves in, while making sure that each company trying to choose just has to use what all their business partners are using.

Those monthly per-user fees start looking a lot like a tax on doing business.

There’s an easy fix: ensure that Shared Channels use a documented protocol and that Slack teams are freely able to link to other products, not just teams hosted by Slack.

-The Grumpy Troll

Git Aliases & Shell

Phil Pennock — Sun, 22 Jul 2018 21:20:00 -0400

Today I took a look at one particular git repository’s configuration and saw something slightly off in the configuration for a credential helper, dating from an old experiment with AWS CodeCommit. I decided to dig deeper to figure out what the actual rules are for shell commands inside git configuration files.

This side-diversion took a bit longer than expected. It’s a Sunday. Ah well. I’ve seen too much cargo-culted incorrect information online, so it was time to figure out an accurate answer. If you see Leaning Toothpick Syndrome markedly increasing the count of backslashes, then a little suspicion is warranted.

In fairness to those who are confused, including me before today: the git logic tries to implement magic “do the right thing” fix-ups at a level beneath the GIT_TRACE reporting, so that it’s pretty hidden until you either read the source or use a strace(1) style of tool. This makes the simple cases right, even when they’re wrong, but makes the fancier cases obtuse.

I decided to focus on looking into how [alias] rules are handled, on the (correct) assumption that it’s the same mechanisms involved.

For clarity: avoid cleverness inside configuration files!

If the steps stated here matter, then the complexity really should be moved out of ~/.gitconfig and into a separate helper script. When you start having to worry about where strings are parsed and how they’re handled in which situations, the correct response is to back away slowly, not breaking eye-contact, and create a very short script to handle things instead.

Notwithstanding correctness, sometimes you need to know how to tame the beast.

`git config` documentation

Let’s start by reading the actual documentation; this is found in git-config(5):

alias.*
Command aliases for the git(1) command wrapper - e.g. after defining alias.last = cat-file commit HEAD, the invocation git last is equivalent to git cat-file commit HEAD. To avoid confusion and troubles with script usage, aliases that hide existing Git commands are ignored. Arguments are split by spaces, the usual shell quoting and escaping is supported. A quote pair or a backslash can be used to quote them.

If the alias expansion is prefixed with an exclamation point, it will be treated as a shell command. For example, defining alias.new = !gitk --all --not ORIG_HEAD, the invocation git new is equivalent to running the shell command gitk --all --not ORIG_HEAD. Note that shell commands will be executed from the top-level directory of a repository, which may not necessarily be the current directory. GIT_PREFIX is set as returned by running git rev-parse –show-prefix from the original current directory. See git-rev-parse(1).

That’s a decent start but things go wrong mysteriously when we try to do something non-trivial. A simple for-loop appears to fail at the "$@" stage, but why?

Also under “Syntax”:

The following escape sequences (beside \" and \\) are recognized: \n for newline character (NL), \t for horizontal tabulation (HT, TAB) and \b for backspace (BS). Other char escape sequences (including octal escape sequences) are invalid.

Shell side-digression

Something which will become important below is a quirky corner of POSIX shell handling. Despite POSIX giving us the convention of -- to terminate option processing and treat following parameters as non-option entries, when you invoke sh -c FOO -- arg1 you are not using -- to terminate options.

Instead, per the specification, in -c handling, the first non-option parameter is the string providing the input to be parsed for shell commands (here FOO) and the next parameter provides the name of the command! It’s argv[0] pass-through. Thus sh -c FOO -- arg1 will invoke a shell, with $0 equal to --, $1 of arg1 and then parse and handle FOO.

After the first non-option, no other strings are examined to see if they might be options; there is no permutation.

Thus sh -c FOO BAR -u does not risk -u telling the shell to treat unset variable references as errors. Instead, FOO is invoked in a context where argv=["BAR", "-u"].

What actually happens with git

Git’s config parser and the shell-or-other invocation are the two layers to worry about.

The config parser uses both ; and # as comment markers, so an ; if unquoted will terminate things. Given that ; is a sub-list terminator in shell syntax, this is a little unfortunate.

A shell is not necessarily used when an alias is treated as a shell command. That’s very careful documentation wording, above. “it will be treated as a shell command” does not mean that a shell will be used, merely that it’s a shell command to be handled. Git will often resort to using a shell, but it’s not a commitment to do so.

Git parses the configuration file handling basic stripping of comments and handling of the \n substitutions and quoting, before the alias mechanisms come into play.

Git can split a string into separate fields, for invoking as a command, without needing to go near a shell to do so. The alias.c:split_cmdline() function handles this, splitting on whitespace while not splitting within quoted strings. It handles single and double-quotes, with double-quotes supporting a backslash escape purely for avoiding breaking out of quoted state. No other escapes are supported, but the configuration parsing has already handled some. You can have:

[alias]
  foo = !"bar\nbaz"

and the value of stored for the alias foo will be:

!bar
baz

Instead, the quotes handling for split_cmdline are for quotes inside the original quotes; this is what lets us write:

[alias]
  foo = "!printf '%s\\n' first \"sec ond\" third"

and invoke:

% GIT_TRACE=true git foo
20:50:18.430074 git.c:654               trace: exec: git-foo
20:50:18.430779 run-command.c:637       trace: run_command: git-foo
20:50:18.432151 run-command.c:637       trace: run_command: 'printf '\''%s\n'\'' first "sec ond" third'
first
sec ond
third

Note here that we used \\ to avoid having the git config parser handle the \n. A strace(1) shows us:

% strace -ff git foo
[...]
[pid 16724] execve("/bin/sh", ["/bin/sh", "-c", "printf '%s\\n' first \"sec ond\" th"..., "printf '%s\\n' first \"sec ond\" th"...], [/* 62 vars */] <unfinished ...>
[...]
% strace -ff sh -c "printf '%s\n'" first 'sec ond' third
execve("/bin/sh", ["sh", "-c", "printf '%s\\n'", "first", "sec ond", "third"], [/* 61 vars */]) = 0

Here, the sequence \\n is simply how strace is showing that \n as a two-character sequence is the string being passed through.

Returning to Git’s codebase: Without an exclamation mark, split_cmdline() is used and the results put into the current process’s argv[] vector after the initial git, and argument processing effectively then restarts, letting git decide again what should be done.

Without an exclamation mark, that’s it. All done, quick and clean and simple. Everything after here assumes that the entry starts with an exclamation mark.

What the exclamation mark means is really “run this entry as a sub-command now, and exit”. This is found in git.c:handle_alias(). The invocation is then run_command() on a struct with .use_shell set. The entire string of the alias value is passed into this as a single string, no whitespace handling. split_cmdline() does not apply.

But just having .use_shell set still doesn’t mean that a shell is used. What it means is that run-command.c:prepare_shell_cmd() will be used to construct the shell command, which might mean that a shell will be used.

What prepare_shell_cmd() does is look to see if the value of the command might be a single word without any special characters. Those special characters are backtick itself or any of: |&;<>()$\"' \t\n*?[#~=%

Since a whitespace is in the list of characters, simply having two words is enough to trigger invoking the shell.

Once the shell is invoked, the exact form invoked depends upon whether or not the invoker of the command provided parameters on the command-line.

In all cases, the shell is invoked with at least four parameters in argv. The first two are ["sh", "-c"]. The third will be the string from the configuration, if and only if no parameters were supplied to git by the invoker. If parameters were provided, then instead the third parameter for the shell is modified, by adding the five characters "$@" to the end of it. (Five: SPACE, QUOTATION MARK, DOLLAR SIGN, COMMERCIAL AT, QUOTATION MARK.) Git tries to be clever and assumes that you’ll want the arguments available at the end of whatever string is given.

This works for simple commands. But for a text which tries to handle arguments itself, it’s a hindrance to be worked around.

The fourth of the always-present parameters for the shell is the text from the alias definition. Repeated, as the name of the shell.

Any elements in the new shell’s argv after that are passed through from the invoker of git.

What this means for us

Handling the auto-inserted "$@" is simple enough, once you know that it needs to be handled: simply end your alias definition with a shell comment character, the octothorpe # (Unicode NUMBER SIGN).

This is correct alias definition for ~/.gitconfig:

[alias]
  wibble = "!set -x\necho $#\nfor x in \"$@\"; do echo \": {$x}\"; done #"
  wobble = "!for x in \"$@\"; do echo \": {$x}\"; done #"

Here, the git configuration file parsing has resulted in the list of aliases containing an entry for wibble and one for wobble, where the stored strings are:

!set -x
echo $#
for x in "$@"; do echo ": {$x}"; done #

and the same but without the first two lines.

We can invoke git wibble:

% git wibble
+ echo 0
0
% git wibble foo 'bar baz'
+ echo 2
2
+ for x in '"$@"'
+ echo ': {foo}'
: {foo}
+ for x in '"$@"'
+ echo ': {bar baz}'
: {bar baz}

Those two invocations using git wobble instead (to make this a little more condensed and easier to scan) would have resulted in these argv arrays being processed (using single-quotes for strings to avoid introducing backslashes):

first = [
  'sh',
  '-c',
  'for x in "$@"; do echo ": {$x}"; done #',
  'for x in "$@"; do echo ": {$x}"; done #',  # ignored, shell $0
  NULL,
  ]

second = [
  'sh',
  '-c',
  'for x in "$@"; do echo ": {$x}"; done # "$@"',
  'for x in "$@"; do echo ": {$x}"; done #',  # ignored, shell $0
  'foo',      # $1
  'bar baz',  # $2
  NULL,
  ]

Credential Helpers

The AWS documentation currently up at https://docs.aws.amazon.com/codecommit/latest/userguide/setting-up-https-unixes.html has you run:

git config --global credential.helper '!aws codecommit credential-helper $@'

I’ve not used CodeCommit in a while and am not setting it up again now to confirm, but I believe this is wrong. The same mechanisms are in play for how git invokes commands, (except that without an exclamation mark, if not given a complete path, the command tried for "foo" will be "git credential-foo") so you’ll end up with git invoking:

['sh', '-c',
 'aws codecommit credential-helper $@ "$@"',
 'aws codecommit credential-helper $@',  # ignored, shell $0
 'get']

and the shell then invoking:

['aws', 'codecommit', 'credential-helper', 'get', 'get']

Clearly the aws codecommit credential-helper is ignoring extraneous parameters, and relying upon the available sub-commands being any of ["get", "store", "erase"], none of which contain whitespace, so $@ degenerating to $* here is harmless.

Conclusion

It’s easier than expected to have arbitrary shell in a git alias, you just need to know about the undocumented implicit sometimes-added "$@". That still doesn’t mean you should do so.

-The Grumpy Troll

Alpine Linux on ARM (Turris Router)

Phil Pennock — Sun, 18 Mar 2018 13:25:00 -0400

My home router is a Turris Omnia, which provides the option for running LXC containers; I use this for SSH jumphosts and other such things as belong “on the router itself”.

Last night I decided that it was time to install an Alpine Linux container, to complement the Debian container which has been predominantly used to date. This presented a few issues, but all was done. In this post: networking from no-network, CIDR (classless) routes accepted over DHCP, and other quirks.

Apologies in advance for use of cat straight to a terminal; busybox’s cat(1) does not support -v and I’m choosing to trust the state of the OS at the point in time where I’m running these commands, such that I consider the risk acceptable.

The version presented here is cleaned up, with debugging repetitions removed, showing only what I believe to be the steps needed.

Getting Started

I live in Pittsburgh, PA. The OS is “Alpine”, the nearest semi-famous mountain is Mount Washington (popular with tourists) so we have a container name: washington.

Create the container, I happened to use the Web UI here, as it presented a convenient drop-down of options. The interface is through LuCI, the current web front-end to the UCI configuration system. Thus for me, the URL was https://turris.lan/cgi-bin/luci/admin/services/lxc – I mention this so as to note in passing that one of the nice things about the Turris was how easy it was to install a key/cert signed by my personal Certificate Authority.

I chose Alpine 3.7. I did not start it. This was the end of the use of the web interface: CLI all the way from here. With logbooks, so that if I ever need to automate this then I have a playbook to use for building the configuration management system rules. From logbooks of previous installs, I see that I could have avoided the Web UI here too with:

lxc-create -t download -n x -- -l
lxc-create -t download -n washington -- -d Alpine -r 3.7 -a armv7l

Logged into the turris system as root, the next step is some basic configuration:

grep '^lxc\.network\.hwaddr' /srv/lxc/washington/config
cat /srv/lxc/washington/rootfs/etc/hostname
echo washington > /srv/lxc/washington/rootfs/etc/hostname

I used the MAC address from the first line to populate the Single Source of Truth used for generating my home DHCP and DNS server configs; this is easier if you use whatever’s on the Turris, I forget the details, but I disabled much of that stuff because I already have ISC dhcpd and Unbound running at home.

Next, the basic configuration to get running:

lxc-start -n washington
uci show lxc-auto
uci add lxc-auto container
uci set lxc-auto.@container[-1].name=washington
uci set lxc-auto.@container[-1].timeout=30
uci show lxc-auto
uci commit lxc-auto
lxc-attach -n washington

At this point, assuming lxc-auto is enabled for system boot, the container should start automatically on next boot; I seem to recall that this took much poking and prodding when I first set up a container on this host, because the usual LXC auto-start stuff all exists but is unused, with UCI instead being the only working mechanism.

Networking

Our next problem is that we have no networking, no /etc/network/interfaces, very little in the way of networking tools, and very little installed:

# apk info | sort | xargs
alpine-baselayout alpine-keys apk-tools busybox libc-utils
  libressl2.6-libcrypto libressl2.6-libssl musl musl-utils
  scanelf zlib
[ manual linebreaks added ]

While https://wiki.alpinelinux.org/wiki/Configure_Networking provides some guidance, it still assumes more of a system than we have. Why are things so bare? https://doc.turris.cz/doc/en/public/lxc_alpine explains that the original package source Turris used dropped all their armhf packages without notice, so Turris switched to the minimal OS images directly from Alpine, which unfortunately are intended for use in other scenarios. So, manual IP configuration and route-table manipulation it is. Adding in the steps we need from both sources and adapting for being inside a container, we have this; I’ve changed some IP details, so we’re assuming that our IP (per DHCP reservation) is 192.168.1.201 in a /24 network with DNS servers on the .10 and .11 IPs and default gateway (the Turris itself) on the .1 IP.

cat /etc/hosts
printf '::1\t\tlocalhost ipv6-localhost ipv6-loopback\nfe00::0\t\tipv6-localnet\n' >> /etc/hosts
for x in 0:mcastprefix 1:allnodes 2:allrouters 3:allhosts; do printf 'ff02::%s\t\tipv6-%s\n' ${x%:*} ${x#*:} ; done >> /etc/hosts
cat /etc/hosts

(printf 'auto lo eth0\n';
 printf 'iface lo inet%s loopback\n' '' 6;
 printf 'iface eth0 inet%s %s\n' 6 manual '' dhcp;
 printf '\tudhcpc_opts -O staticroutes\n' ) \
 > /etc/network/interfaces
mount -t proc proc /proc
ifconfig eth0 192.168.1.201 netmask 255.255.255.0 up
route add default gw 192.168.1.1
( printf 'domain lan\n';
  printf 'nameserver 192.168.1.%s\n' 10 11;
  printf 'options timeout 1 attempts 1 rotate\n') \
  > /etc/resolv.conf

apk update
apk upgrade
apk add busybox-initscripts
rc-update add networking && rc-update add bootmisc boot && rc-update add syslog boot
apk add zsh vim curl git acl attr iputils rsync
reboot

As is normal for containers, reboot merely restarts the container, not the outside OS. Above I enable IPv6 and IPv4. Note that Alpine does not appear to support NDP for IP configuration. Note too that for IPv4, I’ve used dhcpc_opts -O staticroutes as an option line in /etc/network/interfaces. I’ve confirmed that this adds to the options on the command-line, instead of replacing them. This is how we tell the Alpine DHCP client that we want to request an extra option, DHCP option 121 per RFC 3442. At this point, the option will be requested and details made available in environment variables to the configuration scripts, but nothing else done.

We can then look at less /var/log/messages and see a bunch of spam filling the logs quickly because /etc/inittab is configured to request getty processes listening on TTYs which don’t exist. vi /etc/inittab and remove the tty5 and tty6 entries. I also commented out 2, 3 and 4: we don’t need these things. One getty should be enough: in fact, it’s probably more than enough, since lxc-attach -n CONTAINER does not use a serial line to connect in. We can comment out all of the getty lines quite safely and reclaim a little RAM.

To debug what was needed for udhcpc, I read /usr/share/udhcpc/default.script to see how things fit together. Then, taking advantage of “not yet permitting remote login” to do unsafe FS-race-attack-prone things in /tmp, I just ran:

mkdir /etc/udhcpc
echo 'env > /tmp/troll.$$' > /etc/udhcpc/udhcpc.conf
service networking restart
rm /etc/udhcpc/udhcpc.conf

This yields a file in /tmp for each invocation of the udhcpc script, letting us see exactly which variables are exposed each time, and at what stages we’re called. At this point, there was much invocation of strings upon binaries and other debugging techniques to get to where I figured out the option name required: it has been renamed at least once, many search results online are outdated. What I did get from search-fu was that dhcpc_opts is the relevant option in /etc/network/interfaces.

Time to create some shell scripts to manage bringing up the CIDR routes. Here’s what I created and the scripts I wrote; please note, that renew and deconfig have not been heavily tested. I know that bringing the interface up works well enough.

Of note: DHCP option 121, aka cidr-routes, aka rfc3442-classless-static-routes, is required to include all routes, including the default route, as clients should ignore the normal route option in the presence of DHCP option 121.

(This is in contrast to ms-cidr-routes aka ms-classless-static-routes aka private option space usage of code 249 for Microsoft platforms: although it’s documented in MSDN as differing only in option code, various resources online claim that the default route should be omitted from that option. Fun.)

In my code, I do the opposite: I leave the route from DHCP option 3 (RFC 2132) in place, and skip any default route found in the DHCP option 121. This removes a window of turning down a freshly turned up interface, or a bunch of other complexity to handle this. In practice, if your default route differs between option 3 and option 121, then you’re crazy and on your own.

cd /etc/udhcpc
for D in post-bound post-renew pre-deconfig; do
  touch $D/cidr-routes
  chmod 755 $D/cidr-routes
  vi $D/cidr-routes
done

`/etc/udhcpc/post-bound/cidr-routes`

#!/bin/sh
statedir=/var/lib/udhcpc
togglefile="$statedir/cidr-routes"
cachefile="$statedir/latest.staticroutes"
[ -n "${RETURN_AFTER_SETTINGS}" ] && return 0
[ -n "${staticroutes:-}" ] || exit 0
[ -d "$statedir" ] || mkdir "$statedir"

printf > "$cachefile" '%s\n' "$staticroutes"

# the sed relies upon a GNU extension, which is available (addr,+N)
case $staticroutes in
  *\ 0.0.0.0/0\ *) use_routes="$(echo $staticroutes | xargs -n 1 | sed '/^0\.0\.0\.0\/0$/,+1d' | xargs)" ;;
  *) use_routes="$staticroutes" ;;
esac

{
  echo '#!/bin/sh'
  echo 'enable() {'
  printf '  ip route add %s via %s\n' $use_routes
  echo '}'
  echo 'disable() {'
  printf '  ip route del %s via %s\n' $use_routes
  echo '}'
  cat <<'EOTAIL'
case "${1:-missing}" in
  missing) ;;
  enable) enable ;;
  disable) disable ;;
  *) echo >&2 'bad arg'; exit 1 ;;
esac
EOTAIL
} > "$togglefile"
chmod 0755 "$togglefile"

exec "$togglefile" enable

`/etc/udhcpc/post-renew/cidr-routes`

#!/bin/sh
[ -n "${staticroutes:-}" ] || exit 0
RETURN_AFTER_SETTINGS=t
. /etc/udhcpc/post-bound/cidr-routes
unset RETURN_AFTER_SETTINGS

[ -f "$cachefile" ] || exec /etc/udhcpc/post-bound/cidr-routes
previous="$(cat "$cachefile")"
if [ "$staticroutes" = "$previous" ]; then
  # don't force back on; if disabled, leave disabled
  exit 0
fi

"$togglefile" disable
exec /etc/udhcpc/post-bound/cidr-routes

`/etc/udhcpc/pre-deconfig/cidr-routes`

#!/bin/sh
RETURN_AFTER_SETTINGS=t
. /etc/udhcpc/post-bound/cidr-routes
unset RETURN_AFTER_SETTINGS

[ -n "$togglefile" ] || exit 0
exec "$togglefile" disable

Security Setup

Time for CA certificate configuration, SSH, and so forth. I don’t enable DSA, or RSA, and don’t want to have those sitting around on disk; I also don’t like components auto-creating SSH hostkeys if they think they’re missing: if it gets used, it encourages sloppiness around hostkey verification. If the files disappear, it’s time to use console and restore from backups, or create new ones and distribute the new public keys via secure mechanisms.

PKIX, SSL/TLS

The directory /usr/local/share/ca-certificates exists, is empty, and any .crt files will be picked up by update-ca-certificates and trusted immediately. Note though that the symlink created in /etc/ssl/certs/ has ca-cert- prepended to the filename (of the symlink itself), which was an unexpected difference and caused a few moments of head-scratching as I wondered why my certs hadn’t been picked up.

Before making changes, I used curl to verify that certs issued by public CAs were working, and that those issued by my CA were resulting in failure. I pasted in the certificate of my household Certificate Authority, then ran update-ca-certificates. I then re-ran the curl tests and All Was Good.

OpenSSH

I tend to stick to OpenSSH at this time. Note though that OpenSSH 7.5, as included in Alpine 3.7, has a bug where use of -o VerifyHostKeyDNS=ask (or anything other than =no) will cause a segfault for any host missing SSHFP records in DNS. As far as I can tell, this is entirely an upstream bug and the fix is to upgrade to OpenSSH 7.6. I just left it with this known bug in place.

I also tend to place all authorized keys files into /etc/ssh/userauth/USERNAME instead of ~/.ssh/authorized_keys, as this is more amenable both to central distribution and to using systems such as etckeeper to detect and record changes in how authentication can happen.

apk add openssh
printf '\n\nSSHD_DISABLE_KEYGEN=yes\n' >> /etc/conf.d/sshd
for want in ed25519 ecdsa; do
  ssh-keygen -t $want -f /etc/ssh/ssh_host_${want}_key -N '' -C washington.lan ;
done
tail -n +0 /etc/ssh/ssh_host_*_key.pub
KEYDIR=/etc/ssh/userauth; mkdir $KEYDIR; vi $KEYDIR/root $KEYDIR/troll
vi /etc/ssh/sshd_config

Changes to sshd_config:

# uncomment *only* ECDSA and Ed25519 keys
# Double-check that "PermitRootLogin prohibit-password" is the default, else
# set it.
LogLevel VERBOSE
AuthorizedKeysFile      /etc/ssh/userauth/%u
PasswordAuthentication no
ChallengeResponseAuthentication no
AcceptEnv WINDOW TMUX_PANE TZ ITERM_PROFILE COLORFGBG
UseDNS no
AllowUsers *@127.0.0.1 *@::1
AllowUsers root@192.0.2.7 root@198.51.100.49
AllowUsers troll@192.0.2.7 troll@198.51.100.49

Then basic user setup. I couldn’t log in as the troll user at first; after running /usr/sbin/sshd -ddd -p 24 and connecting to that, I saw the permission denied errors trying to access the file; a little more investigation led to this gem:

# ls -ld /
drwx------    1 root     root           108 Mar 18 09:46 /

I have no idea what led to that … “most peculiar setting”.

rc-update add sshd
rc-status
service sshd start
addgroup -g 1000 human
adduser -h /home/troll -g 'Grumpy Troll' -s /bin/zsh -u 3000 -G human troll
  # hit enter a couple of times for password
vi /etc/shadow
  # on the line for `troll`, replace the second field with just a single '*'
chmod 0755 /

Fin

To get an openssl binary, use apk add libressl
The tput command is in ncurses, but I reworked the personal config files git repo (not covered above) to check for existence of the tput command and fallback to an assumption of ANSI escape sequences for color, which seemed eminently sane. If a TTY doesn’t support the ANSI sequences, then the system should have a curses system and working tput installed. If every TTY does, let’s default to sane strings instead of re-deriving them on startup.
https://turris.lan/cgi-bin/luci/admin/network/firewall/forwards to enable a port-forward from the WAN on a non-standard port, to the IP of the container on port 22
- Edit /etc/ssh/sshd_config inside the container to change AllowUsers accordingly.

That’s about it. The complexity was all in the debugging to find the key pieces of information needed.

-The Grumpy Troll

boot2docker xhyve DNS

Phil Pennock — Tue, 14 Nov 2017 15:30:00 -0500

Using macOS with Docker can be “interesting”. When I got started, I followed the useful advice at https://pilsniak.com/how-to-install-docker-on-mac-os-using-brew/. This approach appealed to me, especially the use of xhyve. Because sometimes I just make life difficult for myself.

Thus my initial setup was:

brew install docker docker-machine xhyve docker-machine-driver-xhyve
f=/usr/local/opt/docker-machine-driver-xhyve/bin/docker-machine-driver-xhyve
sudo chown root:wheel $f; sudo chmod u+s $f
  # because yay, more setuid root binaries; it's written in Go, which is
  # something at least.

docker-machine create default --driver xhyve --xhyve-experimental-nfs-share
eval "$(docker-machine env default)"

tmutil addexclusion $HOME/.docker ; sudo tmutil addexclusion -p $HOME/.docker

(Warning: that docker-machine create line will fail for releases of the xhyve driver after v0.3.3; you’ll need to add an area to be shared to the end of the command-line; use /Users to match previous behavior.)

Excluding the Docker VM area from regular backups is a useful addition. Note that the usual docker-machine eval guidance from the maintainers is wrong: it will result in exporting an empty environment variable named export. Wrap the command-substitution in double-quotes to preserve newlines. This is why decent tools emitting export commands for eval will insert semi-colons at the end of each line, and avoid comments: to make life easier for sloppy command-line usage and missing quotes.

This gives us a VM running the fairly minimal OS named boot2docker, which runs Docker and exposes that to Docker on the host macOS via some environment variables pointing the client tool at the VM instead of a local socket.

The biggest problem I had was that my DNS server as seen by the VM kept getting reset to be the IP of my laptop on that virtual network, and my laptop’s DNS resolver was not set to be exposed over the network. I’m reluctant to do so, because I do attend conferences and use untrusted WiFi, but exposing the DNS server to only also listen to an arbitrary list of dynamically created virtual networks, but not to the WiFi, is a fragile pain I don’t want. So the laptop is not itself a suitable DNS server for the VM running boot2docker.

Alas, the docker-machine-driver-xhyve driver’s DHCP server simply doesn’t set a DNS field and ignores whatever’s in the config elsewhere, so the VMs end up with the IP address of the server being used as the DNS server IP too. Always.

In addition, the latest tagged version of that driver, 0.3.3, predates a fix from August wherein that driver was also stomping over bootlocal.sh on each boot, removing the ability of users to work around the first issue. My grateful thanks to Hugues Alary for committing the fix, which I found in git when I went looking to figure out why boot2local.sh changes weren’t persisting.

Really, the xhyve driver is neat, but not yet ready for prime time. In fairness, it does call itself version 0.3.3. That’s pretty clear. So let’s show how to stick with it.

docker-machine stop
brew unlink docker-machine-driver-xhyve
brew install --HEAD docker-machine-driver-xhyve
f=/usr/local/opt/docker-machine-driver-xhyve/bin/docker-machine-driver-xhyve
sudo chown root:wheel $f; sudo chmod u+s $f

sudo true
  # to get the password in the cache, because I/O is messed up through docker-machine

docker-machine start
# json: cannot unmarshal bool into Go struct field Driver.Virtio9p of type []string

Ah, JSON config changes. I have Docker images I really don’t want to have to reload. So instead of nuking the machine and recreating, we study commit dff9c59d of the driver, bring up a temporary second machine to have a template to compare against, and then:

C="$HOME/.docker/machine/machines/default/config.json"
jq < "$C" > "$C.new" '
  .Driver.Virtio9p = null |
  .Driver.NFSShares = [.Driver.Virtio9pFolder] |
  .Driver.NFSShare = false |
  .Driver.NFSSharesRoot = "/xhyve-nfsshares" |
  .Driver.Virtio9pRoot = "/xhyve-virtio9p" |
  del(.Driver.Virtio9pFolder)
  '
ls -ld "$C" "$C.new"
chmod 0600 "$C.new"
mv "$C.new" "$C"

That should get us a bootable VM. At this point, we can once more boot boot2docker, and bootlocal.sh won’t be stomped over on each boot (but has cruft left in it).

sudo true
docker-machine start

docker-machine ssh
  sudo -Hi
    cd /var/lib/boot2docker
    cp /usr/share/udhcpc/default.script ./dhcp.script
    vi dhcp.script
        # near top, after other checks but before the 'case':
      [ -f /var/lib/boot2docker/dns-servers ] && dns="$(cat /var/lib/boot2docker/dns-servers)"
    vi dns-servers
        # whatever IPs you want, separated by whitespace; no comments allowed
        # for Google's Public DNS servers:
      8.8.4.4 8.8.8.8
    vi bootlocal.sh
        # everything in this script is now junk, left behind by old versions of
        # the xhyve driver, so bring it down to:
      #!/bin/sh
      sed -i -e 's,/sbin/udhcpc,& -s /var/lib/boot2docker/dhcp.script,' /etc/rc.d/dhcp.sh
      # and to trigger one run now, because we're too late
      (
        DEVICE=eth0
        /sbin/udhcpc -s /var/lib/boot2docker/dhcp.script -b -i $DEVICE -x hostname:$(/bin/hostname) -p /var/run/udhcpc.$DEVICE.pid
      )
  exit; exit

docker-machine stop
sudo true
docker-machine start

And lo, working DNS.

-The Grumpy Troll

Golang SSH Redux

Phil Pennock — Tue, 25 Apr 2017 20:20:00 -0400

I’d like to set a couple of things straight, for the record.

I’ll cover the post/blog, and then I’d like to counter some misconceptions. While part of me thinks “I must’ve been very unclear to have so many people misunderstand”, I also saw how many people commented without bothering to read, so really there’s a limit to how much self-flagellation will happen.

I am not a security researcher. I do not try to get bug bounties. I normally care about things like CVEs when I’m coordinating the Exim project’s response to some problem and talking to CERTs, packagers, etc. I’m thus either on the receiving side of the CVE report or requesting a CVE for software for which I am somehow responsible.

The Post

In late March, while working on a project for a client, I encountered something surprising in the state of the world around Golang (the programming language) and its SSH support (Secure SHell, for secure connections to remote computers by limited sets of authorized users, often with the goal of running arbitrary commands on those computers: so remote access, rather than remote retrieval of resources).

On April 2nd, I wrote about what happened. On April 3rd, I added the CVE number when it was assigned. On April 4th, I started a Twitter poll and linked to it. The poll ran for the maximum time allowed (10 days). After it was over, I updated the blog post again with the results. Like most of my blog posts, a couple of friends read it and commented but nothing more came of the issue.

Then on April 15th, out of the blue, a former colleague tagged me in a message with “your article is tops on HN, in the unlikely event you were not already aware”.

I was not aware. I had received one message earlier that day from someone I vaguely knew, and who had read the post and had some more concerns about their own experiences with the security handling of a certain vendor. I didn’t know there was anything else happening.

In fact, the post went to the top of Hacker News, then also appeared on Lobsters, Slashdot, two different /r/netsec posts on Reddit and all sorts of other places. (Yes, this Grumpy Troll goes by syscomet in a few fora).

This blog is hosted in an AWS S3 bucket, with AWS CloudFront in front of it. That setup handled the load without blinking; it would handle a far higher load than anything thrown at it that weekend. Total added cost to me for around 64k post views, 608k HTTP requests and 21GB of traffic has been just under $3. That I don’t need to care that this was over one weekend instead of spread over the month is an advantage of using infrastructure built to scale for giants and riding along as a minnow.

Also: I need to get around to using HTML Subresource Integrity checksums and offloading serving jQuery and some CSS stuff to common hosting. That would have saved me most of that $3, while speeding access for most people and with a “probably acceptable” privacy trade-off for resources which are “probably loaded anyway, and use a cryptographic checksum to avoid even having to leak that we might need to load it again”.

In the post, I aimed for my usual level of dry humor, for which folks should be able to see a warning in the sidebar. Expeditions are still failing to report.

Golang

The Go Programming Language never had an insecure SSH library.

What they had was a library with an API prone to misuse. There was never a documentation flaw. The requirement for secure use was always documented.

I know this, because I read the API docs and clearly understood that it was on me to implement hostkey verification. After some brief playing around to sketch out what this would look like, I decided to search to see if there were common debugged libraries which already handled the corner cases, and so looked to see what other projects were doing. Not Invented Here (NIH) is a real problem and I try to avoid it.

At this point, I discovered how few people (none which I saw) were bothering to implement the hostkey verification. They got working connections, they never checked that things failed when they should fail.

The handling of this issue by the Go maintainers was exemplary. They had no security problem but accepted that so many people misusing the library was a security problem and that they could do something about this. They changed the library defaults so as to force programmers to make a decision about how to handle verification. Instead of naive use leading to code which doesn’t fail when it should, naive use now leads to code which doesn’t work until programmers decide how to handle ensuring their application fails when it should.

Further, the Go maintainers have continued to improve the available facilities in this area, with features such as a new sub-package knownhosts to make it easier to Do What OpenSSH Does.

The CVE added to the blog-post, CVE-2017-3204 is an identifier for the issue, so that other programmers fixing their library-using code have a common identifier to use in describing what they’re responding to, and communicating this to their customers and users and having folks able to clearly link things together instead of having to puzzle and decode which set of security issues some release is claiming to be addressing. This is, to my understanding, part of the point of the CVE system.

There was one tiny snafu in reporting the issue, which I described without naming names, in how PGP was handled. The person who had asked me to resend my email without PGP provided the following within a comment on the HN post:

99% of the PGP-encrypted emails we get to security@golang.org are bogus security reports. Whereas “cleartext” security reports are only about 5-10% bogus. Getting a PGP-encrypted email to security@golang.org has basically become a reliable signal that the report is going to be bogus, so I stopped caring about spending the 5 minutes decrypting the damn thing (logging in to the key server to get the key, remembering how to use gpg).

That is a somewhat depressing indictment upon the state of affairs with use of secure communications tool. I can’t fault him for anything he said. I did get a laugh out of:

I’d say it was worth it. This thread was more fun than using gpg, even if the time savings is a wash.

Context: I have patches in GnuPG, in SKS (the keyserver software) and am somewhat heavily involved in the PGP keyserver community (wrote the setup guide which everyone uses today, etc). I use GnuPG and it’s (1) better than the alternatives; (2) high quality cryptographic engineering; (3) not created by UX professionals.

If I’d known how bad the stats were for PGP mail to security addresses, I wouldn’t have bothered. Instead, I’ve now pinned golang.org in my mail-server as a domain which requires validated TLS when sending mail to the MX.

The Vendor

The vendor I mentioned in the original blog-post stepped into the fray and described their view. Please remember that this is not the only vendor with a security problem here. I found no examples in my searches at the time of anything doing this right. I picked one vendor of popular devops tooling, whose work I respected, and reported it to them. They’re already far better than most, or I wouldn’t have been recommending their tools to start with.

I’ll summarize the vendor’s response on HN as “we didn’t communicate clearly; we said some things but changed our minds and didn’t tell the person reporting; we have better ways to handle SSH and have always said to use them”.

I was pleased that they repeatedly noted that I was accurate in my portrayal of their response.

Their stance about the impact of the problem has improved.

About Vault, they wrote:

With the addition of the ability to generate SSH certificates (which was on our roadmap for a long time and added in 0.7, prior to both the original report and the blog post).

About Packer, they describe that the availability of SSH Certificates makes it less of an issue.

I know quite a few people using Packer. None are willing to say that they’re using SSH certificate generation in image generation. There’s a better way available but it’s not used by default. I emphasize this point because this is the same core issue, reframed, which was in the Golang package: it was possible to use it correctly, but almost nobody did so and it wasn’t the easy path.

One small aside: I was an attendee at the vendor’s first conference when the feature of Vault handling SSH for users was introduced. At the conference party, I grabbed the author of the feature and probably edged across the line into rudeness as I described why what they had was not something I’d risk deploying and pointed him at SSH Certificates as a saner path forward without the same operational failure modes and a much smaller attack surface for those who want that.

So I’m glad that the vendor put SSH certificates on their roadmap after I explained what they were and why their first pass implementation was problematic. It’s good when people listen, put aside the problems incurred by my forgetting about human social graces, and work to improve things anyway. That Vault can use SSH Certificates for host and user management is a good thing. It’s not a replacement for hostkey verification unless and until it’s almost as easy to operationally deploy as is non-CA TOFU SSH where clients ignore the “F” in “TOFU”.

Other Thoughts

The quality of the cryptography in Golang is better than in most other languages. I’ve read the source to a few TLS libraries in my time and Golang’s is by far the nicest, with good encapsulation via crypto/subtle of issues such as key-variant timing protection.

Golang isn’t perfect and is sometimes frustrating, but pragmatically it’s the only language I’ll use today for a variety of systems programming and security sensitive tasks. There are a number of other options in this space too (eg, Rust) and they’re all gaining some traction. We finally have a realistic shot at migrating a lot of security-boundary services on Unix-heritage systems to code written in languages not particularly prone to buffer overflow attacks.

Things have reached the point where I can take a stance and decline to deploy on my own systems any new Internet facing servers written in C. Some of those which exist now will remain for a while; servers from groups with an excellent track record (OpenBSD) might cause me to make an exception, based upon their reputation.

Sure, there are more security problems than buffer overflows. But you know what? When systemically purging that one class of flaw gets rid of at least 95% of the remote code execution flaws, I’ll take it.

I look at two approaches to programming language security. On the one hand, the Golang maintainers have a type-safe language immune from buffer overruns, with text templating systems designed to auto-escape data appropriately to avoid various web-based attacks (JS injections etc) and where an API which is being heavily misused gets changed to force folks to confront their problem space more fully. On the other hand, we have C compiler maintainers who assert that any behavior not defined in the C standard is not something where they have to pick one behavior and stick with it, but can instead declare it an impossible situation, that code which relies upon common assumptions is buggy, and proceed to optimize away the security checks.

If I program in C, I need to defend against the compiler maintainers.
If I program in Go, the language maintainers defend me from my mistakes.

If you’re choosing software to deploy on your systems, which language would you rather that the vendors be writing in?

2024-12: A server-side vulnerability in misuse of the `PublicKeyCallback` API has echoes of this issue:

The Grumpy Troll: The Grumpy Troll

(Go)Hugo Generated Text File

Motivation: robots.txt

Hugo Textfile Setup

My robots.txt template and data

Configuration Objects

Kubernetes

OCI Image References

Summary

Other posts

Tailscale and Docker Remote

No TLS??

Appearance to an end user

The Deployment

The setup once written

The Multiple Ansible Invocations Without Pre-Auth Keys

The systemd service file template

The ACLs

The Icing on the Cake

Feedback

ACLs include ports, use daemon.json

Small Mailserver Best Current Practices

Deliverability fixes

Convenience Stuff

Addenda

Shell Locality

The Test

The Results

Zsh

Bash (5)

Dash

Ksh

FreeBSD sh

Conclusion

Slack Tax

Git Aliases & Shell

git config documentation

Shell side-digression

What actually happens with git

What this means for us

Credential Helpers

Conclusion

Alpine Linux on ARM (Turris Router)

Getting Started

Networking

/etc/udhcpc/post-bound/cidr-routes

/etc/udhcpc/post-renew/cidr-routes

/etc/udhcpc/pre-deconfig/cidr-routes

Security Setup

PKIX, SSL/TLS

OpenSSH

Fin

boot2docker xhyve DNS

Golang SSH Redux

The Post

Golang

The Vendor

Other Thoughts

Motivation: `robots.txt`

My `robots.txt` template and data

`git config` documentation

`/etc/udhcpc/post-bound/cidr-routes`

`/etc/udhcpc/post-renew/cidr-routes`

`/etc/udhcpc/pre-deconfig/cidr-routes`