My custom pipeline to upgrade Docker containers with a Telegram approval

If you are building your homelab or you are dabbling with self-hosting, you might have come across the burden of keeping things up to date. I know I did.

And the thing is, I'm lazy, and I don't like burdens.

As I mentioned, I use Caprover to manage my setup, which makes things quite easy to deal with, but I still have some manual steps when it comes to upgrading containers. My setup is not Caprover dependent, up until the very last step, so you might have to find a different solution for actually deploying the new versions, but the rest of the flow can be portable.

But let's do things in order.

Discovering an updated container

To be honest, in my Synology, I just use Watchtower because the docker version is old and I run automatic upgrades (I don't have anything that would make me cry if it breaks), but it's unmaintained and it doesn't work with the latest version of Docker, making it a no go for any other environment.

So I looked around and there are a couple of decent options out there, as I'm writing: WUD (What's Up Docker) and DIUN (Docker Image Update Notifier).

The former has a GUI with a one button upgrade functionality, but it would conflict with my Caprover setup, as they would end up out of sync, and I also noticed it would exhaust the API limit to fetch the new versions quite quickly on start up. On the other hand DIUN is a CLI tool, which is focused specifically on notifying for new versions of the images.

DIUN can be run as a standalone container, but I figured that it works best in combination with DIUN Boost. I really hope its features will end up being merged in DIUN as they belong together, but TL;DR, Boost is a library that parses automatically the running versions of the containers, infers the versioning scheme, and generates a regex matching only semantically newer versions into a configuration file for DIUN to work with.

- name: image1/image1:11.16.1
notify_on:
- new
- update
metadata:
current_tag: 11.16.1
watch_repo: true
include_tags:
- ^((?:12|[2-9]\d|1[3-9]|\d{3,})\.\d+\.\d+|11\.(?:17|[2-9]\d|1[8-9]|\d{3,})\.\d+|11.16\.(?:1|[2-9]|\d{2,}))$

- name: image2/image2:6.21.2
notify_on:
- new
- update
metadata:
current_tag: 6.21.2
watch_repo: true
include_tags:
- ^((?:7|[8-9]|\d{2,})\.\d+\.\d+|6\.(?:22|[3-9]\d|2[3-9]|\d{3,})\.\d+|6.21\.(?:2|[3-9]|\d{2,}))$

- name: image3/image3:latest
notify_on:
- update
metadata:
current_tag: latest

an example of the autogenerated config dealing with different patterns of tags

Whether to use DIUN Boost or not is a personal choice, their configuration is fairly similar as Boost introduces itself as a "middleman", to takeover some of DIUN responsibilities and let it focus on the notification bit.

Either way, they both have an option to watch all the containers running – WATCHBYDEFAULT – but I preferred to ignore it and deliberately add a diun.enable label equal to true to the containers I wanted: I have some containers – for example the databases – which I know I won't be able to update automatically, and in most cases I'm not interested in updating anyway as they come with long term support.

So we have 2 containers, one (diun boost) connected to docker and on a schedule to read the containers and generate the config file, and one (diun) running a few minutes later, off the config file.

name: diun

services:
diun-boost:
container_name: diun-boost
image: harshbaldwa/diun-boost:latest
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- diun-config:/config
environment:
- DIUN_YAML_PATH=/config/config.yml
- CRON_SCHEDULE=0 */6 * * *
restart: unless-stopped

diun:
container_name: diun
image: crazymax/diun:latest
volumes:
- diun-data:/data
- diun-config:/config:ro
environment:
- "TZ=Europe/London"
- "DIUN_PROVIDERS_FILE_FILENAME=/config/config.yml"
- "DIUN_WATCH_SCHEDULE=5 */6 * * *"
restart: unless-stopped

We haven't set up the notification yet, though.

The notification

DIUN has a rich set of options when it comes to notifications, but in order to set them up, we need to understand what we want to do with the information.

I initially was quite happy with just receiving a Telegram message, but I realised I wanted something slightly more refined quite quickly after finding myself dealing with a lot of manual updates.

And that's where Windmill came back handy.

I set up a flow with a webhook trigger, and 3 steps: find the images to update, ask for confirmation, update the images.

Find the images to update

In my case this step relies on Caprover's api, and it's rather straightforward:

import * as wmill from "windmill-client"

const baseUrl = "https://captain.my.domain.example"

async function getCaproverToken() {
const password = await wmill.getVariable("f/cedmax/CAPROVER")

const res = await fetch(`${baseUrl}/api/v2/login`, {
method: "POST",
headers: { "Content-Type": "application/json", "x-namespace": "captain" },
body: JSON.stringify({ password }),
})
const data = await res.json()

return data.data?.token
}

export async function main(image: string) {
const withoutRegistry = image.replace(/^docker\.io\/(?:library\/)?/, "")
const [imageName, newTag] = withoutRegistry.split(":")

const res = await fetch(`${baseUrl}/api/v2/user/apps/appDefinitions`, {
headers: {
"Content-Type": "application/json",
"x-captain-auth": await getCaproverToken(),
"x-namespace": "captain",
},
})
const data = await res.json()

const apps = (data.data?.appDefinitions ?? []).filter((a: any) => {
const versions = a.versions ?? []
if (!versions.length) return false
const latest = versions[versions.length - 1]
return (latest.deployedImageName ?? "").startsWith(imageName + ":")
})

if (!apps.length)
throw new Error(`No CapRover app found using image: ${imageName}`)

return { appNames: apps.map((a: any) => a.appName), imageName, newTag }
}

Ask for Confirmation

This is where the magic happens

import * as wmill from "windmill-client"

export async function main(
appNames: string[],
imageName: string,
newTag: string
) {
const urls = await wmill.getResumeUrls("approver")

const botToken = await wmill.getVariable("f/cedmax/TELEGRAM_BOT_TOKEN")
const chatId = await wmill.getVariable("f/cedmax/TELEGRAM_CHAT_ID")

const appList = appNames.map((n) => `• ${n}`).join("\n")

const message = [
`🐳 Docker update available`,
``,
`Image: ${imageName}:${newTag}`,
`Affected apps:`,
appList,
``,
`✅ Deploy all | ❌ Skip`,
].join("\n")

await fetch(`https://api.telegram.org/bot${botToken}/sendMessage`, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
chat_id: chatId,
text: message,
parse_mode: "HTML",
disable_web_page_preview: true,
}),
})
}

The important thing to remember is to configure this step to suspend execution

As you probably noticed I'm still using Telegram, which makes for a nice experience too:

The arguments are actually more, but for the purpose of this I cut the screenshot

What happens when I approve

import * as wmill from "windmill-client"

const baseUrl = "https://captain.my.domain.example"

// I do log in again instead of passing the token as
// the token might expire while waiting for the approval
async function getCaproverToken() {
const password = await wmill.getVariable("f/cedmax/CAPROVER")

const res = await fetch(`${baseUrl}/api/v2/login`, {
method: "POST",
headers: { "Content-Type": "application/json", "x-namespace": "captain" },
body: JSON.stringify({ password }),
})
const data = await res.json()

return data.data?.token
}

export async function main(
appNames: string[],
imageName: string,
newTag: string
) {
const headers = {
"Content-Type": "application/json",
"x-captain-auth": await getCaproverToken(),
"x-namespace": "captain",
}

const captainDefinition = JSON.stringify({
schemaVersion: 2,
imageName: `${imageName}:${newTag}`,
})

const results = []

for (const appName of appNames) {
const res = await fetch(
// worth noting the `detached=1` to avoid the script hanging
// while waiting for the deployment to go through
`${baseUrl}/api/v2/user/apps/appData/${appName}?detached=1`,
{
method: "POST",
headers,
body: JSON.stringify({
captainDefinitionContent: captainDefinition,
gitHash: "",
}),
}
)

const data = await res.json()
results.push({
appName,
status: data.status,
deployed: `${imageName}:${newTag}`,
})
}

return results
}

Once the flow was in place, I had to add to DIUN the appropriate ENV vars to enable the webhook to run, kicking off the process whenever a new version is available

- "DIUN_NOTIF_WEBHOOK_ENDPOINT=https://windmill.domain.example/api/w/my/job/webhook"
- "DIUN_NOTIF_WEBHOOK_METHOD=POST"
- "DIUN_NOTIF_WEBHOOK_HEADERS_CONTENT-TYPE=application/json"
- "DIUN_NOTIF_WEBHOOK_HEADERS_AUTHORIZATION=Bearer windmill-bearer"
- "DIUN_NOTIF_WEBHOOK_TIMEOUT=10s"

And that's about it. Quite happy with the result so far.

I kind of buried the only "unresolved" bit in a comment in the last chunk of code: to avoid having the script hanging, the deployment is thrown in a "fire and forget" fashion, with that detached=1 parameter passed to Caprover.

Not having proper feedback on whether the deployment actually went through is not a problem for me, but I appreciate it's not ideal 🤷‍♀️