2026-02-19 20:48:00
Two weeks ago I was hopeful of contributing to Uptime Kuma, by making a new notification provider.
Yesterday that hope became reality, with my pull request being accepted and the feature going live in version 2.1.2. :D
From now on, UK can send SMS alerts via Teltonika RUT devices with the SMS gateway software.
I feel pretty great about this! I've never worked with Javascript before, so at times it felt like banging rocks together to make something happen. But it did! I'm very grateful to Frank Elsinga for all his help in reviewing and improving my code.
I can only applaud both Uptime Kuma and Teltonika for their thorough documentation! With UK, their guidance for contributing to the project is excellent! Their codebase is also very clean, which makes it easy to understand for a novice programmer. It took me a little while to find Teltonika's OpenAPI docu, but once I got there it was excellent! Between that and some Burp Suite traffic interception I soon had a good idea on how to make it work.
Just now I've upgraded my UK instance at the office and I'm now using my own code to get SMS alerts if some of my business infra goes down again. #FeelsGoodMan!
kilala.nl tags: work, programming, sysadmin,
View or add comments (curr. 0)
2026-02-13 07:51:00
I'll be submitting a CFP for NLUUG and WICCON this year, called "I'm self-hosting email and the world did not end." In this talk I will definitely cover my chase for HA: high availability.
Because so far:
And it's all been tested, because new things keep going wrong. :)
Like this morning! When my Ubiquiti switch/router did a software upgrade.
Because both nodes, and the quorum device, are all connected to the same switch they decided the cluster was unhealthy and did a reboot. Both nodes did a reboot, but one of them didn't come up because I've set an UEFI boot password. The other one was back within one minute and took all VMs for itself.
I know I can attain the absolute high availability that big companies need, because right now I can't have dual Internet connections for incoming email. I have a fallback line for outgoing mail, but that's on 4G/5G.
But I've figured out a few more things to arrange:
Anyway...
A lot of people always say to not self-host business critical email. So far it's been very doable and I'm learning a lot, but the returns are definitely not there. I could've paid TransIP or Procolix an annual fee, or I could've stuck with MS365, and have paid much less money. But I'm stubborn, I want to learn and I want to hold all of my data myself.
kilala.nl tags: sysadmin, work,
View or add comments (curr. 0)
2026-02-09 15:37:00
It's like I was setting myself up to fail, in January, when I remarked that I should look into monitoring and alerting for my company's network. Well, as things go when you're putting them off: what had been foreshadowed, occured!
Last night I noticed I couldn't receive emails on my laptop. Despite iOS Mail (on my phone) not complaining at all, it seems that the mail server was completely unreachable. ... weird, that the Mail app doesn't complain about that!
Anyway, one of my Proxmox nodes had fallen prey to the e1000 stability issues I'd tackled before. Turns out I forgot to reapply that fix when I rebuilt the cluster. Doh!
So now I do have monitoring and alerting setup in my network.
I've deployed Uptime Kuma via Docker Compose, on a tertiary system which will simply monitor all systems and services in my network. If something goes down, I'll know within minutes! And the alert will go through an external email system, not my own.
I'm also hoping to write a notification extension for Uptime Kuma, so it can use my Teltonika RUT241 SMS Gateway.
kilala.nl tags: work, sysadmin,
View or add comments (curr. 0)
2026-01-28 17:41:00
Remember how I setup a nice Systemd timer+service, to automate certificate renewal and rotation? Works wonderfully!
Until it doesn't. :D
I had totally forgotten to set up monitoring, or alerting, just in case the renewal went wrong. And it did! Not because of the timer, not because I broke things, but because the TransIP API had changed some of its workings and I was running too old a copy of acme.sh. =_= Way to go, kid.
So, this time I've adjusted the Systemd service to include an ExecStartPre command which pulls the latest version of acme.sh.
And I need to find a way to alert myself in case things break.
kilala.nl tags: work, sysadmin,
View or add comments (curr. 0)
2026-01-26 21:55:00
I've had my M2-based Macbook Air for a while now. Unfortunately I went for the smaller storage, at 256GB. Generally speaking, that's good enough for me.
However. I've noticed that MacOS, since 14.x but including 15.x and now 26.x is doing something really fucky with my storage space, if you'll excuse the technical term.
DaisyDisk frequently shows me that "mutexes" are taking up huge amounts of space; double of what they're supposed to be doing. And today, while building a VM, I got a warning that my disk is full.
Where, pray tell, does the space go?
DaisyDisk tells me that out of 242GB of used space, 126GB is going to "hidden space". "Hidden", i.e. system files and Time Machine snapshots. There's no reason why my Time Machine should be taking up this much space, as I run daily backups to my NAS. So that's confusing.
But then the really weird part starts: of the 126GB "hidden space", 118GB goes to snapshots (which according to DaisyDisk actually only add up to 45GB. And another 124GB go to "still hidden".
...
What? What the what the what now?
It's time for a full wipe and reinstall, because this is BS.
EDIT:
Once again, for the third time in twenty years, has Carbon Copy Cloner come to my rescue.
Despite my drive having 1% of disk space left, CCC managed to clone the drive nicely to an external USB device. Took close to 1.5h, but I'm blaming that on the cramped source drive.
After some checking of the target device, a reinstall of MacOS took another 1 hour. Why? Because the recovery partition on my MBA will only install MacOS 15.x, not26.x. So I had to immediately do the upgrade as well. Then Migration Assistant copied my data back in under 15 minutes.
CCC is, and has been for over 20 years, the best MacOS backup and recovery tool I've worked with. Even Time Machine failed to backup the laptop to the external USB. Their blog is also very educational and I now better understand the weird disk layout MacOS uses these days.
kilala.nl tags: sysadmin,
View or add comments (curr. 0)
2026-01-19 06:45:00
I use Proxmox as the virtualization platform for my companies' IT services, like email and backups of the cloud environments. So far it's been great and a nice learning experience.
Despite having proper, full-on backups it's always felt weird to have my production email depend on a single MFF PC, employing a single storage device. So I tackled that!
Last week I took delivery of two newer MFF PCs (Dell 7020 Micro) and with a little bit of hassle I moved all of my VMs to a cluster of two 7020s, both with ZFS RAID1 across two NVMe drives.
The biggest delays in the migration and clustering were backup speeds to my NAS meaning the migration from the first, old node to one of the new ones was a long wait.
At this point I was left with two challenges:
As to the first, there's not much I can do about it right now. I do have two separate Internet connections (fibre and 4G), but for the email server to work with both of those I'll have to make some arrangements.
As to the second, Proxmox themselves have a solution for it: with a Corosync QDevice it's possible to have an additional "external vote". With two out of three votes, one cluster node plus the QDev can have "quorum" (the majority vote) and the cluster should work fine.
One of the few other always-on devices at my office is a Synology NAS. I didn't want to "just install" Corosync on it, so the next best best is containerization. Many people use this option (Corosync in a container) and most often you'll see people refer to bcleonard/proxmox-qdevice.
Bradley Leonard's containerized corosync-qnetd is obviously popular, with over 50.000 downloads. The only downsides to it are a few security issues (everything runs as root) and more importantly: the container image hasn't been updated in over a year. This means that vulnerabilities have potentially crept in there.
Doing a docker scout cves bcleonard/proxmox-qdevice confirms this:
50 vulnerabilities found in 25 packages
LOW 21
MEDIUM 18
HIGH 11
CRITICAL 0
There are many other container images with Corosync-qnetd on Docker Hub, but I don't trust any of them. You have no idea who made them and all of them also have not had updates in more than a year.
So I've built my own container image, one that does weekly updates! unixerius/proxmox-device, which is here on Github as well.
And just this morning I've noticed that there's a new, high CVE! Last time I built it, it was 37 low and 1 medium, nothing else.
39 vulnerabilities found in 15 packages
docker scout cves unixerius/proxmox-qdevice
LOW 37
MEDIUM 1
HIGH 1
CRITICAL 0
The new "high" is in Python module Jaraco, GHSA-58pv-8j8x-9vj2.
So: time to kick off a new build, once Debian have released an upgraded container parent image for Bookworm (v12) and Trixie (v13). Or I can double-check if Jaraco is even needed, so I can kick it out of the image entirely.
Speaking of hardening: I'd really want to use DHI (Docker Hardened Images) as parent images instead of the usual Debian ones. DHI is a relatively new initiative from Docker, to create more secure parent images. For now, there's two blockers:
kilala.nl tags: work, sysadmin,
View or add comments (curr. 0)
2026-01-15 22:57:00
Two weeks ago I decided on my career development plans for 2026. My first, short-term goal was to get some foundations in GRC (governance regulations and compliance) with regards to AI implementations.
Today I provisionally passed ISACA's AAISM (advanced in AI security management) certification exam.
AAISM is a relatively new certification (introduced and beta-tested last year), so people are still finding their footing. There are two solid sources for training materials: ISACA themselves and Destination Certification (DestCert). ISACA has a book, an online training and practice exams; all of them are bloody expensive. For now DestCert has a three day online training, which is even more expensive!
Udemy, endless font of mixed-quality content, has some practice exams and training materials but I really do not recommend them. I bought this set of six practice exams and rated it 2/5 stars. In short: the questions are nowhere close to what you'd expect on the exam and they are much too easy. They are also composed in such a way that you're not being tested: every question has one answer which clearly stands out as "the correct answer".
So how did I prepare?
In total I spent twenty hours on my preparations. Others on Reddit have spent quite a bit more, apparently. Could I have prepared better / more? Yes! But a pass is a pass.
I've been asked: was it worth it?
Well. No. But yes, but no.
And what about the quality of the materials?
Do I recommend AAISM? No. Only do it if you can get it fully expensed by your employer or company and if you actually are involved with AI GRC. And maybe hold off until we know if companies will value the certification.
kilala.nl tags: work, studies,
View or add comments (curr. 0)
2025-12-30 20:37:00
2025 was a year out-of-the-ordinary for me: aside from a beta exam or two, I didn't do any real certification exams or trainings. Well, I did one training (but didn't finish it yet) and one exam (and failed it). I burned out on it, by brute-forcing a training and exam which I wasn't ready for and actually didn't really feel motivated for.
Heck, I even presented about it at Wiccon this year! :)
My worries and doubts haven't fully subsided, especially where it comes to future-proofing my career. So I'm following my own advice and doing more introspection.
And I've made some study plans to go along with it!
I've also literally just now finished reading the book "ReguLEER!", which is a collection of articles and interviews on "selfregulating learning". Combining that with an e-learning on diagnostic questions, I intend to do a rigorous rewrite of my training materials for Linux+, Linux Essentials and DevSecOps Foundations.
kilala.nl tags: studies,
View or add comments (curr. 0)
All content, with exception of "borrowed" blogpost images, or unless otherwise indicated, is copyright of Tess Sluijter. The character Kilala the cat-demon is copyright of Rumiko Takahashi and used here without permission.