lo.calho.st

The nature of AI discourse has changed

Sat, 13 Sep 2025 00:00:00 +0000

I recently made a post in which I complained about AI and said I wouldn’t be posting about it again. That was a lie. I’m back. But this time instead of dumping my own thoughts on the subject I’m just going to be linking other blogs and articles. These aren’t all strictly about AI – some related topics are included here. Hopefully you can infer why.

Why am I doing this? A couple months ago, I felt myself under attack by a constant barrage of AI hype and blind optimism. Over the past month or two, the negative press seems to have overtaken it. This feels like a historic moment to me and I want to document it.

This is a merely collection of articles I have read or intend to read in the near future – that’s right, I have not read all of them yet. However, I have skimmed them enough to create a coarse taxonomy so I could group related topics here. I am excluding articles that may seem relevant to the thesis if I already linked to them in my previous blog.

For the purposes of this post, note that “AI” refers to generative AI, particularly LLMs.

AI Is Breaking Shit, Making Us Dumb, or Otherwise Contributing to Shit Being Broken

AI Is Evil, Hellbent on Destroying Our Careers and Lives

AI Is Stupid, Expensive, and Useless

AI Outright Sucks

n.b.: These authors have provided comprehensive lists of reasons why AI sucks. In a way, it therefore it spans all of the categories above, and then some.

Homelab updates

Sat, 19 Jul 2025 00:00:00 +0000

It’s been a couple years since my last homelab post and many things have changed.

Overview

There are more VLANs now. Retained from last time, we have:

One for IoT crap.
One for surveilance.
One for Home Assistant.
One for work.
One for the Steam Link and gaming VM.

The new additions are:

One for a media server, and
One for WAN.

The SSIDs haven’t changed from last time, and it’s all still Unifi gear (excluding the modem). I upgraded from a USG to a UDM-Pro (and got rid of the UNVR in the process), and consolidated the other infra extensively.

My new place has Ethernet in the walls, so I stuck the old 8p POE switch in the media box where all the drops terminate. All the other major stuff gets to live on the wall or floor next to it.

One exception is the modem, which is in the garage because that’s where the fiber was dropped. Fortunately there was also Ethernet back to the media box from there, so I VLAN’d the WAN and am POEing the modem.

My VM server is the only persistent piece of iron on the network now, so I was able to sell off the rackmount switches.

This is my first time trying out rackstuds and they’re actually pretty nice.

Unfortunately I wasn’t smart enough with the positioning to be able to also stick the UPS on the wall.

I still need 3 APs to get decent coverage. The UAP-AC-Pro is in that same closet, while the IW-HD is by the TV and the FlexHD is upstairs in the bedroom.

I tried AI and all I got was disappointed

Sat, 19 Jul 2025 00:00:00 +0000

This is another post that I began a long time ago and never finished: it was originally dated 2024-05-05 but I think I started working on it even earlier.

At the time, the GPT hype cycle was already well underway, so I finally gave in and decided to give AI a serious try.

Prologue: Setting up a local chatbot

I didn’t want to give OpenAI any of my money, so I set up a VM with a GPU passed through.

I originally wrote “I want to be clear that the GPU was already in the box from when I did my gaming VM; I did not have the ambition to go that far out of my way to do this.” – however when I rebuilt my server I decided to add a Tesla dedicated to this project.

Ultimately, I settled on llama-cpp-python for the backend. I tried LocalAI, too, but ultimately ditched it because:

The prebuilt image assumes you’re not running on decade-old hardware, so I got SIGILL’d
So I had to build from source, which took forever
It very badly wanted to be run in Docker
Docker images with CUDA in them are like 50 GB

It was therefore way easier to just build and install llama-cpp-python and run it as a systemd service, since I didn’t really need any of the additional features.

For the frontend, I ended up with big-AGI. I liked librechat too, but it had some UX issues (kept logging me out, didn’t like to persist model preferences between chats, etc).

I ended up sticking with Mistral-7B-instruct for most use - I tried some other Mistral variants but this one seemed to be the least likely to start hallucinating when given short prompts.

Chapter 1: AI helps me create a node.js project

Now that I had a robot to ask questions to, I embarked on a project: getting AI to help me build a web app for a random domain I purchased on a whim because it sounded cool. I am not a web developer at all, and haven’t done any web stuff since when jQuery was considered a Javascript framework, the best CSS we had was 2.1, and we all still put those “Valid XHTML 1.1” badges in our footers. Therefore, I thought this would be a good exercise: I could pretend to be a newbie and AI could help me do things I otherwise couldn’t. If it went well, then I’d know my job really is at risk.

There is no chapter 2

I spent a week really trying to get Mistral to teach me React. It failed at providing up-to-date guidance on initial setup for a project, so I ended up doing it the old fashioned way (googling). I tried to get it to produce the equivalent of a hello world page, but it couldn’t do that either. I flailed for a while longer then abandoned the project.

If you are about to say “skill issue,” just shut up. I am told time and time again that LLMs can do great things. But if I try to do something complicated and it screws up, the story is “that’s too hard, it’s better at simple tasks.” I gave it the simplest task I could think of – an ideal use case per all the discourse online – and it couldn’t handle it.

If you were gonna say “that model sucks,” also shut up. What reason do I have to expect a paid service to be better, and not just a waste of my money? If you think it’s because a newer model would be better at the job, then why were we already hyping this shit up? This has been discussed time and time again.

Believe me, I really tried

I admit my stupid experiment had limitations, so I tried using an AI coding assistant at my day job for a few months, too. We had a model trained on internal code, touted to be able to deal with the nuances of our frameworks, etc., but it also sucked. If it didn’t outright hallucinate, the code it generated usually smelled; it failed at all but basic tasks, and the IDE integration was buggy and kept crashing.

It was at least somewhat competent at generating boilerplate, but as I began to rely on it for that I could feel it making me dumb. This wasn’t just hypochondria – there is evidence of this phenomena. I uninstalled it and never looked back.

To this day, generative AI forces itself into my daily life in ways I do not consent to. I have to review slop PRs submitted by my coworkers. They’re getting worse and worse. I have to read slop summaries of my own PRs and tickets, forcibly inserted by the tools themselves. The analysis, if it is even accurate, is consistently surface level at best. It’s usually just verbosely paraphrasing the commit message. It provides no value.

You can’t google a basic question without risk of getting a hallucinated answer. Even duckduckgo has inserted slop above all the actual search results.

I do not believe in this technology

LLMs are a brute force approach to approximately accomplishing anything. I won’t rattle on about the energy concerns, because this has already been discussed. But for what, a cheap parlor trick? It fundamentally cannot reason. It cannot exercise creativity, except in the eerie way it confidently lies about things that should be fact. I’m tired of hearing about it and I no longer want to write about it, either.

Raptor Lake iGPU SR-IOV

Sat, 19 Jul 2025 00:00:00 +0000

A madman embarks on a quest

The first line of this post was originally going to be “Last week I posted about getting my wacky new server working.”

It is no longer last week, but rather last year. Yes, I started writing this in June 2024 and it is now July 2025.

This server originally had two discrete GPUs for VMs to use, but I also wanted to leverage the i9’s iGPU to handle video transcoding for my media host.

Even before this post was delayed by a year, I had decided that this would not be a guide, but rather a rant.

Will this be easy?

My original goal was to just load a driver, create a VirGL device, and use it for stuff. It wasn’t that easy. During boot, I got this cool message:

[   21.856166] xe 0000:00:02.0: Your graphics device a780 is not officially supported
               by xe driver in this kernel version. To force Xe probe,
               use xe.force_probe='a780' and i915.force_probe='!a780'
               module parameters or CONFIG_DRM_XE_FORCE_PROBE='a780' and
               CONFIG_DRM_I915_FORCE_PROBE='!a780' configuration options.

Alright, sounds easy. Let’s just try this out: modprobe xe force_probe='a780'.

Nice, a device appears in /dev/dri. Time to attach a VirGL device to a VM and boot it up.

Spoiler alert: it didn’t work

The tl;dr is that this afforded me no VA-API capabilities. I did some googling, but have now lost the links I found. If I recall correctly, I think the conclusion was that the xe driver can’t do what I want when paravirtualized. I tried using i915 too, but it won’t even load – even with the force_probe stuff.

This iGPU supports SR-IOV: some magic to make it look like multiple GPUs so each can be passed through into its own VM. Perfect, then I don’t have to worry about this paravirtualization nonsense.

The kernel doesn’t support this by default, but there’s a DKMS module. Let’s go ahead and build it!

Compromises

When I first started trying to get SR-IOV working, I was upset that the aforementioned DKMS package didn’t support the 6.8 kernel. I followed instructions in the repo to pin to an earlier one. I figured I could live with 6.5. I rebooted the system and turned on SR-IOV in the bios.

I cloned the repo and followed the steps in the readme, adjusting for my kernel version.

root@kagrenac:~# cd i915-sriov-dkms
root@kagrenac:~/i915-sriov-dkms# uname -a
Linux kagrenac 6.5.13-5-pve #1 SMP PREEMPT_DYNAMIC PMX 6.5.13-5 (2024-04-05T11:03Z) x86_64 GNU/Linux
root@kagrenac:~/i915-sriov-dkms# vi dkms.conf # change the package name and version
root@kagrenac:~/i915-sriov-dkms# cd ..
root@kagrenac:~# mv i915-sriov-dkms /usr/src/i915-sriov-dkms-6.5.13-5-pve
root@kagrenac:~# apt install proxmox-headers-6.5.13-5-pve
...
root@kagrenac:~# dkms install -m i915-sriov-dkms -v 6.5.13-5-pve
Sign command: /lib/modules/6.5.13-5-pve/build/scripts/sign-file
Signing key: /var/lib/dkms/mok.key
Public certificate (MOK): /var/lib/dkms/mok.pub
Creating symlink /var/lib/dkms/i915-sriov-dkms/6.5.13-5-pve/source -> /usr/src/i915-sriov-dkms-6.5.13-5-pve

Building module:
Cleaning build area...
Building module(s)......(bad exit status: 2)
Failed command:
make -j32 KERNELRELEASE=6.5.13-5-pve -C /lib/modules/6.5.13-5-pve/build M=/var/lib/dkms/i915-sriov-dkms/6.5.13-5-pve/build KVER=6.5.13-5-pve
Error! Bad return status for module build on kernel: 6.5.13-5-pve (x86_64)
Consult /var/lib/dkms/i915-sriov-dkms/6.5.13-5-pve/build/make.log for more information

Uncharted territory

I checked out the log and found the error:

/var/lib/dkms/i915-sriov-dkms/6.5.13-5-pve/build/drivers/gpu/drm/i915/display/intel_dp_mst.c: In function ‘intel_dp_mst_find_vcpi_slots_for_bpp’:
/var/lib/dkms/i915-sriov-dkms/6.5.13-5-pve/build/drivers/gpu/drm/i915/display/intel_dp_mst.c:86:43: error: too many arguments to function ‘drm_dp_calc_pbn_mode’
   86 |                         crtc_state->pbn = drm_dp_calc_pbn_mode(adjusted_mode->crtc_clock,
      |                                           ^~~~~~~~~~~~~~~~~~~~

I tried to brush off the “kernel module dev” hat that I hadn’t worn since probably 2015.

root@kagrenac:/usr/src/i915-sriov-dkms-6.5.13-5-pve# git grep drm_dp_calc_pbn_mode
drivers/gpu/drm/i915/display/intel_dp_mst.c:                    crtc_state->pbn = drm_dp_calc_pbn_mode(adjusted_mode->crtc_clock,
drivers/gpu/drm/i915/display/intel_dp_mst.c:                            crtc_state->pbn = drm_dp_calc_pbn_mode(adjusted_mode->crtc_clock,
drivers/gpu/drm/i915/display/intel_dp_mst.c:            drm_dp_calc_pbn_mode(mode->clock, min_bpp, false) > port->full_pbn) {
drivers/gpu/drm/i915/display/intel_dp_mst.c:            drm_dp_calc_pbn_mode(mode->clock, min_bpp) > port->full_pbn) {
root@kagrenac:/usr/src/i915-sriov-dkms-6.5.13-5-pve# grep -R drm_dp_calc_pbn_mode /usr/src/linux-headers-6.5.13-5-pve/
/usr/src/linux-headers-6.5.13-5-pve/Module.symvers:0xa47826e4   drm_dp_calc_pbn_mode    drivers/gpu/drm/display/drm_display_helper      EXPORT_SYMBOL
/usr/src/linux-headers-6.5.13-5-pve/include/drm/display/drm_dp_mst_helper.h:int drm_dp_calc_pbn_mode(int clock, int bpp);

It looks like there are a few uses of drm_dp_calc_pbn_mode. Some of them have this extra third argument which is apparently not real.

Let’s take a look at the file and see wh-

#if LINUX_VERSION_CODE < KERNEL_VERSION(6,6,14)
        crtc_state->pbn = drm_dp_calc_pbn_mode(adjusted_mode->crtc_clock,
                                        dsc ? bpp << 4 : bpp,
                                        dsc);
#endif
#if LINUX_VERSION_CODE >= KERNEL_VERSION(6,6,14)
        crtc_state->pbn = drm_dp_calc_pbn_mode(adjusted_mode->crtc_clock,
                                        dsc ? bpp << 4 : bpp);
#endif

Alright, so these guards are wrong. It might be a PVE thing where a patch was backported or something. I’ll just nuke the first chunk and unconditionally use the >= 6.6.14 stuff.

I got it to build now.

Prayer

I added i915.enable_guc=3 i915.max_vfs=7 to my kernel cmdline, called update-grub, rebuilt my initramfs, and rebooted with my fingers crossed.

It booted, but I realized I forgot to do the thing from the Readme to enable the VFs. Rebooted again.

The kernel module loaded, but no VFs were created. Manually writing to sriov_numvfs just fails.

Fortunately, somebody else also wrote some instructions on how to get this thing working.

There are only some slight modifications to the configuration and build process here. I was totally out of my depth but it seemed like this apt install sysfsutils step might also be important to get the VFs actually created on boot.

I rebuilt the module and did the initramfs thing again.

The kernel logs now presented this error:

[   17.593755] pci 0000:00:02.0: no driver bound to device; cannot configure SR-IOV

From lspci -k we can confirm that i915 is not in use on the device, though it did load.

If I rmmod and modprobe it again, nothing really happened except it logged the same stuff it did on boot:

[  187.619871] Setting dangerous option enable_guc - tainting kernel

However, it still wasn’t bound to the device. Trying force_probe again had no effect.

I found yet another post about setting it up, and still I could find nothing different.

Deep in the weeds

I did some more googling, and landed on something different. I unloaded the custom i915, uninstalled it from dkms, and tried building this Intel backport thing instead.

It failed with the same drm_dp_calc_pbn_mode stuff again, so I went back in and made basically those same changes as before. The module eventually built successfully, then…. it loaded but did not probe the device, same as before.

Some sources say kernel 6.11 might do the stuff I want it to do. I considered just waiting until it was released.

A year later, I tried again

At first, I had no idea why I had such a hard time with this, since it seems like success came easy for others with 13th gen chips.

One day, I had a spare couple of hours and decided to update my BIOS. I figured that maybe this would fix SR-IOV too. Support for 6.8 was available in the DKMS repo now, and I didn’t have to do anything weird to get it to build. It still didn’t work.

My motherboard manufacturer suggested to update the Intel ME along with the BIOS update. This was a nightmare, requiring me to build WinPE boot media in order to run the tool, since there was absolutely no Linux support. However, after doing so, suddenly SR-IOV was working.

Bask in the glory of my really funny looking fetch due to the quantity of VFs.

Why?

I have no idea what other firmware lives in the ME image but it seems like it was necessary to enable SR-IOV on my chip.

I had never considered updating ME before, and nobody on the Internet had advised that it might be necessary for this.

After this, I followed the other common guidance online to set everything else up and it just worked.

I guess the moral of this story is to always keep ME up to date? I hate it.

An ultra-low profile Chiffre: Le Oeuf

Fri, 22 Nov 2024 00:00:00 +0000

I recently acquired some Kailh PG1425 (“X Switches”) to experiment with. As my first foray into ultra-low profile, I designed a riff on the popular Le Chiffre design.

This keyboard is ridiculously thin, measuring only 1.05 cm at the thickest point, and quite comfortable to type on.

The open-source design is available on GitHub.

I originally picked up these switches for a different project, but ultimately decided that PG1316S was a better candidate for that one and needed to find something to do with these. In lieu of any other ideas, I decided to draw from the Chiffre. Since only 1u caps are available for the X switch, I replaced the 1.25u+2u thumbs with 3x1u.

As before, I wanted this board to have support for wired and wireless builds. However, to keep it ultra-thin, I chose a Xiao footprint over the Pro Micro due to its support to be mounted SMT-style. As it has less IO, this meant I had to use a shift register to drive the matrix. Fortunately, ZMK made this super easy.

Though this project started out as a mere experiment, Le Oeuf has turned out to be one of my favorite boards to type on. It has replaced the tented split board that I have been dailying for work. As an added bonus, it is ultra-portable.

The community has shown a lot of interest in this board, and as a result a redesign using PG1316S is in development. This version will also feature a custom aluminum case for a more premium look and feel. More updates soon!

My latest ergo design: the Solanum

Sun, 22 Sep 2024 00:00:00 +0000

I haven’t had much time to devote to my hobbies since moving, but I’m quite proud to have finally finished a project that I’ve had in flight for the past several months. This is my latest keyboard design: a 34-key split that I call the Solanum.

In some respects, it represents a return to my roots with many familiar features from the egg58: widened spacing, a Pro Micro, TRRS for wired split, and reversible PCBs. However, 34 keys is the smallest key count for any of my designs yet. I’ve been using a layout of this size for most of the year, but this is the first creation of my own to be designed for it.

I have integrated some quality-of-life features that I first tested on other boards: tenting puck support (from the Chicklet), and an onboard JST and power switch for wireless boards (as on the Tamago60).

I also tried to make this board as easy as possible to hand solder. I stuck to through-hole components (with the exception of the hotswap sockets) and opted for a diodeless design.

After almost three years of building and testing weird little keyboards, I think I have mostly settled on what I like. While I don’t think I’m likely to deviate too far from the features I’ve described above, I am still willing to try new things.

In that vein, the Solanum introduces some changes I consider a bit more experimental like a reachy pinky key, more aggressive stagger, and some mild splay. It’s too early to say whether these will all stick around in the next revision, but so far it feels like a pretty comfortable keyboard.

As always, this design is open-source, and can be found on my GitHub.

The server has been brought to life

Sat, 01 Jun 2024 00:00:00 +0000

My last post talked about speccing out a server and buying a bunch of parts. This is a follow up in which I declare success and complain about incompatible parts I had originally selected.

When I first got started with the build, I was pretty confident, both that it would work and that I would do a good job with cable management.

The good news is that it did eventually work. The bad news is it was a huge pain in my ass and took over a week of debugging. As we’ll see later, I also stopped being so careful with cable management.

A series of problems arose:

My Intel SSD didn’t work
The Teamgroup NVMe didn’t work
The NICs didn’t work
The PCIe switches didn’t work
I had to 3D print a cooling duct for the Tesla (at least I didn’t have to model it myself)
Thermal throttling (at least I expected this one)

Here’s the final BOM we ended up with:

Item	Qty	Condition	$/each	Total
Sliger CX2137b	1	New	$199.00	$199.00
Arctic F8 PWM Fan	3	New	$8.00	$24.00
Asus Pro WS W680-ACE IPMI	1	New	$399.99	$399.99
Intel i9-13900K	1	New	$490.00	$490.00
Dynatron Q5 LGA1700 Cooler	1	New	$49.95	$49.95
Kingston DDR5-4800 32GB	4	New	$118.75	$475.00
GIGABYTE GeForce RTX 4060	1	New	$319.99	$319.99
Silverstone FX600	1	New	$174.99	$174.99
NVIDIA Tesla P4	1	Used	$126.13	$126.13
Intel DC S3500 480GB	2	Used	On hand	$0.00
Crucial P3 4TB	6	New	$236.99	$1421.94
Startech PEX8M2E2	1	New	$162.67	$162.67
Startech PEX4M2E1	1	New	$22.93	$22.93
Noctua NF-A6x25	1	New	$14.95	$14.95

Grand total before tax and shipping: $3881.54. Not really too bad.

Now that we’ve established how this build hurt my wallet, let’s talk about how it hurt me.

Chapter 1: My intel SSD didn’t work

I installed Debian. I rebooted. The boot device was no longer there. The previous build I used this Intel NVMe in had a big chonky heatsink on it and a huge fan to keep it cool. This build did not really have sufficient cooling.

The drive kept appearing and disappearing, and I kept fiddling with flags in the UEFI thinking it was just mad about something. But it was just a thermal issue.

I don’t think the controller got completely cooked, as it did appear again later once it cooled down. However, this was a non-starter. I gave up on my all-NVMe ideal and installed some SATA SSDs I had lying around: Intel DC S3500 480GB. I had two of em, so figured why not make it root-on-mirrored-ZFS.

I installed Debian again.

Chapter 2: The Teamgroup NVMe didn’t work

It’s quite disappointing to buy 6x4TB of NVMe and not be able to use it. They did appear in the UEFI, but the NVMe behind the PCIe switches on the Glotrends carrier cards just wouldn’t show up in Linux. This was a disappointment. The ones installed directly on the motherboard, however, did work.

I tested the Intel drive I had just removed and found that it did appear when installed behind the switch. I figure this is a firmware bug on the Team stuff, or in the Realtek controller they use. I sent the drives back and got Crucial P3 4TB as replacements. These ones appeared to work.

I went ahead and installed Proxmox and set up a big ZFS pool to use for my VMs.

I started migrating stuff from my old server, then I hit another problem.

Chapter 3: The NICs didn’t work

While I was copying stuff over, I set the MTU to 9000. Then later, I was debugging some stuff on my desk and plugged the server into a 100mbit dumbswitch to get it connected. It did not like this.

The kernel kept panicking when systemd decided to turn on the NIC. I could boot into a livecd, but not my installed system. I once again started playing with UEFI toggles. I started playing with kernel flags too. It took me way too long to make the connection that it was because of the jumbo frames and dumbswitch having a bad time with one another.

I didn’t feel like running a new cable across the room to the real switch, but I knew I had more desk-based debugging and testing to do. So, I just mounted / in the livecd and changed the MTU back to 1500.

Chapter 4: The PCIe switches didn’t work

I noticed my disk utilization frequently dropping to zero during the prior rsync’ing. I checked dmesg to see what was going on.

[  988.525697] pcieport 0000:00:1d.0: AER: Multiple Uncorrectable (Non-Fatal) error message received from 0000:12:00.0
[  988.536154] nvme 0000:12:00.0: PCIe Bus Error: severity=Uncorrectable (Non-Fatal), type=Transaction Layer, (Requester ID)
[  988.547103] nvme 0000:12:00.0:   device [c0a9:540a] error status/mask=00004000/00400000
[  988.555098] nvme 0000:12:00.0:    [14] CmpltTO
[  988.560907] pcieport 0000:10:08.0: AER: device recovery successful
[  990.619281] nvme nvme3: I/O tag 204 (d0cc) opcode 0x1 (I/O Cmd) QID 2 timeout, aborting req_op:WRITE(1) size:8192
[  992.667249] nvme nvme3: I/O tag 204 (d0cc) opcode 0x1 (I/O Cmd) QID 2 timeout, reset controller
[  992.688623] nvme nvme3: Abort status: 0x371
[  992.721812] nvme nvme3: Shutdown timeout set to 2 seconds
[  992.933183] nvme nvme3: 8/0/0 default/read/poll queues
[  992.956487] nvme nvme3: Ignoring bogus Namespace Identifiers
[  999.730279] pcieport 0000:00:1d.0: AER: Multiple Uncorrectable (Non-Fatal) error message received from 0000:12:00.0
[  999.740791] nvme 0000:12:00.0: PCIe Bus Error: severity=Uncorrectable (Non-Fatal), type=Transaction Layer, (Requester ID)
[  999.751728] nvme 0000:12:00.0:   device [c0a9:540a] error status/mask=00004000/00400000
[  999.759723] nvme 0000:12:00.0:    [14] CmpltTO
[  999.765495] pcieport 0000:10:08.0: AER: device recovery successful
[ 1000.859080] nvme nvme3: I/O tag 891 (d37b) opcode 0x1 (I/O Cmd) QID 2 timeout, aborting req_op:WRITE(1) size:122880
[ 1002.910082] nvme nvme3: I/O tag 891 (d37b) opcode 0x1 (I/O Cmd) QID 2 timeout, reset controller
[ 1002.932441] nvme nvme3: Abort status: 0x371
[ 1002.961263] nvme nvme3: Shutdown timeout set to 2 seconds
[ 1003.172257] nvme nvme3: 8/0/0 default/read/poll queues
[ 1003.193600] nvme nvme3: Ignoring bogus Namespace Identifiers
[ 1004.128115] pcieport 0000:00:1d.0: AER: Multiple Uncorrectable (Non-Fatal) error message received from 0000:12:00.0
[ 1004.138653] nvme 0000:12:00.0: PCIe Bus Error: severity=Uncorrectable (Non-Fatal), type=Transaction Layer, (Requester ID)
[ 1004.149592] nvme 0000:12:00.0:   device [c0a9:540a] error status/mask=00004000/00400000
[ 1004.157602] nvme 0000:12:00.0:    [14] CmpltTO
[ 1004.259772] pcieport 0000:10:08.0: AER: device recovery successful
[ 1005.211047] nvme nvme3: I/O tag 487 (d1e7) opcode 0x2 (I/O Cmd) QID 3 timeout, aborting req_op:READ(0) size:45056
[ 1006.235030] nvme nvme3: I/O tag 487 (d1e7) opcode 0x2 (I/O Cmd) QID 3 timeout, reset controller
[ 1006.255197] nvme nvme3: Abort status: 0x371
[ 1007.287473] nvme nvme3: Shutdown timeout set to 2 seconds
[ 1007.498696] nvme nvme3: 8/0/0 default/read/poll queues
[ 1007.519380] nvme nvme3: Ignoring bogus Namespace Identifiers

This is bad, awful, and not good. However, it was only occurring for the NVMe on the carrier cards. I sent them back and got a new one with a PLX switch instead of an Asmedia one.

Before I made that decision, I spent an entire day playing with kernel flags to change NVMe timeouts, turn off APST and ASPM, etc., all to no avail.

Since I had ditched root-on-NVMe, I had a spare slot on the board, so I now only needed one card with a switch, and one that was basically just a dumb adapter.

I didn’t really go out of my way to optimize this (e.g., the PCIe switch carrying two of the NVMe is installed in a x4 slot instead of a x8) but it gets pretty decent performance.

A quick test in fio returned some surprisingly round numbers:

Read: 7500MiB/s, 30k IOPS
Write: 5000MiB/s, 20k IOPS

This is way more than good enough for my use case.

Chapter 5: I had to 3D print a cooling duct for the Tesla

The Tesla P4 is designed for real servers that have a bunch of beefy fans to push a ton of air through it. This server is not one of those.

I ran into a bunch of problems with my printer that I don’t feel like elaborating on, but ultimately I got a part made.

It’s ugly but it works.

Aside: This Tesla was sold as open box, but I could see evidence of a really janky refurb job, like poorly-cut thermal pads sticking out under the backplate. I decided to take it apart and redo all the pads and paste myself. Fortunately I did not brick the card.

Chapter 6: Thermal throttling

I knew I would have thermal issues with the 13900K, since I had originally specced a run-of-the-mill 13900.

Ultimately, I:

Disable hyperthreading (not sure if this was really necessary)
Set a max boost clock of 4.3GHz
Set lower power caps and slightly undervolted
Disabled the intel_pstate kernel module and used cpupower to load the conservative governor and ensure the lowest-power states were enabled

This got me down to an idle package temp (with VMs idling) of 35C, at 98W total system power draw. It briefly hits 90C and pulls 330W during stress -c 24, but then once the “long duration” power cap kicks in it drops to 80C/265W.

While we’re here, I’ll also mention that:

The Tesla pulls 25W and sits at 60C while idling with a model loaded,
…and hits 65W/75C during inferencing.
I set a 75% power cap and 70C thermal limit on the 4090 and it idles at 40C,
…and hit 69C during a benchmark in Borderlands 3 at 1080p60 (in which it achieved an average of 58.85fps with a 17ms frame time).

I find this perfectly acceptable.

Conclusion

In the end, the inside of the thing looked like this:

My awful cable management job is fortunately hidden behind a rather pleasant facade.

Ignore the mess on my desk and the fact that the lid wasn’t closed in that picture.

I now have a nice 2U machine to replace both my ancient HP workstation tower and my old Synology NAS.

It's time for a new server

Sun, 12 May 2024 00:00:00 +0000

Setting out to waste some money

It’s only been about two years since I built an atrocity of a VM server in a decade-old workstation box. It still gets the job done, but there is empty space to fill in my homelab cabinet so it’s time for an upgrade. In the interest of killing two birds with one stone, I’d like it to replace my old Synology NAS too, which is also perfectly adequate despite its age.

The constraints and use case of this system make it a pretty challenging build to spec out:

Two GPUs: one for the gaming VM, one for an AI VM. Both must have at least PCIe4 x8 bandwidth.
Shallow 2u chassis: so it fits in my cabinet. I’ve already been though too many different sized ones.
At least 16 cores. Ideally, more.
At least 128 GB of RAM.
ECC support. This limits us to server/workstation SKUs and only the most recent consumer ones.
Room for enough SSDs to get at least 10 TB of usable capacity. I’ve currently got about 5 TB used so this gives me decent headroom.

Let’s start picking some parts.

Case and GPUs

These choices end up being intrinsically linked, since the shallow cabinet ends up being the biggest problem. Nothing with more than about 14" depth will really fit. I considered 3u and 4u chassis as well, but somehow 2u ended up working out the best.

While 3u ostensibly supports a full-height card, consumer GPUs don’t really fit because they tend to extend a bit above the PCI bracket. You’d think 4u avoids this problem, but the only 4u case I can find that is 14" deep or less has the motherboard on an elevated tray that makes it no better than a 3u.

There are way more options available for 2u cases, but this constrains me to half-height GPUs. There aren’t many options with risers to circumvent this, and none of them are good. There also aren’t many good half-height GPUs. Thus, it was time to lock in our first two part selections:

Gigabyte GeForce RTX 4060 OC Low Profile 8G, for the gaming VM
~~Nvidia RTX A2000 12GB, for the AI VM~~ (Update: I got a P4 instead. It was cheaper.)

The 4060 is basically the only “gaming” GPU available in this form factor that isn’t a decade old. I acknowledge that the A2000 is a weirder choice here, but ultimately I chose it because:

It has a decent amount of RAM.
~~It’s way cheaper than any of the half-height Teslas.~~ (Update: Actually, it wasn’t.)

To fit both of these dual-slot cards in a 2u case, now one of two things must be true:

It takes a full ATX motherboard, so there are more than 4 bays, or
I find a mATX board that actually puts two (physically) x16 slots in the 1st and 3rd positions.

After hours of searching, I could not really find any mATX board that worked here, so I settled for the only shallow 2u full-ATX case I could find, locking in our third part:

Sliger CX2137b

It’s only 13" deep, so no problem there. It has 7 PCI bays and takes a flexATX PSU. Looks pretty decent too.

Only one issue. I originally planned to find a case with room for two 5.25" bays so I could put a backplane in them and fill it with 2.5" SSDs. This case doesn’t have the 5.25", so I had to find another option. I could fill the interior of the case with SATA SSDs, but it only really fits five. I’d end up with a 4-member RAID-10, a non redundant boot disk, and a huge mess of cables. Time to figure out how to solve this problem.

Motherboard

At this point, I decided to commit to slapping a bunch of NVMe in it. Thus, a new constraint has been introduced: we need a buttload of M.2, or at least lots of PCIe slots we can stick risers in.

After looking at basically every motherboard SKU produced for the latest 3 or 4 generations of both AMD and Intel chips, I figured out that there is basically only one chipset that will get the job done: Intel W680.

There aren’t a ton of boards with these chips in them, since it targets the pretty-niche part of the workstation segment that isn’t on Xeons. However, the workstation application is ultimately the consideration that makes this the best chip for the job: they are all intended to support multiple GPUs.

All of the W680 boards I found let you run the 16 PCIe Gen5 lanes off the CPU as either a single x16 or two x8’s. We’ll be choosing the latter option.

We must now compare boards for their other features, particularly:

How many ethernet ports does it have?
Does the firmware actually support ECC?
How much storage can I attach to it?
Any extra goodies like IPMI?

At this point, I had three serious candidates: one from Supermicro (X13SAE-F), one from Asus (Pro WS W680-ACE IPMI), and one from Gigabyte (MW34-SP0).

Though I’m a pretty big Gigabyte stan, unfortunately there’s enough FUD on the internet to make me wary that they flaked pretty hard on the ECC support for this one. It also only has one 2.5GbE port - which is enough bandwidth, but I’ve only got gigabit switches so was betting on two 1GbE so I could LACP them. With two strikes against it, I decided to eliminate it from the running.

Both had IPMI (though the Asus is weird, we’ll get to that in a second) and ECC support that apparently works. The decision ultimately came down to the Asus being $100 cheaper and having two 2.5GbE ports instead of just one.

Indeed, we have finally locked in our fourth part:

Asus Pro WS W680-ACE IPMI

It has a PCIe layout that looks something like this:

CPU
- PCIe 5.0 x8 (in a x16 slot)
- PCIe 5.0 x8 (in a x16 slot)
- PCIe 4.0 x4 M.2
PCH
- 2x PCIe 3.0 x4 (in a x16 slot)
- PCIe 3.0 x1 (for the IPMI card)
- 2x PCIe 4.0 x4 M.2

Yeah. That’s the weird part - the IPMI is on an expansion card. It seems like the motherboard does have the BMC on it, but this card is needed to give it ethernet and VGA to turn it into a full-fledged IPMI. Odd choice, but whatever. Fortunately it doesn’t take up any of the PCIe I actually want to use. Still plenty of PCIe lanes for a bunch of NVMe.

Storage

Update: Don’t buy the parts listed in this section. They didn’t work. Refer to my new post.

We’re going to jam 6 M.2 drives in this thing, using the two onboard M.2’s attached to the PCH, and two carrier cards to add two more in each of the x4 slots. Since the slots are x4, this means we need active cards:

~~2x GLOTRENDS PA20 Dual M.2 NVMe to PCIe 3.0 X4 Adapter~~

~~After considering several options, the smart move is to just make the bulk of the storage something cheap, reliable, and high density, even if it doesn’t have the highest performance:~~

~~6x TEAMGROUP MP34 4TB~~

~~We were limited to Gen3 speeds and bottlenecked through those switches anyway, so the only real loss here is in random IOPS. Maybe we can make up for it with the 3-way stripe or the mirroring.~~

~~We’ll run some random spare SSD I have around for boot:~~

~~Intel DC P4511 1 TB~~

Power supply

We’re stuck with a flex PSU thanks to the case we had to choose. Let’s just pick the biggest one available from a respectable brand:

Silverstone FX600 Platinum

CPU

We want a CPU that supports ECC and has a ton of cores. However, we’re pretty limited in terms of power. After doing some math, I figured I can afford a 65W TDP CPU. This means an i9-13900 is the best choice. However, it so happens that the -K is cheaper right now. I’ll get one of those and clock it down to non-K speeds.

Intel i9-13900K

Not opting to pay extra for a 14900, since it doesn’t really seem to perform any better.

RAM

We need 4x 32GB DDR5 ECC UDIMMs. There is only one SKU on the QVL that I can actually seem to purchase right now, and I don’t feel like playing games:

Kingston DDR5-4800 32GB ECC

Adding it all up

The last touches are fans and a CPU cooler. Let’s find those, recap the BOM and add up some numbers. I planned to get a bunch of stuff from eBay to save money, but it turns out that most of these parts aren’t exactly plentiful on the secondary market.

Item	Qty	Condition	$/each	Total
Sliger CX2137b	1	New	$199.00	$199.00
Arctic F8 PWM Fan	3	New	$8.00	$24.00
Asus Pro WS W680-ACE IPMI	1	New	$399.99	$399.99
Intel i9-13900K	1	New	$490.00	$490.00
Dynatron Q5 LGA1700 Cooler	1	New	$49.95	$49.95
Kingston DDR5-4800 32GB	4	New	$118.75	$475.00
GIGABYTE GeForce RTX 4060	1	New	$319.99	$319.99
Silverstone FX600	1	New	$174.99	$174.99
~~NVIDIA RTX A2000 12 GB ECC~~	1	~~Used~~	~~$450.00~~	~~$450.00~~
~~Intel P4511 1TB~~	1	~~Used~~	~~On hand~~	~~$0.00~~
~~GLOTRENDS PA20~~	2	~~New~~	~~$97.99~~	~~$195.98~~
~~TEAMGROUP MP34 4TB~~	6	~~New~~	~~$225.99~~	~~$903.96~~

Update: I ended up deviating from my original plan. An explanation and new BOM is in this new post. tl;dr: I got a cheaper GPU, the Team NVMe was buggy, my Intel NVMe failed, and the PCIe switches sucked.

Everything’s been ordered except the A2000, which I’m going to put off for a little bit to make sure everything else works first.

[a previous version of this post had a pie chart of the cost breakdown here, but I didn’t feel like updating it after changing the BOM, so it has been removed]

My power budget is also pretty tight, so let’s see what kind of shape we’re in. I could not find actual power data for some of these parts, so it’s estimated based on similar ones (or wild guesses).

Item	Power
Intel i9-13900K (throttled)	219 W
GIGABYTE GeForce RTX 4060	115 W
~~NVIDIA RTX A2000 12 GB~~	~~70 W~~
~~6x TEAMGROUP MP34 4TB~~	~~48 W~~
Asus Pro WS W680-ACE IPMI	30 W
4x Kingston DDR5-4800 32GB	20 W
~~2x GLOTRENDS PA20~~	~~10 W~~
~~Intel P4511 1TB~~	~~8 W~~

Max draw looks like 520 W, so we still have a decent amount of headroom with the 600 W PSU. If I end up running into any problems I can just further limit the CPU or one of the GPUs.

Update: I changed the BOM, but I don’t feel like finding new power numbers. It should be basically the same.

Now we wait

The parts have yet to arrive. Will update with a post about my struggles when they do.

Adapting to just 34 keys

Fri, 03 May 2024 00:00:00 +0000

A while back I posted about my 38-key layout, and mentioned that the 34-key form factor is quite a bit more common. As it happens, I’ve been dailying 34-key layouts for the past few months and wanted to share the layout I settled on.

There are a lot of hyperoptimized layouts for 34-key boards floating around, but they are not really easy to adapt to when coming from a larger board. My layout developed pretty much organically over the past 4-5 months - the starting point was basically a 60%, and I moved stuff around in a way that felt intuitive to me.

The main tenets of this layout are:

Home row mods
Only two layer-tap keys, on the inner thumbs
Most symbols can be reached with only one hold, though a few need two
Common function keys (F1-F5) can be reached with only one held key
Common shortcuts (Cmd-C, Cmd-V, Cmd-X) can be performed on one hand
No combos besides mod-tap and layer-tap

Without further ado, here it is:

The visualization was produced with keymap-drawer.

It’s drawn here on a Sweep, but I’ve got it running on my Claudia and Corne too.

(I removed the tucky thumb keys since taking that photo of the Corne.)

These keymaps are available in my QMK fork.

Expanding LVM PVs and LVs

Fri, 26 Apr 2024 00:00:00 +0000

I’m writing this down because I keep having to expand disks in PVE, but can’t remember the commands for the guest and keep having to google it.

Extend the disk in PVE by however much

Use fdisk to expand (really, delete then recreate) the partition

# fdisk /dev/vda
Command: p
Disk /dev/vda: 192 GiB, 206158430208 bytes, 402653184 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 2283D992-048D-407F-A13C-63AFDA9CC224

Device       Start       End   Sectors  Size Type
/dev/vda1     2048   1050623   1048576  512M EFI System
/dev/vda2  1050624   2050047    999424  488M Linux filesystem
/dev/vda3  2050048 201324543 199274496   95G Linux filesystem

Command: d
Partition number (1-3, default 3):

Partition 3 has been deleted.

Command: n
Partition number (3-128, default 3):
First sector (2050048-402653150, default 2050048):
Last sector, +/-sectors or +/-size{K,M,G,T,P} (2050048-402653150, default 402651135):

Created a new partition 3 of type 'Linux filesystem' and of size 191 GiB.
Partition #3 contains a LVM2_member signature.

Do you want to remove the signature? [Y]es/[N]o: n

Command:  w
The partition table has been altered.
Syncing disks.

Extend the PV

# pvresize /dev/vda3
Physical volume "/dev/vda3" changed
1 physical volume(s) resized or updated / 0 physical volume(s) not resized

Extend the LV

# lvresize -l +100%FREE /dev/mapper/vaermina--vg-root
Size of logical volume vaermina-vg/root changed from <95.02 GiB (24325 extents) to <191.02 GiB (48901 extents).
Logical volume vaermina-vg/root successfully resized.

Extend the ext4 partition

# resize2fs /dev/mapper/vaermina--vg-root
resize2fs 1.47.0 (5-Feb-2023)
Filesystem at /dev/mapper/vaermina--vg-root is mounted on /; on-line resizing required
old_desc_blocks = 12, new_desc_blocks = 24
The filesystem on /dev/mapper/vaermina--vg-root is now 50074624 (4k) blocks long.

Stick a trackball on your keyboard

Sun, 14 Apr 2024 00:00:00 +0000

Despite having designed several of my own ergonomic columnar split keyboards, I have been daily driving my Corne (crkbd) for the past several months. I removed the tucked-most thumb keys and outer pinky column and thus am using it with a 34-key layout.

I usually set up with my mouse (a Logitech MX Vertical) in between the two halves of the board. Recently, I found that the repetitive motion back and forth between the keyboard and mouse to be somewhat irritating. I figured it was time for an integrated pointing device.

I had experimented in the past with embedding joysticks, and later Cirque trackpads in keyboards, but ultimately was not a huge fan of how they felt to use. I’ve never used a trackball before, but decided to give the Pimoroni trackball a shot since it was sort of closer in operation to a trackpoint, which I don’t mind using on my laptop.

Fortunately, there was already support for it in QMK, and an esteemed member of the ergo keyboard community had posted an STL file to embed the module into the Corne’s MCU cover.

One poorly-calibrated print and a few bodge wires later, and my Corne had a trackball on it.

I ended up writing some firmware logic to use an auto mouse layer, with a few cheap tricks to give me a hold-to-scroll modifier. For fun, I also used the RGBW LED in the trackball to indicate the active layer.

A few hours of testing later, and the trackball feels pretty natural to use, and is much less annoying to flick compared to reaching over to grab a mouse.

Audio controls in Home Assistant using MQTT

Wed, 03 Apr 2024 00:00:00 +0000

In a previous post, I shared some software I wrote to get my amp to listen to power and volume commands from my TV. Now, I’ve added the ability to control it from MQTT as well. It integrates with Home Assistant, so I can have a nice view that essentially serves as a remote control. It’s quite handy for adjusting volume if I’ve put some music on then walked away.

It’s a bit limited right now since the MQTT feature is just bolted onto the driver that was intended for the CEC use case, but there’s definitely room to expand it in the future.

More info on the project is available on GitHub.

It's really easy to make a bike electric

Mon, 01 Apr 2024 00:00:00 +0000

Over the weekend, I converted my seldom-used mountain bike to an e-bike.

I never used my MTB much to begin with, but since moving it is even harder to find the time (or a place) for a ride. However, the hills here are pretty steep and this city’s roads seem adversarial to cars. It’s a great use case for a utilitarian e-bike, but I didn’t want to let my old frame go to waste.

I chose to purchase a mid-drive conversion instead of a hub motor kit. My original thinking was that retaining use of the bike’s existing transmission would be beneficial. Starting with a MTB not just granted the comfort of a front suspension to lessen the blow of all the potholes, but also meant I’d have a cogset with a good selection of low gears for uphill rides.

Contrary to my expectations, it turns out that many of my gears became useless. The CYC Photon, at least in my 52V setup, is plenty capable of starting up a steep hill in even my 4th (28T) or 5th (24T) gear (with a 38T chainring). Even the upper gears are too close for them all to be super useful under assist, and my top speed (in 11T) is quite limited. A more compact cassette (and probably a larger chainring) are in my future.

Downsizing with this move meant I had to give up my project car, but it seems like I’ve found something else to tinker with.

Introducing another split keyboard: the Chicklet

Wed, 27 Dec 2023 00:00:00 +0000

It’s time for another keyboard post! My latest creation is a 38-key split and today we’ll be talking about its design and development.

Layout

Key counts for split keyboards are essentially bimodal - there many around 60%, and again many with just 34 keys. A 60%ish split presents ease of use, as it can quickly be adapted to from a standard keyboard. The 34-key split meanwhile begins to manifest the ideal of 1DFH, or one distance from home — meaning that each key is no further than one position from home position, minimizing the amount of finger travel while typing. Each hand of a 34-key split will typically have a 3x5 body and two thumb keys.

The 38-key form factor is a bit less popular and doesn’t seem to be well represented in the hobby. It has just a few more keys than 34, but I have found that those extra keys provide quite a bit of utility. It strays only slightly from 1DFH, but allows more keycodes to be accessible with fewer layers.

Consider the following groups of keycodes which you find represented on a 100% keyboard:

Alpha A-Z
Numerals 1-0
Symbols -=[];',./`\
Space, tab, enter, and backspace
Shifted numeral symbols !@#$%^&*()
Modifiers (shift, ctrl, caps, alt, super)
Navigation (up, left, right, down, pgup, pgdn, home, end, del, esc)
Functions F1-F12

We can consider the shifted numerals here as already being on a de-facto layer, even on a 100% keyboard, as you must hold shift to access them. On a 60%, the last two groups are moved to layers: function keys are accessed through Fn, and navigation keys are often on the Fn layer as well. On a 40%, the numerals themselves are removed from the default layer too. Often, you now see the “navigation” and “numeral-symbol” groups being split into two separate layers. On most 30% boards, you now throw away dedicated keys for modifiers, and instead use a tap-hold capability to have them share keys with alphas. Often, layers will also be accessed via tap-hold on keys shared with codes like space or enter.

Once you get down to 30%, you find yourself having several raised layers. For example, the keymap I use for 34-key boards has three: numbers and symbols, navigation, and functions. There are many ways to access layers, but the most common is momentary (held) layer keys located in the thumb cluster. This is great if you have multiple thumb keys, however I prefer to use only one per hand.

With just one key per hand, we become somewhat limited in how many layers we can have. On my 34-key keymap, the left thumb key activates the num-sym layer, the right activates nav, and both simultaneously activates functions. However, it would be nice to remove the need to press both at the same time to access that layer – and I did not want to add another layer-tap key.

This is the problem solved by going up to 38 keys: numbers, symbols that don’t fit on the main layer, and function keycodes can all fit together.

Another idiosyncrasy of this layout is that the three keys below the main block are inline with body columns rather than being part of the thumb cluster. This is another personal preference of mine — I deviate from 1DFH slightly here, but find that the mobility of those three fingers allows easier access to these keys than my thumb would.

Hardware

Another goal for this board was to make building it relatively inexpensive. I used a Seeed XIAO microcontroller and two PCA9555 I2C I/O expanders. Strictly speaking, the primary (left) side of the keyboard did not need the I/O chip but could have been matrixed directly on the XIAO – however, I wanted to have access to the interrupt signal from the peripheral PCA9555 so I could build a wireless version of this board which does not need to constantly scan. The XIAO did not have the I/O to spare, so I added another chip on that side, too.

The addition of two PCA9555s at around $1 each still makes this cheaper than a dual-XIAO split board. It also makes a wired split link possible while retaining the possibility of using BLE to connect to a computer - which is much more battery friendly than a dual-MCU split can be given the current limitations of ZMK.

Since this was meant to be a more budget oriented board, I omitted RGB this time around. However, I retained hotswap sockets to make assembly quick and easy. I also added features to allow attaching a tenting puck, so it can be attached to a mini tripod for customizable positioning.

One final hardware feature worth mentioning here is a set of pads designed to enable a modular peripheral system. I would like to be able to add extra features like OLEDs and trackpads, but wanted to do so in a way that would not require soldering. I placed four pads, exposing power and I2C, and intend the modules to use pogo pins to connect to these.

I had all the SMT components, except the XIAO, assembled overseas, and the PCBs and switchplates arrived in their panels.

Firmware

For now, I have only done the work for QMK but intend to set up ZMK in the future. While trying to get the firmware working, I noticed one small error in the design. Fortunately, it ended up not being a blocker. The PCA9555 expects logic-low on its inputs, however I set up the diodes for logic-high. This only meant I had to swap the roles of pins, i.e. between rows and columns, in firmware.

The firmware is available in the eggsworks QMK fork.

Build

The full build is described on my documentation site. In summary, the XIAO needs to be soldered to the board, one only needs to attach the switchplate then install switches and keycaps before plugging it in and flashing firmware.

The board is open source.

Blue tape or Kapton? Why not both?

Fri, 20 Oct 2023 00:00:00 +0000

In the 3D printing community, there is endless debate about the best bed surface. In any case, blue (painter’s) tape and Kapton (polyimide) tape are two top contenders for many materials.

In my laziness, I have discovered that the best bed surface is, indeed, both.

How it works is quite simple: you put the blue tape on the bed, then you put the Kap tape over it. The painter’s tape provides a better surface to apply the Kapton to than the raw bed. It is far less prone to wrinkling or bubbling during application this way.

I developed this technique by accident when I wanted to change the surface from blue tape to Kapton. I was too lazy to peel the blue off first, and thereby inadvertently discovered this hack.

Of course, since Kapton is the actual surface, this technique is suitable for any materials which already like to be printed on it. I print in TPU, PLA, and PETG, and have found that it works great for them all.

The Kap tape does transfer a bit of the texture from the layer underneath, however you still get the overall shiny surface finish on the bottom that Kap alone normally provides.

It is fairly easy to remove the Kapton, then the blue tape can be reused with a new Kap surface put over it again. I prefer to apply the Kap tape in the opposite direction to the blue, in order to make it easier to remove without taking the blue with it.

A 60% ortho keyboard with a trackpad: tamago v2

Thu, 28 Sep 2023 00:00:00 +0000

It’s been about five months since I released the tamago60. In this dramatically delayed follow-up, I’ve made a few improvements. It’s not 100% done yet, but here’s a preview.

We’ve got an aluminum switchplate, a 40mm Cirque trackpad, and 58 hotswap sockets. All on a board that fits in a standard 60% tray-mount case.

There’s a KB2040 module stuck in the middle of it, which conveniently breaks out USB D+/D-, thus allowing me to relocate the USB port to the normal location on a 60%.

The trackpad is secured firmly to the switchplate by a 3D-printed mount.

I decided to finally try a PCBA service, and it went pretty seamlessly, except for the part where I put the Choc socket part number on the BOM instead of the MX one, thus delaying production by an extra week.

There are a few changes I’d like to make before the full release of this one.

The plan is to add some physical buttons for the trackpad, stick an RP2040 onboard, and maybe add an OLED. I’d like to switch to Choc sockets as well, but I think I recall that they would have interfered with the mounting features.

The design needs a bit of cosmetic polish before it looks like a premium keyboard. However, the typing feel is great. The sound is pretty good too, even without any lube or foam. The stabilizers rattle a little, but that can be easily fixed.

Even in its current form, it is a very usable keyboard. At least the trackpad is more practical than the last version’s joystick.

Streaming QR code detection in Home Assistant

Wed, 27 Sep 2023 01:00:00 +0000

A while back, I set up keyfob access using Home Assistant and ESPHome. It required bodging an RFID reader onto my doorbell, and after some time its 3D printed enclosure has yellowed. Today, out of the blue, an idea: ditch the keyfobs and scanner for QR codes.

Important: Ignore the configuration suggested in this post if you’re trying to install this add-on in Home Assistant itself. This is about an earlier prototype. Follow the readme on GitHub instead.

Scroll to the bottom if you just want the GitHub link.

The plan was simple. First, eat the doorbell’s RTSP stream. Feed it into OpenCV or something and detect QR codes. Then, throw the codes at the Home Assistant API.

It ended up being easier than I thought.

Here’s the code for the initial proof of concept. Yep, that’s all of it. It took about 15 minutes to write and test.

import sys
import re
from datetime import datetime, timedelta

import cv2
import homeassistant_api as ha
import yaml
try:
    from yaml import CLoader as Loader, CDumper as Dumper
except ImportError:
    from yaml import Loader, Dumper

DEBOUNCE_PERIOD = timedelta(seconds=5)
TAG_ID_PATTERN = re.compile('https://www.home-assistant.io/tag/([0-9a-f-]+)')

def main(config):
    with open(config, 'r') as fh:
        config = yaml.load(fh, Loader=Loader)['ha-cam-tag']
    
    stream = cv2.VideoCapture(config['camera']['stream'])
    detector = cv2.QRCodeDetector()

    last_time, last_tag = None, None

    with ha.Client(config['homeassistant']['uri'], config['homeassistant']['auth-token']) as client:
        while stream.isOpened():
            _, frame = stream.read()
            if data := detector.detectAndDecode(frame)[0]:
                if m := TAG_ID_PATTERN.match(data):
                    cur_time, tag_id = datetime.now(), m.group(1)
                    if last_tag != tag_id or last_time < cur_time - DEBOUNCE_PERIOD:
                        client.fire_event("tag_scanned", tag_id=tag_id, device_id=config['camera']['device-id'])
                        last_time, last_tag = cur_time, tag_id

    stream.release()
    cv2.destroyAllWindows()
    return 0
        
if __name__ == "__main__":
    sys.exit(main(*sys.argv[1:]))

The final version has the detector running in a separate thread (this helps with latency), and some boilerplate to make it easier to run, but the core concept is the same.

This requires just three packages: pyyaml, opencv-python, and homeassistant_api.

It eats a config file that looks like this:

ha-cam-tag:
  homeassistant:
    uri: https://homeassistant.example.com/api/
    auth-token: "secret-token"
  camera:
    device-id: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
    stream: rtsp://10.1.20.xxx:7447/xxxxxxxxxxxxxxxx

I’m using the high resolution RTSP stream on my Unifi doorbell. The medium level worked too, but was a bit less reliable.

I’ve got this running in a separate container from Home Assistant on my Proxmox box for now, but I want to port it to a proper extension eventually.

Here’s the repo: https://github.com/tmick0/ha-cam-tag

It's always DNS; or: why I closed my Comcast Business account

Wed, 27 Sep 2023 00:00:00 +0000

Warning: this post consists of a lot of grumbling and not much technical merit.

I don’t usually make blog posts just to complain. However today is different. I’m getting a new roof installed and this means I had to temporarily disconnect the coax that brings internet service into my house, due to the way the cable is routed across it. I was hoping this would be minimally disruptive, because I pay Comcast Business $210/mo for a level of service that includes a backup LTE modem.

As you can assume from the fact that I’m writing this post, it was disruptive anyway.

Comcast’s “Connection Pro” LTE modem sits on the AT&T network and in the past has behaved pretty well. However, for the past day it has been doing something weird. I cannot get to many websites, all failing with PR_END_OF_FILE_ERROR in Firefox.

So far, I have found that behavior on the following sites:

amazon.com
bestbuy.com
ebay.com
mozilla.org
newegg.com
store.ui.com

Upon further investigation, I discovered that these all resolve to the same IP.

;; ANSWER SECTION:
aws.amazon.com.         0       IN      A       192.73.252.25
aws.amazon.com.         0       IN      A       192.73.252.18
;; ANSWER SECTION:
bestbuy.com.            0       IN      A       192.73.252.18
bestbuy.com.            0       IN      A       192.73.252.25
;; ANSWER SECTION:
ebay.com.               0       IN      A       192.73.252.25
ebay.com.               0       IN      A       192.73.252.18
;; ANSWER SECTION:
mozilla.org.            0       IN      A       192.73.252.25
mozilla.org.            0       IN      A       192.73.252.18
;; ANSWER SECTION:
newegg.com.             0       IN      A       192.73.252.18
newegg.com.             0       IN      A       192.73.252.25

And no, this should not be the case.

This isn’t just the behavior of some default resolver that was gifted to me in DHCP - I have hardcoded an upstream.

It doesn’t matter what upstream resolver I use, the DNS requests are intercepted and I get the same bogus reply:

; <<>> DiG 9.16.1-Ubuntu <<>> store.ui.com @8.8.8.8
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 7391
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;store.ui.com.                  IN      A

;; ANSWER SECTION:
store.ui.com.           0       IN      A       192.73.252.25
store.ui.com.           0       IN      A       192.73.252.18

;; Query time: 116 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Wed Sep 27 11:00:40 MDT 2023
;; MSG SIZE  rcvd: 73

; <<>> DiG 9.16.1-Ubuntu <<>> store.ui.com @1.1.1.1
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 1691
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;store.ui.com.                  IN      A

;; ANSWER SECTION:
store.ui.com.           0       IN      A       192.73.252.18
store.ui.com.           0       IN      A       192.73.252.25

;; Query time: 69 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; WHEN: Wed Sep 27 11:00:43 MDT 2023
;; MSG SIZE  rcvd: 73

Let’s find out who they belong to.

25.252.73.192.in-addr.arpa. 14400 IN    PTR     pxy02-nsjc-c2szps.001.prd.c2szps.spscld.net.

Domain Name: SPSCLD.NET
Registry Domain ID: 2551237048_DOMAIN_NET-VRSN
Registrar WHOIS Server: whois.markmonitor.com
Registrar URL: http://www.markmonitor.com
Updated Date: 2022-04-11T16:44:25Z
Creation Date: 2020-08-06T17:43:40Z
Registry Expiry Date: 2024-08-06T17:43:40Z
Registrar: MarkMonitor Inc.
Registrar IANA ID: 292
Registrar Abuse Contact Email: abusecomplaints@markmonitor.com
Registrar Abuse Contact Phone: +1.2086851750

MarkMonitor appears to offer some kind of network security service. Further digging reveals that they provide the product that Comcast calls “SecurityEdge.” I explicitly turned this off on my DOCSIS modem on day one. I cannot configure the LTE modem myself, so have to contact support to have it disabled.

Frankly, I find it unacceptable that my ISP is attempting to intercept traffic to these sites, and has enabled this feature without my consent. It would be different if it were blocking known malware or abuse sites, but there is no reason for intercepting these. The fact that this behavior changed on its own is alarming.

I have lost all trust in Comcast and have given the required 30 day notice to close my account. I have to pay a $770 penalty to break my contract. I will be finding another ISP.

It is not just this sketchy behavior making me do this, either. Before I diagnosed the issue myself, both chat and phone support techs said there was no way to help me. Only after this investigation was I able to call again and tell them exactly how to fix it.

I also recall an interaction I had with support about a year ago, when my internet was going out daily for hours at a time. Upon complaining about this, they stated:

We don’t cause outages, we fix them.

Right.

Geocircles

Thu, 24 Aug 2023 00:00:00 +0000

It’s time to write a post about Geocircles. According to whois, I registered the domain over a year ago, in July 2022. The first commit was made even earlier, in February of that year.

History

If you haven’t seen it before, Geocircles is a game where you try to place a Street View location on the map. It was originally born as a two-player, turn based game. One player would pick a location; the other would guess. Then, they’d switch.

I’m not a frontend guy, so the UI was pretty clunky at first.

I originally stood up a proof-of-concept for this on my VPS, written in Python with the Flask framework. It couldn’t really support more than one game at a time. While I was working on scaling up the infrastructure, I released the daily-challenge version that is currently playable.

Over the course of a year, I collected 365 locations for the daily challenge. It recently ran out and started to repeat.

The UI has improved a little bit since then. Still not perfect, though.

So what happened to the two-player version? Well, I did rewrite the whole backend in Go. I designed a scalable architecture and deployed the thing in AWS. And, well, nobody played it. It only cost me about $10 a month to run, but when nobody actually played a game after two months I shut it down. That happened about a year ago now.

The daily challenge, at least, has a few loyal participants.

This post isn’t for me to complain about the lack of interest, though. It’s to look at the architecture.

Overview

The backend of Geocircles was split into two parts: an API service and a session service. The API service handled creation of games, and the session service hosted the games themselves.

The API service was entirely stateless. Instead of storing anything in a database, it just signed tokens that enabled players to connect to their sessions. This is basically all that it was responsible for.

The session service needed state, but this state was only needed for the duration of a particular game and thus could be stored in memory.

Both services were deployed as fleets of ECS containers. In front of them, a router service and a load balancer. Another service served the static frontend assets. Nothing too interesting to talk about there.

Events

Nearly every interaction with the frontend resulted in some state being transmitted to the game’s assigned session worker. These events were then relayed to the other player in the session.

For example, if one player moves their panorama view, it generates a pano_update event with the new pitch, heading, and zoom level. Then, the other player would receive it so they could watch them explore. Submitting a challenge or a guess also generated an event. A guess would also have a flag added by the backend, to mark it as correct or incorrect.

Based on the events it received, the session would transition between four states: player one choosing a challenge, player two solving the challenge, player two choosing, then player one solving. This would then repeat.

In the initial state, one player just waits for the other to connect. The state names here hint that the player who created the session was internally designated as the “host,” and the other as the “guest.”

Routing

Websockets handled this event transmission and state synchronization, in order to stay real-time. This presented a challenge. I didn’t want the backend to need to do any kind of synchronization, so it was desirable to somehow route both players in any particular session to the same session worker.

To achieve this without storing any shared state at all, I implemented a custom router service. It frequently polled ECS for an up-to-date set of session workers. This wasn’t perfect, but worked well enough. Then, requests were routed using consistent hashing, on the session ID. Once I figured that out, the rest was easy. Golang provides some pretty nice utilities for implementing a reverse proxy.

However, it meant I had to structure URLs such that the session ID was accessible to the router.

APIs and tokens

All told, the API only really has a few unique endpoints, looking like this.

/api/v1
- /new-game
- /join-game
/session
- /<game-id>
  - /<user-id>

The new-game endpoint would generate a game-id, and a user-id for each player. Each would call join-game to generate an auth token before connecting to the session. The auth token essentially consists of game-id, user-id, and a timestamp – all wrapped in an AES-GCM envelope. The tokens are then passed over the websocket in order to authenticate to the session worker.

This extra step was added because the auth token itself is too long to include in a nicely shareable URL. In retrospect, it isn’t necessary because the user-id is essentially treated like a credential anyway in this design. I think I had some plans to eventually prevent reuse, but it ended up not being worthwhile. It’s not like there’s any serious consequence here anyway.

Future

I might still re-launch multiplayer someday. I have some ideas to reduce the infrastructure cost to near-zero. In fact, its original launch was quite over-engineered. Everything is deployed by CloudFormation and supports autoscaling.

I haven’t load tested it, but when I do I expect to find that a single process can serve hundreds, if not thousands, of simultaneous sessions. At least I know that it’ll scale when I need it to.

Moving to Hugo and Cloudflare Pages

Wed, 23 Aug 2023 00:00:00 +0000

tl;dr: I migrated my blog from Jekyll to Hugo. It seems more better. I did my best to fix everything, but you find something broken, do let me know.

Previously, I had set my blog up on Github Pages with Jekyll as the SSG. I found Jekyll increasingly difficult to work with since then, so decided it was time to modernize by switching to Hugo. The directory structure makes a lot more sense with Hugo, and builds are noticeably faster. Getting it set up on a new box is also way easier, since you don’t have to fight with all the Ruby dependency nonsense. It’s pretty much a self-contained package.

GitHub pages was also somewhat annoying to get set up, but at least once it was configured it was set-and-forget. However, I took this opportunity to migrate anyway. I had set up some other blogs on Cloudflare Pages and found it very easy to work with. Builds of my site are faster there than they were with GitHub, but that might just be because of Hugo anyway. Since I already manage my DNS with Cloudflare, it only took a few clicks to get set up with a custom domain and all.

I did have to make a few changes to content to get everything working nicely.

I moved all my posts from standalone markdown files to directories, so I could co-locate the related images with them. First, this meant I had to change all of the image references from absolute to relative URLs – that was simple enough.

Next, I had to do something to avoid breaking existing links to posts. With Jekyll, my yyyy-mm-dd-slug.md filenames for posts would result in urls like /posts/slug, so I added the corresponding slug to the front matter of each post with this awful one-liner:

$ for f in ./*; do slug=$(echo $f | sed -re 's/^.\/[0-9-]+//'); sed -ire "/date: /e echo slug: $slug" $f/index.md; done

The final major fix was to get all the inline math rendering nicely with KaTex. Previously, I used one macro for all math, and had it all rendering as inline. That macro now made it all render as block, so I had to configure a custom one for inline and update all the posts that needed to use it.

Of course, theme customization was also necessary, and is largely still in progress.

The homelab post

Wed, 23 Aug 2023 00:00:00 +0000

It’s finally time for a post about my homelab. Yes, it’s overly complicated. No, it’s not the most extensive.

I had planned this post months ago (maybe a year ago) and never got around to writing it. I even made a diagram.

Overview

We have five (5!) VLANs on our home network.

One for random IoT crap, restricted from talking to the rest of the LAN.
One for the surveilance system.
One for Home Assistant, its tablet UIs, various ESPHome devices.
One for my work laptop.
One for a Steam Link to talk to a gaming VM.

Each of these, except for the Steam Link one, also has a corresponding SSID. Then, there is one more SSID for the default VLAN.

The majority of the infrastructural hardware is from Ubiquiti. It plays nice together and is easy enough to configure. The only exceptions are our two modems, provided by Comcast.

The “core” box

The coaxial drop for the internet comes down into a centrally located closet. There, one can currently find a 9u wall-mounted cabinet containing a 16-port PoE switch, a USG serving as a router, an LTE modem, a DOCSIS modem, some PoE injectors, a CloudKey, and a PDU – all hooked up to a little 500VA UPS.

The LTE modem serves as a backup when the primary cable internet goes down. Both modems are leased from Comcast Business, because for some reason they do not allow you to provide your own as they do for residential service.

There is a dedicated 20 amp circuit dropped at this box, as in a previous incarnation this closet was also home to a few beefier servers. This is the fourth different enclosure that this closet has held in three years. I do not keep it tidy.

From here, ethernet branches out to the roof, my office, the living room, and the primary bedroom.

The office

In my office, we find another enclosure – this one only 6u. It contains a 24-port switch, a UNVR (surveillance video recorder), and a Synology NAS, all attached to a 1500VA UPS.

This enclosure was once wall-mounted, but since has had casters jury-rigged to it and now lives on the floor. This switch used to sit in the core, but now most of its ports remain unutilized.

Atop this cabinet, we find an HP Z820 workstation with Proxmox installed. Inside are two E5-2670v2’s, 8TB worth of Intel DC SSDs, and an RTX 3060.

In addition to running some services like Home Assistant and a DNS server, there is also a Windows VM on this box to which the video card is attached. This box has its own dedicated 1500VA UPS, identical to the one in the cabinet.

My cat likes to sit on top of it. The dongles hanging haphazardly off the workstation are for Zigbee and Bluetooth.

Also located in my office is a little 8-port POE switch. Attached to it is an AP, a printer, and everything on my desk.

The roof

There’s a little three-port switch, powered by passive PoE, on the roof. It provides passthrough power to three Unifi cameras.

Another one of the ethernet drops on the roof crosses over to go into the garage. A third port connects an outdoor AP.

The living room

A UAP-IW-HD connects a Roku and the Steam Link, and also provides wifi coverage for the rest of the house. A very convenient piece of hardware.

Home Assistant stuff

The main Home Assistant VM runs on that HP box. The ZigBee dongle attached to it lets it communicate with 40 other devices such as buttons, sensors, lightbulbs, relays, and switches.

An enclosure formerly home to an alarm system mainboard now hosts a Rasperry Pi, which monitors the old system’s sensors and transmits their state via MQTT.

I have three Raspberry Pis with touchscreens serving as interfaces to Home Assistant. Two are wired in, powered by PoE splitters. A third was not in a convenient place for a new ethernet drop, so that one’s on wifi.

Other fun things on the Home Assistant VLAN include an ESPHome-based access reader, air quality sensors, smart speakers, a garage door open, power monitors for the washer and dryer, and an irrigation controller.

Others

In order to avoid TLS warnings on every internal service, I’ve set up Traefik VM to front them all with a custom domain. It gets a certificate from Let’s Encryupt using a DNS-based challenge, as otherwise the host is not accessible from WAN.

DNS is provided by Pi-hole, filtering requests for undesirable domains.

I designed another weird keyboard: tamago60

Mon, 24 Apr 2023 00:00:00 +0000

The original egg58 prototype used Choc V2 switches, which have MX-style stems and therefore support a wide selection of keycaps. However, it turned out that DSA, the shortest keycap profile widely available, would still bottom out if the switches were plate-mounted. The egg58 layout was not really conducive to PCB mounting, so I switched to original Choc switches.

This left me with a supply of Choc V2 switches and DSA caps which have gone unused for over a year now. I thought the egg58 was my endgame. It probably still is, but nonetheless I have made another keyboard. Just to put those spare parts to use.

I decided early on to go with an ortholinear layout, approximately 60% but arranged similarly to the egg58. At some point, I started considering adding a trackpoint to the board, like what they have on ThinkPads. It turned out that the availability of trackpoint modules isn’t too great, and there’s difficult to integrate, so I settled for a small joystick instead.

I stuck the joystick right in the middle of the keyboard, added two additional keys for left and right click, and landed on a semi-split design.

After a few hours in KiCad, I ended up with this.

The only difficult part was figuring out a solution for the 2u positions. As you can see here, there are three switch positions for the 2u keys. I found that some of my 2u keycaps had the stem in the middle, while others had two stems placed 1u apart. Therefore, I needed the provide the option to populate switches to support either.

I wanted to target using a Pro Micro for the MCU, since I had a bunch of spares of those lying around too. It turns out that it has just enough pins to support a 10x6 matrix and two ADC channels for the joystick. There is no wasted I/O on this board.

The next step was to order the prototypes. PCBWay kindly reached out to sponsor PCB prototyping for this project. Their ordering process was straightforward, with a lot of options available. Notably, they offer a matte black solder mask, which was not available from my previous PCB supplier. Having seen it on other peoples’ projects, I knew it would look great for this keyboard.

Unfortunately, right after submitting my order, I noticed some errors in the design. I missed copying some of the mounting holes when altering the footprint for those weird 2u positions. Fortunately, once I fixed it, their support team was able to quickly replace the Gerber files before the boards went to production.

It only took a week for the boards to arrive. The matte black finish looks great, and there is a noticible improvement in the quality of details compared to the PCB producer I used previously. The cuts are smoother than I’m used to, and the silkscreen resolution seems higher.

I also got an SMT stencil from PCBWay, which made it incredibly easy to assemble the board.

After adding switches, keycaps, and a 3D printed case, the tamago60 is complete.

Of course, the design is open source. I’ve also shared the project on PCBWay so you can easily order your own board:

I made my custom keyboard wireless

Mon, 20 Mar 2023 00:00:00 +0000

I don’t have a particular need for a wireless keyboard. I’m usually planted at my desk, with my laptop plugged into a dock and a wired keyboard readily available.

But, I thought it would be cool to make a Bluetooth version of the egg58.

It turns out, making a keyboard you’ve already designed wireless is pretty straightforward these days. Using a nice!nano microcontroller and ZMK, you can turn basically any keyboard based around the Pro Micro wireless almost instantly.

The nice!nano is a microcontroller module pin-compatible with the ubiquitous Pro Micro, but based around the nRF52840. That means it comes with built-in wireless connectivity. (Obligatory disclaimer: not sponsored, I just think it’s really neat.)

ZMK, as you could guess, is a keyboard firmware, but unlike QMK has support for BLE (and nRF chips). Of course, it even works for split keyboards.

According to ZMK’s power profiler, I can expect the left (main) side of my keyboard to last almost two weeks on a charge, with just a tiny 110mAh battery that fits conveniently under the nice!nano. The right half lasts much longer, allegedly only needing a charge every 3 months

Even though I didn’t need to, I altered my PCB design a bit, just to remove the positions for components that were no longer required. Of course, RGB had to go, since it would kill the battery immediately.

Anyway, the design files are here and firmware stuff here.

The intransmissible packet

Mon, 06 Mar 2023 00:00:00 +0000

In any network, it is a given that there will be loss. We’ve been making computers talk to each other for decades now, and that whole time scientists and engineers have been searching for better ways to compensate for weird things the network does to data in flight.

DALL-E, prompted “a lost packet in a computer network. minimal illustration, contemporary style, no text."

It’s been nearly 50 years since TCP was bestowed upon us, and for the most part it’s still the best we can do. Ignoring witchcraft like erasure codes, reliability is usually achieved through detection and retransmission. That is, if the sender thinks the receiver didn’t get a packet, it will just resend it. So, even when you roll your own reliability on top of UDP, it still usually ends up looking pretty similar to TCP. Even with all the improvements we’ve made, it’s still TCP.

Historically, this has worked pretty well. Network loss tends to be either sporadic or persistent. Once you’ve figured out collision and congestion, if you experience loss and a few retries don’t help then you’re probably out of luck.

But, what if you can’t retransmit that packet? What if that specific packet just can’t be delivered? The next packet arrives just fine. But retransmission isn’t helping the first one.

I was working on a project that involved processing high-bandwidth streaming data. Neither the data nor the processing matter, because today we were just playing some recorded data back to test network throughput. We had just upgraded the infrastructure, so we wanted to see what it could do.

A file was stored on one machine, and some software to eat it was running somewhere else. Because we wanted to avoid head-of-line blocking in this application, we had a custom reliability protocol over UDP. We could tolerate a loss if it meant avoiding a latency spike.

We had a huge library of test data, and had been playing most of it back without problems. Occasional loss was normal and expected. However, there was one dataset that we found would always drop the first packet. Normally, we wouldn’t even notice this. But, this dataset had a lower packet rate so the delayed start was perceptible.

The first few times we observed this, we thought nothing of it. The network did something weird. But, it was repeatable. And the reliability protocol had more than enough time to compensate for a random loss event. Why didn’t it in this case?

We were convinced we had discovered a bug. We started turning knobs to figure out under which conditions it could be reproduced. The first thing we tried tuning was the packet size. Originally, we were sending packets with a payload of something like 32 KiB. What happens if it’s only 8 KiB? No bug. 64 KiB? Still bugged.

I figured there must be something wrong with how we’re handling large buffers in either the sender or receiver. I started staring at the code. I analyzed the unit tests for a gap. But both programs were simple enough and tested comprehensively enough that I was at a loss.

We eventually took the new code and ran it on the old infrastructure. It worked there. This is truly an enigma.

Something inspired another engineer to start twiddling bits. Somehow, he figured out that changing either of two adjacent bytes at a certain offset in the file could bypass the bug. I knew this had been a crucial discovery.

I gave up on analyzing the code and convinced someone with more permissions than myself to collect a packet capture from both the source and destination hosts. I opened it in Wireshark and the first thing I saw was an IP packet that failed to be reassembled.

Sometimes, you start a packet capture at a bad time and miss something. That wasn’t the case here. On the sender side, I could see four fragments in the first datagram, sent about five seconds into the recording. On the receiver, I could see the first, third, and fourth fragments. The second fragment was nowhere to be seen.

Yes, the 32 KiB payload required fragmentation even with jumbo frames - I am not sure why that was the default size, and didn’t care to find out. But, I had to figure out why fragmentation was breaking the transmission of this file.

I continued looking at the packet captures - I could see several attempts at retransmission of this problem datagram, all unsuccessful. The second fragment never made it to its destination.

Going back to the other engineer’s discovery, I tried to find those two bytes he had been able to manipulate to evade the bug. Looking back to the sender capture, I found they landed on the third and fourth bytes of the IP payload in the frame that was getting dropped.

0x0e 0xc8

As a 16-bit big endian integer, that’s 3784. And it landed right where a UDP port number should go.

Port 3784 belongs to a protocol called Bidirectional Forwarding Detection, or BFD. In short, it allows network devices to send each other high-rate pings in order to detect when a link fails. That should be irrelevant, because our packets were not destined for port 3784. And yet, under fragmentation, that port number ended up in the spot that it would if it were.

Thus, we finally had a hunch. Something on the network is eating this packet, because it erroneously thinks it’s a BFD message.

We tested some other scenarios. What if the destination port were indeed 3784? The failure is reproduced every time, even without fragmentation. If we put those two bytes at the same offset in the third fragment instead? The same. We took this as confirmation of our hypothesis.

Indeed, even without a valid UDP header, this IP fragment was being interpreted as BFD and taken up into the control plane, just on the basis of “yep, that’s where the port number should go.”

The new infrastructure’s switches supported BFD, while the old ones didn’t. We weren’t using the feature, and in fact it was turned off, but the problem occurred nonetheless.

We filed a support case with Cisco, who let us know that they were already aware of the bug. It was related to another issue reported to them by another customer. This was a bit anticlimactic. But, fortunately that meant it was only a couple of weeks before a firmware update was available to fix this.

Along with the firmware update, we were given a link to a security advisory regarding it. Apparently there was actually a DOS vulnerability hiding somewhere behind this bug, now known as CVE-2022-20623.

I am not a security researcher, so I wasn’t concerned with finding a vulnerability but rather just fixing a bug. I wasn’t disappointed to not be first to discover it - I was just happy to have had an interesting problem to solve. Nonetheless, sometimes when I recount this story to friends I may take a bit more credit than I deserve, hinting toward Cisco’s attribution,

This vulnerability was found during the resolution of a Cisco TAC support case.

and taking it as a small claim to fame.

Keycard door access in Home Assistant

Thu, 12 Jan 2023 00:00:00 +0000

I first started fantasizing about keycard access to my home when Ubiquiti announced the G4 Doorbell Pro. I’ve had the original G4 Doorbell for a few years now, and it’s been great. However, when the G4 Doorbell Pro finally came out of Early Access, it lacked the RFID feature that was originally advertised.

Based on community consensus, it doesn’t seem like we’ll be getting that feature anytime soon, so I decided to build my own RFID reader that would semi-integrate with the G4 Doorbell I already had.

Using ESPHome, an inexpensive NFC module, and a 3D printed enclosure, I sort of got what I wanted.

Home Assistant has built-in support for RFID tags, which I could leverage to trigger an automation. All I really needed to do was figure out how to feed it tags, and find a lock that would integrate.

I ended up using a Kwikset Convert to add Zigbee control to the back of my Schlage deadbolt, since it was less ugly than Schlage’s own options and I didn’t want to rekey everything again.

I borrowed the BOM from an existing Home Assistant tag reader project and added some components to power it. I changed the WS2812 for an SK9822, since I already had some on the shelf. I also removed the buzzer since I wouldn’t really want to use it anyway.

My original plan was to make this device wireless, but the ESP8266 makes deep-sleep hard to work with, so the 820 mAh battery I originally speced only lasted 6 hours. Thus, I resorted to hardwiring.

Using the 1N4001 as a half-wave rectifier with some buck converters I already had, I was able to get a usable 5V DC from the doorbell’s 18V AC circuit. The converter’s built-in 100uF capacitor was sufficient to smooth the rectified voltage.

I forgot to take a picture of the hardwired version, so here’s the guts from the battery-powered iteration:

After assembly, I had to figure out the correct incantations to get everything working in ESPHome. I modified the yaml from the above project, added some other stuff from the internet, and ended up with this:

esphome:
  name: door_card_reader
  on_boot:
    priority: -10
    then:
    - wait_until:
        api.connected:
    - light.turn_on:
        id: status_led
        brightness: 100%
        red: 0%
        green: 0%
        blue: 100%
        flash_length: 2000ms

esp8266:
  board: esp01_1m
  framework:
    version: 2.7.4 # required for fastled on this platform

logger:

api:
  encryption:
    key: (removed)

ota:
  password: (removed)

wifi:
  ssid: !secret wifi_ssid
  password: !secret wifi_password

captive_portal:

i2c:
  scan: False
  frequency: 400kHz

time:
- platform: sntp
  on_time:
  - seconds: /10
    then:
      if:
        condition:
          api.connected:
        then:
        - light.turn_on:
            id: status_led
            brightness: 50%
            red: 0%
            green: 0%
            blue: 100%
            flash_length: 250ms
        else:
        - light.turn_on:
            id: status_led
            brightness: 50%
            red: 100%
            green: 0%
            blue: 0%
            flash_length: 250ms

pn532_i2c:
  id: pn532_board
  on_tag:
    then:
    - if:
        condition:
          api.connected:
        then:
        - light.turn_on:
            id: status_led
            brightness: 100%
            red: 0%
            green: 100%
            blue: 0%
            flash_length: 500ms
        - delay: 0.15s
        - homeassistant.tag_scanned: !lambda |-
            return x;
        else:
        - light.turn_on:
            id: status_led
            brightness: 100%
            red: 100%
            green: 0%
            blue: 0%
            flash_length: 500ms
        - delay: 0.15s
  on_tag_removed:
    then:
    - homeassistant.event:
        event: esphome.tag_removed

light:
  - platform: fastled_spi
    id: status_led
    internal: true
    chipset: APA102
    clock_pin: 14
    data_pin: 13
    num_leds: 1
    rgb_order: BGR
    name: "Door Card Reader Status LED"

sensor:
  - platform: wifi_signal
    name: "Door Card Reader Wifi Signal dB"
    id: wifi_signal_db
    update_interval: 60s
    entity_category: "diagnostic"

  - platform: copy
    source_id: wifi_signal_db
    name: "Door Card Reader Wifi Signal Percent"
    filters:
      - lambda: return min(max(2 * (x + 100.0), 0.0), 100.0);
    unit_of_measurement: "Signal %"
    entity_category: "diagnostic"

It’ll do nothing special with tags, and just report them directly to Home Assistant. It also has a nice status LED functionality.

To avoid drilling a second hole in my door trim, I designed a bracket which would mount behind the Unifi Doorbell and give me some space to splice the wires. The enclosure for the card reader would essentially just be a box that I’d shove everything into, because I was too lazy to design any mounting features.

The model is available on Thingiverse.

This is what it looks like installed:

Ignore the doorbell’s complaint about Wifi - it had just booted up and apparently hadn’t found the AP yet.

So far it is serving me well. It gives me the ability to hand out dirt-cheap tags instead of get traditional keys copied every time I need someone to be able to get into my house. This is great for guests and petsitters who you’d otherwise have to trust to return the spare key before you need to give it to the next person.

These tags also have the benefit of being easily revocable credentials. It’s a lot harder to rekey all my locks than it is to remove the automation for a tag in Home Assistant. This would mean a cloned credential also has less impact than a cloned key.

Typing on a weird keyboard: a retrospective

Thu, 29 Dec 2022 00:00:00 +0000

It has been just over a year since I finished assembling the first version of the egg58, my custom split keyboard. Thus, it seems like an appropriate time to make note of how it has treated me.

To recap, I once doubted the idea of a split or ortho keyboard. Eventually, I tried out an Ergodox and came to realize it really does offer increased comfort. However, eternally picky, I decided I needed to design my own layout. Thus, the egg58 was born.

So, how has the keyboard performed for the past year? We’ll look at my typing speed, how it has impacted my use of “normal” keyboards, and its ergonomics.

The typing test results discussed here are from keyhero and speedcoder. The egg58 tests were performed using Kailh Choc Red Pro switches, and the standard layout tests were on a 60% ANSI with Kailh Box Hako Violets, so it isn’t quite an apples-to-apples comparison.

Speed and progress

At first, I was discouraged as I could barely type on the thing. In my first day using it, I averaged a spectacular 40 words per minute, down from my benchmark of 120 wpm on my 60% ANSI. However, a wise waterfowl told me,

split isn’t about speed
it’s about not f***ing up your wrists

and I preservered.

Unfortunately, being able to type at a reasonable pace is ultimately part of my job description. Therefore, over the next few months, I took a series of typing tests to gauge my progress.

I took dozens on the first day, trying to see if I could build the muscle memory. I could not. My very first test measured 24 wpm, and only twenty minutes later I reached 40 wpm. This is where I stayed for the rest of the day.

A week later, I was averaging 60 wpm. The next, 75 wpm. After a month, 100 wpm.

Now, after a year, I am able to comfortably reach 120 words per minute.

However, regular typing exams are not a complete measure of performance in this case. Being a software engineer, I probably type a lot more symbols than the average bear. And, the egg58’s biggest deviation from a standard ANSI layout is in where those keys are located. To that end, I scored 70 words per minute in both C++ and Python tests.

Back and forth

One of the complaints I heard about split keyboards is that it is difficult to switch between them between them and a standard layout. For the first five months of the egg58’s tenure, I was forced to use a normal keyboard at work. I suppose going back and forth every day allowed me to keep the muscle memory during that time, as I did not really struggle to switch between them.

However, in May I began a new remote job and began using the egg58 almost full time. The only exceptions are gaming, where I was not able to adapt my muscle memory of where WASD are located, and the occasional use of my laptop, which unfortunately has a staggered-row keyboard physically attached.

Since then, my aptitude with a “normal” keyboard has somewhat faded – I can only achieve 95 wpm on the same keyboard where I generated my benchmark of 120 wpm one year ago. More notably, my accuracy is much lower – about 97% versus 99.9% with the split.

Comfort

I’m not an expert in ergonomics, but it certainly feels like using a split keyboard full time is more comfortable. My wrists used to feel strained after a day of work, but no longer. I also had a weird shoulder pain creep up every once in a while, but it too has ceased.

The key point here seems to be that the angle of my wrists is not constrained by the layout of the keyboard. I think that the staggered-column layout also helps. My fingers have to do less lateral movement than with a staggered-row design.

Conclusion

The time I invested into the design of the egg58 was definitely worthwhile. In addition to being a valuable learning experience in electronics design, it has given me a tangible benefit in not f***ing up my wrists.

Also, it serves as a conversation piece whenever someone sees it in my office. But I’ve yet to have anyone ask me to make them one.

Adapting CEC controls to RS-232

Mon, 26 Dec 2022 00:00:00 +0000

In this article, we will instill a Raspberry Pi with some magic in order for a TV to talk to a hifi amplifier. By the end of the journey, we’ll have the Pi pretending to be an AV receiver, intercepting commands from the HDMI CEC bus, and relaying them to the amp via a combination of serial and IR control outputs.

Don’t feel like reading? Here’s the GitHub link.

I have always been a fan of the sound of an analog amplifier. I’m not the kind of audiophile who is going to claim that I can discern the imperfections in audio reproduction from a standard home theater receiver, but they certainly lack a warmth that analog equipment offers.

Because of this, I’ve always used vintage audio gear. For the past five years or so, my main integrated amplifier has been a Yamaha A-760. Unfortunately, it has been showing its age, needing repairs of some sort every six months or so. I have finally grown tired of opening it up and decided to modernize my setup.

I found a modern hifi amp from Cambridge Audio, the CXA61. It has a TOSLINK DAC, A/B stereo pairs, a builtin bluetooth receiver, and a design that meets my aesthetic preferences. It was missing two things from my wishlist: CEC control and a phono preamp. The phono preamp was an easy fix, just requiring adding a separate component to the setup. Achieving CEC control, however, would be a small challenge.

It’s worth noting that previously, I was using a standalone ARC DAC which attenuated the line-level output based on CEC volume controls. The amplifier itself predated even CDs, so full digital control was out of the question. Therefore, the feature was limited. Migrating from ARC to TOSLINK caused me to lose even that, though at least now I could benefit from a much better DAC.

Anyway, the CXA61 has an RS-232 serial port. Unfortunately, it only exposes power, source and mute controls over this interface. The IR remote control, however, is capable of controlling volume. I wrote to Cambridge Audio about this discrepancy, and apparently it is due to there being no feedback capability from the volume knob, so it might not behave properly in some serial control systems. At least it has a 3.5mm TRS jack for IR repeater input, so the remote control capabilities can be emulated relatively easily. I figured by hacking both the IR and serial inputs, I should be able to create a full-featured CEC bridge.

My requirements were as follows:

When the TV turns on, the amp should turn on and switch to the correct input.
When the TV turns off, the amp should turn off.
Volume and mute controls on the original TV remote work as intended.

The HDMI CEC standard defines a common protocol for devices to interoperate and handle these inputs. CEC is a 1-wire serial bus that sits on its own pin in an HDMI port, and seemed relatively easy to interface with. I was originally planning building some AVR-based monstrosity to interact with the CEC bus, but then I realized I still had a spare Raspberry Pi sitting on the bench. The Raspberry Pi’s HDMI port has built-in CEC support. There would be no real hardware work required for this project.

I hooked up the Pi to my TV, and used cec-client to figure out what messages it emitted. Decoding the messages with CEC-o-Matic, I learned that:

If I present as an audio system, the TV will send a System Audio Mode Request when it turns on.
The TV will broadcast a Standby message when it turns off.
Volume controls and mute will be passed through as keypresses.
Occasionally the TV will query for the audio system’s volume level and mute status.

I found pyCEC which would allow me to easily interface with the CEC bus, handle these commands, and send the necessary replies. Now, it was simply a matter of translating the controls to the CXA61’s native protocol.

I used pyserial with an FTDI-based USB-to-RS-232 adapter to handle the power and mute functions, and PiIR to bitbang IR codes over a GPIO pin which I spliced into a TRS cable for the IR port.

The serial protocol was fortunately well documented, and PiIR could learn the IR codes off the original remote with the help of an IR receiver module.

After implementing these controls, I just had to tie everything together with the right CEC events. Finally, I had created cec2rs232.

In theory, this project should be flexible enough to work with other devices. For the time being, I’ve only implemented the protocol of the one I own. Pull requests, however, are welcome.

Zip tie to smartify: the dumbest way to upgrade your appliance

Fri, 16 Sep 2022 00:00:00 +0000

Any owners of Samsung laundry appliances out there will know they play a distinctive jingle when finishing a job. I find it incredibly annoying (mostly because it becomes stuck in my head for at least two days after hearing it), however disabling the chime causes me to forget to unload my clothes until I run out of boxers.

I’m not a fan of most smart home products, because I distrust the cloud and I like to have control of my data. The only off-the-shelf “smart” products I have are simple Zigbee (802.15.4) devices, and a Roomba (because it can be configured for local control, with its cloud functionality disabled). Thus, when I bought a new washer and dryer a few years ago, I opted for non-smart variants, thus sacrificing a built-in notification capability.

In this post, I’ll exhibit my stubbornness by explaining how I used Home Assistant, ESPHome, and some magic to get my dryer to tell me when it’s done without playing that awful song.

Home Assistant is open source smart home management software, and I’ve been using it for over a year now to control and monitor various things in my home. The ESPHome is a related project that lets you easily integrate Espressif microcontrollers to monitor things that might not have native integrations.

My first attempt to get monitoring for my washer and dryer in Home Assistant involved vibration sensors, however it turned out that modern appliances are simply too well-balanced for this to be a reliable method of detecting if they are running. Thus I switched my focus to determining whether they were actively using any power.

Due to pixie magic (i.e., electromagnetics, physics, etc.), it is possible to detect an electric field when there is current flowing through a wire. Ideally, you can clamp something around a wire and the job is done. This is incredibly reliable, however requires separating the hot and neutral conductors, and I didn’t feel like putting in that kind of effort.

Instead, I redirected my effort into searching for a sensor that can detect current in a cord without splitting it apart. I found this module from Modern Device, which uses a pair of Hall effect sensors to be able to differentially sense a pair of conductors. Magic.

I purchased a few NodeMCU ESP8266 to hook them up to. In retrospect, ESP32 would have been a better choice, as it has multiple ADC lanes, and I could have used one MCU for both appliances.

I connected the sensor module to one, configured ESPHome, and flashed the firmware. The configuration is incredibly straightforward.

sensor:
  - platform: adc
    pin: A0
    name: "Dryer current sensor voltage"
    update_interval: 1s
    filters:
    - sliding_window_moving_average:
        window_size: 15
        send_every: 15

Then, the module is simply zip-tied to the dryer power cord.

I did some laundry, and it got me a graph like this in Home Assistant:

I did not care to do any absolute calibration, so just set up an automation to notify when this voltage goes below a threshold:

alias: Dryer notify
description: ""
trigger:
  - platform: numeric_state
    entity_id: sensor.dryer_current_sensor_voltage
    for:
      hours: 0
      minutes: 2
      seconds: 0
    below: 0.5
condition: []
action:
  - service: notify.notify
    data:
      message: "Dryer has finished"
mode: single

That’s it. A lot of effort exerted, all for this:

The process for the washer was largely the same, but with a different voltage threshold.

More Home Assistant content coming soon, maybe, if I’m motivated.

Scraping wildfire timeseries data with NLP

Sun, 05 Jun 2022 00:00:00 +0000

New Mexico is facing a severe fire season, and I wanted to visualize the progression of the major fires that are currently ongoing. However, I could not find a usable source of data on daily size and containment statistics.

A repository of daily updates exists, however as articles rather than structured data. I ended up scraping the website and applying natural language processing to extract the desired data.

The library and utility is available on my GitHub.

InciWeb contains a collection of the daily updates for the fires of interest, and by using Beautiful Soup I was able to collect and scrape all the articles. This was fairly straightforward.

Extracting the information of interest proved to be the more difficult task. Posts on the site come from various agencies and thus do not follow a consistent format. In order to handle all posts in generality, I turned to a NLP technique.

Using NLTK, I converted the words of the articles into their stems, then found the concordance lists of keywords pertaining to the data I needed.

After tagging the concordance strings with parts of speech, I was able to find the closest number to each keyword; this proved to be a fairly reliable heuristic.

def get_best_value(text, value_tag, label_values):
    """ Apply the heuristic to extract the closest string with the specified tag
        to one of the label values
    """
    val = None
    dist = 0

    concordance_list = []
    for l in label_values:
        concordance_list += text.concordance_list(l)

    for s in concordance_list:
        tags = nltk.pos_tag(nltk.tokenize.word_tokenize(s.line))

        label_candidates = []
        value_candidates = []
        
        for i, (token, tag) in enumerate(tags):
            if tag == value_tag:
                value_candidates.append((token, i))
            elif token.lower() in label_values:
                label_candidates.append((token, i))

        for l, i in label_candidates:
            for v, j in value_candidates:
                d = abs(i - j)
                if val is None or d <= dist:
                    val = v
                    dist = d
        
    return val

def extract_data(raw_article_text):
    """ Returns the size and containment of the fire by applying the heuristics to the text
    """
    stemmer = nltk.stem.WordNetLemmatizer()
    words = nltk.tokenize.word_tokenize(raw_article_text)
    tokens = [stemmer.lemmatize(w) for w in words]
    text = nltk.text.Text(tokens)
    size = get_best_value(text, 'CD', ['acre', 'acres'])
    containment = get_best_value(text, "CD", ['contained', 'containment'])
    return size, containment

The rest of the code is not interesting, but eventually I was able to automate gathering most of the data I needed to produce this plot.

I used the utility in the above GitHub repository to dump the timeseries data from InciWeb to CSV, gathered some historical numbers from SWCC, and wrote a script to put it all together with matplotlib.

Designing a split keyboard: the egg58

Tue, 12 Apr 2022 00:00:00 +0000

Late last year, I was finally exposed to the comfort of the Ergodox – a staggered-column, split keyboard. However, having been using 60% keyboards for the previous decade, I felt like it had too many keys. While some minimalist split ortho options did exist, I found many of them too minimal or too diverged from the Ergodox design, which I did like in theory.

Like any sane person, I proceeded to learn an entirely new skill in order to fulfill my need for a keyboard that was kinda like an Ergodox but with fewer keys.

Settling on a layout

There exists a tool for designing a custom keyboard layout: the appropriately named keyboard-layout-editor.com. I messed around with it for a while until I landed on a layout.

The stagger mimics that from the ergodox, but the thumb cluster is reduced, the innermost column is removed, and the left column is 1u.

Because the tool doesn’t handle split keyboards well, I settled with having just designed half. I managed to generate a switchplate SVG from this design, import it into a CAD tool, and 3D print a “test plate” of sorts so I could feel out the layout.

It felt pretty comfortable, so I proceeded to figure out how to make this into something real. I decided I wanted the board to be low-profile but not basically-a-laptop-profile, so I decided on Kailh Choc version 2 switches.

A crash course in circuit design

The easy way out would have been to hand-wire a keyboard in a 3D printed case. However, I wanted some luxury features: hotswap sockets and RGB.

Neither of these would have been possible with a hand wired build, so I proceeded to install KiCad, an open source electronic design automation suite. I have emphasized several times on this blog that I am not an electrical engineer. Therefore, this new flavor of CAD was to be an entirely new skill.

I at least had some vague idea of how schematics worked, so I managed to put something together. I won’t bore you with a figure here, because it’s basically just “a bunch of switches connect to a microcontroller.” The PCB layout and trace routing are a little bit more interesting, but I appear to have lost the original design files, so I can’t provide an image of that for the first version either.

After a couple weeks, I managed to complete my design and sent the Gerber files off to One Of Those Overseas Manufacturers for prototype production.

After about a month, some PCBs arrived. I built the keyboard. I forked QMK. I flashed firmware. It worked.

Ignore my dirty mousepad.

Getting it working is not enough

It turns out that the only keycap profile that really works with Choc v2 is DSA. There are also no shine-through keycaps in DSA profile. And it’s hard to find a legended keycap set that will work well for a custom layout like this.

I eventually spent Too Much Money on a set of keycaps and still wasn’t satisfied, as with a switchplate DSA still bottoms out. I then found a nice set of shine-through caps for Choc v1, and decided to redesign the entire board around them.

A new design cycle started, around the new switch footprint and targeted keycap set. A few other minor changes were made. A logo was placed on the PCB, some prettification was done, and options for other MCUs and communication protocols were added.

Several months later, some PCBs showed up at my door. I built the board. I installed the keycaps.

Epilogue

It’s been less than twenty four hours with the new keyboard. So far, I have nothing to complain about, except the fact that this keycap set didnt have + and - keys.

I’m happy with it for now.

Here are some Github links:

A demonstration of TDOA multilateration

Sun, 20 Feb 2022 00:00:00 +0000

Yesterday, a thread popped up on r/Albuquerque where users attempted to identify the source of a loud explosion in the early morning. A few individuals provided their locations and timestamps at which their surveillance cameras detected the sound. I thought I could use this information to pinpoint its source using multilateration.

What is multilateration?

Multilateration is the process by which an energy source can be located based on observations of the energy at different locations. The energy can be a light, radio, acoustic, or seismic wave; the time the wave takes to travel to each observer allows us to solve a system of equations to identify the unknown source. The process works mostly the same for all types of waves, just the speed of transmission varies. In this case, we’ll be using acoustic energy, which is fairly resilient to timing errors. Sound takes about five seconds to travel one mile, so even if the timestamps are not accurate we can get fairly close to the source.

Multilateration has many real-world applications, for example the acoustic form we’re studying today is used in law enforcement and defense, and its radio counterparts are used for aircraft tracking and navigation. Its wide range applicability has resulted in this being a very well-studied problem. However, one might expect it to be difficult to understand. In this article, we’ll look at the mathematics of multilateration and find in it a surprising simplicity.

How does it work?

Since we want to locate the event based on the differences in times of detection, what we’re doing is actually TDOA (time difference of arrival) multilateration. For simplicity, we’ll assume a flat Earth and attempt to solve the problem in the Cartesian plane. If we know the difference in arrival time between two points, we can describe the possible locations of the event as a hyperbola. In fact, one of the ways a hyperbola can be defined is as the set of points which have a fixed distance between two fixed points.

The hyperbola we’re interested in can be described as

$$\sqrt{(x - x_1)^2 + (y - y_1)^2} - \sqrt{(x - x_0)^2 + (y - y_0)^2} - v(t_1 - t_0) = 0$$

where $x_0, y_0$ is the position of the first observer, $t_0$ is its observation time, $x_1, y_1, t_1$ are those parameters of the second observer, and $v$ is the propagation speed of the wave we’re interested in.

It’s easy to identify the two instances of the Cartesian distance formula here. In fact, the first square root represents distance between the event and the second observer, and the second gives the distance between the event and the first observer. Therefore, the equation can easily be parsed as providing the set of points that have distance $v(t_1 - t_0)$ from both observers. We’ll refer to this quantity proportional to the difference in arrival times as a pseudodistance.

For example, consider the case where the two observers notice the event at the same time. We can intuit that we should observe a line.

If we were to determine that the TDOA between the observers correlates to a pseudodistance of 3, then we can begin to observe the development of the hyperbolic curvature. In this case, we’ve made the event closer to the top-left observer.

By adding a third observer, we can draw a second hyperbola.

In this case, we’ve set a pseudodistance of 2 between the first and third observers’ observations. The intersection between the two hyperbolae gives us a solution indicating the source of the event.

A practical example

To protect the privacy of the users who contributed, we’ll walk through a fake example instead of the one discussed on Reddit. I’ve generated some points in the middle of the desert with a synthetic event. We’ll be using Python code to solve the multilateration.

First, let’s define the observation locations and timestamps.

measurements = [
    [34.888, -103.826, 0.0],
    [34.931, -103.805, 0.6],
    [34.921, -103.781, 2.5]
]

This is simply a list of the latitudes and longitudes of the observers, with timestamps representing the TDOA from the first observation in seconds.

Now that we have input data, the first step is to convert the geodetic coordinates to something we can work with more easily. To keep things simple, we’ll go to ENU (east-north-up) coordinates centered at the first observer, and discard the up component to keep things in the plane.

import pymap3d as pm

lat0, lon0, t0 = measurements[0]
for i in range(len(measurements)):
    lat, lon, t = measurements[i]
    e, n, _ = pm.geodetic2enu(lat, lon, 0, lat0, lon0, 0)
    measurements[i] = [e, n, t]

After this, measurements has the content [[0.0, 0.0, 0.0], [1918.6593908214895, 4770.574154674195, 0.6], [4111.911492926018, 3661.9046769939555, 2.5]], where each entry gives us the distance east, north from the first observer (in meters) and its time of observation.

Since we’re working with meters and seconds, let’s define the speed of sound with respect to those units:

speed = 343. # m/s

Next, we need to figure out how to define and solve the system of hyperbolic equations. I’m going to use scipy’s least-squares solver so we’ll specify the system in a way that we can use with it.

import numpy as np

def functions(x0, y0, x1, y1, x2, y2, d01, d02, d12):
    """ Given observers at (x0, y0), (x1, y1), (x2, y2) and TDOA between observers d01, d02, d12, this closure
        returns a function that evaluates the system of three hyperbolae for given event x, y.
    """
    def fn(args):
        x, y = args
        a = np.sqrt(np.power(x - x1, 2.) + np.power(y - y1, 2.)) - np.sqrt(np.power(x - x0, 2.) + np.power(y - y0, 2.)) - d01
        b = np.sqrt(np.power(x - x2, 2.) + np.power(y - y2, 2.)) - np.sqrt(np.power(x - x0, 2.) + np.power(y - y0, 2.)) - d02
        c = np.sqrt(np.power(x - x2, 2.) + np.power(y - y2, 2.)) - np.sqrt(np.power(x - x1, 2.) + np.power(y - y1, 2.)) - d12
        return [a, b, c]
    return fn

We’ve added the third hyperbola which represents the TDOA between the second and third observers, in addition to the two we have already discussed.

To help out the solver a bit, we can also specify the Jacobian matrix of this system. This is just a collection of the partial derivatives of each function with respect to each independent variable.

def jacobian(x0, y0, x1, y1, x2, y2, d01, d02, d12):
    def fn(args):
        x, y = args
        adx = (x - x1) / np.sqrt(np.power(x - x1, 2.) + np.power(y - y1, 2.)) - (x - x0) / np.sqrt(np.power(x - x0, 2.) + np.power(y - y0, 2.))
        bdx = (x - x2) / np.sqrt(np.power(x - x2, 2.) + np.power(y - y2, 2.)) - (x - x0) / np.sqrt(np.power(x - x0, 2.) + np.power(y - y0, 2.))
        cdx = (x - x2) / np.sqrt(np.power(x - x2, 2.) + np.power(y - y2, 2.)) - (x - x1) / np.sqrt(np.power(x - x1, 2.) + np.power(y - y1, 2.))
        ady = (y - y1) / np.sqrt(np.power(x - x1, 2.) + np.power(y - y1, 2.)) - (y - y0) / np.sqrt(np.power(x - x0, 2.) + np.power(y - y0, 2.))
        bdy = (y - y2) / np.sqrt(np.power(x - x2, 2.) + np.power(y - y2, 2.)) - (y - y0) / np.sqrt(np.power(x - x0, 2.) + np.power(y - y0, 2.))
        cdy = (y - y2) / np.sqrt(np.power(x - x2, 2.) + np.power(y - y2, 2.)) - (y - y1) / np.sqrt(np.power(x - x1, 2.) + np.power(y - y1, 2.))

        return [
            [adx, ady],
            [bdx, bdy],
            [cdx, cdy]
        ]
    return fn

As a last preparation before solving, we can provide an initial guess for the location of the event. We’ll just use the average of the three observer locations. This helps avoid issues with points for which the function is not well-defined.

xp = np.mean([x for x,y,t in measurements])
yp = np.mean([y for x,y,t in measurements])

Finally, we can run the solver:

import scipy.optimize as opt

x0, y0, t0 = measurements[0]
x1, y1, t1 = measurements[1]
x2, y2, t2 = measurements[2]

F = functions(x0, y0, x1, y1, x2, y2, (t1 - t0) * speed, (t2 - t0) * speed, (t2 - t1) * speed)
J = jacobian(x0, y0, x1, y1, x2, y2, (t1 - t0) * speed, (t2 - t0) * speed, (t2 - t1) * speed)

x, y, _ = opt.leastsq(F, x0=[xp, yp], Dfun=J)

This gives us a solution of 1109.306162, 2213.711852 in ENU space relative to the first receiver. We can convert that back to geodetic coordinates to make it more useful:

lat, lon, _ = pm.enu2geodetic(x, y, 0, lat0, lon0, 0)

We have now determined the location of the event to be 34.907954, -103.813862.

The system of hyperbolae and the solution can be visualized:

import matplotlib.pyplot as plt

# Create reasonable x, y bounds for visualization
max_x = max(x0, x1, x2, x)
min_x = min(x0, x1, x2, x)
range_x = max_x - min_x
min_x -= range_x * .2
max_x += range_x * .2

max_y = max(y0, y1, y2, y)
min_y = min(y0, y1, y2, y)
range_y = max_y - min_y
min_y -= range_y * .2
max_y += range_y * .2

# Create a grid of input coordinates
xs = np.linspace(min_x, max_x, 100)
ys = np.linspace(min_y, max_y, 100)
xs, ys = np.meshgrid(xs, ys)

# Evaluate the system across the grid
A, B, C = F((xs, ys, np.zeros(xs.shape)))

# Plot the results
plt.scatter(x0, y0, color='r')
plt.scatter(x1, y1, color='g')
plt.scatter(x2, y2, color='b')
plt.scatter(x, y, color='k')
plt.contour(xs, ys, A, [0], colors='y')
plt.contour(xs, ys, B, [0], colors='m')
plt.contour(xs, ys, C, [0], colors='c')
plt.show()

We’ve somewhat abused matplotlib’s contour function here by evaluating the grid and plotting the contour for just the zeros of the function. The result is the following:

This allows us to visually confirm that the reported solution is at the intersection of the hyperbolae.

We can also derive each observer’s distance from the event and the time-of-flight of the energy.

d0 = np.sqrt(np.power(x - x0, 2.) + np.power(y - y0, 2.))
d1 = np.sqrt(np.power(x - x1, 2.) + np.power(y - y1, 2.))
d2 = np.sqrt(np.power(x - x2, 2.) + np.power(y - y2, 2.))
t0 = d0 / speed
t1 = d1 / speed
t2 = d2 / speed

This lets us know that the distances from each observer are 2476 m, 2682 m, and 3334 m respectively; the corresponding times of flight are 7.2 s, 7.8 s, and 9.7 s. This confirms the TDOAs of 0.6 s and 2.5 s that we provided as input.

We can also use this information to plot circles around each observer, which gives us a more traditional triangulation diagram, though in this case it is synthetic.

def circle(cx, cy, r):
    def fn(x, y):
        return np.sqrt(np.power(x - cx, 2.) + np.power(y - cy, 2.)) - r
    return fn

c0 = circle(x0, y0, d0)
c1 = circle(x1, y1, d1)
c2 = circle(x2, y2, d2)

plt.scatter(x0, y0, color='r')
plt.scatter(x1, y1, color='g')
plt.scatter(x2, y2, color='b')
plt.scatter(x, y, color='k')

plt.contour(xs, ys, c0(xs, ys), [0], colors='r')
plt.contour(xs, ys, c1(xs, ys), [0], colors='g')
plt.contour(xs, ys, c2(xs, ys), [0], colors='b')
plt.show()

We can see that the intersection of the circles also lies at the event.

Conclusion

I apologize that this is only a shallow overview of multilateration (mostly because I only have a shallow understanding) and for the simplicity of the example. However, two-dimensional multilateration still provides the groundwork for the three-dimensional applications I mentioned earlier. In those cases, we’re solving for the intersection of hyperboloids instead of hyperbolae, for which at least four observers are required. In applications like low-frequency radio waves, the curvature of the earth also needs to be considered, making the mathematics much more complicated. In seismology, variations in Earth’s density also introduce additional complexity. I like the acoustic example given here because of its simplicity, and hopefully some readers found the exercise interesting nonetheless.

How not to build a gaming PC

Sat, 05 Feb 2022 00:00:00 +0000

Over the past month and a half, I’ve been battling the various ghosts and demons that have apparently taken residence in the ancient server chassis that I am now using as a backend for a Steam Link. Since I doubt anyone else on the Internet is insane enough to do this, I suppose I am mostly making this post for myself to refer back to when it inevitably breaks. If, by chance, you do find this useful, then that is merely a coincidence.

Prologue

My main PC runs Ubuntu, and I have been maining Linux for over a decade now. Most of the games I play have native Linux support or at least work well in Proton. Sometimes, however, a game is more suited to sitting on the couch with a controller than sitting at my desk with a keyboard and mouse. Even in those cases, streaming to a Steam Link attached to my TV works well.

Fall Guys was one such couch game, and unfortunately it also ended my run of smooth sailing with my gaming experience on Linux. With the introduction of an anticheat in late 2020, it would no longer work under Proton. To remedy this situation, I installed Windows on another SSD and I found myself dual booting for the first time in at least eight years.

Now, to play Fall Guys, I just had to get up off the couch, walk to my office, reboot into Windows, log in, then connect to it from the Steam Link.

This was clearly too much work, so it was obviously time to exert even more effort doing something stupid to solve it.

Chapter 1: I buy some hardware

Should the reader of the distant future not recall, it was impossible to buy a GPU from 2020 til at least 2022. New cards were going out of stock instantly only to be slung on eBay for double the MSRP. Even old cards, which might have gone for $100 used a few years ago, now cost hundreds more. Anyway, that’s how I ended up with a Dell OEM Radeon RX 580, a card that is now approaching five years in age (I do not want to discuss how much I paid for it).

What will I be doing with this hunk of old silicon somehow worth its weight in gold, you ask? Why, shoving it in an even older server, of course.

I acquired an HP ProLiant DL380p Gen 8 chassis (for far less than I paid for the GPU, I might add) and attempted to stick the card in it.

Chapter 2: This is the part of the story where I start to dislike HP

The PCI risers that came with the chassis would not accomodate the card. Each (one per socket) held a x16 and two x8 PCIe slots. The x16 was at the top, and thus my two-slot card could not physically be installed.

A few other varieties of risers were available for purchase. One has two x16 slots, and it turns out that is the one that I needed to use. Upon acquiring it (after paying serveral times the value of the three-slot riser I already had) I discovered that for some reason, it can only be installed on the second socket. Yes, for some reason the second x16 slot of this riser would land on a slot that HP for some reason decided would only be x8 on the first socket. Whatever, this isn’t the end of the world. It’s just slightly ridiculous that my gaming VM will have to access resources on both NUMA nodes (since of course the network and SCSI adapters reside on the first socket).

Chapter 3: Did I mention the virtualization?

Ah, yes – virtualization. I installed Proxmox VE on the iron, after a relatively smooth install with ZFS for root. It is only somewhat annoying that my /boot partition lies on a flash drive now. Did I mention this chassis doesn’t support UEFI?

I configured the Windows VM with ovmf, pc-q35, host, Virtio SCSI, and everything else the Internet told me to do to get the best performance. I did the install and configured the basic necessities, then it became time to attach the GPU.

To passthrough the GPU to the VM, I jumped through several hoops, only some of which were well-documented; some, I found, were unique to the experience of dealing with, well, whatever HP did to this server.

Proxmox provides pretty good documentation on setting up PCI passthrough, so it was fairly easy to figure out that I needed to add intel_iommu=on iommu=pt vfio-pci-ids=blah,blah to my kernel commandline. I neglected to read the part about blacklisting the graphics drivers, so it took a few more reboots (with very, very long POSTs) to remember to do that in order to get the vfio module to actually handle the card.

I attempted to attach the card to the VM, and it failed to start with Operation not permitted. Several hours of Googling led me to this post, which is terrifyingly recent and makes me think I would not have been able to accomplish this feat if I had attempted it only a year earlier, despite the age of the hardware.

The page he linked to explained some problem I didn’t understand, but sure sounded like what was happening to me. Thus, I spent quite some time struggling to install these HP utilities and after eventual success, started blindly typing commands to flip bits in the BIOS.

Why would HP not make these things configurable in the regular BIOS ROM? Why would I have to acquire a separate XML file that tells it which addresses to poke bits in? *shakes fist at HP*

These mysterious, and frankly frightening, incantations fortunately allowed the VM to finally boot.

Chapter 4: Performance anxiety

After getting the GPU drivers and everything set up, most of the games I was interested in playing (themselves about as old as the server I was running them on) Just Worked – but only half the time. The other half, the VM would boot in a state where everything was unbearably slow.

As previously alluded to, the GPU was bound to NUMA1, so the “half the time” it was slow is when the VM landed on NUMA0. I Googled for some incantations to pin the VM to a node, and typed hookscript: local:snippets/pin-numa1.sh into /etc/pve/qemu-server/${vm}.conf.

The hookscript itself, living at /var/lib/vz/snippets/pin-numa1.sh was the following kludge:

#!/bin/bash
if [[ "$2" == "post-start" ]]; then
    pid="$(< /run/qemu-server/${1}.pid)"
    cpus="8-15,24-31"
    taskset --cpu-list --all-tasks --pid $cpus $pid
fi

I found this actually made the problem worse, because for some reason my system preferred assigning the VM memory that belonged to to NUMA0 rather than NUMA1 now. Further Googling, I obtained the following spell to put in the VM config: numa0: cpus=0-15,hostnodes=1,memory=12288,policy=bind.

Now, quite entertainingly, the VM would fail to start because for some reason, other stuff was now consuming too much of NUMA1’s memory to be able to make this reservation.

Fortunately, this machine is old enough to be DDR3 and thus I was able to obtain enough RAM (with DIMMS identical to those I was already running) to expand to 64 GB per socket for, well, less than I paid for anything else that went into this box. This had the added benefit of using all four of the channels available to each socket, instead of just one as was the case before.

Finally, I was able to boot the VM and confirm with numastat that the VM was running on NUMA1, and its memory was local. Games were running smoothly… most of them, at least.

Chapter 5: Two reasons this post is a month late

There was still something wrong.

Dolphin emulator was running like utter garbage. GameCube games were almost passable, averaging about 25 FPS with occasional lag spikes. Wii games were entirely unplayable. I looked at performance metrics – nothing would indicate that this was too much for the box to handle. I ran again to Google. I found this post which suggested it was a CPU scaling issue. I looked again at the numbers. Indeed, my CPUs were not boosting above their most economical clock. But why? The performance governor was in place, but it did nothing.

Hours of fruitless debugging later, I recalled something I saw in the BIOS. Yes, the actual BIOS, not the feed-me-some-XML-so-I-can-twiddle-bits commandline utility…

This option was set to “High Performance Mode,” which is code for “fast when HP’s heuristic thinks you’re doing something hard.” Setting it to “OS Control Mode” allowed the governor to work as expected. The emulator now worked (after switching it to DX11 mode – for some reason OpenGL still ran poorly).

One problem remained.

Fall Guys, which inspired this whole adventure, would spontaneously crash. Not just crash, but cause the entire VM to reset. Fortunately, the iron survived these events, and its dmesg indicated that the graphics card generated a NMI. Don’t worry, I don’t know what that means either, besides that it’s bad. Bad enough to also show up in iLO logs with a big red icon next to it.

I could have simply ignored this issue and played all the other games in the meantime instead. But no, I can’t give up now, I’ve done too much to quit so soon.

I don’t even know what I typed into Google to find it, but this kernel patch gave me a vague sense of hope.

I typed echo '1' > /sys/module/kvm/parameters/ignore_msrs into the iron and booted the VM back up. I launched Fall Guys. I qualified from the first round. No crash. Surely, I will crash out in the next round… Yet, I did not. I played all the way through to the final, where I didn’t stand a chance for the crown.

However, what I had obtained instead was far more valuable: validation. I added this parameter to GRUB.

Epilogue

About a year ago, I first had the thought that “it would be neat to have a gaming server for the Steam Link to connect to.” Had I known how arduous this journey would be, I likely wouldn’t have embarked. Was it worth it? Probably not. But maybe the real gaming server is the arcane knowledge we picked up along the way.

Revised smart doorbell adapter circuit

Sat, 11 Dec 2021 00:00:00 +0000

In a previous post, I described issues encountered when integrating the Unifi G4 Doorbell with a digital chimebox. I designed an adapter circuit, and it worked for a few months; however, eventually I started encountering problems and upon further investigation discovered that some of the assumptions I made about the behavior of the Unifi chime adapter module were incorrect. With this new information, I designed a new circuit which is actually simpler and should be more reliable.

After about five months in operation, the circuit I detailed in the earlier post started emitting a buzzing sound. The relay contacts were in fact vibrating, but why? The first stage of this investigation involved hooking up the “output” leads from the module to my oscilloscope.

To the left of the trigger, we can see some bizarre modulated AC signal. This represents the “idle” state, where the doorbell has not been pressed. The RMS voltage of this was too low for my cheap multimeter to register, and thus I had incorrectly thought it was a zero volt output. While initially there were no consequences for this error, after some time apparently the relay I chose decided this was enough voltage to cause it to oscillate.

This discovery also made me realize that the characterization of these two leads as “outputs” was incorrect. I popped open the chimebox module and discovered that there are really only two terminals inside; in fact, the two pairs of leads sticking out of the box were mapped two these two terminals in parallel. That is, in the intended deployment scenario, this module sits parallel to the chimebox.

I didn’t take time to fully reverse engineer the board, but identified a comparator and a relay. Nothing else in here can effect any logic. Something strange is going on. Time for more fun with the scope.

Here we can see the behavior when the ring period ends. There is a zero-volt period prior to the idle signal resuming. My best guess at this point is that the adapter module clamps the voltage across the chime low (yielding that idle signal), then when it detects a voltage increase effected by the doorbell itself, triggers the relay to stop clamping. The doorbell later opens the circuit, resetting the module, then the idle resumes.

I came up with a few ideas to filter out the idle stage (RC filter, comparator, attiny85) but landed on a fairly simple design that I think should work reliably.

Now armed with some knowledge of how the Unifi chime module works, the new adapter board can be designed with only three terminals: one for each lead of the Unifi module, and one connecting to the chimebox. The two module leads are identified in the schematic as “AC,” connecting back to the transformer, and “TRIGGER,” connecting to the doorbell.

Using a half-bridge (single diode) rectifier (D1) and some smoothing (C1), we can get an estimate of the current voltage on the TRIGGER line, and rely on a zener diode (D2) to determine when this passes the activated threshold. The zener is connected to the gate of a mosfet (Q1), pulled low; the mosfet can then switch the relay (K1). When the relay is open, another half-bridge (D4) provides the idle supply to the chimebox. A flyback diode (D3) has also been added across the coil of the relay.

After some testing, this circuit seemed to work, so I put something together on protoboard.

The first terminal connects to the transformer (AC), the second to the doorbell (TRIGGER), and the third to the chime. The other terminal of the chime then connects back to the transformer (on the opposite side of the circuit).

Note that the reverse face of the board has been flipped as to correspond directly to the front.

Pulseaudio configuration for soundboard routing

Fri, 25 Jun 2021 00:00:00 +0000

Pulseaudio has been the default audio engine in Ubuntu for several years now, and fortunately it has a very flexible module system that lets you route and mix audio. Today, we’ll be looking at how we can set it up to mix mic and soundboard inputs into a virtual source.

I recently wrote a simple soundboard app in Python that uses a MIDI controller to trigger clips, and relied on the fact that gstreamer would default to Pulseaudio output to later allow me to configure routing and mixing without having to do anything specific in the code.

The main system-wide Pulseaudio configuration lives in /etc/pulse/default.pa; since I’m the only user on my system I decided to just throw everything in there, but it is also possible to set this up per-user.

The relevant lines in my default.pa are:

load-module module-null-sink sink_name=soundboard_mix sink_properties=device.description=SoundboardMix
load-module module-combine-sink sink_name=soundboard_router slaves=alsa_output.pci-0000_00_1f.3.analog-stereo,soundboard_mix sink_properties=device.description="SoundboardRouter"
load-module module-loopback sink=soundboard_mix source=mic_ec

Respectively, these lines will

Create a virtual output called soundboard_mix where we will combine the soundboard and mic sources together; Pulseaudio will also automatically create a monitor called soundboard_mix.monitor that we can use as a source.
Create another virtual output called soundboard_router that we can use to route the soundboard to both our main output (in my case, alsa_output.pci-0000_00_1f.3.analog-stereo) as well as soundboard_mix.
Route our mic (in my case, mic_ec, since I have an echo cancellation module configured) into the soundboard_mix.

To find the names for the mic source and main output, use pactl list sources and pactl list sinks, respectively.

Now, all we have to do is make sure the soundboard application and whatever app in which we want to use our audio source are passed environment variables to choose the correct source and sink. Pulseaudio respects variables PULSE_SOURCE and PULSE_SINK, so in my case I can launch my soundboard app with PULSE_SINK=soundboard_router and launch the app in which I want to use it with PULSE_SOURCE=soundboard_mix.monitor and everything will function as desired.

As an added bonus, here’s the line I’m using to create my echo-cancelled mic source:

load-module module-echo-cancel source_name="mic_ec" use_master_format=1 aec_method="webrtc" source_master="alsa_input.pci-0000_00_1f.3.analog-stereo" sink_master="alsa_output.pci-0000_00_1f.3.analog-stereo" aec_args="analog_gain_control=0 digital_gain_control=0 mobile=1 routing_mode=quiet-earpiece-or-headset comfort_noise=0 intelligibility_enhancer=1 high_pass_filter=0"

Change alsa_input.pci-0000_00_1f.3.analog-stereo to the correct input as listed by pactl list sources. I have a lot of extra options supplied here, which are documented nicely here.

Also, feel free to check out my soundboard app if you’re interested in something very simple to trigger clips. It features a pyqt5 GUI for configuration and automatic MIDI note detection for setting triggers.

Adapting a smart doorbell to an incompatible digital chime

Sat, 19 Jun 2021 00:00:00 +0000

Update (2021-12-11): It turns out some of the assumptions I initially made were incorrect, and the adapter circuit described within eventually stopped working. After further investigation, I have made a revised circuit. Refer to the new post

For a few months, I’ve been using an Ubiquiti G4 Doorbell, but recently chose to upgrade my mechanical chimebox to a digital one. Though The Ubiquiti store lists the NuTone LA600WH as compatible, it turns out there are some caveats which I managed to overcome by conjuring up an adapter from parts I had lying around. This technique can likely be adapted to work with other doorbells or chimeboxes that don’t play nice with each other, and avoids spending money on a commercially-available adapter.

The main issue when using a smart doorbell with a digital chimebox seems to be that both want to continuously draw power. A normal “dumb” doorbell will simply close a circuit to trigger a mechanical chime. In the case of my digital chime, it required a diode in order to continuously receive power when the button isn’t being pressed. However, my smart doorbell also requires a special hack to receive power – an adapter that Ubiquiti provides which sits between the doorbell and the chimebox, and triggers the chimebox when the button is pressed. Unfortunately, these two hacks are inherently incompatible with one another. Based on some searches, it seems that this is the case for a lot of doorbells and digital chimes.

There were two “easy” ways to get the doorbell working, in the case of the LA600WH. Since it supports completely wireless operation, there is an option to install three D-cell batteries, however who wants to change batteries when a wired installation is possible? The second hack would have been to configure a long enough ring duration in the Unifi Protect app for the chime to respond; in this case, there was unfortunately a long delay that seemed to be due to the chime “booting up.”

It was at this point that I consulted Google: surely, someone has run into the same problem as me, right? Well, yes. I managed to find a how-to post in the Ubiquiti community explaining how to get the G4 to play nice with a digital chime. Unfortunately, it required something called the VDBI to serve as an adapter. Not wanting to spend an additional $20 or wait for shipping, I turned to my parts shelf.

After searching a few bins and finding a relay and some diodes, it occurred to me that I could easily build something that would allow the G4 to pretend to be a regular, mechanical doorbell. Doorbell circuits use low-voltage AC (in my case, 18V); the fact that my chimebox came with a diode led me to believe that it wants half-wave power in order to stay powered on but idle. Then, full wave power should cause it to ring. The adapter provided with the G4 simply seemed to provide power to the doorbell continuously, while also closing the circuit intended to ring the chime when the button is pressed. If I could exploit the “output” of the Ubiquiti adapter to trigger a relay (or a triac, probably, but I didn’t have one) that provides full-wave power, and use a diode to provide half-wave power continuously (as originally intended by the manufacturer), the chimebox should work as it was meant to. Unfortunately, relays can typically only be triggered by DC (the inductive properties of the internal coil don’t play nice with AC) but this could be remedied with a bridge rectifier. An integrated one would work, but I didn’t have one, so I built one with discrete diodes. I considered omitting the relay and just feeding whatever comes out of the Ubiquiti adapter directly to the chime, but not being sure what else is going on inside there I didn’t want to backfeed it from the constant supply.

The circuit I came up with looks like this:

The AC provided by the Ubiquiti adapter when the doorbell is pressed will be rectified in order to trigger the relay, causing full-wave AC to be provided to the chime. When it isn’t being triggered, the 1N4002 diode that was provided with the chime will provide it half-wave current, allowing it to stay powered-on but idle. As I mentioned, I ended up building the rectifier out of discrete diodes; for what it’s worth, I used 1N4007s. It’s important that you use a relay that will tolerate the voltage coming out of your doorbell transformer; in the case of 18VAC fully rectified, this would be about 16.2VDC. In some cases, you may also need to add a smoothing capacitor; if the relay isn’t staying latched as long as you would hope, try adding one. You might also be able to get away with a half-bridge rectifier, but I didn’t bother.

I wasn’t originally intending to do this write-up, so unfortunately I didn’t take any pictures prior the installation, but what I built ended up looking like this (after being wrapped in Kapton tape).

Yes, I know that’s a 5V relay, and I just told you not to do that. This is what I had around, but it hasn’t caught fire yet and I’ll replace it eventually.

Hooked into the Ubiquiti adapter and the chime, everything ends up looking like this:

It even all fits nicely into the battery compartment of the chime:

As mentioned in previous posts, I’m certainly not an electrical engineer, but fortunately I was able to summon enough ability to solve this problem. Hopefully my solution can benefit others as well.

Dependency and build management for sourcemod with sourceknight

Mon, 14 Jun 2021 00:00:00 +0000

Though I haven’t been playing with it for long, I’ve found that compiling plugins for sourcemod is a huge hassle, mostly due to how dependencies are managed – that is, they really aren’t.

For those who don’t know what sourcemod is, you probably have no interest in reading this article. But, if for whatever reason you want to read it anyway, sourcemod is a mod for Source engine game servers that provides support for custom scripts and plugins.

The Source engine is fairly old, dating back to Half-Life 2, but many games based on it such as Team Fortress 2 and Counter-Strike: Global Offensive still have a thriving community with active custom servers. Many of these community servers rely on sourcemod to provide various features as well as support for custom game modes.

Unfortunately, the state of sourcemod plugin development is largely “download a bunch of files, put them somewhere, hopefully have them organized correctly, and pray.” I aimed to fix that through the creation of the tool I’m writing about today: sourceknight.

The core concept of sourceknight is to allow you to specify the dependencies of your project with a simple configuration file:

project:
  name: myplugin-example
  dependencies:
    - name: sourcemod
      type: tar
      version: 1.10.0-git6503
      location: https://sm.alliedmods.net/smdrop/1.10/sourcemod-1.10.0-git6503-linux.tar.gz
      unpack:
      - source: /addons
        dest: /addons
  root: /myplugin
  targets:
    - example

Then, all you have to do is type sourceknight build, and your plugins will magically be compiled.

Maybe you aren’t writing your own plugins, but manage a server and hate the process of updating and compiling all the plugins you use? Well, sourceknight can still help – a sourceknight project doesn’t need to contain any scripts of its own, and can also just be used to compile plugins you specify as dependencies.

I did find a few other projects which alleged to help manage sourcemod based projects, however I believe sourceknight, at least in vision, provides a much more streamlined experience. As of today, sourceknight is mostly a proof of concept – however, it is certainly capable of managing dependencies and compiling plugins. I’m writing this post mostly in order to measure interest, find deficiencies, and identify what features are desired moving forward. But nonetheless, hopefully someone will find it useful.

If you missed the link above, the GitHub repository is here: https://github.com/tmick0/sourceknight

Tools to check for compromised Keepass passwords

Thu, 19 Nov 2020 00:00:00 +0000

It’s 2020, and by now everyone should know the basics of password hygiene: use strong passwords, don’t reuse them across sites, etc. A password manager such as Keepass is an essential tool for keeping track of all the passwords you inevitably end up with under this set of rules.

Unfortunately, no matter how good your passwords are, there’s always a chance that they’ll get compromised. For example, a website might have a breach and someone can obtain your password regardless of how hard it would have been to guess. In cases like these, services like Have I Been Pwned can let you know this has happened. However, if your password is leaked in something like a large credential-stuffing dump, it might not be obvious which sites you may need to change your password on.

For situations like this, Have I Been Pwned also provides password lists which contain the cryptographic hashes of all known compromised passwords. However, you still can’t immediately identify which of your passwords have been breached.

For this purpose, I have written a set of very simple tools to dump hashes of your own Keepass-managed passwords and check them against the Have I Been Pwned list.

Here’s an example using a database that only contains Keepass’s builtin example passwords.

The first script simply dumps a CSV of your password hashes and some metadata:

$ kdbx2csv ~/Desktop/Database.kdbx my_passwords.csv
Password: ****
$ cat my_passwords.csv 
8BE3C943B1609FFFBFC51AAD666D0A04ADF83C9D,User Name,Sample Entry
8CB2237D0679CA88DB6464EAC60DA96345513964,Michael321,Sample Entry #2

The second script then checks this CSV against the list from Have I Been Pwned:

$ checkhashes my_passwords.csv pwned-passwords-sha1-ordered-by-hash-v7.txt 
8BE3C943B1609FFFBFC51AAD666D0A04ADF83C9D,136802,User Name,Sample Entry
8CB2237D0679CA88DB6464EAC60DA96345513964,2493390,Michael321,Sample Entry #2

We can see that both of the example entries’ passwords (“Password” and “12345”) are unsafe, occurring in breaches 136,802 and 2,493,390 times respectively.

The code and further instructions are available on Github: https://github.com/tmick0/kdbxtools

A highly customizable RGB controller implementation for Arduino

Sat, 28 Mar 2020 00:00:00 +0000

I recently decided that I needed to add some color to my workspace, but didn’t want to just use any off-the-shelf RGB controller. My first thought was to use an Arduino to control some RGB strips, but I didn’t want to have to to open the Arduino IDE and modify the firmware every time I wanted to change the program.

This desire eventually escalated to implementing a virtual machine on top of the Arduino with an application-specific instruction set designed to easily manipulate LEDs.

For the impatient reader, here’s the Github link.

Controller build

Though I did my initial testing with an Arduino Uno, I chose to go with 16MHz Pro Micros (like this one available from Sparkfun) for the actual build in order to keep the footprint small. Other than that, the supporting circuitry is dependent on the application. I’ve deployed two controllers: one inside my PC, and one for backlight behind my monitors. The setup I’m using for the PC is the simplest, so I’ll explain that one first.

The controller inside my PC is driving an SK9822-based RGB strip, so it only needs a 5V supply. Given that it’s inside a PC, there is easy access to 5V via a Molex connector from the PSU. Therefore, I’ve used that to power both the Arduino and the RGB strip. The controller module was built by soldering two female header rows to a proto board to accept the Arduino, wiring the Molex to the RAW and GND pins, and attaching a JST socket to the data, clock, and power pins. It ended up fitting nicely in the back of my case and I was able to secure it with a Velcro strap.

(Sorry, I forgot to take a better photo before installing it.)

The controller for my monitor backlight is a bit more involved. For this application, wanted an on/off switch as well as a few potentiometer inputs to control the color of the strip. It’s running a WS2811 strip that takes 12V input, but I still needed a 5V reference and I didn’t want to have to keep the Arduino connected over USB. Therefore, I included a voltage regulator to step down the 12V from the strip’s power supply and a MOSFET to handle the on/off capability (since I didn’t expect the latching pushbutton switch I was using to be rated for much current).

The result was still a reasonable size and fit nicely into a decently sized project box, with just the barrel jack for the 12V and a JST connector for the strip protruding from opposite sides.

Since the circuit was a bit involved for me (having little to no experience doing such things), I drew up a schematic first.

I then prototyped the design on a breadboard.

I tested it out and everything worked smoothly. I decided to leave out the decoupling capacitor depicted in the schematic as I found it wasn’t necessary.

For the final build, I ended up placing the potentiometers and the power switch on their own boards, connecting them to the main board with some jumpers.

Testing it out prior to closing up the case, everything still worked as intended.

(Excuse the assorted debris on my workbench.)

For completeness, the final product:

Virtual machine

As I mentioned, the RGB controller itself is implemented in a virtual machine. It’s an 8 bit machine with 15 general purpose registers, and otherwise unremarkable except for a few special instructions for analog input, RGB output, and HSV-to-RGB color space conversion. I wrote an assembler for the machine in Python so I could write code with mnemonics instead of manually constructing bytecode.

The program memory of the VM persists in the EEPROM of the Arduino and is read into RAM at startup. It can be reprogrammed over a simple serial protocol.

The VM doesn’t have function calls (though there is branching) or any form of data memory; instead, programs only operate via I/O and registers. This was my first time implementing a virtual machine, and I wanted to keep it simple.

Drivers

The VM uses three instructions to interact with RGB devices through a set of drivers that have been implemented in the firmware. Namely, there’s an init instruction to activate a driver, a write instruction to store an RGB value to a buffer, and a send instruction to activate the buffered values.

I’ve implemented three drivers: a simple PWM-based driver for analog RGB, a driver for the WS281x family of RGB chips, and a driver for the APA102 family of RGB chips.

Of the three, the WS281x driver was the most challenging, as it involved bitbanging the control signal with precise timings. Writing simple C or C++ code would not have met the timing constraints, thus necessitating the use of inline assembly. I found a few implementations online, but they were all designed to have a hardcoded pin number known at compile time. Since I wanted my driver to have the flexibility to choose an arbitrary pin, or even run multiple strips simultaneously, I needed to relearn AVR assembly and modify the code to use a non-constant value. Other than that, everything went pretty smoothly.

The send instruction ended up being implemented as follows:

// get the IO port number for the requested pin
const uint8_t PORT_NUM = digitalPinToPort(driver->arg);

// create a mask to enable toggling the correct pin
const uint8_t PORT_MASK = digitalPinToBitMask(driver->arg);

// get the pointer to the IO port register
volatile uint8_t *PORT = portOutputRegister(PORT_NUM);

// disable interrupts to allow for precise timing
cli();

// delay to trigger "data latch"
const uint32_t t = micros();
while ((micros() - t) < 50L) {}

// pointer to the data buffer
volatile uint8_t *p = (uint8_t *)driver->buffer;

// get the first value (subsequent values will be fetched in the asm)
volatile uint8_t val = *p++;

// apply PORT_MASK to get the register value for the high/low states
volatile uint8_t high = *PORT | PORT_MASK;
volatile uint8_t low = *PORT & ~PORT_MASK;

// tmp register will swap between high and low when need be
volatile uint8_t tmp = low;

// the number of bits remaining in the current byte
volatile uint8_t nbits = 8;

// length of the output buffer
volatile uint16_t nbytes = 3 * driver->ptr;

asm volatile("nextbit:\n\t"
            "st %a0, %1\n\t"    // store high
            "sbrc %2, 7\n\t"    // don't set tmp high if next bit zero
            "mov  %4, %1\n\t"   // otherwise, do
            "dec  %3\n\t"       // decrement bit counter
            "nop\n\t"           // nop for keeping timing
            "st   %a0, %4\n\t"  // write tmp to the IO register
            "mov  %4, %5\n\t"   // next, we'll set the line low
            "breq nextbyte\n\t" // load the next byte if done with last
            "rol  %2\n\t"       // otherwise, rotate in the next bit
            "rjmp .+0\n\t"      // nop for keeping timing
            "st %a0, %5\n\t"    // set the line low
            "rjmp .+0\n\t"      // nop for keeping timing
            "nop\n\t"           // 
            "rjmp nextbit\n\t"  // send the next bit
            "nextbyte:\n\t"
            "ldi  %3, 8\n\t"    // reset the bit counter
            "ld   %2, %a6+\n\t" // load the next byte
            "st %a0, %5\n\t"    // set the line low
            "rjmp .+0\n\t"      // nop for keeping timing
            "nop\n\t"           // 
            "dec %7\n\t"        // decrement the byte counter
            "brne nextbit\n\t"  // loop if there are more bytes
            ::"e"(PORT), "r"(high), "r"(val), "r"(nbits), "r"(tmp),
              "r"(low), "e"(p), "w"(nbytes));

// re-enable interrupts
sei();

// clear the output buffer
driver->ptr = 0;

The assembly seen here is derived from this post. Note that the chip expects a GRB byte order. To keep the assembly simple, the reordering is handled in the write instruction.

The APA102 driver was much easier to implement, since it uses a clock signal and thus does not rely on precise timings.

const uint8_t clock = driver->arg, data = driver->arg + 1;

digitalWrite(data, LOW);

// start frame
apa102_byte(clock, data, 0);
apa102_byte(clock, data, 0);
apa102_byte(clock, data, 0);
apa102_byte(clock, data, 0);

// send data for each entry in the buffer
for (int i = 0; i < driver->ptr; ++i) {
    // header (currently hardcoded max brightness)
    apa102_byte(clock, data, 0xff);

    // the chip expects b, g, r byte order
    apa102_byte(clock, data, driver->buffer[i].b);
    apa102_byte(clock, data, driver->buffer[i].g);
    apa102_byte(clock, data, driver->buffer[i].r);
}

// end frame
for (int i = 0; i < driver->ptr; ++i) {
    apa102_byte(clock, data, 0x00);
}

// leave the line low when done
digitalWrite(data, LOW);

driver->ptr = 0;

The apa102_byte call simply sets the data line to the correct state then toggles the clock line high then low.

Instruction set

The VM implements 19 mnemonic instructions. In reality, the branch instructions are all implemented under a single opcode followed by a mode flag which determines whether and how to inspect the flag register.

Mnemonic	Operands	Description
nop	[imm]	no operation – if imm specified and nonzero, sleep 2^(imm-1) ms
set	rdst (rsrc\|imm)	load register rdst from rsrc or immediate
add	rdst (rsrc\|imm)	add to rdst from rsrc or immediate
mul	rdst (rsrc\|imm)	multiply rdst by rsrc or immediate
div	rdst (rsrc\|imm)	divide rdst by rsrc or immediate
mod	rdst (rsrc\|imm)	modulo rdst by rsrc or immediate
cmp	r0 (r1\|imm)	compare r0 to r1 or imm and store the result in flags
goto	address	move the instruction pointer to address
brne	address	move the ip to address if last comparison was not equal
breq	address	move the ip to address if last comparison was equal
brlt	address	move the ip to address if last comparison was less than
brle	address	move the ip to address if last comparison was less or equal
brgt	address	move the ip to address if last comparison was greater than
brge	address	move the ip to address if last comparison was greater or equal
hsv2rgb	rh rs rv	convert hsv values in registers to rgb (in place), each of hsv in [0, 255]
init	immd immc [immf]	initialize output channel immc with driver number immd and optional flags immf
write	rr rg rb immc	buffer rgb value from registers on output channel immc
send	immc	activate the buffered output of immc
input	rdst immpin	load rdst with the value from the analog pin numbered by the immediate

Implementation

Instructions

Instructions for rgbvm are of varying size. Starting with a 4-bit opcode is the only thing they have in common.

struct rgbvm_instruction {
    enum rgbvm_opcode opcode : 4;
};

Each class of instruction then has its own derived structure. For brevity, I’ll only include the structures of the arithmetic and driver interaction instructions, but this should be sufficient to provide a general idea of the structure of the bytecode.

struct rgbvm_arithmetic_instruction {
  enum rgbvm_opcode opcode : 4;
  enum rgbvm_reg dst : 4;
  enum rgbvm_reg src : 4;
  uint8_t padding : 4;
  uint8_t imm[];
};

struct rgbvm_init_instruction {
  enum rgbvm_opcode opcode : 4;
  enum driver_type driver : 4;
  uint8_t channel : 2;
  uint8_t arg : 6;
};

struct rgbvm_write_instruction {
  enum rgbvm_opcode opcode : 4;
  enum rgbvm_reg srcr : 4;
  enum rgbvm_reg srcg : 4;
  enum rgbvm_reg srcb : 4;
  uint8_t channel : 2;
  uint8_t padding : 6;
};

struct rgbvm_send_instruction {
  enum rgbvm_opcode : 4;
  uint8_t channel : 2;
  uint8_t padding : 2;
};

Arithmetic instructions have an optional immediate, thus the 0-length array at the end of that structure.

VM state and execution

The structure describing the state of the VM is simple:

struct rgbvm_state {
  // instruction pointer
  uint16_t ip;

  // length of code segment
  uint16_t ip_max;

  // 15 general purpose registers
  uint8_t reg[15];

  // flag register
  int flag;

  // output buffers
  struct rgbvm_driver outputs[4];
};

In setup, the first two bytes of EEPROM are read to determine the length of the code segment (thus determining ip_max).

const uint16_t code_len = (EEPROM.read(0) << 0) | (EEPROM.read(1) << 8);

The bytecode is then read from EEPROM into RAM:

for (int i = 0; i < code_len; ++i) {
    s.code[i] = EEPROM.read(i + sizeof(uint16_t));
}

In loop, when executing code, the instruction at the current instruction pointer is passed to rgbvm_apply.

rgbvm_apply(delay, &s.vm, (const rgbvm_instruction *)&s.code[s.vm.ip]);

We pass a pointer to the delay function so the VM can call it when a nop instruction with a nonzero argument is encountered.

The implementation of rgbvm_apply revolves around a switch on the opcode of the current instruction. For example, the arithmetic instructions are handled as follows.

case RGBVM_OP_SET:
case RGBVM_OP_ADD:
case RGBVM_OP_MUL:
case RGBVM_OP_DIV:
case RGBVM_OP_MOD:
case RGBVM_OP_CMP: {
    uint8_t *dest;
    uint8_t src;
    uint8_t size;
    rgbvm_arith_op_impl op;
    rgbvm_decode_arithmetic(vm,
                            (struct rgbvm_arithmetic_instruction *)inst,
                            &dest, &src, &op, &size);
    op(vm, dest, src);
    rgbvm_increment_ip(vm, size);
    return RGBVM_STATUS_OK;
}

These instructions are all implemented as functions taking arguments of a pointer to the VM state, a pointer to the destination operand, and the value of the source operand. The rgbvm_decode_arithmetic function is decoding the operands and setting the corresponding pointer and value, as well as setting a function pointer to the instruction implementation and determining the length of that instruction.

Decoding the destination operand simply involves returning a pointer to the correct register, while decoding the source operand also has logic handling immediate values.


int rgbvm_decode_val(struct rgbvm_state *vm, const enum rgbvm_reg reg,
                     const struct rgbvm_arithmetic_instruction *inst,
                     uint8_t *dest, uint8_t *imm_flag) {

    if (reg == RGBVM_REG_IM) {
        *dest = inst->imm[0];
        *imm_flag = 1;
        return 0;
    }

    *imm_flag = 0;
    const uint8_t *src = rgbvm_decode_reg(vm, reg);
    *dest = *src;
    return 0;
}

The imm_flag allows rgbvm_decode_arithmetic to determine the length of the instruction and is used to increment the instruction pointer appropriately.

The instructions for communicating with the drivers rely on a set of functions with a common interface for initializing, writing, and outputting data.

typedef void (*driver_write)(struct rgbvm_driver *, const uint8_t,
                             const uint8_t, const uint8_t);
typedef void (*driver_init)(struct rgbvm_driver *, const uint8_t arg);
typedef void (*driver_send)(struct rgbvm_driver *);

Each driver defines an init function which sets pointers to write and send within the VM’s outputs array.

The corresponding instructions are implemented as follows.

case RGBVM_OP_INIT: {
    const struct rgbvm_init_instruction *i =
        (const struct rgbvm_init_instruction *)inst;
    driver_init init = get_driver(i->driver);
    init(&vm->outputs[i->channel], i->arg);
    rgbvm_increment_ip(vm, 2);
    return RGBVM_STATUS_OK;
}
case RGBVM_OP_WRITE: {
    const struct rgbvm_write_instruction *i =
        (const struct rgbvm_write_instruction *)inst;
    const uint8_t *r = rgbvm_decode_reg(vm, i->srcr);
    const uint8_t *g = rgbvm_decode_reg(vm, i->srcg);
    const uint8_t *b = rgbvm_decode_reg(vm, i->srcb);
    driver_write fn = vm->outputs[i->channel].write;
    fn(&vm->outputs[i->channel], *r, *g, *b);
    rgbvm_increment_ip(vm, 3);
    return RGBVM_STATUS_OK;
}
case RGBVM_OP_SEND: {
    const struct rgbvm_send_instruction *i =
        (const struct rgbvm_send_instruction *)inst;
    driver_send fn = vm->outputs[i->channel].send;
    fn(&vm->outputs[i->channel]);
    rgbvm_increment_ip(vm, 1);
    return RGBVM_STATUS_OK;
}

The drivers implement write by storing the r, g, b values in the output channel’s buffer; send then implements the protocol for sending this data to the LEDs.

The branch instructions follow a similar pattern to the arithmetic instructions. These and the remaining instructions’ implementations are omitted for brevity

Serial protocol

The state machine handling bytecode updates from serial ingests a single byte at a time.

if (Serial.available() > 0) {
    proto_msg msg;
    if (proto_state_machine_ingest(&s.vm, s.code, &s.psm, Serial.read(),
                                   &msg, write_eeprom) {
        Serial.write((uint8_t)msg);
    }
}

The return of proto_state_machine_ingest indicates whether msg has been modified, thereby indicating a response that should be written to serial.

The state machine itself supports four states:

enum proto_state {
  PROTO_STATE_INIT,
  PROTO_STATE_SIZEH,
  PROTO_STATE_SIZEL,
  PROTO_STATE_CODE
};

Initially, it is in the INIT state. When in any other state, the VM pauses execution of bytecode until the transaction is completed.

The state machine first expects a HELLO message from the peer; upon receipt it transitions to the SIZEL state, indicating that it is ready to receive the low byte of the length of the bytecode.

case PROTO_STATE_INIT: {
    if (byte == PROTO_MSG_HELLO) {
        psm->state = PROTO_STATE_SIZEL;
        *res = PROTO_MSG_OK;
        return 1;
    }
    *res = PROTO_MSG_ERR;
    return 1;
}

After that byte is received, the state is set to SIZEH, indicating that the high byte of the bytecode length is expected. After receiving that, the state machine then transitions to CODE, where it stays until the specified number of bytes are received.

case PROTO_STATE_SIZEL: {
    ((uint8_t *)&vm->ip_max)[0] = byte;
    psm->state = PROTO_STATE_SIZEH;
    return 0;
}
case PROTO_STATE_SIZEH: {
    ((uint8_t *)&vm->ip_max)[1] = byte;
    psm->state = PROTO_STATE_CODE;
    psm->offset = 0;
    *res = PROTO_MSG_OK;
    return 1;
}

The serial ingest implementation writes the bytecode directly to the VM’s code array in order to save memory. After receiving the full bytecode, the state machine returns to INIT and returns an OK message.

After receiving a bytecode update, a callback is invoked to write the new bytecode to EEPROM, then the VM state is updated with the new length of the code segment and the instruction pointer is reset to zero.

case PROTO_STATE_CODE: {
    code[psm->offset++] = byte;
    if (psm->offset == vm->ip_max) {
        cb(psm);
        vm->ip = 0;
        psm->state = PROTO_STATE_INIT;
        *res = PROTO_MSG_OK;
        return 1;
    }
    return 0;
}

Because writing the bytecode to EEPROM is delayed until after receiving the full stream, the device can simply be reset to revert back to the old code if an error occurs in transmission.

Outcome

This was a challenging project for me because it involved a lot of new things. However, I learned a lot and it payed off. After about a week of working on this for a couple of hours every day, I’ve finally got a colorful battlestation.

The source code for this project is available on Github.

Timelapses from Landsat data

Sun, 08 Mar 2020 00:00:00 +0000

I thought it would be cool to extend the scripts I posted about yesterday with the ability to create a timelapse.

The above video shows all of the Landsat 8 scenes of WRS cell 33/36 over the course of 2019.

The repository has been updated with the code to generate this: http://github.com/tmick0/landsat_fetch

An automated workflow for fetching and mosaicing Landsat imagery

Sat, 07 Mar 2020 00:00:00 +0000

One extension I had been planning for my previous project involving terrain meshes was the addition of satellite imagery to add color to the models. Landsat, in addition to enabling important climatology research and other science missions, turned out to be a great source of open data to use for this.

As a step toward integrating this data with my terrain mesh generator, I’ve implemented an automated workflow for fetching, radiometrically correcting, and mosaicing these images.

AWS generously provides free access to its S3-based Landsat mirror, however the Worldwide Reference System Landsat uses to index products makes it a bit less than straightforward to identify the right data to fetch. Therefore, I implemented a parser for the provided scene list that resolves requested geodetic extents to a set of footprints which cover them. The script then fetches all of the data needed to generate the requested output product.

After obtaining the data, a bit of massaging is required to make it useful. The data has to be radiometrically calibrated and then warped to a more flexible map projection.

Finally, I’ve implemented a pansharpening operation which uses Landsat 8’s panchromatic band to improve the spatial resolution of its spectral bands and produce a high-fidelity output product.

The code that handles all of this has been released to Github for all to use.

Generating terrain meshes for 3D printing

Sat, 05 Oct 2019 00:00:00 +0000

A while back, I thought 3D printed models of the local terrain might be a cool gift idea. To make this a reality, I have implemented a simple Python utility to convert publicly-available terrain data into a format suitable for 3D printing.

Most 3D printers expect STL models, which define a solid in terms of vertices in 3D Cartesian space and faces (triangles) which connect them. However, terrain models are typically distributed as rasters of heights indexed by latitudes and longitudes. Fortunately, libraries exist to convert latitude, longitude, altitude tuples to standardized Cartesian coordinates. However, some additional massaging of the data is required.

If you’re interested in the details of this process, read on. Otherwise, the project is available on Github for immediate use, without you needing to worry about it.

Getting the data

Before we can start, we need to find terrain data to work with. Fortunately, NASA has been kind enough to make its SRTMGL3 dataset public, which gives us fairly high-quality terrain data for (mostly) the entire world. I don’t know how the radar magic that they used to create these models works, but fortunately the form they are distributed in is fairly easy to work with.

Each raster, or “granule,” covers a square measuring 1 degree of latitude and 1 degree of longitude. At the 3 arcsecond resolution we’ll be using, this gives us a raster of 1201 by 1201 pixels. The data is stored as a matrix of big-endian signed shorts without any sort of metadata, and thus we can read it with a simple one-liner in numpy:

altitudes = np.fromfile(filename, dtype='>i2', count=1201 * 1201).reshape((1201, 1201))

Now we’ve got a matrix representing the altitudes associated with the latitude and longitude of each pixel. The latitudes and longitudes themselves are determined by the pixel coordinates as well as the filename. The filename itself reveals the latitude and longitude of the bottom-left (southwest) corner of the raster; each pixel is 3 arcseconds by 3 arcseconds in size.

Again, we can use a few simple numpy incantations to give us arrays of latitudes and longitudes that we can index in parallel with the altitudes we extracted:

lats = np.linspace(north_latitude, south_latitude, num_rows, False)
lons = np.linspace(west_longitude, east_longitude, num_cols, False)
longrid, latgrid = np.meshgrid(lons, lats)

If the region we want to 3D print lies entirely within one SRTM granule, this is almost all we have to do. However, we must cover an edge case where the target area spans more than a single raster.

Merging granules and extracting the target area

Unless we’re 3D printing an area that itself spans more than 1 square degree (which I wouldn’t recommend due to the limited visibility of terrain detail at this size), the worst case scenario we will encounter requires us to merge four adjacent rasters.

If we determine that the requested area spans more than one raster, we simply load all of the required rasters and merge them as follows:

merged = np.block(blocks)

where blocks is a 2D list representing the physical layout of the rasters we are merging, and each element is a loaded raster. It is important to note that you must trim off the first row and last column in this step, as they are overlapped with the adjacent granules.

Now that we have aggregated the granules of interest, we can extract the region we want:

r0 = 1200 - int((north - floor(north)) * 1200)
c0 = int((west - floor(west)) * 1200)
rows = int((north - south) * 3600 // 3 + 1)
cols = int((east - west) * 3600 // 3 + 1)
result = merged[r0:r0+rows, c0:c0+cols]

At this point, we can also apply a scaling factor to the altitudes in order to exaggerate the altitudes and make the 3D printed result more interesting to look at.

Converting to Cartesian coordinates

At this stage, we have extracted vectors of altitudes, latitudes, and longitudes corresponding to the data we wish to 3D print. However, we must combine these three parallel vectors into a coherent set of 3D vertices.

Latitudes and longitudes are ellipsoidal coordinates that give us a point on the surface of the Earth, as opposed to a point on a plane. While it is reasonable to exploit the local linearity of the surface of the ellipsoid and use simple scaling factors to generate a set of vertices above a fixed plane, I actually found it easier to use the “real” conversion between ellipsoidal and Cartesian coordinates for this step.

I found that the most convenient coordinate system to use was ECEF, or Earth Centered Earth Fixed. This is a simple 3D Cartesian system where the origin (0, 0, 0) is located at the center of the Earth, and vectors are expressed in meters relative to this.

I used the Python library pyproj to perform the conversion as follows:

ECEF = pyproj.Proj(proj='geocent', ellps='WGS84', datum='WGS84')
LLA = pyproj.Proj(proj='latlong', ellps='WGS84', datum='WGS84')
x, y, z = pyproj.transform(LLA, ECEF, longrid, latgrid, altitudes, radians=False)

Now we have three parallel arrays representing our vertices.

Correcting orientation, positioning, and scale

Unfortunately, the ellipsoidal to ECEF conversion essentially gives us a patch somewhere on the surface of the Earth, and thus it is not oriented correctly for 3D printing. To correct the orientation, I exploited some facts about the geometry of the Earth ellipsoid the ECEF coordinate system.

First, we must ensure that the model is “right side up.” We can use an ellipsoidal normal vector to approximate “up” and rotate our vertices such that this vector aligns with the z-axis. To avoid re-learning geometry, I simply used pyproj to find the axis:

def get_axis(lat0, lon0, alt0, lat1, lon1, alt1):
    v0 = pyproj.transform(LLA, ECEF, lon0, lat0, alt0, radians=False)
    v1 = pyproj.transform(LLA, ECEF, lon1, lat1, alt1, radians=False)
    v = np.array(v1) - np.array(v0)
    return v

z_axis = get_axis(center_lat, center_lon, 0., center_lat, center_lon, 1000.)

I did something similar to align the north and south edges of the model relative to the x axis:

x_axis = get_axis(south, west, 0., south, east, 0.)

Next, we can generate rotation matrices and use the Scipy spatial library to perform the rotation:

def get_rotation(A, B):
    au = A / np.linalg.norm(A)
    bu = B / np.linalg.norm(B)
    R = np.array([[bu[0]*au[0], bu[0]*au[1], bu[0]*au[2]],
                  [bu[1]*au[0], bu[1]*au[1], bu[1]*au[2]],
                  [bu[2]*au[0], bu[2]*au[1], bu[2]*au[2]]])
    return scipy.spatial.transform.Rotation.from_dcm(R)

r0 = get_rotation(z_axis, [1., 0., 0.])
r1 = get_rotation(r0.apply(x_axis), [1., 0., 0.])
r = r1 * r0
vertices = r.apply(vertices)

Our vertices are now oriented above the x-y plane in a way that kind of makes sense. However, the data is still “life size” and must be scaled down. First, we must center them around the origin in x-y space:

x, _, z = np.mean(vertices, axis=0)
_, y, _ = vertices.min(axis=0)
vertices = np.subtract(vertices, [x, y, z])

Now, they can be scaled:

vertices = vertices * scaling_factor

The last step is now to create the faces connecting these vertices, and our 3D model will be complete.

Creating the solid

Despite the amount of math we saw in the previous section, I found this step to be the most difficult. Building faces over the vertices along the “top” of the model, where the terrain is, was relatively straightforward; however, making the model solid at the sides and bottom was a bit trickier.

At this point, it’s been long enough since I wrote the code that I don’t remember what exactly I was doing. To make things worse, it would appear that I didn’t write any comments. Therefore, all I can do is link the relevant section of code and wish you the best, if you wish to attempt understanding it.

Exporting the mesh

At this point, we are ready to export our mesh as an STL file. This is yet another relatively straightforward file format, consisting of a mostly-useless header and a list of vertices that form each face. Each face also has some optional auxiliary information and a normal vector, however I found that populating this was unnecessary. Using Python’s builtin struct module, we can easily pack our floating-point coordinates into the file:

fh.write(b'\0' * 80)
fh.write(struct.pack('<I', len(triangles)))
for a, b, c in triangles:
    fh.write(struct.pack('<fff', 0, 0, 0))
    fh.write(struct.pack('<fff', *vertices[a]))
    fh.write(struct.pack('<fff', *vertices[b]))
    fh.write(struct.pack('<fff', *vertices[c]))
    fh.write(struct.pack('<H', 0))

Now we’re finally ready to print. However, not having my own 3D printer, I ended up sending the model over to Shapeways. It turned out pretty great:

Again, the code is available here on Github: https://github.com/tmick0/topostl

A graphical tool for configuring Alesis V-Series MIDI controllers on Linux.

Sun, 12 Nov 2017 00:00:00 +0000

In my last post, I explained how I reverse-engineered the Alesis SysEx protocol and detailed my findings. Now, two months later, I’ve finally decided to implement a tool for configuring these controllers on Linux.

The entirety of this work was done over the past two days, so it likely contains some hidden bugs, but as far as I can tell it is entirely usable.

Unfortunately, this is my first attempt at writing any kind of GUI, so it’s not beautiful. But hey, it works.

You can find the project on GitHub. Contributions are welcome.

Reverse engineering the Alesis V-series SysEx protocol.

Fri, 08 Sep 2017 00:00:00 +0000

I recently got back into music production and decided to order myself a MIDI controller. I got a few recommendations for the the Alesis V25, so I went ahead and ordered it. However, I was less than pleased to find that its configuration software wouldn’t run on Linux, even under Wine. Of course, this prompted me to reverse engineer the protocol that lets the software talk to the keyboard.

Overview of MIDI

Musical Instrument Digital Interface (MIDI) is a long-lived standard for interactions both among musical instruments and between instruments and software. The Alesis V25 is a MIDI controller, which is essentially just a keyboard that lets you send notes to a Digital Audio Workstation (DAW) such as Bitwig (which is just my personal favorite).

MIDI messages are always three bytes, and can look something like this (in hexadecimal):

09 30 6a

The first byte is called the “status byte,” and the second two are simply called “data bytes.”

The meaning of the particular bytes in the above message are as follows:


`09`	Channel 0, note on
`30`	Note 48 (C#4)
`6a`	Velocity 106

In addition to note messages, there are also Control Change (CC) messages which indicate that a knob was turned or a button was pressed. However, these standard messages aren’t going to be very important for the adventure we’re going on today. Reconfiguring MIDI devices uses an extended protocol called SysEx.

Introduction to SysEx

System Exclusive (SysEx) is an extension to MIDI that lets instrument manufacturers define custom commands for their devices. The specification is quite simple: it only requires that SysEx messages start with a two byte header, and end with a one-byte trailer:

f0 xx ... f7

Here, f0 is the start byte, xx is a placeholder for a manufacturer ID, and f7 is the end byte. The bytes in between can be anything, as long as their high bits aren’t set (i.e., they are less than or equal to 7f). Their meaning is interpreted on a manufacturer-by-manufacturer basis.

Overview of the Alesis V25 Editor

Alesis’ editor lets you do things such as reassign the control numbers on the keyboard’s knobs and buttons. It doesn’t do much – it mostly just supports reading the configuration from the keyboard, editing it, and writing it back. The interface looks something like this:

Since the software doesn’t work on Linux, I had it running in a virtual Windows machine under Virtualbox, with USB forwarding enabled for the device.

Intercepting SysEx Messages

We want to see what this software is doing in the backend to update the MIDI device’s configuration. To do that, we’ll need to snoop on the SysEx messages that it exchanges with the device.

It turns out that the popular network-capture software Wireshark supports snooping on USB devices. That’s exactly what we’ll need to do today.

First, I had to enable the usbmon kernel module:

sudo modprobe usbmon

Then, I started up Wireshark and started listening for packets. The display was initially quite noisy due to other USB devices (mostly my mouse). I found the bus and device IDs via lsusb and filtered on those.

In this particular capture I’ve simply queried for the controller’s current configuration and received a reply. The 80-byte packet is the query, and the 128-byte packets are chunks of the reply. The 64-byte messages in between appear to be acknowledgements occurring at the USB protocol level.

We can dissect one of the packets to see what’s going on:

We can see that the SysEx message has been divided up into 3-byte segments in accordance with the traditional MIDI specification. At the USB level, each 3-byte segment is given a header to indicate which device it is from and how many of the SysEx bytes are meaningful. The details of that aren’t important, but the result is that we are able to pull out the actual SysEx message for the query:

f0 00 00 0e 00 41 62 00 5d f7

We can identify the SysEx start and end bytes, and conveniently the manufacturer ID is 00. A bit of research has driven me toward the conclusion that because Alesis hasn’t officially registered a MIDI manufacturer ID, it uses 00 with a two-byte extension of 00 0e. I haven’t deciphered all of this message, however have been able to determine that the byte 62 is the important part; it indicates that this packet is a query for the current configuration.

Decoding the Reply

To determine which part of the SysEx messages do what, I had to play around with the GUI for a while, changing options to see how they affected the data going over the wire. This was a tedious process, however in the end it only took about three hours.

The reply is quite long, and is split over several USB packets, however I’ve copied and annotated the final SysEx message here:

Raw bytes	Section	Interpretation
`f0`	SysEx start byte
`00 00 0e`	Alesis manufacturer ID
`00 41 63 00 5d`	Some sort of header	The 3rd byte indicates message type: `61`: Set configuration `62`: Query configuration `63`: Current configuration (reply)
`0c 02 00 00`	Keys configuration	Base note Current octave Channel Velocity curve
`00`	Pitch wheel configuration	Channel
`00 01 00 7f`	Mod wheel configuration	Channel CC number Minimum value Maximum value
`40 00 7f 00`	Sustain pedal configuration	CC number Minimum value Maximum value Channel
`00 14 00 7f 00` `00 15 00 7f 00` `00 16 00 7f 00` `00 17 00 7f 00`	Knobs configuration	Operation mode `00`: CC `01`: Aftertouch CC number Minimum value Maximum value Channel
`00 31 00 00 09` `00 20 00 00 09` `00 2a 00 00 09` `00 2e 00 00 09` `00 24 00 00 09` `00 25 00 00 09` `00 26 00 00 09` `00 27 00 00 09`	Pads configuration	Operation mode `00`: Note `01`: Toggle CC `02`: Momentary CC Note / CC number Fixed (?) / Minimum CC value Velocity curve / Maximum CC value Channel
`00 30 7f 00 00` `00 31 7f 00 00` `00 32 7f 00 00` `00 33 7f 00 00`	Buttons configuration	Operation mode `00`: Toggle `01`: Momentary CC number On value Off value Channel
`f7`	SysEx end byte

Note that I’ve neglected exploring some of the options provided by the GUI, such as assigning the buttons to Program Change events. I also have no idea what the “Fixed” field does for the drum pads, as the word “Fixed” is the only description offered by the GUI. I suspect it enables fixed velocity, but I haven’t bothered to check.

Next Steps

Now that I’ve decoded the SysEx protocol, I plan to make a tool to enable editing the controller’s configuration under Linux. I may go as far as writing a controller script to enable modifying it from within Bitwig. Check for updates here to see what’s to come.

Using black magic to make a fast circular buffer.

Tue, 15 Aug 2017 00:00:00 +0000

Yesterday, I took a glance at the Wikipedia page for the circular buffer and was intrigued by an alleged optimization technique that I was not familiar with:

A circular-buffer implementation may be optimized by mapping the underlying buffer to two contiguous regions of virtual memory. (Naturally, the underlying buffer‘s length must then equal some multiple of the system’s page size.) Reading from and writing to the circular buffer may then be carried out with greater efficiency by means of direct memory access; those accesses which fall beyond the end of the first virtual-memory region will automatically wrap around to the beginning of the underlying buffer. When the read offset is advanced into the second virtual-memory region, both offsets—read and write—are decremented by the length of the underlying buffer

When implementing a circular buffer, we need to handle the case where a message spans the “discontinuity” in the queue and wraps around. The naive circular buffer’s write routine might employ a byte-by-byte write and look something like this:

void put(queue_t *q, uint8_t *data, size_t size) {
    for(size_t i = 0; i < size; i++){
        q->buffer[(q->tail + i) % q->buffer_size] = data[i];
    }
    q->tail = (q->tail + size) % q->buffer_size;
}

The fact that a modulo operation is necessary to index into the array makes this function hard (if not impossible) to vectorize, and thus unnecessarily slow. Though there are other optimizations we can make, the technique offered in the above Wikipedia surpasses hardware-agnostic approaches by virtue of the fact that the memory management unit can handle most of the wrap-around logic on our behalf. I was so excited by this idea that I did no further research whatsoever, and implemented it based only on the brief description above.

The next two sections overview the circular buffer’s design and the behavior of virtual memory, respectively. If you don’t need the refresher, feel free to skip ahead.

The Circular Buffer

The circular buffer is a convenient approach to the storage of streaming data which is produced in real-time and consumed shortly thereafter. It “wraps around” so that new data may continually reuse the space previously occupied by data which has since been consumed.

A circular buffer is typically implemented by storing two pointers (or indices): a read pointer (I’ll call it head) and a write pointer (or tail). As is obvious, we write into the buffer at tail, and read from head. The structure might be defined as follows:

typedef struct {
    uint8_t *buffer;
    size_t   buffer_size;
    size_t   head;
    size_t   tail;
    size_t   bytes_avail;
} queue_t;

Given this, we can write simple read (get) and write (put) routines using byte-by-byte accesses:

bool put(queue_t *q, uint8_t *data, size_t size) {
    if(q->buffer_size - q->bytes_avail < size){
        return false;
    }
    for(size_t i = 0; i < size; i++){
        q->buffer[(q->tail + i) % q->buffer_size] = data[i];
    }
    q->tail = (q->tail + size) % q->buffer_size;
    q->bytes_avail += size;
    return true;
}

bool get(queue_t *q, uint8_t *data, size_t size) {
    if(q->bytes_avail < size){
        return false;
    }
    for(size_t i = 0; i < size; i++){
        data[i] = q->buffer[(q->head + i) % q->buffer_size];
    }
    q->head = (q->head + size) % q->buffer_size;
    q->bytes_avail -= size;
    return true;
}

Note that this put routine is identical to the one given above, with the exception that we now check that sufficient space is available before attempting to write. I also have purposefully neglected synchronization logic (which would be absolutely necessary for any real application of the circular buffer).

The byte-by-byte access pattern and modular arithmetic embedded in each iteration of the loop make this code horrendously slow. We can instead implement each read or write operation with two memcpy calls, where one handles the portion of a message at the end of the buffer, and the other handles the portion at the beginning if we wrapped around.

static inline off_t min(off_t a, off_t b) {
    return a < b ? a : b;
}

bool put(queue_t *q, uint8_t *data, size_t size) {
    if(q->buffer_size - q->bytes_avail < size){
        return false;
    }
 
    const size_t part1 = min(size, q->buffer_size - q->tail);
    const size_t part2 = size - part1;
 
    memcpy(q->buffer + q->tail, data,         part1);
    memcpy(q->buffer,           data + part1, part2);
 
    q->tail = (q->tail + size) % q->buffer_size;
    q->bytes_avail += size;
    return true;
}

bool get(queue_t *q, uint8_t *data, size_t size) {
    if(q->bytes_avail < size){
        return false;
    }
 
    const size_t part1 = min(size, q->buffer_size - q->head);
    const size_t part2 = size - part1;
 
    memcpy(data,         q->buffer + q->head, part1);
    memcpy(data + part1, q->buffer,           part2);
 
    q->head = (q->head + size) % q->buffer_size;
    q->bytes_avail -= size;
    return true;
}

An example usage of this circular buffer would be as follows:

int main() {
    queue_t queue;

    queue.buffer      = malloc(128);
    queue.buffer_size = 128;
    queue.head        = 0;
    queue.tail        = 0;
    queue.bytes_avail = 0;

    put(&q, "hello ", 6);
    put(&q, "world\n", 7);

    char s[13];
    get(&q, (uint8_t *) s, 13);

    printf(s); // prints "hello world"
}

Our code is simple, and it works fine. But why not make it more complicated?

Enter the Page Table

In the early days of personal computing, computers could essentially only run one program at a time. Whatever program we ran would have full, direct access to the physical memory. It turns out that if we want to run several programs together, they have a tendency to fight over what regions of that memory they want to use. The solution to this conflict is virtual memory, under which each program thinks it controls all of the memory, but in reality the operating system decides who’s getting memory from where.

The key to virtual memory is the page table, through which the operating system informs the CPU of mappings between virtual pages and physical pages. A page is a contiguous, aligned region of memory of a (typically) fixed size, say around 4 KiB. When a program wants to access some (virtual) memory, the CPU figures out what page it belongs to then uses the page table to index into the physical memory.

The page table for some Program A might look something like this:

Virtual Page Number	Physical Page Number
0	null
1	25
2	12
3	null
4	56
…	…

Meanwhile, Program B’s could be as follows:

Virtual Page Number	Physical Page Number
0	11
1	null
2	92
3	21
4	null
…	…

The key point is that there doesn’t need to be any rhyme or reason to the way physical pages are assigned to virtual pages. They can be out-of-order with respect to the virtual pages, and some virtual pages don’t need physical memory assigned to them at all. Our two programs can use the same virtual page number to refer to different physical pages, and thus they don’t need to worry about conflicting with each other’s memory.

The details behind figuring out the page numbers, assigning memory, and indexing into the page table are out of the scope of this article, but are unnecessary for understanding what’s to come.

A Few Ordinary System Calls, and One That glibc Doesn’t Want You to Know About

Before we proceed to exploit the page table in optimizing our circular buffer, let’s review some common Linux system calls.

System Call	Description
`int getpagesize()`	Returns the number of bytes in a page of memory. (Technically not always a system call, but close enough.)
`int ftruncate(int fd, off_t length)`	Sets the size of a file to `length` using its file descriptor, `fd`.
`void mmap(void addr, size_t length, int prot, int flags, int fd, off_t offset)`	Maps the file described by `fd` into memory and returns a pointer the new memory region. Can be manipulated to do evil through configuration `flags`.

And now, a less common system call that glibc (our friendly userland interface to the kernel) doesn’t expose to us:

System Call	Description
`int memfd_create(const char *name, unsigned int flags)`	Returns a file descriptor for an `anonymous file` which exists only in memory.

Interesting! Because glibc doesn’t implement it, we will need to write a little bit of wrapper code if we want to use it.

int memfd_create(const char *name, unsigned int flags) {
    return syscall(__NR_memfd_create, name, flags);
}

Excellent, now we’ll be able to call it like any other function. This is going to let us allocate memory and manipulate it like a file. Since it acts like a file, we’ll be able to mmap it wherever we want (spoiler alert).

Evil Black Magic Page Table Hacking

Okay, let’s get down to business. Let’s do what Wikipedia said and make two adjacent pages point to the same memory. We’ll need to modify our queue structure and extend its API with a convenient initialization method.

typedef struct {
    uint8_t *buffer;
    size_t   buffer_size;
    int      fd;
    size_t   head;
    size_t   tail;
} queue_t;

bool init(queue *q, size_t size){

    // First, make sure the size is a multiple of the page size
    if(size % getpagesize() != 0){
        return 0;
    }

    // Make an anonymous file and set its size
    q->fd = memfd_create("queue_buffer", 0);
    ftruncate(q->fd, size);

    // Ask mmap for an address at a location where we can put both virtual copies of the buffer
    q->buffer = mmap(NULL, 2 * size, PROT_NONE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);

    // Map the buffer at that address
    mmap(q->buffer, size, PROT_READ | PROT_WRITE, MAP_SHARED | MAP_FIXED, q->fd, 0);

    // Now map it again, in the next virtual page
    mmap(q->buffer + size, size, PROT_READ | PROT_WRITE, MAP_SHARED | MAP_FIXED, q->fd, 0);

    // Initialize our buffer indices
    q->head = 0;
    q->tail = 0;
}

Okay, what’s happening? Well, first we call memfd_create, which does exactly what it says on the tin. We find some space for two copies of the buffer, and store that as our buffer address. Then we map the virtual file into memory, at the address mmap gave us. After that is where our evil kicks in.

We ask mmap to map our file again at the address size bytes past its original location. Now, the page table is going to look something like this (assuming size is larger than one page):

Virtual Address	Physical Address
…	…
`q->buffer`	anonymous file `q->fd`, offset 0
`q->buffer + getpagesize()`	anonymous file `q->fd`, offset `getpagesize()`
…	…
`q->buffer + size`	anonymous file `q->fd`, offset 0
`q->buffer + size + getpagesize()`	anonymous file `q->fd`, offset `getpagesize()`
…	…

Yes, we have two virtual pages that point to the same physical page, for each page in our buffer. Let’s see how this will affect our circular buffer logic:

bool put(queue_t *q, uint8_t *data, size_t size) {
    if(q->buffer_size - (q->tail - q->head) < size){
        return false;
    }
    memcpy(&q->buffer[q->tail], data, size);
    q->tail += size;
    return true;
}

bool get(queue_t *q, uint8_t *data, size_t size) {
    if(q->tail - q->head < size){
        return false;
    }
    memcpy(data, &q->buffer[q->head], size);
    q->head += size;
    if(q->head > q->buffer_size) {
       q->head -= q->buffer_size;
       q->tail -= q->buffer_size;
    }
    return true;
}

We allow our tail offset to advance into the second virtual memory region, past the bounds of the “real” buffer. However, it still writes to the same physical memory. Similarly, we can read past the bounds of the first region, and by going into the second region we end up reading the same physical memory as if we had manually wrapped around.

Notice that we need to make only one memcpy call instead of two. We check after reading data that the head pointer hasn’t moved into the second virtual copy of the buffer. If it did, we can decrement both by the buffer size; they will point to the same physical memory, though the virtual addresses will be different. The API remains the same as before.

How Good Is It?

The key difference here is that we can always access contiguous blocks of virtual memory, instead of worrying about wrapping around mid-message. We can write a simple micro-benchmark that isolates this behavior as follows:

#include <stdio.h>
#include <string.h>
#include <stdint.h>
#include <sys/time.h>

#define BUFFER_SIZE (1024)
#define MESSAGE_SIZE (32)
#define NUMBER_RUNS (1000000)

static inline double microtime() {
    struct timeval tv;
    gettimeofday(&tv, NULL);
    return 1e6 * tv.tv_sec + tv.tv_usec;
}

static inline off_t min(off_t a, off_t b) {
    return a < b ? a : b;
}

int main() {
    uint8_t message[MESSAGE_SIZE];
    uint8_t buffer[2 * BUFFER_SIZE];
    size_t offset;
 
    double slow_start = microtime();
    offset = 0;
    for(int i = 0; i < NUMBER_RUNS; i++){
        const size_t part1 = min(MESSAGE_SIZE, BUFFER_SIZE - offset);
        const size_t part2 = MESSAGE_SIZE - part1;
        memcpy(buffer + offset, message, part1);
        memcpy(buffer, message + part1, part2);
        offset = (offset + MESSAGE_SIZE) % BUFFER_SIZE;
    }
    double slow_stop = microtime();
 
    double fast_start = microtime();
    offset = 0;
    for(int i = 0; i < NUMBER_RUNS; i++){
        memcpy(&buffer[offset], message, MESSAGE_SIZE);
        offset = (offset + MESSAGE_SIZE) % BUFFER_SIZE;
    }
    double fast_stop = microtime();
 
    printf("slow: %f microseconds per write\n", (slow_stop - slow_start) / NUMBER_RUNS);
    printf("fast: %f microseconds per write\n", (fast_stop - fast_start) / NUMBER_RUNS);
 
    return 0;
}

On my i5-6400, I get pretty consistent results, looking something like this:

slow: 0.012196 microseconds per write
fast: 0.004024 microseconds per write

Note that the single-memcpy code is about three times faster than the double-memcpy code. We have a much larger margin over the naive byte-by-byte copy, which benchmarked at 0.104943 microseconds per write. This micro-benchmark is not wholly representative of the full queue logic (which could be affected by conditions such as page faults and TLB misses), however gives us a good indication that our optimization was worthwhile.

What Just Happened?

Apparently this trick has been known for a while, but is seldom used. Nonetheless, it’s an excellent example of taking advantage of a machine’s low-level behaviors to optimize software.

If you’re interested in seeing my full circular buffer implementation, you can find it on GitHub. The version there is complete with synchronization, error checking, and access at the granularity of arbitrarily-sized messages.

Proving a mathematical curiosity.

Wed, 21 Jun 2017 00:00:00 +0000

Today, a thread full of cool math facts appeared on Reddit. As usual, someone mentioned the fact that 111111111 * 111111111 = 12345678987654321. In another reply, someone pointed out that this also works in other bases. For some reason, I decided that I needed to prove that it works in all bases.

To begin, I needed a general formula for values of the 111… terms. This was fairly straightforward: for a base $b$, we want $b-1$ base-$b$ digits, all ones. To standardize the base, we multiply each digit by an increasing power of $b$ and sum. Since each digit is one, we get a nice geometric series which can easily be solved.

$$ \sum\limits_{i=1}^{b-1} b^{i-1} = \frac{b-b^b}{b-b^2}$$

When we multiply this number by itself, we are squaring it, so we end up with $\left((b-b^b)/(b-b^2)\right)^2$.

The hard part was writing a general form for the $1234 \dots (b-1) \dots 4321$ number. To deal with this, I broke it down into two parts, as illustrated below.


Digit value	1	2		$(b-2)$	$(b-1)$	$(b-2)$		2	1
	$b^{2b-3}$	$b^{2b-4}$	$\dots$	$b^{2b-(b+1)}$	$b^{2b-(b+2)}$		$\dots$
					$b^{b-2}$	$b^{b-3}$		$b^1$	$b^0$

I calculated the values of the most-significant digits starting at the left, and the values of the least-significant digits starting at a right. To make the math come out nicely, I actually included the center digit in both formulas. That’s okay, since we can subtract it off once to make up for the duplicate. Now we have a summation formula for the value of the square.

$$ \left( \sum\limits_{i=1}^{b-1} ib^{2b-(i+3)} \right) + \left( \sum\limits_{i=1}^{b-1} ib^{i-1} \right) - b^{b-2} $$

With a little thinking (or the help of a computer algebra system), we can get a neat closed form.

$$ \left( \sum\limits_{i=1}^{b-1} ib^{2b-(i+3)} + ib^{i-1} \right) - b^{b-2} = \left(\frac{b-b^b}{b^2-b}\right)^2 $$

We can see that this is quite similar to the expression we got for the square above; the only difference is that the $b-b^2$ denominator has changed to $b^2-b$. Fortunately, this negation goes away when squaring, so we can trivially prove that the two expressions are equal.

And there we have it: proof that this curiosity is true in any base of at least two.

Generating spectrograms the hard way with numpy.

Thu, 18 May 2017 00:00:00 +0000

A spectrogram is a convenient visualization of the frequencies present in an audio clip. Generating one involves obtaining the frequency components of each window of the audio via a Discrete Fourier Transform (DFT) of its waveform. While tools are available to both generate spectrograms and compute DFTs, I thought it would be fun to implement both myself in my language of choice, Python.

In the following, I will discuss computing a DFT (the hard way), processing a WAV file, and rendering a spectrogram (all in Python). If you’re impatient and just want to see the code, you can find it on GitHub.

The Discrete Fourier Transform.

A Fourier Transform converts a signal from the time domain (i.e., a waveform) to the frequency domain, wherein peaks represent dominant frequencies in the signal. The Discrete Fourier Transform (DFT) is the finite-resolution version of the Fourier Transform that we use for sampled signals such as audio clips.

Usually, the DFT is computed using an algorithm called the Fast Fourier Transform (FFT). However, the naive computation of the DFT just involves multiplying a vector of samples with what is known as the DFT matrix.

The values of the DFT matrix are the same for each given size of sample vector. One may refer to the relevant Wikipedia article for examples of the 2-element, 4-element, and 8-element DFT matrices. Each is computable through a general formula. To create a DFT matrix, simply assign each element of the matrix the value $\omega^{jk}$, where $j, k$ are the element indices. The $\omega$ itself is a constant determined by the vector size $N$: $\omega = e^{-2i\pi/N}$.

It turns out that this type of matrix, where a constant is raised to a power depending on its index, is called a Vandermonde matrix. Fortunately, numpy gives us a useful method for computing such matrices. Therefore, in Python, we may simply compute the value of $\omega$:

w = np.exp(-2j * np.pi / N)

And then create a Vandermonde “matrix” which is really just a column vector:

col = np.vander([w], N, True)

And finally build the complete Vandermonde matrix using that column:

W = np.vander(col.flatten(), N, True)

To perform a DFT, we simply compute the dot product between this matrix $W$ and our vector $V$. Putting it all together, we have a very simple function:

def matrix_dft(V):
    N = len(V)
    w = np.exp(-2j * np.pi / N)
    col = np.vander([w], N, True)
    W = np.vander(col.flatten(), N, True) / np.sqrt(N)
    return np.dot(W, V)

Note that we scale by the term $1/\sqrt{N}$, as is conventional.

Catching a WAV.

WAV files are simple, and seemingly timeless. They’re the most simple format for storing uncompressed audio. Python ships with a library for parsing them, so they will be quite convenient for the task at hand.

First, we must open the file and determine important parameters such as the sample rate, number of channels, sample width, and number of frames.

w = wave.open(input_file, 'r')
    
sample_rate  = w.getframerate()
num_channels = w.getnchannels()
sample_width = w.getsampwidth()
num_frames   = w.getnframes()

When we ask to read a frame of audio, it will be given to us in raw form. Therefore, we need to know the sample width and number of channels in order to make sense of it.

x = w.readframes(1)
x = np.fromstring(x, dtype=np.int8 if sample_width == 1 else np.int16)
x = np.reshape(x, (1, num_channels))

We begin by reading the raw frame. This gives us an array of bytes. We pass it to numpy’s fromstring function, which will convert it to an array for us. Note that we must specify the datatype as int8 if the sample width is one byte (8 bits), or int16 if the sample width is two bytes (16 bits). Then, we reshape the array into a matrix wherein the channels are separated.

This single audio sample isn’t incredibly useful, so we can also request a window of some number of samples and process it the same way:

window_size = 1024
x = w.readframes(window_size)
x = np.fromstring(x, dtype=np.int8 if sample_width == 1 else np.int16)
x = np.reshape(x, (window_size, num_channels))

This will be quite necessary shortly, when we begin computing DFTs in order to generate our spectrogram.

Transforming the Window.

Once we have our window of audio, we can do some magic to it to get spectrum data out. If the audio is stereo, we will first mix down to mono before we run a DFT. Then, we must perform some postprocessing to make that output useful.

if num_channels > 1:
    x = (x[:,0] + x[:,1]) / 2
else:
    x = x[:,0]
        
y = np.fft.rfft(x)
y = y[:window_size/2]
y = np.absolute(y) * 2.0 / window_size
y = y / np.power(2.0, 8*sample_width - 1)
y = (20 * np.log10(y)).clip(-120)

To mix together the two channels, we simple take their average. Note that instead of using the DFT function we implemented earlier, we simply called the FFT provided by numpy because it is way faster.

You may be surprised to see that we have discarded half of the output. This is because the DFT result is symmetric, and thus the second half is redundant and we won’t need it. You may have noticed earlier that the DFT matrix itself is symmetric across its diagonal – that explains this result.

Next, we take the absolute magnitude of the DFT, which gives us a real-valued magnitude. We scale by a factor of 2 because we originally discarded half of the spectrum, then divide by the window size to give us a value of magnitude per sample. We divide again by the value $2^{8s-1}$, where $s$ is the sample width, to give us an intensity relative to the maximum intensity possible with a sample of that size (recall that $s$ bytes equals $8s$ bits, and samples are signed integers). Finally, we compute $20log_{10}(y)$ to convert to decibels (dB) and filter out values below -120 dB.

Now, we have successfully computed the spectral content of a single window of audio. We may perform the above computation in a loop in order to generate spectra for each frame. In my implementation, I appended each of the above results y to a list Y. By combining all of these frames, we can produce a spectrogram.

Plotting the Spectrum.

I chose to use matplotlib to visualize my data, since it is the plotting system most familiar to me.

It turns out, the first thing we must do is transpose the list Y. By default, numpy treats each subsequent input y as a row, rather than a column (as we want for the purpose of a spectrogram).

Y = np.array(Y).transpose()

For quick-and-dirty validation purposes, we can visualize this using the imshow function.

plt.imshow(Y)
plt.show()

The result for a 440-Hz sine wave looks like this:

It looks pretty good, but there are a couple problems. First, the frequencies are plotted upside down. Then, notice that our axis labels are simply pixel coordinates. While there are hacks to fix both of these, it will be much easier to use the pcolormesh function instead. However, this requires us to first generate arrays for our time and frequency coordinates, which we will use as our x and y coordinates when plotting.

t = np.arange(0, len(Y), dtype=np.float) * window_size / sample_rate
f = np.arange(0, window_size / 2, dtype=np.float) * sample_rate / window_size

For the time coordinates, we start with an array of indices of each of our window spectra in Y. Then, we multiply by the window size (number of samples per window) and divide by the sample rate (number of samples per second). The result is the time coordinate, in seconds, for each window.

The frequency coordinates are generated similarly. We start with the indices of our window\_size / 2 frequency values, then multiply by the sample rate and divide by the window size. This is because the spacing of frequencies returned by the DFT is given by $f/N$, where $f$ is the sample rate and $N$ is the window size.

Having generated our coordinate arrays, we can finally call pcolormesh. Note that we specify a range of -120 dB to 0 dB for the colormap.

plt.pcolormesh(t, f, Y, vmin=-120, vmax=0)

The result is still not that pretty, however at least now everything is right-side up and our axes are numbered appropriately.

To complete the presentation of our spectrogram, we only have a few more changes to make. We will use a ‘symlog’ scale (which is logarithmic above a certain threshold, and linear below it), set our axes limits, label the axes, and display a colorbar.

plt.yscale('symlog', linthreshy=100, linscaley=0.25)
ax.yaxis.set_major_formatter(matplotlib.ticker.ScalarFormatter())

plt.xlim(0, t[-1])
plt.ylim(0, f[-1])

plt.xlabel("Time (s)")
plt.ylabel("Frequency (Hz)")

cbar = plt.colorbar()
cbar.set_label("Intensity (dB)")

The result is now quite acceptable.

We can even run an analysis on an entire song.

There we have it, a fully-functional spectrogram tool.

Completing the Package.

To make things a bit more useful, I added a progress bar while the DFTs are being calculated, and added an option to change the window size at runtime. With larger windows, you will notice much better resolution in the output.

As previously mentioned, the complete source is available in GitHub. While it isn’t incredibly useful for real applications, this was quite a fun project, especially considering that I had no prior background in practical signals analysis. For real-world applications, you might want to consider matplotlib’s builtin spectrogram capabilities, or a standalone tool such as Spek.

slug: numpy-spectrogram

Update: Window Function.

I was given a suggestion to implement a smooth (rather than rectangular) window function, and after reading this guide I found that it was quite straightforward. The calculation of the Hann window function was a one-liner:

hann = 0.5 - 0.5 * np.cos(2.0 * np.pi * (np.arange(window_size)) / window_size)

I then multiplied these coefficients against the audio window before feeding it to the FFT. Because the window is tapered, I also had to implement a running window with 4x overlap in order to cover the full signal. The result has significantly less spectral leakage.

The plot on the left has a rectangular window, and the plot on the right has the Hann window. It is visibly more crisp. As always, the full updated code is on GitHub.

Integrating GitLab and Google Calendar.

Tue, 25 Apr 2017 00:00:00 +0000

Zeall, like many other software startups, uses GitLab for version control and issue management. We also use the ever-popular Google Calendar to handle meetings, reminders, and deadlines. For several months, we’ve been looking for a way to automatically push GitLab issue deadlines into Google Calendar, and until now it seemed impossible. Only after a recent migration from our own private mailserver to G Suite did we find a solution – or rather, figure out how to feasibly build one.

GitLab’s webhooks make it pretty easy to obtain information about new and modified issue deadlines, however the problem has always been pushing that information into Google Calendar. I had previously explored the Google Calendar API and ultimately decided that implementing our own solution wasn’t worth it, simply because authorization would be a nightmare – there was no easy way to share a calendar with our team’s members. However, once we got a G Suite domain, I found it was possible to use a domain scope for Calendar ACLs. I simply created a service account for our integration, set it up to grant read access to our entire domain, and implemented some basic logic to map GitLab issues to calendar events.

As far as I can tell, I’m the first person to have successfully integrated GitLab issues into Google Calendar; however, I’m sure I’m not the first to want to do so. Therefore, to spare others from sharing in my misery I’ve decided to make the fruit of my labor available on GitHub for anybody to use. Try it out, and be sure to let me know if it breaks.

Adding custom fields to packets in ndnSIM 2.3 without forking the entire repository.

Thu, 16 Mar 2017 00:00:00 +0000

The recommended way to build something on top of ndnSIM is to fork its scenario template repository and work inside there. You still need to download and compile the actual framework, however you will simply install it into /usr/local and link to it instead of actually working inside the main repository.

It turns out that this workflow actually makes certain tasks a lot more difficult. You might think a network simulator would make it easy to add new header fields to packets. Well, think again.

First Steps

What do we want to do? Our goal is to just add one field to the Interest packet header. The ndn::Interest class inherits an interface called ndn::TagHost, which allows you to attach arbitrary tags to it. Defining your own tag can be as simple as a single typedef, if you only need to contain a single value in that tag:

typedef ndn::SimpleTag<uint64_t, 0x60000001> MyCustomTag;

You simply specify the type of the tag and make up an ID for it. However, you must pick an unused tag from the valid range given in the ndn-cxx wiki. My 0x60000001 is the first value in this range.

To attach a tag to an Interest, you simply call the setTag method:

interest.setTag<MyCustomTag>(std::make_shared<MyCustomTag>(54321));

To read a tag from an Interest, there is a corresponding getTag method:

std::shared_ptr<MyCustomTag> tag = interest.getTag<MyCustomTag>();

This gives you a pointer to the tag object, and you can get the value out of it quite easily… But first, check if it is null.

if (tag == nullptr) {
    // no tag
}
else {
    uint64_t tagValue = tag->get();
}

However, now is where we encounter our problem. Our tag will not actually be encoded and sent over the network. That’s right – we can attach a tag to the Interest, but when it arrives at the next hop it will be gone.

How can we fix this?

Investigation

Vanilla ndnSIM uses these sorts of tags itself in a few places. One obvious one is the HopCountTag, which you can use to figure out how far a packet has gone in the network. A grep through the ndnSIM source brings us to a class called GenericLinkService. This class is responsible for actually encoding packets and sending them out on the wire. In particular, we can find the bit responsible for encoding the HopCountTag in a method called encodeLpFields:

shared_ptr<lp::HopCountTag> hopCountTag = netPkt.getTag<lp::HopCountTag>();
if (hopCountTag != nullptr) {
    lpPacket.add<lp::HopCountTagField>(*hopCountTag);
}
else {
    lpPacket.add<lp::HopCountTagField>(0);
}

Clearly, we need to define a MyCustomTagField to be able to encode our new tag.

Declaring a Tag

This is actually pretty easy, but first you need to know what kind of witchcraft is going on. Let’s start with the actual code to define the field, then go on to analyze it:

enum {
 TlvMyCustomTag = 901
};

typedef ndn::lp::detail::FieldDecl<ndn::lp::field_location_tags::Header, uint64_t, TlvMyCustomTag> MyCustomTagField;

First, we define a constant for the TLV type ID… There are actually a few hidden constraints to what we can pick. If we don’t do this right, we get a packet parse error. Why?

Let’s look at ndn::lp::Packet’s wireDecode method:

for (const Block& element : wire.elements()) {
    detail::FieldInfo info(element.type());
    if (!info.isRecognized && !info.canIgnore) {
        BOOST_THROW_EXCEPTION(Error("unrecognized field cannot be ignored"));
    }
    ...
}

Apparently, this FieldInfo class tells the decoder whether the field is recognized, and whether it can ignore it if it isn’t. Let’s peek at the constructor:

FieldInfo::FieldInfo(uint64_t tlv)
: ...
{
   boost::mpl::for_each<FieldSet>(boost::bind(ExtractFieldInfo(), this, _1));
   if (!isRecognized) {
     canIgnore = tlv::HEADER3_MIN <= tlvType
                 && tlvType <= tlv::HEADER3_MAX
                 && (tlvType & 0x01) == 0x01;
  }
}

Now this is interesting… To figure out what a TLV tag is, it iterates over FieldSet (which only contains the built-in tags, and we can’t override). However, if it doesn’t find a match, it determines if it is ignorable based on the value of the TLV type ID. We can’t make the field recognized without forking the actual ndnSIM core, but we can make it ignorable by choosing the right ID.

To save you from looking up tlv::HEADER3_MIN and tlv::HEADER3_MAX, they are 800 and 959, respectively. Also, don’t forget that the low bit has to be set. And don’t pick one of the types that is already used.

Moving on from the TLV ID nonsense, the rest of the FieldDecl is pretty straightforward. We pass a flag that says “this goes in the header,” followed by the type of the value and the TLV ID we just made up.

Note that for some reason, the code won’t compile if the type is specified as anything other than uint64_t. I didn’t care enough to figure this out, but it seems to have something to do with the fact that the only integer EncodeHelper defined is for uint64_t.

Encoding the Tag

So far, we have defined our tag twice: once for the high-level Interest object, and once for the low-level TLV encoding. Now, we need to write code to convert between these two representations.

To do this, we need to create a new LinkService. Sounds intimidating, but really all we need to do is make a copy of GenericLinkService and change a few things. Yes, literally copy generic-link-service.hpp and generic-link-service.cpp out of ns3/ndnSIM/NFD/daemon/face/ and into your own project. Rename the file as you see fit, and carefully rename the class to something like CustomTagLinkService. You will want to be careful because we still need to implement the GenericLinkServiceCounters interface if we don’t want to break anything. We can also avoid redefining the nested Options class by using a typedef to import it from GenericLinkService into the new CustomTagLinkService namespace.

Now that we have an identical clone of the GenericLinkService, let’s fix it. To encode your new field, take a look at the encodeLpFields method. Follow the pattern used by the CongestionMarkTag field to implement your new custom one:

shared_ptr<MyCustomTag> myCustomTag = netPkt.getTag<MyCustomTag>();
if (myCustomTag != nullptr) {
    lpPacket.add<MyCustomTagField>(*myCustomTag);
}

Then, add the corresponding decoding logic to decodeInterest:

if (firstPkt.has<MyCustomTagField>()) {
    interest->setTag(make_shared<MyCustomTag>(firstPkt.get<MyCustomTagField>()));
}

Add the same code to the decodeData and decodeNack methods if you need them.

Using the LinkService

Specifying a custom LinkService isn’t going to do us any good if we don’t tell ndnSIM to use it. We’ll have to replace the callback that sets up a Face in order to do this. We’re going to focus on Faces for PointToPointNetDevices, but the following can be generalized for other types of links.

The call from our scenario file will look something like this:

stackHelper.UpdateFaceCreateCallback(
    PointToPointNetDevice::GetTypeId(),
    MakeCallback(CustomTagNetDeviceCallback)
);

For context, this is a method of the StackHelper that you’re probably already using to install the NDN stack on nodes. To write the callback, copy the logic from the PointToPointNetDeviceCallback in that same class. All you have to change is the instantiation of the LinkService – replace the GenericLinkService with your own. You will also need to copy the constructFaceUri method (verbatim) because your callback will need to refer to it, but it is out of scope.

Other Caveats

By default, the scenario template wants to compile your code in C++11 mode. However, the LinkService uses some C++14 features, so you’ll have to edit the flags in .waf-tools/default-compiler-flags.py. Note that you need to re-run ./waf configure if you edit these flags.

Conclusion

I think this is way too much effort just to add a field to a packet. We’ve duplicated a lot of logic in order to do something so small. I feel like the ndnSIM developers should have made it a bit easier to add fields to a packet… At worst, I might expect a call to the StackHelper to add new fields. It would likely be possible to write a generic enough LinkService which will encode any custom fields as long as mappings between the TLV classes and tag classes are provided. I look forward to this feature, because it would have made the middle part of my week go a lot more smoothly. Until then, I hope that this post can be useful to anyone else trying to do the same thing.

An idiot's guide to fulltext search in PostgreSQL.

Thu, 23 Feb 2017 00:00:00 +0000

I love PostgreSQL. It’s probably the most powerful open-source database system out there. Recent features to handle JSON and geospatial data are allowing it to supplant specialized database systems and become closer to a one-DB-fits-all solution. One feature that I’ve recently been able to exploit is its fulltext search engine. It allowed me to easily move from a terrible search implementation (using regular expressions) to one that actually meets users’ expectations.

In this article, I will walk through a basic fulltext search configuration, as well as highlight a few potential improvements that can be made if you’re so inclined.

Many of the features discussed in this post are only available as of PostgreSQL 9.6. Earlier versions have some rudimentary fulltext functionality, but a lot of the more powerful tools we’ll be using are fairly new.

The Wrong Way To Search

Let’s start with a basic table:

create table test_table (id serial, title text, description text);

And populate it with some bogus data:

insert into test_table (title, description) values ('Hello world!', 'This is an example.'), ('Filler text', 'foo bar baz');

How might we go about doing a simple search on this table? The naive strategy, which I originally might have used, would go something like this:

select * from test_table where title ~* 'hello' or description ~* 'hello';

This is the result (which is correct, in this case):

id  | title        | description
----+--------------+---------------------
1   | Hello world! | This is an example.
(1 row)

However, consider that a human searching for something might not always know exactly what they’re looking for. Let’s put a new row in that table:

insert into test_table (title, description) values ('A helpful example', 'This is where things get complicated');

Let’s say a user kind of knows what they’re looking for: they enter the query “gets complicated”:

select * from test_table where description ~* 'gets complicated';

There is no result. Why? Because we have no way to identify that “gets” and “get” are semantically equivalent.

Also consider that a regex-based search is at least linear in the length of each string we check (unless you enable trigram indices, apparently, but that’s out of scope of this article). If you’re searching a large table, you’re going to be in big trouble.

Fulltext search solves both of these problems: It treats semantically equivalent words as matches, and can be drastically sped up by virtue of indices.

A Better Solution

To handle fulltext searches, there are two main things we have to do: add a tsvector to the table to store keywords, and write a tsquery when we want to perform a search.

A tsvector is essentially a list of keywords. We can assign one to each row in the table. Postgres will automatically filter out common words and replace similar words with roots or synonyms. Here’s an example query, to demonstrate how a human-written string is translated into an indexable tsvector:

select to_tsvector('This is where things get complicated');

to_tsvector
-------------------------------
'complic':6 'get':5 'thing':4
(1 row)

We can see that the input got filtered down into three keywords. One of them, “complicated,” got trimmed down to the root “complic.” The rules for how this translation takes place are defined by the regconfig.

Your default regconfig is probably either “english” or “simple.” It can be retrieved by selecting get_current_ts_config(), or set with the default_text_search_config directive. If you need to change it per-query, you can stick it in any call to to_tsvector or its related functions, e.g. to_tsvector('english', 'hello world'). In most cases, you’ll want “english” (or some other human language) rather than “simple” (which doesn’t handle magic such as removing meaningless words and distilling words into lexemes). Read the documentation if you think you’ll need to do anything funky with regconfig.

Now that we understand what we’re doing, let’s add a tsvector column to the table:

alter table test_table add column tsv tsvector;

Note: It isn’t always necessary to create a dedicated tsvector column (i.e., you can skip right ahead to creating a fulltext index on a concatenation of existing columns), but I’ve done it here because it’s a more flexible option and it’s easier to reason about.

How do we populate this column? Well, it’s quite easy to add a trigger that does it for us. We just tell it the name of our tsvector column, the language to use for processing, and then list all of the fields that should be considered for keywords:

create trigger test_tsv before insert or update on test_table
for each row execute procedure tsvector_update_trigger(
  tsv, 'pg_catalog.english', title, description
);

Note that 'pg_catalog.english' is actually our regconfig showing up again. It is mandatory to supply a regconfig to tsvector_update_trigger, and you must fully qualify it with the schema name (pg_catalog).

Now, any new or updated tuples will be given a tsvector containing the title and description. Let’s see it in action:

insert into test_table (title, description) values ('We have tsvectors now!', 'Pretty cool, eh?') returning tsv;

tsv
-----------------------------------------
'cool':6 'eh':7 'pretti':5 'tsvector':3
(1 row)

We can do a quick hack to trigger the hook and create vectors for our pre-existing rows. Be careful doing this in production.

update test_table set id=id;

Creating the tsvectors was half the battle. Now, how do we query them? The quite intuitive answer: tsquery.

A tsquery, like a tsvector, is essentially a list of keywords. However, we can do some fun things, like apply boolean operations and specify that words have to be in a certain order. Here’s a list of a few useful operators we can abuse:

Syntax	Description
`foo <-> bar`	Matches “foo” followed by “bar”
`foo <2> baz`	Matches “foo” with “baz” two words later (i.e., one word in between). Replace the 2 with another number to match a larger “gap.”
`foo & bar`	Matches a vector that contains both “foo” and “bar,” in any order and with anything in between.
`foo	bar`
`!baz`	Matches a vector that doesn’t contain “baz.”

Here’s an example of a simple tsquery, to see how it is interpreted:

select to_tsquery('foo <-> baz');

to_tsquery
-----------------
'foo' <-> 'baz'

And here’s how we would use it to search (say hello to the @@ operator):

select id, title, description from test_table where tsv @@ to_tsquery('foo <-> bar');

id  | title       | description 
----+-------------+-------------
2   | Filler text | foo bar baz

Of course, we’re not going to get our users to type their queries in this format. Fortunately, Postgres gives us a few helper functions for creating queries. Let’s take a look at both plainto_tsquery and phraseto_tsquery, and see how they differ:

select plainto_tsquery('foo baz'), phraseto_tsquery('foo baz');
 plainto_tsquery | phraseto_tsquery 
-----------------+------------------
 'foo' &amp; 'baz'   | 'foo' <-> 'baz'
(1 row)

Most noticeably, plainto_tsquery just matches for the boolean conjunction of all words in the input, while phraseto_tsquery matches the exact ordering. Choose the one most appropriate for your application. You can easily make queries using a condition like where tsv @@ plainto_tsquery(user_input_here).

At this point, you’re able to handle simple fulltext queries. Congratulations. If you’re interested, read on for some interesting tweaks and optimizations.

Even Better

The next thing you’re going to want to add is an index. The Postgres documentation recommends using GIN indices for tsvectors (the reasons why are outside the scope of this post):

create index tsv_index on test_table using gin (tsv);

This will speed up your queries significantly, and is quite important if your tables are large.

There are several other features you might want to explore. For sake of brevity, I’ll just list a couple interesting ones and provide links to the appropriate documentation:

You can rank results by relevance.
You can apply weights to the different columns that go into your tsvector, for example to make the title more important in ranking than the description.
You can write custom triggers to handle more complicated tsvector constructions.
You can obtain highlighted results to indicate to the user which part of the document matched their query.
You can match on word prefixes, which is useful for implementing a search-box typeahead feature.
There’s all kinds of customization you can do with the dictionaries.

There are many more features I’ve omitted here, so don’t be afraid to read through that entire chapter of the documentation to learn more.

Conclusions

There are lots of reasons why we want to avoid substring-based or regex-based searching in our database. Perhaps most importantly, the results are often unintuitive to the user.

Postgres gives us an easy way to harness powerful fulltext searching using tsvectors and tsqueries. There’s absolutely no reason not to use them – you’ll avoid needing to configure a separate system for searching, you’ll easily be able to keep your keyword vectors and indices consistent with the latest database contents, and most importantly you’ll be giving your users the search results they expect.

Fun with integer division optimizations.

Tue, 21 Feb 2017 00:00:00 +0000

I recently stumbled across a post about some crazy optimization that clang does to divisions by a constant. If you aren’t interested in reading it yourself, the summary is as follows:

Arbitrary integer division is slow.
Division by powers of 2 is fast.
Given a divisor $$n$$, the compiler finds some $$a, b$$ such that $$a/2^b$$ approximates $$1/n$$.
This approximation gives exact results for any 32-bit integer.

I was interested in seeing just how much faster this seemingly-magic optimization was than the regular div instruction, so I set up a simple test framework:

#include <stdio.h>
#include <sys/time.h>

extern unsigned udiv19(unsigned arg);

int main(int argc, char **argv) {

    struct timeval start, finish;
    
    unsigned long long accum = 0;
    int j;

    for(j = 0; j < 1000; j++){
        gettimeofday(&amp;start, NULL);

        int i;
        for(i = 0; i < 1000000; i++){
            udiv19(1000);
        }
        
        gettimeofday(&amp;finish, NULL);

        accum += ((1000000l * finish.tv_sec + finish.tv_usec) - (1000000l * start.tv_sec + start.tv_usec));
    }
    
    printf("%f microseconds per 1M operations\n", (float) accum / 1000);

    return 0;

}

If you don’t feel like reading C, the algorithm is basically as follows:

Repeat 1,000 times:
- Start a stopwatch.
- Repeat 1,000,000 times:
  - Call the assembly function.
- Stop the stopwatch and add the time elapsed to the total.
Spit out the total time elapsed, divided by 1,000. This gives the average time for 1,000,000 calls to the assembly function.

I took the assembly from the other blog post and marked it up to be able to compile it into this scaffolding:

.intel_syntax noprefix
.text
.global udiv19
udiv19:
    mov     ecx, edi
    mov     eax, 2938661835
    imul    rax, rcx
    shr     rax, 32
    sub     edi, eax
    shr     edi
    add     eax, edi
    shr     eax, 4
    ret

I also wrote a naive version using the div instruction (this took a bit of googling to refresh my knowledge of the x86 ABI and instruction set…):

.intel_syntax noprefix
.text
.global udiv19
udiv19:
    mov     edx, 0
    mov     ecx, 19
    mov     eax, edi
    div     ecx
    ret

To compile:

$ gcc -o optimized main.c optimized.s 
$ gcc -o simple main.c simple.s

The results:

$ ./optimized 
1861.276001 microseconds per 1M operations
$ ./simple 
2783.295898 microseconds per 1M operations

We can clearly see a ~1.5x speedup (at least on my i5-6400). However, there’s a caveat: we’re only really testing the latency of the implementation, not its throughput. Modern processors have sophisticated ways to increase performance by pipelining instructions and executing them out of order. In a real application, this would drastically reduce the impact of the div instruction’s latency. However, it’s still interesting to see this optimization technique and prove that it theoretically could be beneficial, maybe, in some particular instances.

The problem with Python's datetime class.

Sat, 14 Jan 2017 00:00:00 +0000

This might sound like a strong opinion, but I’m just going to put it out there: Python should make tzinfo mandatory on all datetime objects.

To be fair, that’s just an overzealous suggestion prompted by my frustration after spending two full days debugging timestamp misbehaviors. There are plenty of practical reasons to keep timezone-agnostic datetimes around. Some projects will never need timestamp localization, and requiring them to use tzinfo everywhere will only needlessly complicate things. However, if you think you might ever need to deal with timezones in your application, then you must plan to deal with them from the start. My real proposition is that a team should assess its needs and set internal standards regarding the use of timestamps before beginning a project. That’s more reasonable, I think.

The problem.

If you’re handling timestamps in Python, chances are you are using its standard datetime class. The datetime honestly has a pretty great feature set: it lets you do arithmetic with dates, stringify dates, etc.; pretty much anything you need to do with a date, datetime will do for you. However, lots of problems arise when you use “naive” datetime objects, i.e., datetimes without any timezone awareness.

Python 2.x had a similar problem differentiating between different types of strings. It’s a long story, but essentially whether a string contained binary or text, it was still a string. People who knew what they were doing with strings didn’t have a problem, but it was far from idiot-proof. In fact, you didn’t really need to be an idiot to fall into the trap – just naive. This caused lots of problems, so eventually Python 3.x decided to make str and bytes into totally different things.

Naivety is also detrimental in the use of the datetime. The only place where it works as intended, without a hassle, is in an application where you never have to do any kind of localization or timezone conversion. Once you start trying to convert naive datetimes between timezones, you’ll find that you’ve been shot in the foot. My personal opinion is that footguns should not exist, or at least not in the standard libraries of high-level languages like Python.

I wish I could seriously propose that Python eliminate the naive datetime, but this would only cause problems. Naive datetimes are great, since they don’t ever require you to look at a timezone database (tzdb). Once you start dealing with timezones, you have to worry about the tzdb being up to date. If you don’t have complete control over the environment your code is running in, then you can expect inconsistent behavior between users. Whether this is a problem depends on the nature of your project, and I’m not about to enumerate all the possibilities — you can weigh the consequences yourself.

In short, I propose that anyone starting a new project should decide – at its very beginning – what to do with timestamps. In most cases, I think that naive datetimes should be avoided altogether — explicit timezone information (tzinfo) should be included absolutely anywhere datetimes are used. You should use naive datetimes only if you will never need to convert between timezones, you can’t trust users to have an up-to-date tzdb, and having inconsistent tzdbs between users would likely create other problems.

Unfortunately, I didn’t have the foresight to disallow naive datetimes in my project at its inception; therefore, I ran into a problem two years down the road at which point I had to do a lot of refactoring. The remainder of this article details the problems I encountered and the subsequent process of eliminating all naive datetimes from my codebase.

The dilemma.

When I first started using datetimes I didn’t know any better. I simply called datetime.now() whenever I needed a timestamp. At that time (no pun intended), my app was only displaying times for a single timezone. Eventually, I realized that I should be converting timestamps to users’ local timezones, and my naivety came back to bite me in the ass.

If you didn’t know already, datetime.now() gives you the current time in your local timezone. However, it does not have this timezone information attached by default: it gives you a naive datetime object.

I tried to convert one of these naive datetimes using the pytz library (which handles timezone magic):

>>> import pytz
>>> from datetime import datetime
>>> now = datetime.now()
>>> now
datetime.datetime(2017, 1, 14, 15, 15, 11, 475618)
>>> pytz.timezone("America/New_York").localize(now)
datetime.datetime(2017, 1, 14, 15, 15, 11, 475618, tzinfo=<DstTzInfo 'America/New_York' EST-1 day, 19:00:00 STD>)
>>> pytz.timezone("Australia/Sydney").localize(now)
datetime.datetime(2017, 1, 14, 15, 15, 11, 475618, tzinfo=<DstTzInfo 'Australia/Sydney' AEDT+11:00:00 DST>)

Note that my local timezone is MST; however, the datetime has no idea about this and therefore doesn’t actually do any conversion when I ask for another timezone. All of the datetimes it returned are the same, except for their attached tzinfos.

My first idea was to inject my local timezone into all the naive datetime objects:

>>> now = datetime.now(pytz.timezone("America/Denver"))
>>> now
datetime.datetime(2017, 1, 14, 15, 20, 8, 410761, tzinfo=<DstTzInfo 'America/Denver' MST-1 day, 17:00:00 STD>)
>>> pytz.timezone("America/New_York").normalize(now)
datetime.datetime(2017, 1, 14, 17, 20, 8, 410761, tzinfo=<DstTzInfo 'America/New_York' EST-1 day, 19:00:00 STD>)
>>> pytz.timezone("Australia/Sydney").normalize(now)
datetime.datetime(2017, 1, 15, 9, 20, 8, 410761, tzinfo=<DstTzInfo 'Australia/Sydney' AEDT+11:00:00 DST>)

Now the conversion works. However, there are also lots of places in my codebase where I’m accepting or returning Unix timestamps. If you don’t know, Unix timestamps are always UTC. If you don’t ask otherwise, datetime will convert them into local time for you, again without a tzinfo:

>>> import time
>>> unixtime = time.time()
>>> unixtime
1484432537.234377
>>> datetime.fromtimestamp(unixtime)
datetime.datetime(2017, 1, 14, 15, 22, 17, 234377)

This isn’t so bad, we can fix it mostly the same way as the datetime.now():

>>> datetime.fromtimestamp(unixtime, pytz.timezone("America/Denver"))
datetime.datetime(2017, 1, 14, 15, 22, 17, 234377, tzinfo=<DstTzInfo 'America/Denver' MST-1 day, 17:00:00 STD>)

You can even convert it to another timezone:

>>> datetime.fromtimestamp(unixtime, pytz.timezone("America/New_York"))
datetime.datetime(2017, 1, 14, 17, 22, 17, 234377, tzinfo=<DstTzInfo 'America/New_York' EST-1 day, 19:00:00 STD>)

But what if we want to convert a localized datetime into a Unix timestamp? If you’re familiar with the C strftime API, you’ll be tempted to use strftime("%s"):

>>> datetime.now().strftime("%s")
'1484432862'

That time, we got the correct result. But watch this:

>>> t = time.time()
>>> datetime.fromtimestamp(time.time(), pytz.timezone("America/Denver")).strftime("%s")
'1484433034'
>>> datetime.fromtimestamp(time.time(), pytz.timezone("America/New_York")).strftime("%s")
'1484440239'

What’s going on here? We created a single Unix timestamp (t), and converted it to two separate datetimes in two different timezones. We already know that conversion from Unix time into any timezone works correctly. We should have gotten the same result when we converted back. However, it turns out that you can only convert a datetime to a Unix timestamp if it is in your local timezone.

Actually, strftime("%s") is unsupported in Python. It ends up just stripping the tzinfo, thereby creating a naive timestamp in an arbitrary timezone, and calling the C strftime which assumes it’s being given a local timestamp. Obviously this doesn’t work.

Now how do you create a Unix timestamp the correct way? It’s ugly:

>>> t = datetime.now(pytz.timezone("America/Denver"))
>>> (t - datetime(1970, 1, 1, tzinfo=pytz.UTC)).total_seconds()
1484433334.448718

In short, you need to take a timezone-aware datetime, subtract the Unix epoch from it (thereby obtaining a timedelta), and convert it to seconds. Luckily for us, any arithmetic done with timezone-aware datetimes is automatically converted to UTC.

Fortunately, it fails if you pass it a naive datetime:

>>> t = datetime.now()
>>> (t - datetime(1970, 1, 1, tzinfo=pytz.UTC)).total_seconds()
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
TypeError: can't subtract offset-naive and offset-aware datetimes

Unfortunately, I’m sure a lot of beginners are still going to get screwed, since the most popular StackOverflow answers for this situation give you incorrect solutions like the following:

>>> t = datetime.now()
>>> (t - datetime(1970, 1, 1)).total_seconds()
1484408399.491469

It doesn’t fail, since both timestamps are naive. However, the result is wrong: since I used my local time, the result is the number of seconds since 1970-1-1 in my timezone, rather than in UTC.

The solution.

Upon discovering how difficult it is to do anything nontrivial with timestamps correctly, I decided to eliminate naive datetimes from my codebase altogether and standardize an API for doing common tasks with timezone-aware datetimes. This would help prevent other contributors to my project from shooting themselves in the foot (and by extension, shooting me).

The timehelper class I created is meant to be used any time you want to:

Get the current time,
Localize and format a timestamp,
Parse a Unix timestamp, or
Create a Unix timestamp.

Any use of the builtin datetime functions to do these things will now result in a failed code review, because they’re all nearly impossible to get right.

The timehelper itself is very simple:

import pytz, psycopg2
from datetime import datetime

class timehelper(object):
 
  @staticmethod
  def localize_and_format(tz, fmt, dt):
 
    # disallow naive datetimes
    if dt.tzinfo is None:
      raise ValueError("Passed datetime object has no tzinfo")
 
    # workaround for psycopg2 tzinfo
    if isinstance(dt.tzinfo, psycopg2.tz.FixedOffsetTimezone):
      dt.tzinfo._utcoffset = dt.tzinfo._offset
 
    return pytz.timezone(tz).normalize(dt).strftime(fmt)
 
  @staticmethod
  def now():
    return datetime.utcnow().replace(tzinfo=pytz.UTC)
 
  @staticmethod
  def to_posix(dt):
    return (dt - datetime(1970, 1, 1, tzinfo=pytz.UTC)).total_seconds()
 
  @staticmethod
  def from_posix(p):
    return datetime.fromtimestamp(p, pytz.UTC)

Its usage is simple, too:

Instead of calling datetime.now(), just call timehelper.now(). You’ll automatically be given a timezone-aware UTC datetime. The goal of this is to use UTC everywhere within the codebase.
To convert from a Unix timestamp to a UTC datetime, use timehelper.from_posix().
To convert from a datetime to a Unix timestamp, use timehelper.to_posix().
To localize a timestamp to a timezone and format it at the same time, use timehelper.localize_and_format(). I decided to always localize and format together in order to help enforce the goal of using UTC everywhere.

You might notice that there’s some special magic in the localize_and_format() method for dealing with tzinfo objects created by psycopg2. For some reason, its API has a slight mismatch against that of pytz. If you aren’t using psycopg2, you can strip out that if statement. But if you are, make sure all the timestamp-containing columns in PostgreSQL are declared as timestamp with time zone, rather than simply timestamp. This is another footgun; traditionally, Postgres used timezones implicitly, but this was reverted in order to comply with SQL standards.

The conclusion.

It took me several hours of research to figure out how to properly deal with timestamps in Python. Its datetime API is full of gotchas, and a naive developer can easily succumb to its apathy. It turns out that I had many subtle bugs in my codebase before I revisited all code pertaining to timestamps.

As it’s unlikely that naive datetimes will ever actually be removed from Python, I recommend that everyone create standards for datetime manipulation within their projects. Doing so may prevent tricky bugs and large rewrites later on.

If you happen to stumble upon this article in your own search for datetime incantations, feel free to use my above timehelper class. Consider it public domain.

Using bcache to back a SSD with a HDD on Ubuntu.

Fri, 06 Jan 2017 00:00:00 +0000

Recently, another student asked me to set up a PostgreSQL instance that they could use for some data mining. I initially put the instance on a HDD, but the dataset was quite large and the import was incredibly slow. I installed the only SSD I had available (120 GB), and it sped up the import for the first few tables. However, this turned out to not be enough space.

I did not want to move the database permanently back to the HDD, as this would mean slow I/O. I also was not about to go buy another SSD. I had heard of bcache, a Linux kernel module that lets a SSD act as a cache for a larger HDD. This seemed like the most appropriate solution – most of the data would fit in the SSD, but the backing HDD would be necessary for the rest of it. This article explains how to set up a bcache instance in this scenario. This tutorial is written for Ubuntu Desktop 16.04.1 (Xenial), but it likely applies to more recent versions as well as Ubuntu Server.

Preparation

If you have any existing data on the SSD or HDD, back it up elsewhere. Remove any associated mounts from /etc/fstab.

If you don’t already have it installed, you need to sudo apt-get install bcache-tools.

Partitions

On my machine, the HDD is /dev/sdb and the SSD is /dev/sdc. With bcache, you can either use entire disks or individual partitions. In my case, I’m using just one HDD partition, /dev/sdb5, but allowing the entire SSD to be used. Note that the backing HDD or partition has to be at least as large as the caching SSD or partition.

Surely your setup is different, so replace /dev/sdb5 with your HDD partition, and /dev/sdc with your caching SSD or partition.

I gave 250 GiB to /dev/sdb5; no partitioning is necessary on /dev/sdc if you are using the entire drive for caching.

You will need to remove any existing filesystem on both devices/partitions:

$ sudo wipefs -a /dev/sdc
$ sudo wipefs -a /dev/sdb5

This is necessary because bcache will refuse to instantiate if it looks like a filesystem already exists on the device.

Creating the bcache

A bcache instance looks and acts like a regular block device; instead of being named /dev/sdXX like a disk partition, it will be named /dev/bcacheX. The bcache kernel module will handle the underlying hardware and magic of the bcache device, we just have to set it up once.

We will be using “writeback” cache mode to enhance write performance; note that this is less safe than the default “writethrough” mode. If you’re worried about this, omit the --writeback flag. We will also enable the TRIM functionality of the SSD, to further enhance long-term write performance. If your SSD does not support TRIM, omit the --discard flag.

The bcache device can be optimized for your disk sector size and your SSD erase block size. In this case, my HDD has 4 KB sectors. I was unable to find the erase block size of my SSD, so I am using the default; however, if you know it you can append, for example, --bucket 2M, if your erase block size is 2 MB. Similarly, you should change the HDD sector size in this command if you know it, or remove the --block 4k flag if you don’t.

$ sudo make-bcache -C /dev/sdc -B /dev/sdb5 --block 4k --discard --writeback

Now you should see that the device /dev/bcache0 has been created.

Create and mount the filesystem

Now, we need to format our new bcache device. I’ll be using ext4.

$ sudo mkfs.ext4 /dev/bcache0

Now add the appropriate fstab line. I’ll be mounting the bcache device on /home/postgres since that’s where my PostgreSQL installation previously lived. Another good place for general use would be, for example, /media/bcache.

First, you will need to create an empty directory for the mountpoint:

$ sudo mkdir -p /home/postgres

Then, open /etc/fstab in your favorite editor (as root, of course) and append the corresponding line (altering the mount options as necessary):

/dev/bcache0 /home/postgres ext4 defaults,noatime,rw 0 0

Now, we can test that we can mount the new device:

$ sudo mount /home/postgres

The device and mount should both persist through reboot. You may copy any data back to the bcache device at this time.

Acknowledgments

This article was adapted from the following resources:

Parallelizing single-threaded batch jobs using Python's multiprocessing library.

Fri, 23 Dec 2016 00:00:00 +0000

Suppose you have to run some program with 100 different sets of parameters. You might automate this job using a bash script like this:

ARGS=("-foo 123" "-bar 456" "-baz 789")
for a in "${ARGS[@]}"; do
  my-program $a
done

The problem with this type of construction in bash is that only one process will run at a time. If your program isn’t already parallel, you can speed up execution by running multiple jobs at a time. This isn’t easy in bash, but fortunately Python’s multiprocessing library makes it quite simple.

One of the most powerful features in multiprocessing is the Pool. You specify the number of concurrent processes you want, a function representing the entry point of the process, and a list of inputs you need evaluated. The inputs are then mapped onto the processes in the Pool, one batch at a time.

You can combine this feature with a subprocess call to invoke an external program. For example:

import subprocess, multiprocessing, functools

ARGS = ["-foo 123", "-bar 456", "-baz 789"]
NUM_CORES = 4

shell = functools.partial(subprocess.call, shell=True)
pool = multiprocessing.Pool(NUM_CORES)

pool.map(shell, ["my-program %s" % a for a in ARGS])

To break it down, we have:

Translated the ARGS array from the bash script to Python syntax,
Used functools.partial to give us a helper function that invokes subprocess.call(..., shell=True),
Created a pool of NUM_CORES processes,
Used list comprehension to prepend the program name to each element in the ARGS list,
And finally mapped the resulting list onto the process pool.

The result is that your program will be executed with each specified set of arguments, parallelized over NUM_CORES processes. There’s only two more lines of code than the bash script, but the performance benefit can be manyfold.

The fruits of some recent Arduino mischief.

Wed, 21 Dec 2016 00:00:00 +0000

I recently consulted on a project involving embedded devices. Like most early-stage embedded endeavors, it currently consists of an Arduino and a bunch of off-the-shelf peripherals. During the project, I developed two small libraries (unrelated to the main focus of the project) which I’m open-sourcing today.

Library 1: A generic median filter implementation.

The first library I’m releasing is a simple, generic median filter. Median filters are useful for smoothing noise out of data, but unfortunately I couldn’t find a good existing library and was forced to write my own. Because I had to apply median filters to several types of data, I implemented it as a template class.

Note that while I developed the library for the Arduino use case, it is actually just plain ol’ C++ and will work literally anywhere.

Github link: https://github.com/tmick0/generic_median

Library 2: A simple driver for a poorly-documented Chinese RFID module.

Next up, we have a barebones driver for DFRobot’s ID01 UHF RFID reader. The sample code released by the vendor is a joke, and the manual is even worse. To put the module to use, I had to reverse-engineer the frame format for tag IDs. I don’t think anyone else should ever have to go through this, so I’m publishing the library here.

Github link: https://github.com/tmick0/dfrobot_rfid

Refer to the READMEs on the Github repos for more information about both libraries.

Optimizing MySQL and Apache for a low-memory VPS.

Mon, 17 Oct 2016 00:00:00 +0000

Diagnosing the problem.

My last post had a plug about the migration of our Wordpress instance to a new server. However, it didn’t go completely smoothly. The site had gone down a few times in the first day after the migration, with Wordpress throwing “Error establishing a database connection.” Sure enough, MySQL had gone down. A simple restart of MySQL would bring the site back up, but what caused the crash in the first place?

A peek into /var/log/mysql/error.log yielded this:

2016-10-12T21:20:50.588667Z 0 [ERROR] InnoDB: mmap(137428992 bytes) failed; errno 12
2016-10-12T21:20:50.588702Z 0 [ERROR] InnoDB: Cannot allocate memory for the buffer pool
2016-10-12T21:20:50.588728Z 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
2016-10-12T21:20:50.588749Z 0 [ERROR] Plugin 'InnoDB' init function returned error.
2016-10-12T21:20:50.588758Z 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2016-10-12T21:20:50.588767Z 0 [ERROR] Failed to initialize plugins.
2016-10-12T21:20:50.588772Z 0 [ERROR] Aborting

So it looks like an out-of-memory error. This VPS only has 512 MB of RAM, so I wouldn’t be surprised. Clearly, some tuning would be necessary. First, we’ll reduce the size of MySQL’s buffer pool, then shrink Apache’s worker pool, and finally add a swapfile just in case memory pressure remains a problem.

Optimizing MySQL.

The error that we saw was for the allocation of a buffer pool for InnoDB, one of MySQL’s storage engines. We can see from the log that it’s trying to allocate somewhere around 128 MB using mmap. This corresponds to the default value of the innodb_buffer_pool_size configuration option. Let’s go ahead and trim this down to about 20 MB – it’ll reduce MySQL’s performance, but we don’t have much of a choice on a machine this small.

On Ubuntu, I put this option in /etc/mysql/mysqld.conf.d/mysqld.cnf:

innodb_buffer_pool_size = 20M

Issue sudo service mysql restart, and rejoice as MySQL no longer uses 25% of your RAM.

Optimizing Apache.

Most of Apache’s memory usage comes from the fact that it preemptively forks worker processes in order to handle requests with low latency. This is handled by the mpm_prefork module, and if enabled, its config can be found in /etc/apache2/mods-enabled/mpm_prefork.conf (on Ubuntu, at least).

By default, Apache will create 5 processes at startup, keep a minimum of 5 idle processes at all times, allow up to 10 idle processes before they’re reaped, and spawn up to 256 processes at a time under load. Let’s reduce these to something more sane given the constraints of our system:

<IfModule mpm_prefork_module>
 StartServers 3
 MinSpareServers 3
 MaxSpareServers 5
 MaxRequestWorkers 25
 MaxConnectionsPerChild 0
</IfModule>

Now, sudo service apache2 restart` and you’re done.

Creating a swapfile.

Most VPSes don’t give you a swap partition by default, like you would probably create on a dedicated server or your desktop. We can create one using a file on an existing filesystem, in order to make sure there’s extra virtual memory available in case our tuning doesn’t handle everything perfectly.

First, let’s pre-allocate space in the filesystem. We can do this using the fallocate command. I made a 2 GB swapfile:

$ sudo fallocate -l 2G /swapfile

Now, give it some sane permissions:

$ sudo chmod 600 /swapfile

Next, format the file so it looks like a swap filesystem:

$ sudo mkswap /swapfile

And finally, tell the OS to use it as swap space:

$ sudo swapon /swapfile

Now, we’ve got swap for the time being, but it won’t persist when we reboot the system. To make it persist, simply add a line to /etc/fstab:

/swapfile none swap sw 0 0

Congratulations, you’re now the proud owner of some swap space.

Hopefully this post will help anyone suffering stability problems with MySQL and Apache on a small VPS. I’ve adapted the instructions here from several sources, most notably this StackOverflow post, and this article from DigitalOcean.

Information-centric networking for laymen.

Sat, 08 Oct 2016 00:00:00 +0000

The design of the current Internet is based on the concept of connections between “hosts”, or individual computers. For example, when you visit a website, your computer (a host) always connects to a particular server (another host) and retrieves content through a session-oriented pipe. However, the amount of content hosted on the Internet and the number of connected devices are both growing. This is a crisis scenario for the current Internet architecture – it won’t scale.

Several proposals for Next-Generation Network (NGN) architectures have been proposed in recent years, aimed at better handling immense amounts of traffic and orders of magnitude more pairwise connections. Information-Centric Networking (ICN) is one NGN paradigm which eschews the concept of connections entirely, removing the host as the basic “unit” of the network and replacing it with content objects.

In other words, the defining feature of an ICN is that instead of asking the network to connect you to a particular server (where you may hope to find a content you desire), you instead ask the network for the content itself.

Several distinct ICN architectures have been proposed, however for the remainder of this article I will focus on Named-Data Networking (NDN) and Content-Centric Networking (CCN), the two most popular designs in recent literature. NDN and CCN both share the core concept of consumer-driven communication, wherein a consumer (or client) issues an Interest packet (a request) for a content object and hopes to receive a Data packet in return. Interest and Data packets are both identified by a Name, which is in essence a immutable, human-readable name for a particular content object.

Whereas current Internet routers rely on only one lookup table (i.e., a forwarding table) in order to route packets toward a destination, NDN/CCN routers use three main data structures in order to locate content objects. A Pending Interest Table (PIT) keeps track of outstanding requests, a Content Store (CS) caches content objects, and a Forwarding Information Base (FIB) stores default forwarding information.

When a router receives an Interest, it will first check its CS to see if it can serve the content immediately from its cache. If it is not found there, then the router checks its PIT to see if there is an outstanding request for the same content; if there is then the request does not need to be forwarded again, since the data for the previous request can satisfy both that request and the new one. Finally, if an existing Interest is not found, the router checks the FIB for a route toward the appropriate content provider; once a route has been identified, the Interest is forwarded and the PIT is updated.

Though ICN requires routers to store more state and make more complicated forwarding decisions, it is still expected to reduce the overall network load by virtue of Interest aggregation and content caching. Caching in particular also benefits the end-user, since the availability of content nearby reduces download time. Since content downloads are independent of any particular connection, ICN also allows multi-RAT (Radio Access Technology) communication to be exploited by mobile devices, further improving the user’s QoE (Quality-of-Experience).

Last week, I presented a collaborative caching scheme for NDN at ACM ICN 2016, the leading conference in the ICN domain (slides, paper) which is able to satisfy up to 20% of Interests without leaving the home ISP’s network. Additionally, we published an article in IEEE Communications Magazine about the advantages of ICN for mobile networks (paper). These works, as well as those of the larger ICN community, have the potential to influence the acceptance of ICN as the foundation of the future Internet. Only with continued research will we find a holistic solution for scalability in the face of billions of connected devices and billions of terabytes of traffic.

Why are tuples greater than lists?

Mon, 03 Oct 2016 00:00:00 +0000

I pose this question in quite a literal sense. Why does Python 2.7 have this behavior?

>>> (1,) > [2]
True

No matter what the tuple, and no matter what the list, the tuple will always be considered greater. On the other hand, Python 3 gives us an error, which actually makes a bit more sense:

>>> (1,) > [2]
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
TypeError: unorderable types: tuple() > list()

The following post is a journey into some CPython internals, with a goal of finding out why 2.7 gives us such a weird comparison result.

Those of you who have implemented nontrivial classes in Python are probably aware of the two different comparison interfaces in the data model: rich comparison, and simple comparison. Rich comparison is implemented by defining the functions __lt__, __le__, __eq__, __ne__, __gt__, and __ge__. That is, there is one function for each possible comparison operator. Simple comparison uses only one function, __cmp__, which has a similar interface to C’s strcmp.

Any comparison operation you write in Python compiles down to the COMPARE_OP bytecode, which itself is handled by a function called cmp_outcome. For the types of comparisons we’re concerned with today (i.e., inequalities rather than exact comparisons), this function will end up calling PyObject_RichCompare, the user-facing comparison function in the C API.

At this point, the runtime will attempt to use the rich comparison interface, if possible. Assuming that neither operand’s class is a subclass of the other, the first class’s comparison functions will be checked first; the second class would be checked if the first class does not yield a useful result. In the case of tuple and list, both calls return NotImplemented.

Having failed to use the rich comparison interface, we now try to call __cmp__. The actual semantics here are quite complicated, but in the case at hand, all attempts fail. One penultimate effort before hitting the last-ditch “default” compare function is to convert both operands to numeric types (which fails here, of course).

CPython’s default_3way_compare is somewhat of a collection of terrible ideas. If the two objects are of the same type, it will try to compare them by address and return that result. Otherwise, we then check if either value is None, which would be considered smaller than anything else. The second-to-last option, which we will actually end up using in the case of tuple vs. list, is to compare the names of the two classes (essentially returning strcmp(v->ob_type->tp_name, w->ob_type->tp_name)). Note, however, that any numeric type would have its type name switched to the empty string here, so a number ends up being considered smaller than anything non-numeric. If we end up in a case where both type names are the same (either they actually have the same name, or they are incomparable numeric types), then we get a final result by comparing pointers to the type definitions.

To validate our findings, consider the following:

>>> tuple() > list()
True
>>> class abc (tuple):
... pass
... 
>>> abc() > list()
False
>>> class xyz (tuple):
... pass
... 
>>> xyz() > list()
True

The only difference between classes abc and xyz (and tuple, even) are their names, however we can see that the instances are compared differently. Now, we have certainly found quite the footgun here, so it’s fortunate that Python 3 has a more sane comparison operation.

Quick postfix & dovecot config with virtual hosts (Ubuntu 16.04)

Tue, 23 Aug 2016 00:00:00 +0000

This morning, I received an email from my VPS host notifying me that they will no longer accept PayPal. Instead, my only payment option would be Bitcoin. Not willing to go through this trouble, I decided to migrate from this host (which I had been using for my personal servers for about five years now) to DigitalOcean (which fortunately accepts normal forms of payment).

Part of my server migration was to move email for two of my domains: le1.ca and lo.calho.st. Setting up a new mailserver is a notoriously arduous task, so I’m documenting the process in this post – mostly for my future reference, but also to benefit anyone who might stumble upon my blog in their own confusion.

Since I’m serving mail for two domains, I will be using a simple “virtual hosts” configuration. I’ll talk about the process in four parts: local setup, postfix, dovecot, and DNS configuration.

Initial Setup

We must prepare a few things before we’re ready to configure the actual mail services. The first thing I’d like to do is install a SSL certificate, so that we can access the mail server securely. This is easy with Let’s Encrypt:

$ sudo apt-get install letsencrypt # (if you don't have it installed already)
$ letsencrypt certonly --standalone -d mail.le1.ca -d mail.lo.calho.st

I’m using the standalone option here, because I do not have an HTTP daemon installed on this server. If you’re also serving HTTP from your mail machine, you will need to follow the appropriate instructions for your configuration.

Now that we have an SSL cert, we can set up the directory structure for our virtual mailboxes. I’m going to make the root for this environment /var/spool/mail, and in that directory, I’m going to create one folder for each domain I’m serving:

$ mkdir -p /var/spool/mail/le1.ca
$ mkdir -p /var/spool/mail/lo.calho.st

The next step is to configure passwords for each of your users. I’m doing this the easy way, since I’m the only person using the mail server. If you need users to be able to change their own passwords, this is going to be more difficult; you’ll need to find an alternate authentication configuration.

You will need to calculate a password hash for each user you want to serve:

$ doveadm pw -u foo@le1.ca

Each time, it should spit out a hash that looks like this:

{CRAM-MD5}a7cb902940b3f6662c48ace840a4e3e410241e875d720cb45b2d95a3e1ddfc8b

For each of your domains, create a shadow file in the corresponding /var/spool/mail/* directory. For example, I will create /var/spool/mail/le1.ca/shadow. Add a username:password line for each virtual user you want to create.

foo@le1.ca:{CRAM-MD5}a7cb902940b3f6662c48ace840a4e3e410241e875d720cb45b2d95a3e1ddfc8

Next, let’s create a “virtual mail” user and configure him as the owner of these mail directories.

$ sudo adduser vmail
$ sudo chown -R vmail:vmail /var/spool/mail/

Find out the UID and GID of this new user and remember it for later. An easy way to do this is grep vmail /etc/passwd. The UID and GID are the third and fourth columns, respectively.

That’s it for initial setup. Now let’s deal with postfix.

Configuring Postfix

Postfix is responsible for delivering received mail to the correct locations on our server, as well as sending mail on our behalf.

There’s some initial configuration we can do interactively before we start poking around in config files. If you haven’t installed postfix yet, this configuration will start automatically when you do apt-get install postfix. If you do already have it installed, then you can run dpkg-reconfigure postfix to get there.

Here are the responses you should enter to this wizard:

Type of mail configuration: Internet Site
System mail name: Use your “primary” mail domain here. I used “le1.ca”
Root and postmaster recipient: Enter your primary email address here. It will be used as the destination for administrative mail.
Destinations to accept mail for: Enter any aliases of your machine here, but don’t enter the domains you’re going to use in your virtual mailboxes. A good list is something like my-hostname, localhost.localdomain, localhost
Everything else: Leave it at the default value

The main configuration file for postfix is /etc/postfix/main.cf. Let’s break down the config into three sections: SSL, virtual mailboxes, and SASL. I’m only including lines that need to be modified here. Everything else should stay at its default value.

First, SSL-related configs:

smtpd_tls_cert_file=/etc/letsencrypt/live/[your_mailserver_name]</strong>/fullchain.pem
smtpd_tls_key_file=/etc/letsencrypt/live/[your_mailserver_name]</strong>/privkey.pem
smtpd_use_tls=yes

Virtual mailbox configs:

virtual_mailbox_domains = /etc/postfix/virtual_domains
virtual_mailbox_maps = hash:/etc/postfix/virtual_mailbox
virtual_mailbox_base = /var/spool/mail
virtual_uid_maps = static:1001 # Replace with the UID of your vmail user
virtual_gid_maps = static:1001 # Replace with the GID of your vmail user

SASL configs (to pass authentication duties to Dovecot):

smtpd_sasl_auth_enable = yes
smtpd_sasl_type = dovecot
smtpd_sasl_path = private/auth
smtpd_sasl_security_options = noanonymous
smtpd_sasl_local_domain =
smtpd_relay_restrictions = permit_mynetworks permit_sasl_authenticated defer_unauth_destination
smtpd_recipient_restrictions = permit_sasl_authenticated, permit_mynetworks, reject_unauth_destination

Hopefully you noticed that the virtual mailbox configs refer to some new files. We’re going to have to create /etc/postfix/virtual_domains and /etc/postfix/virtual_mailbox in order to create the mappings for our domains.

The virtual_domains file is easy, just a list of the domains you’re serving. In my case:

le1.ca
lo.calho.st

The virtual_mailbox file is also pretty easy. It just maps email addresses to mailbox folders, relative to the virtual_mailbox_base directory. For example, if I want both foo@le1.ca and bar@le1.ca to go to the mailbox le1.ca/foobar, and baz@lo.calho.st to go to lo.calho.st/baz:

foo@le1.ca le1.ca/foobar/
bar@le1.ca le1.ca/foobar/
baz@lo.calho.st lo.calho.st/baz/

You can create as many aliases as you want here. Whatever is on the right will end up determining your login username (in this case, foobar@le1.ca and baz@lo.calho.st are my usernames), and the addresses on the left create aliases.

Postfix now requires us to hash this map file. Run the following command from within /etc/postfix:

$ postmap virtual_mailbox

This should create a file called virtual_mailbox.db.

The last part of the postfix config is to enable the “user-facing” SMTP port, 587. Just paste this block in master.cf somewhere:

submission inet n - y - - smtpd
-o smtpd_tls_security_level=encrypt
-o smtpd_sasl_auth_enable=yes
-o smtpd_sasl_type=dovecot
-o smtpd_sasl_path=private/auth
-o smtpd_sasl_security_options=noanonymous
-o smtpd_sasl_local_domain=$myhostname
-o smtpd_client_restrictions=permit_sasl_authenticated,reject
-o smtpd_sender_login_maps=hash:/etc/postfix/virtual_mailbox
-o smtpd_recipient_restrictions=reject_non_fqdn_recipient,reject_unknown_recipient_domain,permit_sasl_authenticated,reject

The next step is to configure dovecot.

Configuring Dovecot

Dovecot is much easier to configure than postfix. Create /etc/dovecot/dovecot.conf with the following content (delete it if it already exists):

protocols = imap
log_path = /var/log/dovecot.log

ssl = required
ssl_cert = </etc/letsencrypt/live/[your_mailserver_name]/fullchain.pem
ssl_key = </etc/letsencrypt/live/[your_mailserver_name]/privkey.pem
ssl_dh_parameters_length=2048
ssl_cipher_list = ALL:!LOW:!SSLv2:!EXP:!aNULL

disable_plaintext_auth = no
mail_location = maildir:/var/spool/mail/%d/%n

auth_verbose = yes
auth_mechanisms = plain login

passdb {
 driver = passwd-file
 args = /var/spool/mail/%d/shadow
}

userdb {
 driver = static
 args = uid=vmail gid=vmail home=/var/spool/mail/%d/%n
}

protocol lda {
 postmaster_address = root@le1.ca
}

service auth {
 unix_listener /var/spool/postfix/private/auth {
 mode = 0660
 user = postfix
 group = postfix
 }
}

Now you should be ready to restart postfix and dovecot:

$ sudo service dovecot restart
$ sudo service postfix restart

If everything went smoothly, we’re ready to configure DNS records.

DNS Records

For each of your domains, you will need three DNS records:

Either a CNAME or an A record pointing to the mailserver
An MX record designating the mailserver as the destination for the domain’s mail
A SPF record designating the server as a permitted origin for mail

The A record is easy. Just create an A record for “mail.yourdomain.com” pointing to the IP address of the server.

The MX record might be even easier: set the mailserver for “yourdomain.com” to “mail.yourdomain.com”

The SPF record is also very straightforward. Many people overlook its importance, but without it, mail from your domain is likely to go into recipients’ spam folders. There exist wizards to create custom SPF records, but I find the one below to be sufficient for most purposes. Just insert this as a TXT record for “yourdomain.com”:

"v=spf1 mx a ptr ~all"

And that’s it. Allow a few minutes for your DNS records to propagate, then test out your new mailserver.

An easy way to visualize git activity

Sat, 11 Jun 2016 00:00:00 +0000

Today, I wrote gitply – a fairly simple Python script for visualizing the weekly activity of each contributor to a git repository.

It started out as a run-once script to get some statistics for one of my projects, but I ended up improving it incrementally until it turned into something friendly enough for other people to use.

What it does:

You’ll get a graph like this for each contributor. What all do we see here?

Lines inserted (green bars, left axis, log scale)
Lines deleted (red bars, left axis, log scale)
Commits made (blue line, right axis, linear scale)

You also get a textual representation of the same data:

History for anon2
  2016, week  6:  2 commits, +366  -26  
  2016, week  7:  4 commits, +325  -5   
  2016, week  8:  2 commits, +224  -3   
  2016, week  9: 21 commits, +2617 -219 
  -- Gap of 2 weeks
  2016, week 12: 10 commits, +4066 -614 
  2016, week 13:  6 commits, +2480 -432 
  2016, week 14:  2 commits, +654  -490 
  -- Gap of 1 week
  2016, week 16:  2 commits, +661  -229 
  -- Gap of 2 weeks
  2016, week 19:  3 commits, +1508 -47  
  -- Gap of 2 weeks
  2016, week 22:  1 commits, +1490 -29  

History for anon1
  2016, week 12:  1 commits, +143  -89  
  2016, week 13:  2 commits, +58   -16  
  2016, week 14:  3 commits, +137  -17  
  2016, week 15:  2 commits, +403  -106 
  -- Gap of 2 weeks
  2016, week 18:  5 commits, +134  -38  
  -- Gap of 3 weeks
  2016, week 22:  1 commits, +13   -1   
  2016, week 23:  1 commits, +485  -1

If you just want to download it and use it already, check out the project on GitHub. For those who are interested, some implementation details follow.

How does it work?

The first thing gitply does is invoke the standard git client to retrieve a log:

$ git log --numstat --date=short --no-notes

Then, it analyzes the log line-by-line in real time (thanks, Python iterators) in order to get the data we need. Essentially, we create an iterator that yields tuples of (email, date, stats) for each commit, where stats is the pair (insertions, deletions).

Once we create this iterator over the commit history, we can attribute each commit to a user and ultimately fill in a dictionary that will help us print the analysis. We sum the insertions, deletions, and commits for each ISO week, then iterate through the weeks and print out the final results.

Now, if for some reason you need an iterator over the commits in a repository, you can import my iterate_commits function from this script and use it yourself.

What’s next?

Right now, gitply supports aggregating statistics from several repositories into one report. One planned feature is to use stacked bars and lines to show the amount of activity per-project, per-user, per-week. Until then, the data from multiple repositories will all be displayed together.

Adventures in image glitching

Thu, 09 Jun 2016 00:00:00 +0000

Databending is a type of glitch art wherein image files are intentionally corrupted in order to produce an aesthetic effect. Traditionally, these effects are produced by manually manipulating the compressed data in an image file. As a result, this is a trial-and-error process; often, edits will result in the file being completely corrupted and unopenable.

Someone recently asked me whether I knew why databending different types of image files produces different effects – and particularly, why PNG glitches are the most interesting. I didn’t know the answer, but the question inspired me to do a little research (mostly reading the Wikipedia articles about the compression techniques used in different image formats). I discovered that most compression techniques are not all that different. Most of them just employ some kind of run-length encoding or dictionary encoding, and then a prefix-free coding step. The subtle differences between the compression algorithms could not explain the wildly different effects we observed (except for in JPEGs, perhaps, since the compression is done in the frequency domain). However, PNG used a pre-filtering step which made it stand out.

PNG’s pre-filtering is basically a heuristic to try to make the image data more compressible. It tries to translate each pixel of the image into some delta based on pixels in the previous row or column. Since the core compression algorithms have no idea about the two-dimensional nature of an image, this two-dimensional filtering step actually makes a pretty big impact.

There are five main line filters defined by the PNG format. Most PNG encoders will try to optimize their choices of filters for each line in the image, in order to reduce the compressed size of the image. However, each filter also yields its own unique glitches.

Thus, an idea was spawned, and I wrote a Python script that glitches images not by manipulating their raw representations, but by applying one of the PNG filters and modifying the pixels in that representation.

PNG Filter Glitches

Since this article is about glitching – not image compression – we’re going to focus on the different visual effects we get from each filter. Usually, PNG codecs alternate between filters for each line, however for the sake of demonstration I configured my script to use one filter throughout the whole image.

All of the following glitches are made from a scaled-down version of this wallpaper from Wallhaven.

Two of the simplest PNG filters are the Up filter and the Sub filter. While Up creates a delta from the adjacent pixel in the previous row, Sub creates a delta from the previous pixel in the same row. The results of glitching on the filters are similar – Up gives us vertical lines, and Sub gives us horizontal lines.

Glitch result from the Up filter

Glitch result from the Sub filter

A more interesting one is the Average filter. This filter looks at the both the above-pixel and the left-pixel and takes their average, then computes a delta using that value. Glitching on the Average filter results in smears of altered color that slowly defuse away across the image.

Average

My favorite is actually the Paeth filter, which heuristically chooses either the above-pixel, left-pixel, or above-and-to-the-left pixel as the basis for the delta, depending on a function of all three. Glitches applied to this filter give us rectangular disturbances which slowly fade out, however stack together to create harsher effects.

Paeth

I mentioned that there are five filters, but so far I’ve only shown four. The reason for this is simple – the fifth filter is the Null filter, which has no effect whatsoever.

Mixing Filters

To simulate the mixed effects one can observe in a typical PNG glitch, I implemented a glitching procedure which uses a random filter on each line. The results are pretty intriguing:

Random line filters

In this example, I’ve only allowed a 5% chance for the filter to change from one line to the next; without this provision, the results are quite boring.

To make the glitches even cooler, I started stacking filters on top of each other. Here’s an example with an extra pass through the Average filter applied after the randomized filtering stage:

Randomized + Average

Glitch Glitches

Overall, my favorite filters were the ones that had bugs when I wrote them. My original implementation of the Paeth filter had an error due to the use of unsigned integers in the predictor (when really I needed signed ones). However, this version resulted in some cool effects that acted like a combination of the normal Paeth and Average behaviors:

Buggy Paeth filter

I also wrote a buggy Average filter that resulted in some bold vertical lines and some subtler horizontal artifacts:

Buggy Average filter

Try it Yourself

I’ve posted my glitch scripts on GitHub, so you can try them out too. The original version was pure Python and incredibly slow, but I’ve now converted the core functionality into a C module and it is several orders of magnitude faster. This is also my first foray into the intricacies of the Python C API, but that’s a topic for another post…

To conclude, have two more glitches achieved by mixing various filters:

What's inside a PEM file?

Thu, 28 Apr 2016 00:00:00 +0000

As you probably know, PEM is a base64-encoded format with human-readable headers, so you can kind of figure out what you’re looking at if you open it in a text editor.

For example, let’s look at an RSA public key:

We can see that we have a public key just from the header. But what’s all the base64 stuff? If it’s a public key, we know that there should be a modulus and an exponent hidden in there. And there’s also probably some kind of hint that this is an RSA key, as opposed to some other type of key.

That base-64 payload actually has the potential to contain a lot of information. It’s an ASN.1 (Abstract Syntax Notation One) data structure, encoded in a format called DER (Distinguished Encoding Rules), and then finally base64-encoded before the header and footer are attached.

ASN.1 is used for a lot of stuff besides keys and certificates; it is a generic file format that can be used to serialize any kind of hierarchical data. DER is just one of many encoding formats for an ASN.1 structure – e.g. there is an older format called BER (Basic Encoding Rules) and an XML-based format called XER (you can probably guess what that stands for).

But anyway, what is inside that public key? How can we find out?

Parsing PEM files

OpenSSL comes with a utility called asn1parse. It can understand plain DER or PEM-encoded ASN.1. Let’s feed our public key into it and take a look:

$ openssl asn1parse -in public.pem -i
    0:d=0  hl=4 l= 546 cons: SEQUENCE          
    4:d=1  hl=2 l=  13 cons:  SEQUENCE          
    6:d=2  hl=2 l=   9 prim:   OBJECT            :rsaEncryption
   17:d=2  hl=2 l=   0 prim:   NULL              
   19:d=1  hl=4 l= 527 prim:  BIT STRING

The column that contains “cons” and “prim” gives us information about the hierarchical structure of the data. “Cons” stands for a “constructed” field, i.e., a field that encapsulates other fields; on the other hand, “prim” means “primitive.”

The column after “cons” or “prim” tells us what type of data is in that field. The -i flag I’ve supplied causes that column to be indented according to how deep we are in the hierarchical structure. So overall, what are we looking at?

There is one root SEQUENCE object. That SEQUENCE contains another SEQUENCE and a BIT STRING. That internal SEQUENCE has an OBJECT and a NULL terminator. The OBJECT field is actually an Object Identifier – it contains some constant that tells us what kind of object we’re decoding. As we may have suspected, it tells us that we’re decoding an RSA key.

Here’s where stuff gets really interesting: That BIT STRING field actually contains more ASN.1 data. Let’s jump in and parse it:

$ openssl asn1parse -in public.pem -strparse 19 -i 
    0:d=0  hl=4 l= 522 cons: SEQUENCE          
    4:d=1  hl=4 l= 513 prim:  INTEGER           :E18160C0DAC490C7729E...
  521:d=1  hl=2 l=   3 prim:  INTEGER           :010001

(The -strparse 19 flag means “parse the data in the field located at offset 19 in the original structure.” If there was another BIT STRING inside this one, we could add another -strparse argument to recurse into it.)

What do we have here? A SEQUENCE containing two INTEGERs. It turns out that the first INTEGER is our modulus, and the second INTEGER is the exponent.

You can actually infer the effective length of the key from this information. The third column indicates the length of the modulus is 513 bytes; one of these bytes is just padding, so that means our key is 512 bytes (or 4096 bits) in strength.

Reverting to DER

As I explained earlier, PEM is just a wrapper around DER. So can we dump the raw DER and parse it that way? You sure can. You just cut out the PEM headers and base64-decode the payload, and what comes out is DER.

$ grep -v "PUBLIC KEY" public.pem | openssl base64 -d > public.der

Now we can inspect the raw DER:

$ hexdump public.der 
0000000 8230 2202 0d30 0906 862a 8648 0df7 0101
0000010 0501 0300 0282 000f 8230 0a02 8202 0102
0000020 e100 6081 dac0 90c4 72c7 3b9e b043 2acd
0000030 e16f 76de 294f 9ce2 7f92 a416 3761 40d2
0000040 32f1 1c1b 4067 1567 5d31 1299 d133 76e2
0000050 a85e ed9a 3935 c662 b612 6a61 9cae 884e
0000060 8a2a 0e55 d6ad d3be c1ca 44e3 9f79 da01
0000070 3650 c123 ae45 5296 8f8c a2d3 bcca c6fc
...

Or, we can use asn1parse if we actually want to understand it:

$ openssl asn1parse -in public.der -inform DER -i
    0:d=0  hl=4 l= 546 cons: SEQUENCE          
    4:d=1  hl=2 l=  13 cons:  SEQUENCE          
    6:d=2  hl=2 l=   9 prim:   OBJECT            :rsaEncryption
   17:d=2  hl=2 l=   0 prim:   NULL              
   19:d=1  hl=4 l= 527 prim:  BIT STRING   

$ openssl asn1parse -in public.der -inform DER -strparse 19 -i
    0:d=0  hl=4 l= 522 cons: SEQUENCE          
    4:d=1  hl=4 l= 513 prim:  INTEGER           :E18160C0DAC490C7729E...
  521:d=1  hl=2 l=   3 prim:  INTEGER           :010001

This is the exact same data we saw earlier, but there’s your proof that PEM really is just DER underneath a layer of base64.

My first adventure with Let's Encrypt on nginx, dovecot, and postfix

Mon, 28 Mar 2016 00:00:00 +0000

Let’s Encrypt is old news by now. It launched back in December, so it has been giving away free DV certificates for nearly four months now. Being a TA for a Computer Security course, it’s about time that I actually tried it out.

Prologue

Let’s Encrypt is a free certificate authority. They grant TLS certificates that you can use to secure your webserver. They are Domain Validated (DV) certificates, which means they will verify that you control the domain name you are trying to certify.

To test out Let’s Encrypt, I decided to deploy new certificates on some internal Zeall.us services (Gitlab and Roundcube). Previously, we were using self-signed certs, so we would have to approve a security exception on every visit.

Unfortunately, I chose the wrong day to test out Let’s Encrypt. For some reason, their DNS client decided to die at the very moment I tried to get a certificate. I wasn’t the only one having this problem, but Google hadn’t yet crawled the support site so I didn’t see these other reports until the problem had already subsided. Once it cleared up, everything else was a breeze.

Obtaining certificates

Obtaining certificates for our Gitlab and webmail instances was incredibly easy. I just cloned the Let’s Encrypt repository from Github and used their super-simple command-line utility. It doesn’t have built-in support for nginx (our webserver of choice), but you can still easily use the webroot plugin to validate your domain. You just tell it the directory from which your webserver serves, and it puts a file there which it uses for the validation. I ran it once for each of the services:

$ ./letsencrypt-auto certonly --email example@example.com --text --agree-tos --renew-by-default --webroot -w /opt/gitlab/embedded/service/gitlab-rails/public -d <our_gitlab_domain>
$ ./letsencrypt-auto certonly --email example@example.com --text --agree-tos --renew-by-default --webroot -w /srv/roundcube -d <our_webmail_domain>

It handles everything automatically, and sticks your new certificates and keys in their respective /etc/letsencrypt/live/<domain_name>/ directories.

Configuring nginx

Prior to setting up the new certificates in nginx, I generated strong Diffie-Hellman key exchange parameters, as suggested by this guide to nginx hardening:

$ sudo openssl dhparam -out /etc/ssl/certs/dhparam.pem 4096

Then I just put the following lines in each of my nginx site configs:

ssl on;
ssl_certificate /etc/letsencrypt/live/<domain>/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/<domain>/privkey.pem;
ssl_session_timeout 5m;
ssl_session_cache shared:SSL:10m;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers 'EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH';
ssl_prefer_server_ciphers on;
ssl_dhparam /etc/ssl/certs/dhparam.pem;

Follow up with a sudo service nginx reload, and you’re good to go. With this configuration, my domain got an A+ from Qualys’ SSL Labs Server Test.

Configuring Dovecot

To supply the new certificate to my Dovecot IMAP server, I set the following configs in /etc/dovecot/conf.d/10-ssl.conf:

ssl_cert = </etc/letsencrypt/live/<domain>/fullchain.pem
ssl_key = </etc/letsencrypt/live/<domain>/privkey.pem

This is the same certificate I used for webmail, since the domain is the same. I have not yet investigated hardening techniques for Dovecot, but at a glance the settings seem to be quite similar to nginx.

This configuration was followed up with a sudo service dovecot restart, and then everything worked.

Configuring Postfix

Finally, I supplied the same certificate to Postfix (SMTP) by putting the following in /etc/postfix/main.cf:

smtpd_tls_cert_file = /etc/letsencrypt/live/<domain>/fullchain.pem
smtpd_tls_key_file = /etc/letsencrypt/live/<domain>/privkey.pem

After a sudo service postfix restart, the certificates are live.

Auto-renew

Let’s Encrypt provides certificates with a term of only 90 days, so it is necessary to write a script to renew them automatically. I wrote the following script:

/opt/letsencrypt/letsencrypt-auto renew | tee -a ./lerenew.log
service nginx reload
service postfix reload
service dovecot reload

And a corresponding crontab:

0 0 * * 0 ~/lerenew.sh

Now once a week, any eligible certificates will be renewed and the services which use them will be reloaded.

Conclusion

It has never been this easy to deploy TLS certificates. I highly recommend that anyone who is currently using a self-signed certificate switch to Let’s Encrypt. If you’re currently paying for a DV certificate, consider switching to Let’s Encrypt to save some money – but keep compatibility issues in mind.

Demonstrating the double-DES meet-in-the-middle attack

Thu, 04 Feb 2016 00:00:00 +0000

Introduction

DES (Data Encryption Standard) is an old-school block cipher which has been around since the 1970s. It only uses a 56-bit key, which is undeniably too short for use in the modern day. Between the realization that DES is weak in the late 90s and the invention of AES in the early 2000’s, Triple-DES had a brief time to shine.

Triple-DES is just what it sounds like: you run the DES algorithm three times. You use two 56-bit keys, K1 and K2, and apply them in the order K1-K2-K1. The result is a cipher with 112 bits of key strength.

Students often ask me, why not just encrypt twice: K1, K2, done? The reason is that this construction is vulnerable to a particular chosen-plaintext attack, which we call the meet-in-the-middle attack. That is, if the attacker knows your plaintext in addition to your ciphertext, he doesn’t have to try all $2^{112}$ keys. The maximum amount of work he has to perform is actually only $2^{56}$ – not much more than to break single DES.

Notation

Before I proceed to an explanation of the meet-in-the-middle attack, let’s get some basic cryptography notation out of the way.

Let’s call our plaintext P, and our ciphertext C. Let’s call our DES encryption function E. It takes in a plaintext and a key; in return, it produces a ciphertext. Similarly, there is a decryption function D which takes a ciphertext and a key and spits out a plaintext.

We can then say:

E(P, K) = C
D(C, K) = P
D(E(P, K), K) = P
E(D(C, K), K) = C

You get the idea. Let’s move on.

Double-DES

Double DES uses two keys, K1 and K2 We produce a ciphertext using the following formula:

C = E(E(P, K1), K2)

We just encrypt twice, using our two keys. Easy.

Since each key is 56 bits, an attacker that doesn’t know the plaintext would need to guess a total of 112 bits of key to break the cipher. Therefore, the total keyspace is $2^{112}$, and an average of $2^{111}$ keys would have to be guessed before the true K1 and K2 are obtained.

Meet-in-the-middle

Let’s redefine Double-DES as a two-step process:

M = E(P, K1)
C = E(M, K2)

We will refer to the result of the first encryption as M, as it is our “middle” value. Now, for this attack to work, we assume our adversary has access to both P and C, but wants to determine K1 and K2

Consider the following statement – we can easily derive it from the second formula we wrote above:

M = D(C, K2)

An attacker can easily exploit this fact, and essentially make K1 independent of K2. He can guess them each separately, instead of trying to guess them at the same time. But how?

First, the attacker guesses every possible K1. Let’s call each of his guesses K1’. For each guess, he computes M’ = E(P, K1’) to obtain a potential “middle” value, and stores the result in a table along with the corresponding K1’.

After filling out the table, with each of the $2^{56}$ possible K1’ keys, the attacker moves on to guessing K2. Again, let’s refer to each guess as K2’ He computes M’ = D(C, K2’) then checks whether this M’ matches any M’ he stored in the table previously.

If a match is found, then both keys have been broken. That is, if E(P, K1’) = D(C, K2’), then K1 = K1’ and K2 = K2’.

How much work has the attacker done? Well, generating the table required $2^{56}$ encryptions, then finding a match would require $2^{55}$ decryptions on average. Overall, we will only have done only $2^{56.6}$ cryptographic operations. This is in stark contrast to the $2^{111}$ operations which we would expect to perform in a ciphertext-only attack with two 56-bit keys.

Pseudocode

In short, this is the attack:

def meet_in_the_middle(C, P):
  table = {}
  for i in range(0, 2^56):
    table[E(P, i)] = i
  for i in range(0, 2^56):
    if D(C, i) in table:
      return table[D(C, i)], i

That’s it.

Demonstration

I’ve written a quick demonstration of this attack in Python using the pycrypto library. The source code is on my Github. For the demo, I have only used 18 bits of the key (just so you don’t have to wait forever to see the results). There are also some optimizations which can be made (e.g., DES takes the key as a 64-bit input even though only 56 of those bits are used… we can skip some of the inputs that we have tested to make it faster). However, it works.

Triple-DES

So why isn’t Triple-DES vulnerable to this attack? Well, it’s easy. We can only “skip” one encryption using this strategy. Since Triple-DES is defined as E(D(E(P,K1),K2),K1), we can’t split it up in such a way that we won’t, at some point, have to guess both K1 and K2 at the same time. A formal proof of this claim is, unfortunately, out of scope of this article.

Last word

This attack isn’t specific to DES. It works on any composition of two ciphers which is constructed as E(E(P, K1), K2). So don’t ever, ever, ever try to do that.

A fun experiment with Twilio

Sat, 09 Jan 2016 00:00:00 +0000

I first heard about Twilio a long, long time ago. As Google Voice faded out of relevance, it took the lead in the mobile-communication-as-a-service market. However, I had never had the chance (or inclination) to play around with its API until today.

About 12 hours after we landed back in the US from our holiday in Mexico, Lynsey departed once again – this time to the Plant and Animal Genome conference (PAG) in San Diego. She asked me to supply her with pictures of our cats for the duration of her trip. I told her I would send her a cat pic every hour, on the hour.

I didn’t realize what I had gotten myself into until I had already deposited $20 into a new Twilio account and spent 2 hours coding away… Though my goal was just to send some photos of cats, I had developed a pretty general application that lets you build a queue of MMSes to be disseminated at a constant rate.

This was not only my first experiment with Twilio – it was also my first adventure with Flask, a framework for building simple HTTP services in Python. I must say that both Flask and Twilio were extremely easy to use, and I would not hesitate to use them again.

I will certainly be finding some more uses for Twilio in the future – it will be invaluable for the SMS-based 2FA we plan to integrate into Zeall. But for the time being, it will be serving as a medium for the delivery of photos of cats.

If anyone is interested in my Twilio-based app, here is a link again: https://github.com/tmick0/twiliq

No, fingerprints are not secure

Wed, 02 Dec 2015 00:00:00 +0000

Authentication is the process by which a system determines whether a particular user is allowed to access it. There are three widely agreed-upon methods to authenticate a user:

Something you have.
Something you know.
Something you are.

When you use your key to unlock your front door, you are authenticating yourself using something you have. In information security, passwords are the most popular method of authentication; they are something you know. Authentication by something you are (i.e., biometrics) has historically been only a niche practice, but in recent years it has caught on in the realm of consumer electronics.

When Apple announced Touch ID in late 2013, security experts immediately voiced their concern. The authentication mechanism was quickly compromised, and there is still very little that Apple can do about it. Why, you ask? Because fingerprints are inherently insecure.

An authentication mechanism must meet three major requirements to be considered secure:

It is secret.
It is unique.
It is revocable.

While fingerprints are unique (in theory – more on that later), they are neither secret nor revocable.

You wear your fingerprints every day, and everybody can see them. But what implications could this possibly have? About a year ago, a German hacker famously announced that he had compromised the fingerprint of defense minister Ursula von der Layen using only a photograph. After obtaining a photograph of someone’s fingerprint, it is trivial to create a fake. And that easily, your biometric security has been defeated – permanently.

Permanently? Yes – biometrics aren’t revocable. If someone steals your password, you can easily change it. If they steal the keys to your house, you can change the locks. But if someone steals your fingerprint, you can’t get new fingers. Just take a minute and let that sink in.

Now, more on the uniqueness aspect. We are all taught that our fingerprints are unique, which is essentially true. However, the way they are interpreted by computers is not exactly unique. When a fingerprint is used for authentication, an algorithm typically selects a handful of features it thinks are unique and distinct; these points are called minutia and are the only portions of the fingerprint considered for identity matching.

Because the environment will not always be the same when your fingerprint is scanned, each read will probably contain a different set of minutia. Whether it be caused by random noise, damage to the imaging lens, or something on your finger (like some dirt, or perhaps a burn or a cut), two scans will never be exactly the same. Thus, fingerprints can not be compared for an exact match the way that passwords are.

What are the implications of this kind of comparison? Well, it means there must be some acceptance threshold which determines how “alike” two fingerprints must be to be considered a match. And thus there will always be a chance that someone else’s fingerprint will be able to authenticate your identity. We can always raise the threshold to reduce this probability, but it will in turn make it more likely that a valid fingerprint will be rejected. In the current state of the art, a false positive rate (FPR) of 0.1% can correspond to a false negative rate (FNR) anywhere between 0.02% and 15%. Standard consumer devices probably lie on the higher end of that FNR range. I doubt that users would tolerate getting rejected 15% of the time, so it would be a safe bet to assume that the threshold is set so that the false positive rate is much higher than 1 in 1000.

So what have we learned here? Well, that’s up to you to decide. In my opinion, the integration of fingerprint scanners into consumer devices is nothing more than a shifty marketing scheme. The general public thinks biometrics are super high-tech, and thus they are a means to inspire consumers to invest in a new gadget.

If you have any sensitive data on your phone, I suggest you at least put it behind a PIN. I reiterate, your fingerprint can be trivially stolen. It might be a good choice for second authentication mechanism (i.e., for two-factor authentication), but it should not be used by itself.

Time clocking at the command line

Wed, 02 Sep 2015 00:00:00 +0000

I often feel inclined to start new projects to avoid working on old ones. In a particularly ironic display of procrastination, I have written a productivity-oriented application in order to avoid actual productivity. The app is called InSTiL, and its goal is to make it easy to log how much time you spend working on various projects. The source is available on Github, and the Readme provides a succinct overview of InSTiL’s functionality.

https://github.com/tmick0/instil

A C++ encapsulation of the Linux inotify API

Sun, 01 Feb 2015 00:00:00 +0000

The inotify API allows you to monitor a file or directory for various events such as file creation, modification, and deletion. It is part of the Linux kernel and the glibc userspace library, however its C API can be cumbersome to use in a C++ application. A C++ binding of inotify does exist, but it still requires the application developer to write an unsightly wait-and-handle loop. My goal for this project was to create an asynchronous event-driven API through which filesystem events can be processed.

My inotify encapsulation, Inotify_cpp, hides inotify’s read loop behind a simple system of event callbacks. A programmer who seeks to monitor the filesystem only needs to write an event callback (which can be as simple or as complicated as desired) and attach it to a watch. It will then receive events as they occur, all while allowing the main application thread to attend to more important matters. An example of the library’s ease of use is included.

Currently, Inotify_cpp implements all the functionality of the underlying inotify API, only with the exception of cookie handling for move events. However, this will be implemented soon, and I even plan to extend my library beyond the capabilities of traditional inotify. Most importantly, I will add support for recursive watching – one of the most sought-after features inotify lacks.

Inotify_cpp is hosted on Github. You can clone or fork it here:

https://github.com/tmick0/Inotify_cpp

RIOT OS ported to TI Tiva C Connected Launchpad

Sat, 31 Jan 2015 00:00:00 +0000

My current project is porting RIOT OS to the EK-TM4C1294XL evaluation board. RIOT is an embedded operating system aimed at the Internet of Things, developed primarily by Free University of Berlin. The EK-TM4C1294XL is a pretty powerful board, featuring an ARM Cortex M4 MCU and built-in Ethernet MAC. So far, I have implemented only the most basic support for the CPU - just timers and UART. However, I’m currently working on the Ethernet drivers (almost done) and my next focus will be drivers for an XBee add-in.

This port of RIOT (and the boards) will be deployed in a Smart Grid project being worked on by myself and the others in the Network Systems and Optimization Lab at NMSU.

The port is forked from the 2014.12 release of RIOT.

My repository is here: https://github.com/nsol-nmsu/RIOT

Roll your own dynamic DNS (Ubuntu)

Thu, 24 Jul 2014 00:00:00 +0000

Dynamic DNS, or DDNS, is a type of DNS configuration which allows hosts with dynamic IP addresses to automatically update their DNS records. Often users will rely on services such as DynDNS or No-IP to manage this type of setup, but it is actually relatively easy to run your own DDNS server. Of course, this requires that you have your own domain name and access to at least one host with a static IP (to use as the DNS server).

Step 1: Set up DNS records for your dynamic subdomain

I do not recommend hosting all of your DNS records yourself. I manage most of my records on Cloudflare, but delegate my dynamic subdomain to my own nameserver. In this step, you’ll need to decide two things: what subdomain you want your dynamic hosts to live on, and what server you will host their DNS records on.

Let’s assume you’ll use ddns.example.com as your dynamic subdomain, and ns1.example.com as the nameserver. With this setup you will need to add a NS record delegating ddns.example.com to ns1.example.com. Then your dynamic host will be named, for example, host1.ddns.example.com. However, you can also add a CNAME pointing ddns.example.com to host1.ddns.example.com, allowing you to use a shorter name to talk to the machine.

Step 2: Install and configure your nameserver

Now, on ns1.example.com, install the bind9 package (as it is named on Ubuntu). Edit /etc/bind/named.conf.local and add the following:

zone "ddns.example.com" {
  type master;
  file "/var/lib/bind/ddns.example.com";
  update-policy {
    grant *.ddns.example.com self ddns.example.com.;
  };
};

This just allows your the hosts on your ddns subdomain to update their own records (after they authenticate themselves). Now to generate the key for authentication:

$ ddns-confgen -r /dev/urandom -q -a hmac-md5 -k host1.ddns.example.com \
  -s host1.ddns.example.com. | tee -a /etc/bind/ddns.example.com.keys   \
   > /etc/bind/key.host1.ddns.example.com

This can be repeated for however many hosts you need on the subdomain. Then copy the key.*.ddns.example.com files to their respective hosts, and delete them off the nameserver. Now, we have to make sure to set secure permissions on the nameserver’s copy of the keys:

$ chown root:bind /etc/bind/ddns.example.com.keys
$ chmod u=rw,g=r,o= /etc/bind/ddns.example.com.keys

Now, add a line to /etc/bind/named.conf.local to use the keys you just created

include "/etc/bind/ddns.example.com.keys";

The server is now configured, but we still have to create a zonefile which will store the DNS records.

Step 3: Create the zonefile

Create /var/lib/bind/ddns.example.com and add the following information:

$ORIGIN .
$TTL 300             ; 5 minutes
ddns.example.com IN SOA  ns1.example.com. root.example.com. (
          1          ; serial
          3600       ; refresh (1 hour)
          600        ; retry (10 minutes)
          604800     ; expire (1 week)
          300        ; minimum (5 minutes)
          )
NS      ns1.example.com.
$ORIGIN ddns.example.com.
$TTL 60 ; 1 minute

That’s it. Now start up bind and we’ll move to the dynamic host to configure it.

Step 4: Configure the dynamic host

First, let’s try to manually write a record to server. Write a simple text file with this content:

server ns1.example.com
zone ddns.example.com.
update delete host1.ddns.example.com.
update add host1.ddns.example.com. 60 A 192.168.0.1
update add host1.ddns.example.com. 60 TXT "Updated on $(date)"
send

Now, pipe it into nsupdate:

$ nsupdate -k /path/to/key.host1.ddns.example.com < file_you_just_wrote.txt

If it worked, then everything is set up correctly! However, it would be nice to have the IP update automatically, right? Fortunately, there’s a script that can do that! It just queries OpenDNS for your IP address, then sends the update to your nameserver. Just set up a cronjob (for example, every 15 minutes) to run it, and your DDNS will be completely automated. (Sorry, the site I was hosting the script on was no longer active, otherwise there would be a link right here.)

Quick and easy OpenVPN server config (Ubuntu)

Mon, 21 Jul 2014 00:00:00 +0000

VPNs allow you to route your Internet traffic through an encrypted tunnel to a remote server, enhancing your privacy while online. Often a VPS is around the same price per month as a dedicated VPN service, but gives you much more freedom and utility as a poweruser. This post will overview a basic routed OpenVPN configuration on an Ubuntu machine.

Step 1: Drivers and packages

First you need to enable TUN/TAP on your VPS. Usually your host will give you an option for this in your Control Panel. If not, you may have to file a support ticket. We will need TUN to emulate a new network device for the VPN to use. To check whether you have TUN enabled, do sudo cat /dev/net/tun. If you get the message “File descriptor in bad state,” then TUN is working. If you get “No such device,” then the TUN driver is missing.

Now we will need to install OpenVPN, as well as Easy-RSA which we will use for our PKI:

$ sudo apt-get install openvpn easy-rsa

Step 2: PKI and keys

Set up the PKI environment and generate a CA certificate:

$ sudo -s # become root
$ mkdir /etc/openvpn/easy-rsa/
$ cp -r /usr/share/easy-rsa/* /etc/openvpn/easy-rsa/
$ editor /etc/openvpn/easy-rsa/vars # set up parameters for your CA
$ cd /etc/openvpn/easy-rsa/
$ source vars
$ ./clean-all
$ ./build-ca # the CA certificate is now built

Build a key and certificate for the server:

$ ./build-key-server SERVER_NAME_HERE
$ ./build-dh
$ cd keys/
$ cp SERVER_NAME_HERE.crt SERVER_NAME_HERE.key ca.crt dh1024.pem /etc/openvpn/

Make keys and certificates for each client, and transfer them to the remote hosts (for example using SCP):

$ ./build-key CLIENT_NAME
$ scp /etc/openvpn/ca.crt CLIENT_MACHINE: 
$ scp /etc/openvpn/easy-rsa/keys/CLIENT_NAME.key CLIENT_MACHINE:
$ scp /etc/openvpn/easy-rsa/keys/CLIENT_NAME.crt CLIENT_MACHINE:

Step 3: Server config

Copy the default the server configuration:

$ cp /usr/share/doc/openvpn/examples/sample-config-files/server.conf.gz /etc/openvpn/
$ gzip -d /etc/openvpn/server.conf.gz

Now edit /etc/openvpn/server.conf and set the correct filenames for the server certificates and keys:

ca ca.crt
cert SERVER_NAME.crt
key SERVER_NAME.key 
dh dh1024.pem

Step 4: iptables

You may have to set up some routes and firewall rules to get your VPN to work as expected (using NAT):

$ /sbin/iptables -A FORWARD -o tun0 -j ACCEPT
$ /sbin/iptables -A FORWARD -m state --state RELATED,ESTABLISHED -j ACCEPT
$ /sbin/iptables -A FORWARD -s 10.8.0.0/24 -j ACCEPT
$ /sbin/iptables -A FORWARD -j REJECT
$ /sbin/iptables -t nat -A POSTROUTING -s 10.8.0.0/24 -o venet0:0 -j MASQUERADE
$ /sbin/iptables -t nat -A POSTROUTING -j SNAT --to-source YOUR_SERVER_IP_HERE

If you’re using an iptables manager, you can easily add these rules to that service’s configuration. You can also put them in a new initscript, in an on-boot cronjob, or some other script that gets run when the server boots.

Step 5: Start the daemon and connect clients

Now just start OpenVPN:

$ service openvpn start

And then configure your client with the keys you scp’d. Everything should be peachy.