I think these notes are idiosyncratic to me and my local WISP, not a commentary on OpenWRT. But recording some notes for my reference.

I switched to an OpenWRT One for my backup Internet connection, a fixed wireless link from Smarter Broadband. Soon after switching link performance got a lot worse for about 9 days, going from 0.2% packet loss to 2.8% packet loss (pings to 8.8.8.8). Much higher latency too, particularly in the evenings. It’s better again but I’m still at 0.5% packet loss.

At first I suspected the new router was the problem, given the co-incidence. But the pattern looks exactly like congestion in my WISP. Also it’s gotten significantly better on its own. Maybe the router switch precipitated the problem though. I had to reboot my end of the wireless link. And IIRC their fixed wireless isn’t adaptive, it picks a channel and sticks with it when it turns on. Maybe I got unlucky.

Update (May 14): I’m blaming my WISP, not the router. It’s back to 0.257% packet loss.

Following up on my efforts for a remote PC console, I gave PiKVM a try. Very impressive! A great first experience: once you find your way to the right docs it is great. This remote console option is a winner and I’m going to stick with it.

Skip down to the section about software if you’re curious what it’s like to use the thing. I spent a lot of time hacking up some hardware to run it so first notes on that.

Hardware

PiKVM runs an Raspberry Pis. The simple thing is to buy an assembled device from them. You get a CM4, video capture device, an ATX power controller, a power supply, and other stuff all in a nice case. I think the V4 Mini at about $300 is what I’d need. It’s not a bad price for the convenience. OTOH at that price you have a choice of other IP KVMs. I’m not sure if PiKVM is the best choice.

Update: I bought the PiKVM Mini v4. It’s a very nicely prepared product, well built and nicely packaged with plenty of cables. The Mini isn’t just cheaper than the Plus, it’s smaller too. Not a huge thing but I appreciated it. I should have just bought this to start, could have saved myself a fair amount of time.

I’m stubborn and wanted to build my own. They have a very nice guide on doing that, including what hardware to buy. I took their recommendations. $50-$100 depending on what parts you may have lying around already.

• Raspberry Pi Zero 2 W. A tiny $15 computer, these things are very versatile. I had one lying around.
• A HDMI-CSI bridge. A $30 HDMI capture device. It takes HDMI as the input. The output is an RPi Camera device (Camera Serial Interface).
• A CSI ribbon cable. $9 for a set of cables, you only need one. This adapts the wider CSI output to a narrower CSI input on the RPi Zero.
• A quality USB power supply for the RPi.
• An ordinary Micro USB to USB A cable (maybe with some tape, see below)
• Optional: an ATX power interface. This seemed complicated to build, I haven’t looked at it.

The drawback with the RPi Zero is both USB ports are busy doing things and it’s not really practical to have an ethernet adapter. I’m OK just using on-board WiFi but obviously that’s not ideal.

Also this camera interface is pretty hacky. They say it’s capable of 1080p@50 maximum, the product page says 1080p@30. Either way it can’t do the bog-standard 1080p@60 that everything expects these days and that could cause a problem. My Linux PC is happily putting out 720p for it and that works fine.

Physical challenges

The ribbon cable is very fiddly to install. this is not a normal consumer connector. I found this video helpful. Basically you have to unsnap something in the connector, stick in the cable, then snap it in again.

It’d be nice to have a case. The parts are all exposed circuit boards and a delicate ribbon cable. And the HDMI and USB cables are heavier than the computer, which ends up putting a lot of strain on that delicate ribbon cable connector.

No one seems to have a case that fits both boards at once but this 3d printed kludge looks plausible. There’s lots of pre-fab cases for the RPi Zero by itself, that might help.

A note on power

The RPI Zero 2 W has pretty squirrely power. The board has one USB 2.0 port just for power and one just for data. But the power lines of both USB ports are hardwired together. This means if you provide power to the power port you may be dumping that power down the data port into the PC you’re plugged in to. Not good! Alternately, if you only plug the data port into the PC then the RPi will more or less manage to power itself off the data port, which isn’t reliable.

For the problem of backfeeding power to the PC, the PiKVM docs suggest taping off the power leads on your USB cable plugged in to your PC. That’s probably a good idea but I haven’t bothered and so far haven’t fried anything. Here’s hoping my PC’s motherboard is robust. OTOH I ordered this USB power blocker: $6.50 for peace of mind.

What about powering the PiKVM just off the data port connected to the PC? It sorta works! In a previous experiment I measured the RPi Zero 2 W using 165 mA when idle at 570 mA at max load. The USB spec promises 500 mA of power from the USB port so it’s almost enough, and in practice probably is enough if you’re not at max load. PiKVM does work and is usable with just data port power. But then it failed on me once and I have no idea why, was it a brownout? Also it makes the PiKVM dependent on the PC’s power and USB ports which is probably not a great thing in a debugging tool. Independent power seems like a good idea.

One last power note: RPis have a WiFi power saving mode that can cause problems for long-running servers like PiKVM. It’s easy to turn off. PiKVM doesn’t by default. (They wisely don’t recommend using WiFi at all.)

Software: Installation and use

The docs are pretty great and easy to follow. Basically you flash an SD card with their OS. Then edit a text file in a FAT32 partition for initial setup options (like wifi password). Then boot and it’s all pretty automatic. From there it’s all a pretty standard Linux experience. Arch, btw.

But I’ll say this, the PiKVM web interface is great. It was very easy to get it up and going and see my PC’s screen and type at the PC. Super easy to do the basic thing. And the screen sharing UI has nice options like shortcuts for sending weird keystrokes (Ctrl-Alt-Del or Alt-PrScr-T).

One weird thing: the PiKVM filesystem is mounted readonly by default. You have to explicitly issue an rw command to switch to read-write, then ro to go back. In practice you almost never need to do this manually, maybe when you’re changing the root password or something.

I’ve noticed the video feed flakes out when the PiKVM system is busy installing upgrades or fetching system logs. (Even when it has a separate power supply). It may be the RPi Zero CPU is just not powerful enough to keep up. Shouldn’t matter in normal use.

Advanced capabilities

There’s lots of cool hacker extras beyond the basic IP KVM. Some things it can do:

• Tailscale
• Cellular backup network connection
• Fancy HDMI and video options. They seem to care about providing a good video feed, not just an adequate one.
• OCR for the video feed so you can get text of what’s on the screen.
• Audio and microphone
• ATX power control so you can reset or power off/on your PC
• Serial and Ethernet over the USB link to the PC
• GPIO access on the RPi for special-purpose hacking
• VNC server
• An HTTP API
• Extra hardware for a KVM switch for four devices

It’s a remarkably impressive product.

I noticed my Linux box sees a bunch of USB devices when I plug in the PiKVM: keyboard, mouse, CDROM. And I guess ethernet and serial are also possible if I enabled them.

The Linux driver e1000e for Intel ethernet adapters has had a persistent problem for years. It can get in a state where it hangs and reports “Detected Hardware Unit Hang”. Sometimes it recovers on its own but recently on my system it started breaking my networking until reset.

The usual advice for fixing it is to disable hardware acceleration features on the adapter. This has a performance impact but is probably small and avoids the problems with the driver and/or hardware. A simple one-off fix of ethtool -K eno1 tso off gso off was enough to help my system, that turns off acceleration for segmentation of packets. But doing that once won’t persist for reboot, for that I like this systemd unit file. Note that script turns off a whole lot of other acceleration features too which may or may not be a good idea. See also discussion here and here and a whole lot of other places.

It’s not clear what the real underlying bug is. Some folks blame bad hardware design. But Linux drivers have lots of code for working around hardware problems, not sure why it’s such a problem in e1000e. The fact that segmentation may be a particular problem doesn’t surprise me. Segmentation is stateful: either the kernel or the hardware has to store some data while it breaks it up into packets and that probably involves locking.

The other confusion is why this bug started happening to me recently. The problem started when I upgraded to Proxmox 8.4.1 with their kernel 6.8.12-9. This bug discussion has a comment noting that one patch to the kernel didn’t get included in that build, maybe that was the problem. Proxmox has since released a 6.8.12-10 which says cherry-pick "bnxt_en: Fix GSO type for HW GRO packets on 5750X chips". But that patch is for “5750X chips”, which I think is a Broadcom part. So maybe not relevant to e1000e at all?

My guess is Proxmox had patched this problem in some kernels but it’s come back. Would love to understand why the upstream Linux release doesn’t have a fix for this once and for all. Even if it’s to default a bunch of hardware acceleration features to off in the driver.

I bought this to replace the Raspberry Pi 4 I’ve been using as a backup router. That’s worked OK but isn’t ideally suited: CPU is overpowered and the networking is underpowered (USB). The main improvement with this hardware should be proper WiFi with nice antennas although at the moment I’m not using the wireless. It’s a little weird that it has a 1GB ethernet port for LAN but 2.5GB for the WAN. Also it has an M.2 port hidden in it that’s not advertised.

It worked fine right out of the box. I went ahead and upgraded to 24.10.1, the latest main release. That mostly worked fine once I worked around some OpenWrt confusion. I initially installed a “snapshot release” but those don’t come with LuCI, the web interface. But I could still ssh so I reinstalled the official release via sysupgrade on the command line. One nice thing about this hardware is it has two full firmware sets. You normally use and boot the NAND firmware but there is also a second NOR firmware that’s a sort of failsafe you can manually select to boot from. (You can also flash the NOR firmware which means you might still be able to brick the device if you try hard enough.)

OpenWrt 24.10 was a major release, the first in over a year. A huge change is groundwork to replace their custom-built opkg package manager with Alpine’s apk. Which should be a big improvement in future development and maintenance. (It’s not live in 24.10 release, but is in the subsequent snapshots). Also lots of big changes like Linux 5.15 to 6.6. They support an impressive 2000 devices, it’s quite an ambitious open source product.

Not much more to say about OpenWrt or the One hardware, it seems to be doing the job. I’t’s’ve had my gripes with OpenWrt but it’s turned into quite a robust hacker-friendly router firmware. If I weren’t bought into the Ubiquiti ecosystem I’d use it for more than a backup.

BTW because of all the changes all the OpenWrt advice and docs are a little out of date. For instance a bunch of previous upgrade tools like sysupgrade and auc have been replaced by owut.

Final note, some packages I added to stock OpenWrt: openssh-sftp-server less bash curl nano joe. Also installed luci-app-statistics and did a one time “save” on the setup to get it started.

I wanted remote access to my Linux PC and decided to try a very old fashioned idea, a serial console. Normally a Linux PC’s console is a keyboard and video device. But it can also just be a serial device. Either an old school DB9 connector on an ancient PC or some modern thing plugged in with a serial connection. Then you need a second device to act as a serial terminal.

In theory the serial device is simpler and is more likely to work even if the network isn’t working. In particular the Linux PC’s kernel is managing the serial console and so as long as Linux is operating, there’s a good chance that serial connection will be.

I tried using a Raspberry Pi Zero 2 W for the terminal, plugged in to my Linux PC via a USB cable.

I got this sort of working and then quit. I realized there’s few cases where a serial connection will really help. Ie: the Linux machine is up but its networking is down, however the whole LAN is still up. It happened to me this week! But I’m going to look for a full IP KVM solution so I can do things even if Linux is busted.

Update: this set of docs may be useful.

Serial connection: USB Gadget / On-the-Go

The first complication: who has a real serial port anymore? I haven’t seen a DB9 in years. It’s all USB ports. And you can’t just plug a USB cable in between two computers because they are both in host mode, not device mode.

There’s hardware hacks for true serial. Plenty of USB-to-Serial adapters out there that give you a DB9 (or an RJ45 with serial). Also RPi hardware has UARTs that you can use with the GPIO pins, hack up your own serial port. All that conversion seemed du mb to me, particularly since I’d need two adapters on either end.

The hack I’m using is OTG / Gadget mode on my RPi Zero 2 W. The micro USB port is an OTG port, you can plug that in directly with a cable to a PC’s USB A socket. The PC is a USB host, the OTG port is a USB device. This actually works! No special cable or adapter needed, just an ordinary USB cable between the two computers. (Mine was male Micro-B to male A.) However, it still takes a lot of doing to get it going.

Getting serial working manually

Here’s how I got a serial connection working manually in Linux.

On the RPi Zero 2 W: steps to enable OTG mode and use the USB as a serial port

• enable dtoverlay=dwc2 in /boot/firmware/config.txt and reboot
• modprobe dwc2
• modprobe g_serial
• verify there’s a new /dev/ttyGS0 on the RPi

On the Linux server (after the modprobe on the RPi)

• lsusb and verify there’s something named “Linux-USB Serial Gadget”
• Verify there’s a new /dev/ttyACM0
• Run getty -L 9600 ttyACM0 vt100 to create a login prompt on the serial port

Back on the RPi, connect to the Linux server.

• Run screen /dev/ttyGS0 9600 as a quick & dirty serial terminal program

This all works! I can now connect from one machine to another via a serial interface.

Proper server setup

The next big step is getting the server to run that console all the time. The simple way is to add ttyACM0 to the list of things that systemd should run a getty on. But that doesn’t catch the kernel. What you really want is to modify the Linux kernel command line to treat this serial port as a second terminal, not just for getty but also for kernel errors, etc. I didn’t do this step.

Proper terminal setup: ConsolePi

Any old serial terminal would work and that screen hack I tried is actually not bad if you only have a single server to connect to. But there’s a fancy OS purpose-built for being a serial terminal with a lot of nice extra features: ConsolePi. The key component is ser2net, that’s a Linux service that proxies all serial ports to TCP network ports. Then ConsolePi wraps that up in a nice UI.

I tinkered with it a bit but didn’t get it working. Honestly I found the docs confusing and was running out of patience. The main snag I ran into is ser2net doesn’t seem to find devices named ttyGS* by default, or at least I couldn’t figure it out. I’m sure that’s hackable.

Update: Conserver may be worth a look as a different app for monitoring serial ports.

I want to be able to remotely access my PC Linux box even if Linux is down. Ie: just like you’d do with a keyboard and monitor plugged in locally, but via the Internet. There’s a zillion ways to do this but they are all fiddly and the simplest ways are expensive or unobtanium.

This post is a mess but discusses three remote options.

I left out one useful technology: power control. Some of these solutions let you power cycle the Linux PC, either via the BIOS or via a hardware control. Me, I’m just using a smart power switch.

Remote management

The Big Boy option is to buy server hardware that has remote management capabilities. Some bulletproof mini-OS in the BIOS that you can access over the network. I have a friend who swears by HPE Proliant MicroServers for this exact reason. This isn’t really a Linux console, it’s just a PC that’s better suited for remote server management.

IP KVM

A common hack is to emulate a keyboard, video card, and mouse. You get a magic device (the “IP KVM”) that has HDMI and USB inputs, plug those in via cables to the PC’s HDMI and USB output, and as far as the Linux PC knows it has a normal local display, keyboard, and mouse. Meanwhile the magic device is accepting the video feed and making it available via a network server somehow, also emulating a keyboard and mouse for the PC.

The nice thing about this solution is it’s really a PC level solution, Linux is unaware of it. You can even use it to access the machine’s BIOS.

So what’s the magic device?

There’s a bunch of enterprise-grade IP KVMs out there for $400 or more. The nicest ones are rackmounted. They were more expensive and capable than I wanted so I kept looking.

JetKVM looks very impressive: $70 for a slick device with a web interface. Unfortunately this is not really purchasable. It’s only for sale on Kickstarter, they are way oversold, and they aren’t taking orders for the US.

PiKVM is another magic device option and you see it mentioned frequently because it’s a bit more hacker friendly. But the simplest thing is to buy it for.. yes, about $400. You can also DIY it with your own RPi and an HDMI capture device. I’m about to try this with a $15 RPi Zero 2 W and a $30 HDMI interface that plugs in to the camera port with a $10 cable.

I’ve also run across NanoKVM, a tiny PiKVM based on the LicheeRV Nano hardware. Interestingly the CPU is RISC V. It looks to be entirely a Chinese product so may be a bit tricky to import and understand the code. I sure wish I could read Chinese. Update: SiPeed has a pre-order page in English for NanoKVM.

Update: GL-iNet has a Comet（GL-RM1） for $90

Serial console

Another choice is to go old school and use Linux’ serial console support. This won’t let you access the BIOS of the PC (probably) but it will let you get to Linux the moment the boot loader starts.

On the Linux server end you’ll want to enable a serial console. Eventually you want this serial device to be a console at boot time. For hacking around you can just run a getty on the serial console.

On the other side of the serial cable you need a device that accepts serial and has an Internet interface. ConsolePi is one device for this purpose. It seems quite capable but also difficult to understand.

I have some much more detailed notes on a serial console in another post.

Some notes on a personal server outage. Highlights:

• Smart power switches are great
• Would love a remote console of some sort
• Having any sort of remote shell other than the server would be nice
• Remote Unifi is helpful
• My Intel ethernet device / e1000e driver may have a bug
• A second network interface?
• I’d sure like a watchdog

I left San Francisco on the morning of Apr 18. At 2:38AM Apr 19 I got notified some of my services running on my personal Linux server there weren’t working. No problem, I thought! This is what Proxmox is for! Only I couldn’t reach the Proxmox hypervisor: the whole machine was down.

Long story short, today (Apr 22) I power-cycled the machine and it came back fine.

Smart power switch

My first thought on the 19th was to reboot the server. Then I went looking for the smart power switch I thought I’d set up and couldn’t find it. I have like 4 brands of switches and apps. And I forgot to check Emporia, the newest one. Mostly that’s my car charger but they have a nice set of smart power switches too. I finally found it this morning (while looking at a list of devices on my LAN), power cycled the machine, and the reboot worked. Yay!

Remote console

I’d sure love a remote console for this server. Something I can log into remotely. I’ve got another blog post coming on options for that, I think PiKVM might be my solution. Next time I buy a server I should buy something like an HPE Proliant with remote management.

Remote Unifi

My Ubiquiti router has remote management with a web interface. I’ve used this more than I thought I would. I used it on the 19th the moment I knew things were wrong. It let me confirm the house Internet was working and that the Proxmox host was no longer on my LAN.

Remote shell

Usually when things go wrong and I’m not home I log into a VM on my server and poke around with diagnostic tools. But with my server down I had no VM to log into. Kinda wish I had a second machine, a little RPi or something. I finally figured out you can get a remote ssh shell on the router via the Unifi console. The Linux environment you get there is very limited but at least I could run ping.

Second network

I should consider adding a second network device, a cheap WiFi adapter, just as a backup. Would be nice to bypass the ethernet switches too.

Watchdog

The root cause of the failure was the networking going down. It’d be nice if my Proxmox server had a watchdog to reboot itself. In this case the computer was fine, just not the network, so that’s a little trickier. In the past every time I’ve set up watchdogs it caused more problems than it solved. The real Proxmox solution is to cluster and use high availability, but I don’t really want to buy more computers.

Root cause / ethernet problem

So why’d it crash? I’m not certain, but here’s the big sign of trouble

Apr 19 09:06:59 sfpve kernel: e1000e 0000:00:1f.6 eno1: Detected Hardware Unit Hang:

That got logged once every two seconds once it started. tailscaled also started complaining it couldn’t connect to the network. Everything else more or less kept running fine, scheduled tasks and backups and stuff. Just no network.

Looking at my logs, this error message has been logged about 150 times in the past year and a half. But in previous times it’s been transitory, a few errors and then it recovers itself without a reboot. This last outage was the first time where it got stuck and didn’t reboot.

Looking online, this error is frequently asked about. I don’t have the stamina to read all the diagnoses of what might be wrong. I did ask some AIs though. The usual cargo cult advice is to do something like ethtool -K eno1 gso off tso off to disable various advanced ethernet device features like hardware supported checksums or segmentation. Also some thought that power management might be a problem. Kind of hate the idea of this. I thought Intel ethernet devices were well regarded and supported in Linux.

I updated Proxmox from 8.3.? to 8.4.1 a few days before this crash. If it happens again it might be a problem with this kernel (6.8.12-9-pve).

Updates

The server crashed again the same way last night at about the same time.

Apr 23 09:08:37 sfpve kernel: e1000e 0000:00:1f.6 eno1: Detected Hardware Unit Hang:

This is happening during the 2am busy cron job window, so it’s possible there’s system load involved. There’s a big backup job running of my primary VM which takes about 8 minutes. The backup does complete! And I’d hope it doesn’t generate network traffic. But who knows. FWIW I was able to run the backup job manually without causing a problem.

After that Apr 23 crash I turned off GSO and TSO and it hasn’t crashed the last two nights. Given the code I’ve read, I’m guessing segmentation is in fact the problem.

I’ve got a separate blog post just about this kernel problem.

Apr 29: got a crash late last night with GSO and TSO turned off. At 04:52:56 UTC (about 10pm local), so a few hours earlier than the previous crashes. So turning those features off helped but didn’t solve the problem. Going to try turning off many more options as in this unit file and see how I do. But still a one-off, I keep hoping Proxmox releases a fixed kernel.
/sbin/ethtool -K eno1 gso off gro off tso off tx off rx off rxvlan off txvlan off sg off

I had an idea for a project and posted to Mastodon

I got an interesting variety of replies, ideas collected here.

I gather there’s at least three different methods that could apply. Most folks suggested #2, RAG.

1. LoRA training, which alters the core LLM weights
2. Retrieval Augmented Generation which augments LLMs with a document store it can select from and use to tune replies. Sort of a formal version of…
3. using a big context window and preloading the query with stuff

I think 2 and 3 are in common use now for consumers. But 1 is still more specialist and requires you have your own LLM inference engine.

One recommendation I got was to use SillyTavern, a consumer LLM wrapper for role playing game players. It does RAG for you: you give it a bunch of documents and it manages LLMs for you and lets you talk to it. I bet it has a lot of the plumbing I need. This suggestion also reminds me there’s a world of porn AI enthusiasts too, all DIY with local models to work around censorship. They may have some interesting tech.

Mark Fletcher (who runs an email service) gave me a great reply about how he mixes RAG with ElasticSearch. Useful notes but pretty involved, would be a lot of work. I should give his feature a try though and see how well it functions.

Update: Justin Mason reminded me of his personal GMail archive project: he indexed 20 years of email from Google in a local system using notmuch. No LLM in this work but could be a useful related component for the email retrieval part.

I’m not interested enough in this to build a whole product, I was more wondering if I could just hack something up by throwing a wall of text at an LLM. I think RAG is the tool to do that, paired with ollama or deepseek.

The other question is where to run it. My email is very private, I don’t want to ship the archive off to some untrusted provider. “A big Mac at home” is one common suggestion. I’ve been thinking about leasing a cloud machine to use on demand. I think a PC at home with an nVidia 3xxx may be sufficient to at least prototype.

Update August 2025: it’s been a few months and some things exist.

• Tutorial for hacking your own (uses LangChain and Chroma DB)
• OnPrem, a Python toolkit
• Some notes from a hacker
• Suggestions from ChatGPT 5

This problem was driving me crazy: Adobe Acrobat was opening PDFs that Chrome downloaded despite every setting I could think of in Chrome or in Windows. This was blocking Chrome’s own PDF reader. Also Sumatra, my favored PDF reader.

The solution is in Acrobat’s preferences. Menu > Preferences > General > Basic Tools > Open PDFs automatically in Reader when they are downloaded in Chrome browser.

I have no idea how an Acrobat preference could be overriding Chrome. I blame Windows, Microsoft built a bunch of hijacking stuff in to Windows to boost their browser. I suspect Adobe is exploiting some thing in Windows.

Adobe is also using malware tactics to reinstall itself as a Chrome extension every month or so. Chrome won’t actually run it, but you do get nagged to use it.

Adobe: 20+ years of terrible, hostile software.

My Pixel 8 phone camera is great. It takes photos in HDR. But very little software supports it. The image displayer on the Pixel does, it has some extra-bright display mode that makes the white too bright on screen. Also Google Photos has some HDR editing support with its “Dynamic” and “Enhance” automated presets, plus a slider for “HDR effect”. But when you download the original image file from the phone, it just looks like a regular 8 bit JPG. What’s going on?

I only spent a few minutes researching this but the images appear to be a special Google format they call “Ultra HDR Image Format v1.1” or maybe “JPEG R”. More info in this article and this Reddit discussion. Also older info here, here, here. There’s even a Wikipedia article.

Basically it’s an ordinary 8 bit JPG with some extra data in th efile. Some of that is general metadata about brightness range, etc. And there’s something like a second raster image, a gain map which “provides a mechanism to store data sufficient to recover the original, higher dynamic range luminance data”.

I can’t find any non-Google software that works with the format. Fortunately the files do work pretty well as a regular 8 bit SDR JPG. And Google Photos lets you edit the tone mapping from HDR to that SDR. I’m also not sure which editing tools preserve the extra HDR metadata even if they can’t do anything with it.