• Proxmox container for a Tailscale exit node

    I want to set up a Tailscale exit node for me on the server with a good home Internet connection. This is a sort of poor-man’s VPN, routing all traffic through my home ISP connection. It’s pretty easy but I ran into a problem with IP forwarding.

    First I set up a tiny Alpine container and used the Tailscale community script to set up Tailscale on it. You can install Tailscale by hand but this script handles the extra config needed to give an unprivileged LXC access to the tunnel device.

    Then I set up Tailscale in the container to advertise as an exit node. The key thing there is sysctl configurations to enable IP forwarding inside the container OS. Very important! You also have to tell tailscale to advertise itself as an exit node.

    Once I did this it worked. At least, I think it did. But it stopped working, maybe after a reboot?

    To fix it, I also enabled IP forwarding via sysctl on the Proxmox server itself. I’m not 100% sure this is required or even a good idea. But when I enabled it, the exit node started working again. I’m a little confused about this but I think it’s OK.

  • Death to Tailscale MagicDNS

    It took several years but I finally disabled Tailscale MagicDNS. It just caused too many problems. It broke Proxmox containers. It broke private DNS on my Android phone. It broke all sorts of stuff. I don’t want tailscale to mess with DNS.

    I’ve replaced it with tailscale-cloudflare-dnssync. This is a simple Python program that uses Tailscale and Cloudflare APIs to read your entire tailnet’s list of hostnames and create DNS records. There’s a zillion versions of this kind of script, including some that support other DNS services. This one works fine for me. I’ve set it up to run daily but since the Tailscape IP assignments are static, the DNS records aren’t going to change much.

    I also set up my local resolver to search the subdomain I put all the tailnet hosts in. So I can use short hostnames like sfrouter instead of FQDNs like sfrouter.tailscale.example.com.

    The downside of this new setup is now all my tailnet hostnames are out there in a public DNS list somewhere. They all point to private unroutable IP addresses, so there’s very little someone can do with that info. But now someone could figure out with some DNS queries that I have, say, a PiKVM and then try to attack it somehow. Then again I remember when AXFR was still a supported DNS query, and a very useful one at that! It just doesn’t seem that big a worry for me.

    MagicDNS was always a kludge. Tailscale would basically try to intercept your system’s DNS resolvers and insert its own at 100.100.100.100. Its primary feature I used was to serve address records for hostname.your-tailnet.ts.net. (These aren’t real DNS records, they were only visible to you from their special nameserver). Over time it has grown a lot of other complex features, none of which I want. And it caused all sorts of problems. Good riddance.

    As a bonus I think this will make it easier for me to get working SSL certificates for my tailscale names. There’s some way to do that with MagicDNS but I never figured it out.

  • Working with Ultra HDR JPGs

    My Pixel phone takes nice HDR images and saves them as UltraHDR. (aka ISO 21496-1 or Adaptive HDR). These files look and act mostly like normal JPEGs but HDR-aware software can find a second raster image in there with a “gain map” which has extra brightness information for boosting the image into 10 bit HDR.

    It’s a bit of a kludge but seems to work well in practice. Chrome and a lot of Android software works with them as true HDR images. Microsoft and Apple both support them, as do all the browsers. Any old software without UltraHDR falls back to an 8 bit SDR image: not great but better than nothing.

    GUI HDR programs

    I’d like to view and edit HDR images on my Linux desktop. I have a nice HDR monitor that is working! Unfortunately I can’t find a single mainstream Linux image viewer or editor that supports Ultra HDR JPGs. A little surprised at that. The only thing that will show them correctly is Chrome, you can drag a local file to it and it displays in HDR.

    There are a couple of specialist viewers that sort of work. tev seems to handle the HDR data but I couldn’t easily get a display that looked right although fiddling with the tone mapping the image looked intact. HDRView displays the image great but I think is only rendering at 8 bits, just nicely tonemapped. Also it doesn’t scale the UI to my display so it’s unusable.

    CLI HDR programs

    Even worse, I can’t find a good command line program to work with UltraHDRs or even convert them intact to other HDR formats like AVIF or EXR. At least, nothing simple seems to work. I tried out a bunch of Gemini suggestions on a Pixel photo and didn’t get very far.

    The winner is hdrify-cli, a pure Javascript node program. It is easy to install and use. I can convert the UltraHDR-JPG to a EXR file that when opened in Okulus looks… different than the source SDR. Like a badly tone mapped image, but maybe that’s Okulus’ display algorithm fault. If I convert it back to a JPEG and view it in Chrome it looks like the original, so actual round trip conversion! hdrify also supports writing 8 bit SDR files like WebPs, it seems to just extract the SDR from the source.

    ImageMagick now has some HDR support but I can’t honestly tell how much. The build on my Nobara Linux system doesn’t include the support. I tried a precompiled build from the project but didn’t get much success with the converter. I got somewhere using this elaborate command, but the resulting tone mapping was too dark. Also this is way too complicated, why doesn’t convert foo.jpg foo.avif just work?

    uhdr2avif seemed promising but the program just spun and ate 2 CPUs.

    Google’s reference libultrahdr has a demo program but it’s more of a swiss army knife for working with UltraHDR images, not a simple converter.

    A lot of these tools all seem very much aimed at experts who understand tone mapping curves and color spaces and gamuts. Not intended for civilian use.

    File formats

    Part of my confusion is what image format I should be using for HDR images.

    UltraHDR JPG is what Google is giving me when I download my photos. It’s a kludgey format but has a lot of support, it seems OK.

    AVIF seems like the “best” choice for a clean image format that happens to also do HDR. Unfortunately hdrify doesn’t support writing it.

    EXR is a very low level HDR format that has some broad support. Main issue is there’s no lossy compression so files are huge.

    JPEG-XL in theory should work but in practice is still a dead-on-arrival format.

    HEIC/HEIF is what Apple has been doing but I don’t think much else supports it.

  • Wake on LAN

    One fun thing about my new Linux desktop is that the computer is useful enough that I want to be able to access it remotely. I never cared with a Windows machine. I’ve got this machine set to go to sleep automatically but just set up Wake on LAN for the first time. Works like a charm.

    The key piece of this is UpSnap. That’s a little Go program that provides a web server with a GUI for monitoring machines and sending Wake-on-LAN packets when you click a button. I installed it with a Proxmox community script and then put Tailscale on it so I can access the webapp remotely, like from my phone.

    Beyond that was verifying my PC would actually Wake on LAN. There’s a BIOS setting to enable (although with that off it will still wake from sleep, just not full power off). There’s also a Linux feature you might have to enable in NetworkManager (it was already on for me). “Wake-on: g” is the important thing, the “g” means “wake on magic packet”

    $ sudo nmcli connection modify "Wired connection 1" 802-3-ethernet.wake-on-lan magic
    $ sudo ethtool enp5s0 | grep -i wake
    Supports Wake-on: pumbg
    Wake-on: g

    Wake on LAN is a 30 year old technology! The core thing I’m using is that most ethernet hardware will wake up if it hears a magic packet sent to its MAC address, even if it’s otherwise mostly turned off. (Gemini says this uses under 1 watt) There’s a WiFi version too although I have to think that’s a lot more complicated given the need to maintain a radio link.

  • CoolerControl / quieting CPU fan

    Awhile back I identified the fan that makes noise on my computer. It’s Fan 2, the Noctua CPU cooler. When it gets over about 55% (1000 RPM) I start to hear it. I just fixed it with a custom cooling curve, with help from Gemini. It was more annoying in Linux than in Windows. Not sure if Windows had some custom fan discipline or if Linux just had a way of spiking heat.

    The result is at idle the cooler runs at about 30-35% speed, I can’t hear it, and the CPU hovers around 60°-70°C. Under heavy CPU load it gets to 80°C and 40% fan. I still can’t hear it!

    I’m using CoolerControl to do this. It overrides the BIOS fan control. I’ve set a custom fan speed curve that’s slower than the BIOS. And then I added a smoothing function with a 3s delay to add a little hysteresis.

    It seems a bit silly to have userspace software controlling this. Maybe I’ll figure out if I can make the BIOS do this for me. This CPU cooler is really quite good and lets me run the fan slower.

    Temperature graph, including a stress period

    Fan curve

    Controls

  • Charging cheap devices with broken USB-C Charging

    I have a tiny portable Bluetooth speaker (EWA A106) that I like but has an infuriating problem: I can’t charge it via its USB-C Charging port if I plug it into a USB-C charger with a C-to-C cable. I can only charge it by plugging into a USB-A charging port with an A-to-C cable or adapter. There’s a lot of cheap electronics that have this charging problem. They ship an A-to-C “charging cable” in the box and consider it good enough.

    USB-C charging is complicated. Fancy devices use USB-PD and a protocol negotiation to charge. Even simple devices require 5.1k resistors on a couple of control pins: without them the USB-C chargers won’t know to charge. USB-A charging is much simpler. A-to-C cables work because USB-A chargers will just put out 5V without any signaling at all.

    Another fix is a special C-to-C adapter that has the resistors. Amazon sells one named Hagibis 5.1k Resistor 5V/3A Charging Adapter. This Reddit teardown says it is reasonably well built. Adafruit sells another that’s not as nicely packaged.

    It’s infuriating so many devices are being sold with these broken USB-C charging ports. The adapter is cheap enough but I really don’t want to have to carry it around!

  • Customizing my Nobara system

    I’ve now set up KDE Plasma desktops under Fedora about 5 times. Here’s my notes on all the things I do. It’s a lot! What’s particularly annoying is there’s no easy way to capture all this config and just replay it all. The config files are too much of a mess.

    Earliest setup

    • Switch to Breeze theme
    • Set wallpaper to solid #001830
    • Install Chrome from downloaded RPM.
    • Install 1Password from downloaded RPM.
    • Install tailscale
      curl -fsSL https://tailscale.com/install.sh | sh
    • Remove the Brave browser and disable its repository.
      sudo dnf remove brave-browser brave

    Nobara Welcome App

    First steps

    • Update my system. The installer will do this for you if you let it.
    • Install the Nobara codec package. It is very aggressive about launching this!
    • Open Driver Manager: I didn’t need to do anything

    Optional Steps

    • Launch Steam. Then use ProtonPlus to Install Proton-GE

    Recommended

    • Install Discord
    • Install Flatpak Bazaar
    • Uninstall OBS Studio, Kdenlive, and Blender 3D

    Misc

    Set up ssh agent stuff

    Install RPMs

    dnf history list is very helpful for reconstructing this list.

    • ghostty
    • joe
    • input-remapper
    • chezmoi
    • sunshine (beta)
      dnf copr enable lizardbyte/beta
      dnf install miniupnpc
      dnf install sunshine –repo=”copr:copr.fedorainfracloud.org:lizardbyte:beta”
    • syncthing
    • coolercontrol
    • adobe-source-sans-pro-fonts adobe-source-code-pro-fonts adobe-source-serif-pro-fonts google-roboto-fonts google-noto-sans-fonts jetbrains-mono-fonts fira-code-fonts

    Install Flatpaks

    flatpak history or flatpak list can help to reconstruct this.

    Use Bazaar (installed above) to find things instead of Flatpost or Discover or whatever.

    • Slack
    • Mission Center. Consider hack to enable wattage info:
      sudo chmod a+r /sys/class/powercap/intel-rapl*/energy_uj
    • Gear Lever
    • Signal. Launch it and follow instructions about kwallet.
    • Plex. Had to install via command line.
    • Calibre
    • SyncThiny. Set up sync for ~/src

    Install AppImages (in Gear Lever)

    • LM Studio
    • noson (for Sonos)

    Customization

    • sudo systemctl enable --now sshd
    • chezmoi init nelson@sf.somebits.com:~/gitrepos/chezmoi
      chezmoi apply
    • Set up apps for favorites for launcher
    • Timeshift to do daily btrfs snapshots
    • Move system tray to left side, 146 pixels wide
    • Set date format to ddd MM/dd
    • Install KIM6 for image conversion in file manager
    • Set up an ssh-agent for the graphical environment. This is complicated and the linked AI answer is wrong in important ways.
    • Log in to every website, there is no way to copy login cookies to other machines.

    Keyboard customizations

    • Fix the function keys
      echo "options hid_apple fnmode=2" | sudo tee /etc/modprobe.d/hid_apple.conf
    • Use Input Remapper to override CapsLock to be F15. Make sure to apply it at boot!
    • Use Shortcuts to bind F15 to launch clipboard in web browser
    • Use Shortcuts to override F13 to take screenshots with Spectacle. (May not be necessary: F13 is in the same spot on my Apple Keyboard.)
    • Set up XCompose key under “Position of Compose Key”
    • Disable the Plasma virtual keyboard. It uses a lot of VRAM (a known bug).

  • Chez Moi: modernizing Linux config

    I’ve used Linux since 1994 and Unix since 1989. I have some very old and crufty configurations I’ve hauled around over the years with some clunky script to set up a new machine. I’ve modernized it with help from Gemini.

    Dotfile management

    The key tool I used is chezmoi. It’s a nicer version of my clunky script. It feels a little like git, but customized for shell configs. For instance I run chezmod add ~/.bashrc and it makes a copy of my current bashrc to ~/.local/share/chezmoi. If I run chezmod apply it copies all the files from that chezmoi repository to live places. It’s a little goofy but it works and it has advanced features for templating, encrypting secrets, etc. Note you’re expected to use git yourself to keep versions of this chezmoi mirror of your config files.

    Setting up chezmoi was a very manual process of deciding what config files I want and manually importing them. In the end I’ve got 31 config files. A bunch of ancient tty stuff from my old configs and some new KDE stuff from my new Linux desktop

    Modern Linux file locations

    XDG defines is the modern standard location for config files. The key thing is ~/.config, which in theory is where all config files belong. This has been predictably corrupted over the years. Chrome spams it with over 6000 files including a bunch of binary junk. But it’s still a pretty useful standard. ~/.config/lesskey for instance lets you configure the less pager. (And in a text format now, hooray!)

    Auditing config files

    It’s very confusing what all your config files are. xdg-ninja is a helpful tool. It looks in your home directory for standard config files for a bunch of programs and advises you ways you can modernize and move things to .config. Not a panacea but a helpful tool!

    Modern bash shell

    There’s some nice tools that make bash really nice to use.

    starship is a lovely dynamic prompt framework that is very fast. The key thing I rely on is it showing me git repo status.

    atuin turns your shell history into a giant database you can query. It’s very nice if you have the habit of building up complex commands interactively.

    Modernizing configs

    I had a surprising amount of fun showing Gemini my config files and asking it to advise me on how to update things. For instance it suggested I don’t need 25 year old Netscape environment variables anymore. And that there’s better / newer ways to do some things with modern tools. It was a fun interactive process for an hour or two sweeping off some of the cobwebs. Particularly impressive that I can ask it things like “what does mpage -a do?” and it remembers some 25 year old BSD utility I long since forgot.

  • Linux gaming and resolution scaling

    Deep into tinkering with gaming on Linux using Bazzite 44. The problem I’m wrestling with is that fullscreen games are guessing a different resolution than they should. my screen is 6016×3384 natively but instead the game is trying to run at 6863×3860 or some other wrong value. This doesn’t happen for all games.

    These tests are for the game Rhythm Storm, a fairly simple but graphically intense game. I’m playing on a ProArt display that’s 6016×3384 natively, with a NVidia 5060 and the 595.71 Linux open drivers installed. Bazzite is pretty much an all-Wayland system, with just a bit of X11 compatibility. I am running the display at 225% scaling. This Gemini chat is all the research I’ve done so far.

    First, a useful tool: xrandr. That shows a set of resolution that X11 apps think are valid. On my system with 225% scaling it shows 6863×3860, the mode the game is picking. If I change scaling this number changes in some mathematically way I can’t quite understand. Anyway this must be what the Steam game is picking up.

    One tool to fix this scaling problem is gamescope. That’s some sort of Steam graphics shim that sits between the game and the Linux display software. It can be told to enforce a specific resolution and you can impose it for a particular game by altering the Steam game launch options. That works! Unfortunately it also hides the Steam overlay. There’s supposed to be a -e or --steam option to work around that, but when I try that the game never launches at all.

    I’ve been surprised at how well games work on Steam / Linux in general. Proton is a marvel! But this kind of problem is exactly the sort of infuriating bullshit that makes Linux seem unacceptable as a desktop OS. Right now I’m still cheerful about trying to figure problems like this out, so I’ll stick with it. Wonder how long that will last.

  • Remote game streaming

    Some things I’ve learned about playing games by running them on a big computer and using my TV as a client. Basically this is an optimized form of desktop screen sharing.

    The client TV is a 2160p Sony Bravia. That runs Android and seems the best of the smart TV platforms. The server running the games is a desktop gaming PC, I’ve tried both Bazzite Linux and Windows. Both devices are wired on the same LAN. (This stuff works on the Internet too and is surprisingly responsive.)

    Steam Remote Play

    Valve has had a remote gaming product for some 10 years. I still have working Steam Link client hardware! But these days it’s easier to run a Steam client on the TV.

    It works pretty well but is surprisingly flaky. The biggest problem is resolution switching. Steam should change the desktop resolution to the TV’s resolution, or down to 1080p if I tell it to. That only works one time out of three so mostly I have to change the resolution on the PC myself. (The TV can display anything, but 3440×1440 looks terrible on a 2160p display.)

    The other knock on Steam Remote Play is that it’s not very optimized. Folks report 20-40ms latency, or maybe 2 frames of a 60fps game. I haven’t really noticed that but I’m not super sensitive to it.

    Sunshine / Moonlight

    The alternative fancy gamers recommend is the Sunshine streaming game server and the Moonlight client. These have been around quite awhile, started as an open source reimplementation of the NVidia Game Shield protocol. The big advantage folks quote over Steam is latency: it delivers 4ms on my LAN, or well under one game frame.

    The server works well on Windows. It seems reliable. It has some nice stuff about launching things when a client connects, like invoking Steam Big Picture. I’m still having to set the Windows desktop resolution manually although I think Sunshine can do that for me.

    The server does not work for me on Bazzite Linux. It sort of does. But the NVidia drivers on my old 1080 card aren’t correctly detecting display modes so I’m stuck streaming 3440×1440.

    Alternate resolutions

    One dumb thing about all these solutions is that they’re displaying what’s on the PC’s screen, at the PC’s resolution. But basically I have a remote screen on the cliente TV: why should I care what’s plugged in on the server?

    One solution is a dummy HDMI plug. This does the EDID handshake to make the computer thing you have a monitor plugged in so you can configure the OS to use it as a primary or secondary screen. These are particularly popular for headless machines: with no monitors plugged in some GPUs refuse to operate at all.

    The dummy plug seemed to work as a second screen on Linux. But caused all sorts of problems. First, the desktop GUI wants to use that screen! But it’s invisible, so stuff gets placed there you can’t see. Second, I couldn’t reliably get all the gaming stuff to happen on that fake screen while leaving my desktop alone for normal stuff. The software is just not designed to do it.

    I went down a rabbit hole of trying to fix the resolution-switching problems on Linux with my 1080. Total disaster. There’s some hack with kernel keyword arguments with display mode lines, yuck! There’s also a hack where you load fake EDID data from a file to trick the graphics card. I couldn’t make any of that work. Honestly why doesn’t the NVidia driver just detect the modes that are there?

    The other solution is a virtual display. Basically have a device driver fake a viritual screen disconnected from actual hardware. This feels like the right solution and reminds me of the ancient X11 virtual desktop hacks. Ironically, virtual displays are reportedly unreliable to impossible on Linux.

    On a Windows server folks recommend Apollo, a Sunshine fork, which has virtual displays as a headline feature. I’ve tried this server and it works, but I haven’t tried the virtual display feature yet.