Perfectly Sane

Kernel Hacking + Device Bringup on NixOS

2024-11-16T00:00:00+00:00

one of my NixOS devices is a PinePhone Pro</a>. at time of writing, it boots to a working display/touchscreen and even WiFi on stock NixOS, however other basic features like audio and battery readouts aren't available. actually, these features are available on distros like postmarketOS</a> which ship kernel forks specifically for this. but maintaining a fork is at least a bit of work, and NixOS gives you lots better ways to tweak your kernel than simply building the whole thing from a different source tree.

so, how best to develop and deploy kernel-level changes on a NixOS system?

Kernel Pain Points</h2>
skip ahead if you're already familiar with the NixOS kernel options. the straightforward way to ship custom kernels on NixOS is something like:
{ pkgs, ... }: { boot.kernelPackages = pkgs.linuxPackagesFor (pkgs.buildLinux { src = fetchgit { # ... }; # ... }); }</code></pre> or if your kernel patches are simple enough, then track the NixOS kernel and patch it instead:
{ ... }: { boot.kernelPatches = [ { name = "my-kernel-patch"; patch = ./my-kernel-patch.patch; } ]; }</code></pre> but the crushing downside to this approach is that tweaking just a single line in my-kernel-patch.patch</code> forces a complete rebuild of the kernel: your iteration cycle time might well be 30 minutes.
Faster Kernel Iterations</h2>until we get something like dynamic derivations</a>, the path out of costly iteration is to ship whichever tweaks you're pursuing via some other mechanism than as part of the top-level kernel derivation. the kernel is fairly pluggable; consider: device tree files can be loaded at runtime.</li> kernel modules can be loaded at runtime.</li> </ol> with the right config, either of these things can be freely modified without forcing a kernel build. Shipping Device Tree Patches</h2> Linux uses device tree</a> files to determine which peripherals are available on your device and how to interface with them. it's more relevant for embedded devices than for traditional x86 machines. the typical boot flow for an embedded device is that the bootloader (U-Boot) knows the name of the device it's running on (perhaps because it was built specifically for that device) and communicates this to the kernel when it hands over control of the device. the kernel ships with device tree definitions for hundreds of different devices, each with a name like samsung,exynos7-espresso</code> or pine64,pinephone-pro</code>, and after learning which device it's actually running on it applies the appropriate device tree, instantiating and configuring device drivers accordingly. device trees are composable by design; the spec defines the concept of an "overlay", such that the actual device tree to apply to a device should be the union of whatever's defined in the kernel source tree plus any overlays that apply to the same device. NixOS makes this feature available via the hardware.deviceTree.overlays</code></a> option, and a certain amount of our kernel changes can be expressed via this option instead of costly .patch</code> files. for example, the stock kernel misidentifies volume-down key presses as volume-up events on the PinePhone Pro. distros like postmarketOS fix this by patching the device tree file like so: --- arch/arm64/boot/dts/rockchip/rk3399-pinephone-pro.dts +++ arch/arm64/boot/dts/rockchip/rk3399-pinephone-pro.dts @@ -48,11 +48,11 @@ / { adc-keys { compatible = "adc-keys"; io-channels = <&saradc 1>; io-channel-names = "buttons"; keyup-threshold-microvolt = <1600000>; poll-interval = <100>; button-down { label = "Volume Down"; linux,code = <KEY_VOLUMEDOWN>; - press-threshold-microvolt = <600000>; + press-threshold-microvolt = <400000>; };</code></pre> this patch can alternately be represented as a Device Tree Overlay</a> (DTO): // file: rk3399-pinephone-pro-lradc-fix.dtso // boilerplate required by the device tree compiler. /dts-v1/; /plugin/; // instruct the system to only apply this overlay // to PinePhone Pro, and not any other devices. / { compatible = "pine64,pinephone-pro"; }; // address the parent node we want to patch, // and then inject a new property value. // this overrides `press-threshold-microvolt` // while leaving all other properties unchanged. &{/adc-keys/button-down} { press-threshold-microvolt = <400000>; };</code></pre> and then we can ship it in NixOS like so: { ... }: { hardware.deviceTree.overlays = [ { name = "rk3399-pinephone-pro-lradc-fix"; dtsFile = ./rk3399-pinephone-pro-lradc-fix.dtso; } ]; }</code></pre> after adding the above to a stock NixOS config, and deploying to a PinePhone Pro, the volume down button should now be fixed! Shipping Custom Kernel Modules</h2> one of the less trivial patches yet to be mainlined is support for battery monitoring on the PinePhone Pro (i.e. viewing the charge level and the charge/discharge rate). this feature is supported in downstream kernels by shipping two new kernel modules: rk818_battery</code> and rk818_charger</code>. these modules are device drivers: they detect a rk818-battery</code> device somewhere on the system, and then run code in response to that (namely, enable some voltage regulators, and create sysfs nodes to let userspace interact with the device). code wise, the typical driver is a single .c file with few if any direct dependencies on other drivers. most driver modules are standalone in the same sense that each nix package is standalone. then, we can ship new kernel modules by: defining a nix derivation that compiles the kernel module.</li> telling nix to deploy that on the kernel's module search path.</li> instructing the kernel to load our module.</li> shipping a device tree overlay to associate the relevant devices with the driver our module provides.</li> </ul> Building a Kernel Module</h3> my kernel module has 3 files, which live in my nix repo inline: rk818_battery.c</a></li> rk818_battery.h</a></li> rk818_charger.c</a></li> </ul> these are compiled into rk818_battery.ko</code> and rk818_charger.ko</code> by a Makefile: obj-m := rk818_battery.o rk818_charger.o all: $(MAKE) -C "$(KERNEL_DIR)" M="$(PWD)" modules install: install -Dm444 rk818_battery.ko \ $(INSTALL_MOD_PATH)/drivers/power/supply/rk818_battery.ko install -Dm444 rk818_charger.ko \ $(INSTALL_MOD_PATH)/drivers/power/supply/rk818_charger.ko</code></pre> this can be packaged for nix as so: # file: pkgs/linux-packages/rk818-charger/default.nix { buildPackages, kernel, lib, stdenv, }: stdenv.mkDerivation { pname = "rk818-charger"; version = "0-unstable-2024-10-01"; src = ./.; hardeningDisable = [ "pic" ]; nativeBuildInputs = kernel.moduleBuildDependencies; makeFlags = [ "KERNEL_DIR=${kernel.dev}/lib/modules/${kernel.modDirVersion}/build" "INSTALL_MOD_PATH=$(out)/lib/modules/${kernel.modDirVersion}/kernel" # from pkgs/os-specific/linux/kernel/manual-config.nix: "O=$(buildRoot)" "CC=${stdenv.cc}/bin/${stdenv.cc.targetPrefix}cc" "HOSTCC=${buildPackages.stdenv.cc}/bin/${buildPackages.stdenv.cc.targetPrefix}cc" "HOSTLD=${buildPackages.stdenv.cc.bintools}/bin/${buildPackages.stdenv.cc.targetPrefix}ld" "ARCH=${stdenv.hostPlatform.linuxArch}" ] ++ lib.optionals (stdenv.hostPlatform != stdenv.buildPlatform) [ "CROSS_COMPILE=${stdenv.cc.targetPrefix}" ]; # NixOS kernel expects compressed kernel modules, so do that here. postInstall = '' find $out -name '*.ko' -exec xz {} \; ''; }</code></pre> then add this package to nixpkgs' linuxKernel</code> package set: { ... }: { nixpkgs.overlays = [(self: super: { linuxKernel = super.linuxKernel // { packagesFor = kernel: ( super.linuxKernel.packagesFor kernel ).extend (kFinal: kPrev: with kFinal; { rk818-charger = callPackage ./pkgs/linux-packages/rk818-charger { }; }); }; })]; }</code></pre> kernel modules must be built against the same kernel version that they'll be loaded by: this linuxKernel</code> package set looks funny at first, but it exists to make that easier. your kernel module can be built like this: nix-build -A linuxPackages_6_11.rk818-charger</code></li> nix-build -A linuxPackages_6_10.rk818-charger</code></li> </ul> and so on, depending on which kernel you're running. Deploying a Kernel Module</h3> add the following to your NixOS config: { config, ... }: { # this builds our modules against the specific kernel in use by the host # and makes them available under /run/current-system/kernel-modules # where they can be found by tools like `modprobe`. boot.extraModulePackages = [ config.boot.kernelPackages.rk818-charger ]; # explicitly load the modules during stage-2 boot. # if they're needed earlier in the boot process, # then use `boot.initrd.kernelModules` instead. boot.kernelModules = [ "rk818_battery" "rk818_charger" ]; }</code></pre> now we've recreated the same behavior as if the kernel module had been shipped in-tree, but we still have to associate the driver with a specific device if we want it to do anything. Device Tree + Module Integration</h3> add the following Device Tree Overlay to hardware.deviceTree.overlays</code>, as we did earlier with the battery DTO: /dts-v1/; /plugin/; // apply the overlay only to PinePhone Pro; no other devices. / { compatible = "pine64,pinephone-pro"; }; &{/} { bat: battery { compatible = "simple-battery"; voltage-min-design-microvolt = <3400000>; voltage-max-design-microvolt = <4350000>; }; }; &rk818 { battery { // this line associates this device with the rk818-battery module // we just shipped compatible = "rockchip,rk818-battery"; // constants were taken from the pine64-org kernel tree: ocv_table = <3400 3675 3689 3716 3740 3756 3768 3780 3793 3807 3827 3853 3896 3937 3974 4007 4066 4110 4161 4217 4308>; design_capacity = <2916>; design_qmax = <2708>; bat_res = <150>; max_input_current = <3000>; max_chrg_current = <2000>; max_chrg_voltage = <4350>; sleep_enter_current = <300>; sleep_exit_current = <300>; power_off_thresd = <3400>; zero_algorithm_vol = <3950>; fb_temperature = <105>; sample_res = <10>; max_soc_offset = <60>; energy_mode = <0>; monitor_sec = <5>; virtual_power = <0>; power_dc2otg = <0>; otg5v_suspend_enable = <1>; }; charger { // this line associates this device with the rk818-charger module // we just shipped compatible = "rockchip,rk818-charger"; monitored-battery = <&bat>; }; };</code></pre>Patching an Upstream Kernel Module</h2> this all works, however the PinePhone Pro battery integration requires not just a new kernel module, but also to patch an existing kernel module (rk8xx-i2c</code>). since the kernel module is defined in-tree, the natural way to do that would force a kernel rebuild every time we adjust the patches. we can build our patched module out-of-tree using the same method we used to build a wholly new module out-of-tree (write the derivation as some patches</code> atop src = linux-latest</code>, or forget about tracking patches and just copy the entire module source into our repo). then we just configure the kernel to prefer our module over its in-tree module. a first approach might be to configure the kernel with CONFIG_MFD_RK8XX_I2C=n</code>, then ship our module as above. this works: { lib, ... }: { boot.kernelPatches = [ { name = "rk8xx-i2c-out-of-tree"; patch = null; extraStructuredConfig = with lib.kernel; { MFD_RK8XX_I2C = no; }; } ]; }</code></pre> this will result in one kernel rebuild, and then you can freely edit your out-of-tree rk8xx-i2c</code> module without costly rebuilds. if you find yourself patching a lot of modules, then it may be preferable to simply build all in-tree modules as dynamic modules, and configure out-of-tree modules to take precedence over the in-tree ones. remove the MFD_RK8XX_I2C = no</code> patch and replace it with this: { ... }: { nixpkgs.config.hostPlatform.linux-kernel = { # build every module known to the mainline kernel autoModules = true; # and build them as dynamically loaded modules (`=m`), not builtins (`=y`) preferBuiltin = false; }; # default nixos behavior is to error if a kernel module is provided by more # than one package. but we're doing that intentionally, so patch the logic # from <nixos/modules/system/boot/kernel.nix> (AKA pkgs.aggregateModules) # to not complain. system.modulesTree = lib.mkForce [( (pkgs.aggregateModules (config.boot.extraModulePackages ++ [ config.boot.kernelPackages.kernel ]) ).overrideAttrs { # when ignoring collisions, order becomes important: # earlier items (config.boot.extraModulePackages) override later items # (config.boot.kernelPackages.kernel). ignoreCollisions = true; } )]; }</code></pre> more examples for this method can be found on the nixos wiki</a>. Conclusion</h2> and there you have it! several techniques for working with device trees and kernel modules without suffering lengthy builds. if you found anything here useful, then my request to you is to upstream your kernel work so that the next reader of this blog doesn't have to go through the same pain 😛 complete nix expressions for my PinePhone Pro system can be found here</a> in case i missed some crucial detail in this writeup.

Wake-on-LAN and Push Notifications in Mobile Linux 2023-12-09T00:00:00+00:00 the more i carry my Pinephone for long stretches, the more i feel limited by battery life. there's a lot one can do to deal with that, and among those options is to reduce idle power draw. if you use a desktop environment like SXMO</a>, it'll keep a charge for the whole day if you were to keep it screen-off in your pocket. it does that by suspending the CPU to RAM, but keeping the modem powered -- great if all you care about is being notified on incoming calls/texts, not so great if you want to be notified by apps running on the CPU. a typical solution to this is to wake the CPU whenever the system gets an IP packet, yield that to userspace & give your applications a chance to sound a ringer, vibrate, etc, and then suspend again when they're done. the more you can decide "this packet isn't urgent" without CPU intervention, the more you can avoid waking the CPU, the more you can extend battery life. and so most systems provide tooling to help you tune IP-based wakeups if you know where to poke. Wake on LAN</h2> a.k.a. "Wake-On-Wireless LAN" or "wowlan". Wake on LAN</a> is an actual established standard for remotely waking one ethernet device from another device on the same LAN. the user or OS enables this feature in the BIOS, the PC suspends, some other device sends a specially crafted packet (a "magic packet"), magic happens, and the BIOS wakes the system. this can work on WiFi, too, but on a mobile system you enable this by speaking directly to the WiFi chip, rather than BIOS/UEFI. if you're lucky, sudo iw phy0 wowlan enable any</code> will do just that. then enter sleep (rtcwake -m mem -s 300</code> to suspend to RAM for 300 seconds), and from a different device on the same LAN, wake your phone with wol <your_phones_mac_address></code> e.g. wol 02:ba:7c:9c:cc:78</code> (find the MAC address with iw dev</code>). for the Pinephone, whether or not this works depends on how your kernel is configured. firstly, your kernel needs to know to keep the WiFi chip powered during suspend. second, there's a GPIO routed from the WiFi chip to the main SoC: the WiFi chip toggles this GPIO when it sees the magic packet, and the SoC needs to be told to recognize that as an interrupt source that the kernel can respond to. megi's</a> kernel does these things, i can't say about other kernels like Mobian's. so if you're on megi's kernel, this should work... at least once if you repeat the process enough. WiFi and Bluetooth share a lot of the same resources and don't always play nicely together. a quick to get WoL to work reliably is to just disable bluetooth: rm /lib/firmware/rtl_bt</code> (back it up first). there's also a CONFIG_BT_COEXIST</code> kernel option you could mess with if you're pursuing a longer-term fix (i don't really use bluetooth, but would appreciate to hear better fixes if anyone pursues them). and now, wol</code> should pretty reliably wake the phone. there's a race condition if you try that during the rtcwake</code> call instead of waiting a second for it to complete -- i'll address that further down. Wowlan for Userspace</h3> you're probably not interested in manually calling wol</code> from some other computer on the same network to wake your phone. you'd probably prefer it to wake anytime you get a Matrix or XMPP message, or something. for that, things diverge quickly from any sort of standard. but wol</code> actually just sends out an ordinary UDP packet with a specific payload, so getting from there to "wake on TCP traffic sent to port 22", or "wake on TCP traffic sent from IP foo" shouldn't be technically difficult. the Pinephone's Realtek WiFi chip exposes a programmable pattern-matching facility. just give it a byte string and a mask, and it'll wake on any packet whose bytes within the masked regions match those within the byte string. for example, IPv4</a> TCP</a> packets specify their destination port at bytes 37-38. it's easy to tell the WiFi chip to "wake on any TCP packet sent to port 22". run this command before the rtcwake</code> call, then try ssh</code>ing into your phone while it's asleep: sudo iwpriv wlan0 wow_set_pattern pattern=-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:00:16</code></pre> with any luck, it should wake! the fields there are hexadecimal, :</code> indicates byte boundaries, -</code> indicates "i don't care what value this byte is set to". but figuring out the byte pattern which achieves what you want is painful, so i've got a script for that here</a> (mirrored</a>). equivalent to the above is rtl8723cs-wowlan tcp --dest-port 22</code>. if you have a chat application which is talking over https, you might try rtl8723cs-wowlan tcp --source-port 443</code> and see that the phone wakes every time you get a message. but then you'll get spurious wakes if you leave a web browser open, etc. you could use lsof -i4</code> to locate the local port(s) your IM app is using and wake more specifically on those. but you'll still hit limits with this approach, especially if you're using a chatty protocol like Matrix, or if you idle in a bunch of channels configured to notify you only on @mention. enter... notification servers! Notification Servers</h2> when an application wants to sound a notification on iOS or Android when it's not in focus, it's not the application running on the phone which does that, but actually some other server (operated by or for the developer) which tells Apple's server (or Google's) about the notification, and then they relay it to your phone. one of the justification for that is precisely to allow the type of power-saving we're aiming for here, but because that approach poses privacy concerns, passionate people have created open source alternatives to the official Apple/Google notification delivery servers. UnifiedPush</a> defines the standards for each confusingly-named portion of this data flow, and ntfy</a> provides hosted and self-hosted implementations for all the major components. most mature applications provide a way for you to integrate all this: i'll show how to do this for Matrix-synapse and Prosody (XMPP). both can be configured server-side if you have access to that, or client-side if your client exposes an option for it. but first, let's prove this push notification system in a simpler CLI workflow. Minimal ntfy Workflow</h3> ntfy operates a free-to-use server for the push gateway component (i.e. the part that's normally routed through Apple/Google), so install the CLI package for the publish/subscribe portions and we can test a wake-on-notification setup trivially. on your phone: $ ntfy sub TEST_WOWLAN_TOPIC</code></pre> then determine the local port that ntfy process connected from with lsof -i4 -P</code>, and create a wake condition for it: # clear out previous rules $ rtl8723cs-wowlan enable-clean $ rtl8723cs-wowlan tcp --dest-port 1234 #^ replace 1234 with the port from `lsof`</code></pre> then sleep the phone with rtcwake -m mem</code>, and from another computer (or the ntfy web interface</a>) send a notification: $ ntfy pub TEST_WOWLAN_TOPIC "hello phone"</code></pre> if all is well, your phone should awake and the ntfy sub</code> command from earlier should have printed "hello phone"! Hosting a ntfy instance</h3> this part is optional. most applications which speak UnifiedPush provide an option to only send a summary (like "new Matrix message") instead of the actual contents, so you only really need to concern yourself with this if you want the control of self-hosting or are worried more specifically about metadata leaks. NixOS ships options</a> for hosting your own ntfy. use it like: services.ntfy-sh.enable = true;</code></pre> ntfy listens on port 2586 by default, so you can subscribe like ntfy sub MY_INSTANCE_ADDRESS:2586/TEST_WOWLAN_TOPIC</code>, and use a wowlan rule that matches the source port instead of checking lsof</code> manually: rtl8723cs-wowlan tcp --source-port 2586</code>. however, running it this way allows anyone to use your server. the easy way to overcome this is to treat the topic as a shared secret, and forbid use of any topic except your secret topic. naturally, that only works if you secure the connection so do all this behind a TLS-capable reverse proxy like nginx: services.ntfy-sh.enable = true; services.nginx.virtualHosts."MY.NTFY.HOST" = { forceSSL = true; listen = [ { addr = "0.0.0.0"; port = 2587; ssl = true; } { addr = "0.0.0.0"; port = 443; ssl = true; } ]; locations."/" = { proxyPass = "http://127.0.0.1:2586"; proxyWebsockets = true; #< support websocket upgrades. without that, `ntfy sub` hangs silently recommendedProxySettings = true; #< adds headers so ntfy logs include the real IP extraConfig = '' # absurdly long timeout (86400s=24h) so that we never hang up on clients. proxy_read_timeout 86400s; ''; }; }; services.ntfy-sh.settings = { base-url = "https://MY.NTFY.HOST"; behind-proxy = true; auth-default-access = "deny-all"; }; systemd.services.ntfy-sh.preStart = '' # note that this will fail upon first run, i.e. before ntfy has created its db. # just restart the service. ${pkgs.ntfy-sh}/bin/ntfy access everyone TEST_WOWLAN_TOPIC read-write '';</code></pre> deploy that and subscribe to the https url this time. note the port change to 2587: you still want a unique port against which you can write a wowlan rule, and it's easier to put nginx on a new port than ntfy's default 2586. if you're thorough, you might notice some spurious wakeups with this setup. ntfy sends keep-alive packets every 45 seconds, but that's no good for us! best is to decrease/disable those keep-alives. i'll revisit the topic of keep-alives & sleep durations near the end of this article. services.ntfy-sh.settings.keepalive-interval = "30m";</code></pre>Synapse (Matrix) ntfy integration</h3> the Synapse Matrix server exposes an API for controlling its push behavior here</a>. some of the larger clients, like Fluffychat, can be seen to host their own push gateways, which they point Synapse to via this API. presumably one could configure these clients to request a different push gateway (i.e. ntfy.sh, or your own instance from just above), either in the UI or with an edit to their source code. but if you have CLI admin access to Synapse, it's easier to do this generically from the CLI. first, grab an auth token for your Matrix account. it looks like 6cC_a03Ty3sTfqvo3FS_x8vEjdvNsxyL5W4mm73</code> and can be found in Element's "Help & About" settings page. then, with this token, open a CLI to wherever you host your synapse server and issue this command to see where it's currently sending notifications (if anywhere): $ curl --header "Authorization: Bearer ACCESS_TOKEN" \ localhost:8008/_matrix/client/v3/pushers \ | jq . { "pushers": [ { "app_display_name": "Element (iOS)", "app_id": "im.vector.app.ios.prod", "data": { "url": "https://matrix.org/_matrix/push/v1/notify", "format": "event_id_only", "default_payload": { "aps": { "mutable-content": 1, "alert": { "loc-key": "Notification", "loc-args": [] } } } }, "device_display_name": "iPhone", "kind": "http", "lang": "en-US", "profile_tag": "1Akvq5DDr159ANj9", "pushkey": "Bap1n7kqGzF9TuPkiDNDy9wW+cuvfDnxy7Cab7AMsIX=" } ] }</code></pre> now, add a new pusher for your ntfy service. this won't replace the existing settings, instead it'll cause push notifications to be sent to two locations simultaneously: $ curl --header "Authorization: Bearer ACCESS_TOKEN" \ --data '{ \ "app_display_name": "ntfy-adapter", \ "app_id": "ntfy.uninsane.org", \ "data": { \ "url": "https://MY.NTFY.HOST/_matrix/push/v1/notify", \ "format": "event_id_only" \ }, \ "device_display_name": "ntfy-adapter", \ "kind": "http", \ "lang": "en-US", \ "profile_tag": "", \ "pushkey": "TEST_WOWLAN_TOPIC" \ }' \ localhost:8008/_matrix/client/v3/pushers/set</code></pre> repeat the first query and you'll see both of these listed. to delete the new pusher, repeat the above curl</code> command with kind</code> set to null</code>. anyway, put your phone to sleep, have someone send you a message that Synapse would alert you on, and now your phone should awake! Prosody (XMPP) ntfy integration</h3> Prosody has mod_cloud_notify</a> and XEP-0357</a>, but at the time of writing, client support is wanting. easier is to hack it in server-side. Prosody has a Lua-based module system, so we can just author our own mod_ntfy_push</code> (available here</a> or here</a>), drop it in the modules directory, and then import it: services.prosody.extraPluginsPath = [ ./folder/containing/mod_ntfy_push ]; services.prosody.extraModules = [ "ntfy_push" ]; services.prosody.extraConfig = '' ntfy_endpoint = "https://MY.NTFY.HOST/TEST_WOWLAN_TOPIC" '';</code></pre> i've only authored this module to alert me on jingle calls. one could presumably alert on DMs, MUC messages, or anything more particular by finding the right Prosody hooks (mod_cloud_notify may be an ok guide for that). SXMO Integration</h2> SXMO</a> is a mobile-friendly Linux desktop notable for being extremely hackable (it's really just a collection of shell scripts layered over sway</a>), and as such we can integrate all the client-side work we did above into something the desktop environment handles for us transparently. SXMO includes an autosuspend</code></a> service. it's just a bash while</code> loop that periodically checks if conditions are suitable to sleep the phone (is the screen off, is the user not in a call, are no media players active, and so on) and if so, calls out to sxmo_suspend.sh</code>. sxmo_suspend.sh</code></a> looks like this (simplified -- it does a little more to make cronjobs reliable) suspend_time=99999999 # far away sxmo_log "calling suspend with suspend_time <$suspend_time>" doas rtcwake -m mem -s "$suspend_time" || exit 1 sxmo_hook_postwake.sh</code></pre> since the different components of SXMO communicate by shelling out to each other, we can patch it just by putting our own sxmo_suspend.sh</code> script somewhere on the PATH. that's intentional behavior: SXMO goes out of its way to add ~/.config/sxmo/hooks</code> as a PATH entry to its services, so we can just drop a file like this into ~/.config/sxmo/hooks/sxmo_suspend.sh</code>: ntfy sub TEST_WOWLAN_TOPIC & NTFY_PID=$! port= while [ -z "$port" ]; do # netstat output will look like: # tcp 0 0 127.0.0.1:12345 10.11.12.13:443 ESTABLISHED 2798004/ntfy # pipe through sed/cut to extract the `12345` local port component # do this in a loop to allow time for ntfy to establish the connection port="$(netstat --tcp --program --numeric-ports | grep ntfy | sed 's/ */ /g' | cut -d' ' -f 4 | cut -d':' -f2)" done # configure to wake on any traffic to that ntfy connection rtl8723cs-wowlan enable-clean rtl8723cs-wowlan tcp --dest-port "$port" sxmo_log "calling suspend with suspend_time 600" doas rtcwake -m mem -s 600 || exit 1 kill $NTFY_PID sxmo_hook_postwake.sh wait # for ntfy to exit</code></pre> and that's it. use your phone as you normally would, and everything should be the same except that now it'll wake up whenever you get a Matrix/XMPP/ntfy notification. Pinephone Wowlan Race Condition</h2> as alluded, sending a wake packet within about a second of sending the SoC to sleep will fail to wake the system. worse, the SoC will be stuck in the sleep state, not waking on any future wake packets (you can still wake it via the power button, and it should behave correctly during the next sleep cycle). one can imagine lots of ways an edge-triggered interrupt could be misimplemented to cause this type of race, but i wasn't able to diagnose it any more than this. in light of that type of thing, i'm being cautious and never sleeping the phone for more than 10 minutes at a time (rtcwake -s 600</code>). so if you get a VoIP call, there's something like a 0.2% chance you'll miss it and see a notification for it 10 minutes later. there's a higher chance than that i miss a call just by forgetting my phone in the other room, so i'm not especially concerned. but if it bothers you, a workaround exists by coordinating your notification server and your phone such that it knows when your phone is about to enter sleep and delays notifications during that time by a couple seconds to avoid triggering the race. here's</a> how i do that. there are other reasons you may want to not sleep for extended durations: NATs. in theory, a TCP connection with no traffic should remain routable for at least 2 hours</a> -- meaning you could safely sleep for 2 hours and the notification server will still be able to reach you at any point. in practice, i don't know how that holds across all networks. UDP has far lower guarantees (30s to 120s</a> based on who you ask): consider that if you're using a UDP-based VPN like Wireguard. there's also lots of gotchas around WiFi connection details (moving from one network to another, GTK/encryption rekeying, DHCP lease renewals, etc). in any case, longer sleep durations give diminishing returns. if you sleep for 600s, and then wake for 15s before going back to sleep, you've already captured 95% of the gains you could get from this sleep method. sleeping for an hour isn't going to get you that much more battery life. Wrapping Up</h2> i only covered the WiFi usecase. the modem also has a line to the CPU and could be made to wake it under similar scenarios. i believe it already does for calls/SMS, but i haven't gotten around to understanding or tuning its behavior for IP traffic (information from anyone who has looked into this would be appreciated!) i'm not convinced this push notification system will have a place in the long-term future of mobile Linux. in my setup, i'm not actually surfacing any of the content from the push gateway onto the phone's display. the push gateway could just as well be sending the phone 0-byte packets. i'm just using it to wake the phone and then let the applications learn about new messages/calls and tell the DE about them in their ordinary manner. there are simpler ways to do that. maybe some XDG spec or Wayland protocol will be developed to let applications tell the OS "incoming data on this TCP connection isn't urgent. but data on this other TCP connection is urgent" and then the OS can program the wowlan rules accordingly, and notification servers have no special place. on the other hand, a lot of messaging systems are having to implement Apple/Google's notification system for their users anyway, so am i describing an actual reduction in complexity there or would it more likely be yet another standard</a> everyone would need to support? maybe you want to help determine that future? if so: Librem's working on their Chatty client, and while i have no special affiliation with them, i'll point out that they're actively considering</a> how to integrate sleep/push notifications into their client recently. besides that, KDE is also looking into</a> similar things</a>. Where to Learn More</h3> the Arch Linux wiki has a page on Wake-on-LAN</a></li> official SoC/peripheral documentation can sometimes be found in the linux-sunxi wiki</a></li> this</a> ntfy ticket illuminated the Matrix interactions for me</li> i found out about the Pinephone's wowlan second-hand from Peetz0r in #linux-sunxi</a> (IRC) and summarized in megi's dev log</a></li> </ul> </li> i found out about BT_COEXIST issues from Aren in #sxmo</a> (IRC/Matrix)</li> Guido hinted me onto Phosh's plans for wake-on-lan/push notifications in #phosh</a> (Matrix)</li> i got some hints about XMPP push notification support from xavi in chat@dino.im</a> (XMPP)</li> more generally, i keep tabs on app-level and ecosystem developments through: LINMob</a></li> This Week in Gnome</a></li> LinuxPhoneApps</a></li> postmarketOS podcast</a></li> </ul> </li> </ul> as always, feel free to message me via any method on my about page</a>. How To Casually Contribute to NixOS/Nixpkgs 2022-10-13T00:00:00+00:00 as a nix/NixOS</a> user, you'll inherently find yourself writing code, and some of this code might be valuable to others: be it new packages, improvements upon existing packages, or new services/options. part of what makes nix successful (IMO) is the ease of sharing work and upstreaming, so i'm setting out here to show some easy workflows for doing so. the number one thing you can do -- if you're not already -- is host your nix config publicly. for example, mine is here</a>, and even without advertising it i occasionally hear from people in my orbit that they've found it useful. as a beginner i had some public configs i found on google or wiby</a> which i routinely consulted, even though many of those authors probably never heard a word from the people like me who found use in their work. Authoring New Packages</h2> let's say you find some software you want that isn't yet packaged. first, confirm it doesn't exist by searching the package name (and probable variants) on github. remove all filters, so that you're searching both PRs and issues: maybe somebody opened a packaging request and there's already something to work off of. for example</a>. having confirmed that, let's make a new package. anywhere in your config, add an overlay</a>: { ... }: { nixpkgs.overlays = [ (final: prev: { zecwallet-lite = prev.callPackage ./zecwallet-lite { }; }) ]; }</code></pre> i'm just going to show the process i followed when upstreaming that zecwallet-lite package. create the package directory so that callPackage</code> knows what to reference: $ mkdir zecwallet-lite $ touch zecwallet-lite/default.nix</code></pre> when you reference a path in nix like ./zecwallet-lite</code>, it'll resolve to ./zecwallet-lite/default.nix</code> if that's valid. if you're using a flake, the path won't resolve unless it's in git: $ git add zecwallet-lite/default.nix</code></pre> now i locate the upstream package distribution. zecwallet-lite is distributed as an appimage, so i search nixpkgs for other appimages to reference: $ cd ~/ $ git clone https://github.com/NixOS/nixpkgs.git $ cd nixpkgs $ rg appimage pkgs/applications -l | xargs wc -l | sort -h 19 pkgs/applications/misc/remnote/default.nix 24 pkgs/applications/video/losslesscut-bin/default.nix 25 pkgs/applications/networking/instant-messengers/caprine-bin/default.nix 26 pkgs/applications/misc/fspy/default.nix 27 pkgs/applications/audio/sonixd/default.nix ... </code></pre> remnote</code> is a simple derivation, only 19 LOC: { lib, fetchurl, appimageTools }: appimageTools.wrapType2 rec { pname = "remnote"; version = "1.7.6"; src = fetchurl { url = "https://download.remnote.io/RemNote-${version}.AppImage"; sha256 = "sha256-yRUpLev/Fr3mOamkFgevArv2UoXgV4e6zlyv7FaQ4RM="; }; meta = with lib; { description = "A note-taking application focused on learning and productivity"; homepage = "https://remnote.com/"; maintainers = with maintainers; [ max-niederman ]; license = licenses.unfree; platforms = platforms.linux; }; }</code></pre> so copy that to zecwallet-lite/default.nix</code> and edit the values to match your appimage package. set sha256 = lib.fakeHash</code>, and nix will tell you the actual hash. then build the package, test it, and tweak it until you're happy. the simplest way to do that is add it to environment.systemPackages</code></a> and nixos-rebuild switch</code>. normally, if it builds, it'll run (and you might just need to patch up some icon/.desktop files, etc). Upstreaming New Packages</h2> so the package builds, you've used it long enough to be confident in it: time to upstream it from your silo into the main nixpkgs. even if you're strictly self-interested, upstreaming means less custom code for you to maintain, less burden keeping the version up-to-date since others may help with that, and less time spent rebuilding it when a dependency updates since it'll be present in the official nixos caches. it often takes all of ten minutes once you're familiar. go back to your nixpkgs</code> repo and check out the master</code> branch. choose a reasonable folder for the package, like pkgs/applications/misc/</code>, and copy your package there: $ cp -R /etc/nixos/zecwallet-lite ~/nixpkgs/pkgs/applications/misc/</code></pre> then take the callPackage</code> invocation from your overlay and place it in pkgs/top-level/all-packages.nix</code> instead. git add</code> the file, and make sure the package builds: ~/nixpkgs$ nix build '.#zecwallet-lite' ~/nixpkgs$ ./result/bin/zecwallet-lite # make sure it runs</code></pre> commit the changes, push to a github account, and open a PR. github should pre-populate the PR description with a checklist for you to complete. near as i can tell, you don't have to check every item: it's just a way for reviewers to know where to focus. Cleanup and DRY</h2> you could leave the duplicate package definition both in your /etc/nixos</code> repo and in your nixpkgs</code> repo and wait until the next release of nixos/nixpkgs before removing the former; or you can remove the package from your nixos repo now and import it by reference to avoid repetition. if you're using a flake for your system config, it likely looks something like this right now: { inputs = { nixpkgs.url = "nixpkgs/nixos-22.05"; }; outputs = { self, nixpkgs }: { nixosConfigurations.mySystem = nixpkgs.lib.nixosSystem { system = "x86_64-linux"; modules = [ ./mySystem.nix ]; }; }; }</code></pre> we want to somehow patch that inputs.nixpkgs</code> so that it includes whatever changes we're trying to upstream (our new package, a package update, etc). the simplest option is to point inputs.nixpkgs.url</code> directly at our open PR (add &rev=<git-commit></code> to do this), but that doesn't work if we have multiple PRs. instead, we use a trick shown here</a>: { inputs = { nixpkgs.url = "nixpkgs/nixos-22.05"; }; outputs = { self, nixpkgs }: let patchedPkgs = nixpkgs.legacyPackages.x86_64-linux.applyPatches { name = "nixpkgs-patched"; src = nixpkgs; patches = [ ./patch1.patch ./patch2.patch ]; }; nixosSystem = import (patchedPkgs + "/nixos/lib/eval-config.nix"); in { nixosConfigurations.mySystem = nixosSystem { system = "x86_64-linux"; modules = [ ./mySystem.nix ]; }; }; }</code></pre> that is, we take the base nixpkgs repo/derivation, create a new derivation which patches the original nixpkgs, and then import the result of that derivation so that the rest of our flake refers to this patched nixpkgs. the version above expects your patches to live in the same repo as your config flake, but if the goal is to avoid duplication we'd rather point these at the PRs hosted on github. you can use fetchpatch</code></a> for this. if your PR is 180960</a>, append .diff</code> to the URL to get a patch. the let</code> block above becomes this: let fetchpatch = nixpkgs.legacyPackages.x86_64-linux.fetchpatch; patchedPkgs = nixpkgs.legacyPackages.x86_64-linux.applyPatches { name = "nixpkgs-patched"; src = nixpkgs; patches = [ (fetchpatch { url = "https://github.com/NixOS/nixpkgs/pull/180960.diff"; sha256 = "sha256-HVVj/T3yQtjYBoxXpoPiG9Zar/eik9IoDVDhTOehBdY="; }) ]; }; nixosSystem = import (patchedPkgs + "/nixos/lib/eval-config.nix");</code></pre> and we're set! as before, if you don't know the hash just set it to nixpkgs.lib.fakeHash</code>. you can freely push to your open PR: to have the changes reflect on your system just reset/update the hash and nixos-rebuild switch</code>. once your PR is merged, the next nix flake update</code> might include your patch already, in which case nix will error when building nixpkgs-patched</code>. at that point you can safely remove this entry from patches</code>. Upstreaming Other Changes</h2> since this approach is ultimately just applying patches onto nixpkgs</code>, it's not limited to only new packages. to update a package version, add a new config option, etc, you can follow effectively the same process: clone nixpkgs</code>, checkout the master branch, make a change and nix build</code> the result, commit & open a PR, and fetchpatch</code> the PR into your flake. if you're making changes in a hot area of the codebase, the diff from your PR (which targets master) might not apply cleanly to your flake (which pulls from a release or nixos-unstable). in that case you can git log</code> the files you're changing and use the same fetchpatch</code> trick to cherry-pick whatever prerequisite commits you need so that your patch applies. Closing Thoughts</h2> i described here the workflow i found easiest to get started with. with something as configurable as nix, many people use many different workflows. some prefer maintaining their own long-running fork of nixpkgs wherein they cherry-pick all these PRs and periodically rebase against nixos-unstable (or a release branch), and point their flake directly at their fork of nixpkgs. your workflow will surely evolve as you settle in, but hopefully this gives you enough inspiration to get started :-) finally, don't be overly intimidated by the nixos-unstable</code> branch. it might not be quite as unstable as the name suggests: i've yet to encounter any show-stopping issues, 90% of the instability is just that packages don't build and need to be pinned or fixed (or you skip the update that day). there's a point where the inconvenience of nixos-unstable is less than the inconvenience of maintaining patches that have already landed upstream. if you're upstreaming enough that you feel that pain point, you'll do fine on the unstable branch. as always, don't hesitate to contact me</a> or reach out on the forums/chat</a>. Bootstrapping a NixOS Pinephone Installation 2022-06-15T00:00:00+00:00 there is no official, easy-to-grab NixOS image to download and flash to devices like the pinephone today. although there is a way</a> to do that via the hydra build cache, it's a bit tortured and the images built in automation don't have a user. the bootstrapping process would require manually editing /etc/fstab</code> (from a different machine) to add a user and then logging in with a USB-C keyboard to setup ssh or a desktop -- neither user-friendly nor easily reproducible. so let's bootstrap from an existing Nix install. first, provision a machine (either a NixOS installation or something with the nix</code> binary installed). the architecture doesn't matter -- i'll assume it's x86_64 and show you how to cross-compile. create a new directory for the work mkdir nixos-pinephone-getting-started && cd nixos-pinephone-getting-started</code></pre> each section in this article corresponds to one commit in this</a> repository in case you want to skip the commentary. Bare-minimal Build</h2> for our Pinephone machine, we'll let mobile-nixos</a> do the heavy lifting. create a minimal flake.nix</code>: { inputs = { # nixpkgs.url = "nixpkgs/nixos-22.05"; nixpkgs.url = "nixpkgs/dfd82985c273aac6eced03625f454b334daae2e8"; mobile-nixos = { # url = "github:nixos/mobile-nixos"; url = "github:nixos/mobile-nixos/efbe2c3c5409c868309ae0770852638e623690b5"; flake = false; }; }; outputs = { self, nixpkgs, mobile-nixos }: { pinephone-img = (nixpkgs.lib.nixosSystem { system = "aarch64-linux"; modules = [ (import "${mobile-nixos}/lib/configuration.nix" { device = "pine64-pinephone"; }) ({ ... }: { nixpkgs.config.allowUnfree = true; }) ]; }).config.mobile.outputs.u-boot.disk-image; }; }</code></pre> i've frozen the mobile-nixos commit here because they're in the process of switching from U-Boot to towboot -- the pinned commit is among the last which support U-Boot, which this guide assumes. pinning nixpkgs</code> may be paranoia: try switching to nixpkgs/nixos-<release></code> after you're bootstrapped. if your host machine isn't aarch64, you'll have to enable cross compiling. the trivial way is emulation (i.e. qemu), though it could make the build take a full afternoon depending on how much of your system is available through nixcache. on the host box, add this somewhere in your config (e.g. /etc/nixos/configuration.nix</code>): boot.binfmt.emulatedSystems = [ "aarch64-linux" ];</code></pre> if you added this emulation, rebuild your host machine to enable it (nixos-rebuild --flake /etc/nixos switch</code>). then build the image with: nixos-pinephone-getting-started$ nix build './#pinephone-img'</code></pre> the resulting image is effectively the same as what Hydra spits out in the automation: no users, and no way to login. if you're brand new to the Pinephone, i recommend flashing the image and booting as a sanity check. otherwise, skip to Making the Image More Usable</a> where we'll build a more usable image. at any time, consult the Troubleshooting</a> section if you hit something unexpected. Flashing the Image</h3> you might be tempted to flash this to the eMMC instead of an SD card, thinking that the former will be more reliable storage. it's not: my eMMC began to fail within 20 write cycles. i highly recommend you use an SD card for this. sudo dd if=$(readlink ./result) of=/dev/sdb bs=4M oflag=direct conv=sync status=progress</code></pre> oflag=direct</code> makes the progress bar actually usable -- otherwise it'll just show how much data has been passed to the kernel -- and conv=sync</code> seems to deal better with low quality SD cards (it feels like paranoia, but if i fsck -f</code> the result it sometimes shows corruption without this flag). insert the card to the Pinephone and boot it. the SD slot takes precedent over the eMMC in its boot sequence, so nothing more is needed. you should see the power indicator turn red (indicating that U-Boot is active), then yellow (u-boot has run the mobile-nixos boot script), then green (NixOS stage 1). you'll see a mobile-NixOS splash screen, it'll take a minute to validate the fs, and then boot into stage 2 where you'll be stuck at a login prompt. Making the Image More Usable</h2> we'll want to rebuild the image and include a user, desktop environment, and some basic applications. i tried Phosh, Plasma Mobile, and Gnome: of these, Phosh works the best OOTB by far (Plasma Mobile is slow and crash-prone, Gnome lacks an on-screen keyboard outside the core apps and its navigation is less tailored to phones). we'll use home-manager to make user setup a bit nicer. add that to flake.nix</code>: inputs = { nixpkgs.url = "nixpkgs/dfd82985c273aac6eced03625f454b334daae2e8"; mobile-nixos = { url = "github:nixos/mobile-nixos/efbe2c3c5409c868309ae0770852638e623690b5"; flake = false; }; + home-manager.url = "github:nix-community/home-manager/release-22.05"; };</code></pre> add a module for this pinephone build: - outputs = { self, nixpkgs, mobile-nixos }: { + outputs = { self, nixpkgs, mobile-nixos, home-manager }: { pinephone-img = (pkgs-mobile.lib.nixosSystem { system = "aarch64-linux"; + specialArgs = { inherit home-manager; }; modules = [ (import "${mobile-nixos}/lib/configuration.nix" { device = "pine64-pinephone"; }) - ({ ... }: { - nixpkgs.config.allowUnfree = true; - }) + ./modules/default.nix ]; }).config.mobile.outputs.u-boot.disk-image; ...</code></pre> and then populate modules/default.nix</code>. you could just throw everything in here, but it's more readable (and reusable -- in case you want to share this config across machines) if you split it up: { ... }: { imports = [ ./hardware.nix ./home-manager.nix ./phosh.nix ./users.nix ]; system.stateVersion = "22.05"; nixpkgs.config.allowUnfree = true; }</code></pre> modules/hardware.nix</code>: { ... }: { ## enable the hardware rotation sensor hardware.sensor.iio.enable = true; hardware.opengl.enable = true; hardware.opengl.driSupport = true; }</code></pre> modules/home-manager.nix</code>: { pkgs, home-manager, ... }: { imports = [ home-manager.nixosModule ]; home-manager.useGlobalPkgs = true; home-manager.useUserPackages = true; home-manager.users.colin = { home.stateVersion = "21.11"; home.username = "colin"; home.homeDirectory = "/home/colin"; programs = { home-manager.enable = true; firefox.enable = true; git.enable = true; }; # a few useful packages to start with home.packages = with pkgs; [ # useful CLI/admin tools to have during setup fatresize gptfdisk networkmanager sudo vim wget # it's good to have a variety of terminals (x11, Qt, GTK) to handle more failures xterm plasma5Packages.konsole gnome.gnome-terminal ]; }; }</code></pre> modules/phosh.nix</code>: { ... }: { services.xserver.desktopManager.phosh = { enable = true; user = "colin"; group = "users"; }; environment.variables = { # Qt apps won't always start unless this env var is set QT_QPA_PLATFORM = "wayland"; }; }</code></pre> modules/users.nix</code>: { ... }: { users.mutableUsers = false; users.users.colin = { isNormalUser = true; home = "/home/colin"; uid = 1000; # make this numeric so that you can enter it in the phosh lockscreen. # DON'T leave this empty: not all greeters support passwordless users. initialPassword = "147147"; extraGroups = [ "wheel" ]; }; security.sudo = { enable = true; wheelNeedsPassword = false; }; services.openssh = { enable = true; permitRootLogin = "no"; passwordAuthentication = true; }; }</code></pre> build and flash the image as before. this is the final image we'll flash, so before you eject the card resize the rootfs to make use of the full device. do NOT use fdisk</code> or parted</code> for this. it disrupts some on-disk structures required by the bootloader (this won't be a worry after mobile-nixos officially switches to tow-boot). instead, use the ncurses frontend to fdisk: cfdisk</code>: $ sudo cfdisk /dev/sdb scroll to /dev/sdb4 > Resize leave at default (28.8G) > Write type "yes" > Quit</code></pre> eject the card and boot the phone, login, and toy around with it. open the display settings and mess with the scale (it defaults to 200%). i find 150% is best, particularly so that Firefox doesn't overflow. Building Generations on the Device</h2> we don't want to re-flash the device every time we change something. let's update the flake to allow on-device updates. - outputs = { self, nixpkgs, mobile-nixos, home-manager }: { - pinephone-img = (nixpkgs.lib.nixosSystem { + outputs = { self, nixpkgs, mobile-nixos, home-manager }: rec { + nixosConfigurations.pinephone = (nixpkgs.lib.nixosSystem { system = "aarch64-linux"; modules = [ (import "${mobile-nixos}/lib/configuration.nix" { device = "pine64-pinephone"; }) ./pinephone.nix ]; - }).config.mobile.outputs.u-boot.disk-image; + }); + pinephone-img = nixosConfigurations.pinephone.config.mobile.outputs.u-boot.disk-image; nixosConfigurations.host = nixpkgs.lib.nixosSystem { system = "x86_64-linux"; ...</code></pre> you may need network access for this. unless you connect a USB-C keyboard, the graphical network manager might not be usable. in that case, open a terminal and run: sudo nmcli device wifi connect <wifiname> password <password></code></pre> copy your flake over to the phone at /etc/nixos/flake.nix</code>. i use git for this. on the phone (or over ssh -- try ssh nixos</code> from the same LAN or use ip addr</code> to find its IP): ~$ git clone https://git.uninsane.org/colin/nixos-pinephone-getting-started.git ~$ sudo rmdir /etc/nixos && sudo mv nixos-pinephone-getting-started /etc/nixos ~$ cd /etc/nixos /etc/nixos$ sudo git config --global --add safe.directory /etc/nixos /etc/nixos$ sudo nixos-rebuild --flake "./#pinephone" switch</code></pre> validate this with a reboot, and you should be golden! i recommend mirroring your /etc/nixos</code> folder to at least one other place: a github repo, rsync</code> to a desktop, duplicity</code> to cloud storage, etc. that way if you ever brick your phone you can restore from your latest config using the dd</code> approach. managing generations and rolling back to a good one doesn't seem quite as easy to do on a phone. happy nixing :-) Troubleshooting</h2> device is unbootable or unflashable (eMMC isn't exposed as a /dev node over USB):</li> </ul> these issues tend to be power-related. flash a minimal, known-good image (like postmarketOS</a>) to the SD card and boot with the battery and USB charger plugged in. wait until the battery gets to a good charge and resume. if it's still not working, try holding the reset button underneat the back cover of the phone for 10s. fs appears to be corrupted:</li> </ul> validate the built, but unflashed, image: # create a loopback device from our .img file: # ./result should be a symlink to /nix/store/<hash>-pine64-pinephone_full-disk-image.img $ sudo losetup -Pf $(readlink ./result) # check the rootfs partition (partition #4): # you might see a single, inconsequential error about "Padding at end of inode bitmap is not set." $ sudo fsck -f /dev/loop0p4 # close the loopback device: $ sudo losetup -d /dev/loop0</code></pre> then flash the image using the dd flags i show in the article. while the SD card is still attached to the host, validate it with sudo fsck -f /dev/sdb4</code>. finally, the eMMC has a dozen or so write cycles: prefer an SD card. the nix config won't build:</li> </ul> nixos moves fast, but does occasionally break. you might need to freeze the nixpkgs to a specific commit. try cloning the repo</a> for this post and building it directly -- the repo includes flake.lock</code> so that it should be more reproducible. Additional Resources:</h2> Tom Fitzhenry building a minimal Pinephone image</a> by dropping his config straight into a mobile-nixos checkout, no flake.nix</code> required.</li> Ana, Hoverbear on making an install image</a> from an existing NixOS configuration.</li> Cole Mickens' public nix config</a> shows how to maintain a shared repo for multiple machines with more advanced things like custom packages, modules, and secrets.</li> my own nix config</a> which targets multiple machines with one of these operating a nix cache to speed up pinephone builds.</li> Pavel Makhov showing a better way to distribute builds across machines</a> which i haven't got around to trying.</li> the NixOS Matrix space</a> which contains chat rooms like #nixos-on-arm:nixos.org</code>.</li> or contact me directly via the links in my about</a> page.</li> </ul> The Jackson Pad: DIY Vibration Damping for Drumsets 2022-05-25T00:00:00+00:00 a few years ago i moved out of student apartments and into the second story of a commuter-centric apartment. i wanted to stay on my neighbors' good side, so i ditched my acoustic set and bought an electronic one. shortly after, i received my first ever noise complaint! although electronic sets are audibly quieter, they still transmit a lot of energy into the floor. when struck, vibrations carry from the drum into the harness and from there into the floor. moreover, the kick and high-hat pedal action mirrors that of a person stomping, so of course that's going to annoy the downstairs neighbor. How to decrease vibrations</h2> the most straightforward way to decrease floor vibrations is to decrease the energy associated with each hit. for example, use beaterless pedals</a>. if there's no beater imparting energy into the kick plate then that's less energy transmitted into the floor. similarly, slamming the high-hat close imparts momentum from the upper cymbal into the stand and from there to the floor. beaterless pedals solve both these issues. but even with beaterless pedals (if you're willing to go that route), you still have the issue of snare/tom hits vibrating through the harness. if you want to stop all these, you'll need an isolation platfrom like this</a>: or you can place the set atop a riser: i was very tempted by that Thomann podium, but it's priced a bit too aggressively for me. cheaper options include placing the set on a large rubber or foam pad, similar to what you might put under a washing machine or an exercise bike ("anti vibration mats"). but after browsing drum forums i couldn't find many success stories from these approaches. maybe that's just a bias from the people who frequent these forums and anti-vibration mats work equally well, but i was intrigued by the other approaches i saw there. DIY solutions</h2> the most infamous DIY approach within these drum communities is the tennis ball riser: sandwich tennis balls between two large panels (or a panel and the floor), put the drumset on top, and you're done. it's a simple suspension system and actually resembles that commercial Thomann podium quite a bit. a more thoroughly engineered solution i came across was The Jackson Pad. the creator (Brian Jackson) provides detailed build instructions, some convincing noise measurements, a believable theory of operation, and i couldn't find anyone who tried this without success. The Jackson Pad</h2> initially proposed on vdrums.com</a>, i've mirrored the original design document</a> and build instructions</a>. the first idea is that the drumset should sit atop a platform that has a lot of mass. because of the high mass, the momentum you impart onto it when stomping your feet causes less physical displacement than a platform with low mass. the second idea is to set this platform atop four air-filled inner tubes. the displacement of the platform will compress the air within the tubes, redirecting some of the motion outward instead of downward and supposedly transforming mechanical energy into heat energy (compressing a gas causes it to heat): the heat will be transformed back into mechnical energy as the pressures balance, but the effect is that sharp impulses of mechanical energy are imparted into the floor over a longer period of time. it's like applying a low-pass filter to the original foot stomp or drum hit. i'm a little skeptical of the heat argument, but overall i'm satisfied enough with the theory of operation to give it a try. Construction</h2> the pad materials are super practical to source. it's all off-the-shelf lumber that you can buy at a hardware store with a minimal number of cuts. the stock 4'x4.7' design is just large enough for a typical 4-drum + highhat + kick + ride + crash configuration and a stool with modest footprint. unfortunately, it's just large enough to not be practical to transport. i move homes every year or two, so i wanted to find a way to make it fit in my stationwagon. the stock plan makes use of four 55.5" trusses which support fourteen 2x4s running perpindicular to them: i decided to make a vertical cut down the center of this entire design such that i would have two separate 48"x27.25" pads. each of the four trusses would be cut in half, and a brace would be added to each one which would secure the halves when bolted. with the bolts in place, you're left with a solid 48"x55.5" pad; with the bolts removed, you have two smaller pads that can be easily transported. the original design leaves four 40" boards of scrap from cutting the trusses. you can trim these down to 2' and use those as the braces. the only addition to Jackson's BOM is sixteen 4" bolts, washers, and wingnuts. here's what the truss + brace system looks like when bolted together: from here on, i just follow Jackson's original plans. when constructing the platform, keep the bolts in place and secure the 2x4s atop each truss as normal (make sure to secure the slats onto the trusses, and not the braces -- otherwise you won't be able to detach the braces!) here's what it looks like from the underside: you'll want to label the braces now so that you know which brace belongs to which truss when you reassemble it later (even though they should have the same cuts, in practice you won't have lined up the holes identically in each one). i just marked them such that they spell a word when correctly paired: if you want carpeting, do that while the platform is assembled. you can cut one piece of carpet to the full 48"x55.5", staple it, and let it fold inward during disassembly. it'll take two people to carry it this way, but it probably looks nicer than if you carpeted each half separately. Disassembly & Reassembly</h2> when you're ready, just unbolt the braces, slide them off, and fold the platform: to setup the pad somewhere more permanent, first situate the isolation pedestals, then reattach the braces and lower the pad onto the pedestals. and you're good to go :) this pad has served me for four years and at three different homes with zero noise complaints. i refill the tubes yearly, and very imprecisely. you really just need to fill them to capacity and the 200-300 pounds sitting atop them should pressurize it all decently. Rescuing a Broken EXT4 System with Ext4Magic and dd 2022-04-14T00:00:00+00:00 while i was setting up this machine, i made some mistakes along the way and observed the dreaded fsck messages on boot: [kernel boot output] ... /dev/sda1 contains a file system with errors, fsck required. dropping into emergency shell > </code></pre> so i ran fsck</code> on the disk like the computer told me to: # fsck /dev/sda1 [...] Inode 5202 has an invalid extent (logical block 0, invalid physical block 2250752, len 2048) Clear ('a' enables 'yes' to all) <y>? yes to all</code></pre> and then i typed a</code> ("yes to all") like a noob, expecting that any cornerstone tool of Linux in 2022 would act sanely. i mean, it didn't give me any other apparent option: i had to complete fsck</code> before it would let me boot, and the only option fsck</code> gives me on each error is yes</code> or yes to all</code>. so, obviously i'm supposed to let it fix everything itself. Fsck fsck</h2> so i let it whir. and it trashed everything. if i had properly read the output, i maybe would have understood that it was simply deleting ("clearing") everything on the disk that it didn't understand. but i'm an imperfect user, and i was in a hurry. so bye-bye /opt/pleroma</code>. bye-bye /home</code>. bye-bye arbitrary chunks of /var/lib/postgres/data/base/50616</code>. i didn't have backups in place: i had naively decided to tackle that after i finished installation and had a clear sense of how this system would be structured long-term. so do i just live with this, and redo two days of work? hell no. i'd rather spend three days diving into EXT4 internals than redo anything. so i saved the fsck output, attached (but didn't mount) the drive to a clean system, and got busy. What Did fsck Actually Do?</h2> at the start of the fsck run was this: Resize inode not valid. Recreate<y>? yes</code></pre> the remainder of the messages were mostly of two forms: Entry 'admin' in /home (8196) has invalid inode #: 2234378. Clear? yes</code></pre> and Inode 4574 has an invalid extent (logical block 0, invalid physical block 4031998, len 1) Clear<y>? yes Inode 4574, i_blocks is 8, should be 0. Fix<y>? yes</code></pre> in fact, i had done an earlier resize2fs</code> to expand the 8 GiB FS to fit its 2 TiB partition. the docs say you can do this on a live filesystem, but... caveat emptor? EXT4 defaults to a block size of 4096B (i.e., a traditional page of RAM). physical blocks are a direct reference to some offset into the underlying device. so "physical block 4031998" corresponds to the byte range on the device of 16515063808 - 16515067904. inodes are indexed by their physical block address as well, so inode # 2234378</code> corresponds to the block starting at byte index of 9152012288. notably, these indexes are both beyond the original 8 GiB fs size. this holds true of every inode and data block fsck complained about. there's a good chance that all/most of the actual data/inode blocks still hold valid data on-disk, and the EXT4 drivers simply didn't understand their address. we have two tasks here: recover the unlinked inodes (i.e. directory entries).</li> recover the cleared extents (i.e. data blocks within a file).</li> </ol> there's a purpose-built tool for #1, and we can script our own thing for #2. Recovering Inodes with Ext4Magic</h2> ext4magic</a> is a tool to manage data loss like this. one of its modes is ext4magic -I <inode> -R <device></code>, wherein you pass it an inode #, it parses the inode data structure off the disk, and then makes a best-effort attempt to recover everything in the fs tree at, or referenced by, that inode. so for the missing /home/admin</code> directory, i need only run: # extmagic -I 2234378 -R -d recovered-inodes /dev/sda1</code></pre> and out pops the entire directory tree for the admin directory. e.g. $ tree recovered-inodes └── <2234378> └── gitea ├── assets │ ├── emoji.json │ └── logo.svg ├── BSDmakefile ├── build │ ├── code-batch-process.go │ ├── codeformat │ │ ├── formatimports.go │ │ └── formatimports_test.go │ ├── generate-bindata.go │ ├── generate-emoji.go ...</code></pre> everything under that <2234378></code> even has the correct group/owner/permission bits. you just need to rename <2234378></code> to admin</code>, chown</code> it to what it was before, and then link it back into /home</code> in your fs (but don't do that yet: put this in some staging directory and link everything back in only after all the data has been recovered). repeat this for all the "invalid inodes" referenced in the fsck output. then we'll recover the data blocks. Recovering Data Blocks: EXT4 Data Structures</h2> the 2nd class of message was: Inode 4574 has an invalid extent (logical block 0, invalid physical block 4031998, len 1) Clear<y>? yes Inode 4574, i_blocks is 8, should be 0. Fix<y>? yes</code></pre> to understand that this even is recoverable, it helps to understand the ext4 inode structure. inodes are on-disk data structures, one for every directory entry on the system. an inode might represent a file or a directory. they look similar in both cases, but we only care about inodes which represent files here. each inode is a fixed-size structure holding metadata, like file type/mtime and -- notably -- file size, and then they link to a dynamically-sized sequence of "extents"; roughly, pointers to where the file data lives on disk. the English translation is like "data bytes 0-32768 occupy the physical blocks starting at block 4156555; bytes 32768-36864 occupy the physical blocks starting at block 6285112". this is all represented in terms of blocks, so based on the file length the last block may only be partially filled with data. EXT4 (and many file systems) largely keeps the file data entirely outside of the inode structure. fsck tells us that it cleared the extent entries, but not the actual data blocks. i_blocks</code> here refers to the blocks allocated to the inode for storing its variably-sized data, i.e. the list of extents (for what seems to be legacy reasons, this is denoted in 512B disk sectors instead of FS blocks). so, all the inode metadata is still here; the data blocks exist but are unlinked, and only the extents were lost. if you try reading the file, it'll still present its original length of data, but will show a block's worth of zeros for every logical block whose extent was cleared. Recovering Data Blocks</h2> so we just need to link the data blocks back into the extents structure. we could dive deeper into EXT4 data structures and twiddle those bits, but that would lead us into having to understand the inode and block allocators. instead, we can just dump the block-level data, and use fs-level APIs to put it back. Ext4Magic -B <block></code> will dump the full 4096 bytes of some physical block. but because the physical block is a direct index into the device, we can also just use dd</code>. for example, let's recover this cleared extent: Inode 4574 has an invalid extent (logical block 0, invalid physical block 4031998, len 1) Clear<y>? yes Inode 4574, i_blocks is 8, should be 0. Fix<y>? yes</code></pre> first, we'll want to know which file this comes from: $ mkdir preserved # mount the drive READ-ONLY: $ sudo mount -o ro /dev/sda1 preserved $ find preserved/ -inum 4574 preserved/etc/passwd</code></pre> that's, uh, an important file. does the data block still hold proper data? $ dd if=/dev/sda1 of=/dev/stdout bs=4096 skip=4031998 count=1 root:x:0:0::/root:/bin/bash bin:x:1:1::/:/usr/bin/nologin daemon:x:2:2::/:/usr/bin/nologin mail:x:8:12::/var/spool/mail:/usr/bin/nologin ftp:x:14:11::/srv/ftp:/usr/bin/nologin http:x:33:33::/srv/http:/usr/bin/nologin nobody:x:65534:65534:Nobody:/:/usr/bin/nologin [...] <NUL><NUL><NUL>[...]</code></pre> yes! let's set up a scratch space. we can construct an overlay of our rootfs where we place all the recovered and patched entries, and then apply that to the original device once we're done recovering. $ mkdir recovered</code></pre> go ahead and manually link all the entries we recovered with ext4magic -I</code> earlier into this recovered</code> directory and fix up their group/owner/permissions. now we can patch individual files by copying them from preserved/<path></code> to recovered/<path></code> and then dd</code>ing specific byte ranges from /dev/sda1</code> into recovered/<path></code>. for example: $ mkdir -p recovered/ext $ sudo cp preserved/ext/passwd recovered/ext/passwd $ sudo dd if=/dev/sda1 of=recovered/ext/passwd bs=4096 skip=4031998 count=1 $ ls -l preserved/ext/passwd -rw-r--r-- 1 root root 3528 /etc/passwd $ sudo truncate --size=3528 recovered/ext/passwd</code></pre> because dd</code> copies the whole block, we have that additional step of truncating the file to its original size. Bringing It Together</h2> we've successfully recovered (into the recovered</code> directory): all unlinked directory entries.</li> the cleared extent in /etc/passwd</code>.</li> </ol> we still need to: recover all other cleared extents.</li> link the recovered data back into the real file system.</li> </ol> step 2 is a simple rsync</code>. step 1 is some nasty dd</code> work. i demoed it for a file with only one cleared extent, but some files have many cleared extents, often non-contiguous. assume the presence of a script patch_file.py</code> (see Appendix</a>) which takes: an inode number (4574</code>)</li> a file path (etc/passwd</code>)</li> the first logical block of the extent which was cleared (0</code>)</li> the length in blocks of the extent which was cleared (1</code>)</li> the first physical block containing the extent's data (4031998</code>)</li> </ul> then we can parse the fsck output and script the rest of step 1. # fsck /dev/sda1 [...] Inode 34215 has an invalid extent (logical block 14, invalid physical block 2207227, len 1) Clear? yes Inode 58213 has an invalid extent (logical block 0, invalid physical block 3456000, len 1024) Clear? yes Inode 58213 has an invalid extent (logical block 1024, invalid physical block 3463168, len 623) Clear? yes Inode 58213, i_blocks is 13176, should be 0. Fix? yes Inode 58222 has an invalid extent (logical block 0, invalid physical block 2207151, len 1) Clear? yes Inode 58222, i_blocks is 8, should be 0. Fix? yes [...]</code></pre> run find -i <inode> preserved/</code> on each of these inodes to find the file they correspond to, and then you can create this script from that snippet of fsck output: ./patch_file.py -i 34215 -f var/log/pacman.log 14,1,2207227 ./patch_file.py -i 58213 -f usr/bin/yay 0,1024,3456000 1024,623,3463168 ./patch_file.py -i 58222 -f etc/fstab 0,1,2207151</code></pre> sometimes find</code> won't find the inode that fsck updated. for example, if you booted the system after running fsck</code>, Linux will notice that certain files have been corrupted and will update them with placeholders, destroying the original inode. these are usually the more important files, so you can dump the data block with that dd</code> command and compare it to notable entries on a good file system to "guess" what it was originally. since we don't have the original inode, we lost the metadata like its length, so use the --auto-len</code> flag to guess the length by trimming zero's off the original data block. take this snippet of fsck output: Inode 4997 has an invalid extent (logical block 0, invalid physical block 3831801, len 1) Clear<y>? yes Inode 4997, i_blocks is 8, should be 0. Fix<y>? yes</code></pre> try to find the file: $ find -i 4997 preserved/ # (no output)</code></pre> but we dump physical block 3831801 and notice that it looks a lot like /etc/shadow</code>. so: ./patch_file.py -i 4997 --auto-len -f etc/shadow 0,1,3831801</code></pre> once you've patched all the files, then bring the file system back online, writeable, and copy over your changes. $ sudo umount preserved $ mkdir sda1 $ sudo mount /dev/sda1 sda1 $ rsync -av --checksum recovered/ sda1/ $ sync && sudo umount sda1</code></pre> if all went well, you can boot the disk now. cheers 🍻 Appendix</h2> the patch_file.py</code> script: #!/usr/bin/env python3 ''' replaces zero-pages, or partial zero-pages within a single file ''' import os import subprocess import sys PAGE_LEN = 4096 IN_DIR = 'preserved' OUT_DIR = 'recovered' def patch_range(file_: str, logical_block: int, n_blocks: int, physical_block: int): ''' patch a whole range of blocks within the file ''' subprocess.check_output([ 'dd', 'if=/dev/sda1', f'of={file_}', 'bs=4096', f'seek={logical_block}', f'skip={physical_block}', f'count={n_blocks}', ]) def copy_for_patch(path: str) -> str: in_path = os.path.join(IN_DIR, '.', path) out_path = os.path.join(OUT_DIR, path) subprocess.check_output(['rsync', '-a', '--relative', in_path, OUT_DIR + '/']) return out_path def estimate_length(path: str) -> int: ''' return the length of the file were there to be no trailing bytes ''' contents = open(path, 'rb').read() l = len(contents) while l and contents[l-1] == 0: l -= 1 return l def main(path: str, auto_len: bool, patches: list): path = copy_for_patch(path) old_size = os.stat(path).st_size for patch in patches: logical_block, n_blocks, physical_block = patch patch_range(path, logical_block, n_blocks, physical_block) if auto_len: os.truncate(path, estimate_length(path)) else: os.truncate(path, old_size) def parse_args(args: list): ''' return: str: the relative file being operated on, bool: auto-estimate len, list: the ranges to patch ''' i = 0 inode = None file_ = None auto_len = False ranges = [] while i < len(args): arg = args[i] if arg == '-i': inode = int(args[i+1]) i += 2 elif arg == '-f': file_ = args[i+1] i += 2 elif arg == '--auto-len': auto_len = True i += 1 else: logical_block, n_blocks, physical_block = map(int, arg.split(',')) #vvv not actually required, but indicative of an error assert logical_block < physical_block ranges.append((logical_block, n_blocks, physical_block)) i += 1 # inode doesn't actually get used # it's useful just to keep the script invocations organized return file_, auto_len, ranges if __name__ == '__main__': main(*parse_args(sys.argv[1:]))</code></pre> A Reasonably Secure Mailserver Installation 2022-04-06T00:00:00+00:00 i need software to receive emails, and possibly to send them too. i.e., a mailserver. the mature mailserver implementations were all written in a time where security was even worse than today. Postfix is among the better ones, but even it has a fair number of CVEs</a>. its intended operation -- where it writes to mailboxes owned by different users -- relies on elevated access control. although the risks are mitigated by its design around separation of concerns -- where only select portions of code get elevated permissions -- and the linux capabilities system, i still would not feel comfortable running this without isolating it from other applications on the same machine. enter systemd-nspawn. nspawn is an extremely lightweight container. it's more of a transparent chroot: package up the userspace of some linux distribution, place it in a directory, and then nspawn uses the host kernel and performs the whole PID 1 boot sequence of that chroot, virtualizing all the fs access and isolating the processes/etc. we'll use this to create a container dedicated to postfix. Installation</h2> start by creating the rootfs and launching it as a container. this assumes the host is running Arch: [root@host /]# pacman -S arch-install-scripts [root@host /]# mkdir /opt/postfix [root@host /]# pacstrap -c /opt/postfix base postfix openbsd-netcat opendkim perl [root@host /]# systemd-nspawn -D /opt/postfix ># passwd # choose some password you can remember for the rest of setup ># exit</code></pre> if you're ssh'd into the host, you need to relax some security settings</a> in the container before it'll let you login: [root@host /opt/postfix]$ mv etc/securetty etc/securetty.OLD [root@host /opt/postfix]$ mv usr/share/factory/etc/securetty \ usr/share/factory/etc/securetty.OLD # then comment out the line containing securetty in usr/lib/tmpfiles.d/arch.conf</code></pre> configure myhostname</code> and mydomain</code> in /opt/postfix/etc/postfix/main.cf</code> (it's fine for these to be the same: i use uninsane.org</code> for both) using the --network-veth</code> flag, systemd will create a NAT'd network and expose the downstream to the container. we can then forward ports across the NAT just like you would forward ports from your router to your PC/server (port 25 here is the SMTP port): [root@host /]# systemd-nspawn -b --network-veth -p 25:25 -D /opt/postfix postfix login: root Password: <enter it> [root@postfix ~]# systemctl enable --now systemd-resolved [root@postfix ~]# systemctl enable --now postfix # then create the db which maps email address to linux user accounts: [root@postfix ~]# newaliases</code></pre> for the record, /etc/postfix/aliases</code> contains the mappings consumed by newaliases</code>. the defaults work for us now but you'll want to tweak them later. like HTTP, the SMTP grammar is human friendly. we can verify our setup with netcat</a>. you can do this from within the container (substitute localhost</code> for <container></code>), or from another box on your LAN (substitute the host's IP/name for <container></code>). you probably don't want to expose this to the WAN yet: $ nc <container> 25 helo uninsane.org mail from:<test@uninsane.org> rcpt to:<root@uninsane.org> data this is a test. . quit</code></pre> mail should show up in the container at var/spool/mail/root</code>. if this is intended as a single-user mailserver, you might want a catch-all mail rule: [root@postfix /]# echo '@uninsane.org root' >> /etc/postfix/virtual [root@postfix /]# echo 'virtual_alias_maps = hash:/etc/postfix/virtual' >> \ /etc/postfix/main.cf [root@postfix /]# postmap /etc/postfix/virtual [root@postfix /]# systemctl restart postfix</code></pre> try the nc</code> command from above again, but use rcpt to:<anything@uninsane.org</code> and the mail should be appended to that same /var/spool/mail/root</code> file. Non-Root User</h2> we'd prefer to be able to read mail without being root. so create a user dedicated to holding the mailboxes: [root@postfix /]# useradd --create-home --user-group vmail</code></pre> edit etc/postfix/main.cf</code>: - mail_owner = postfix + mail_owner = vmail</code></pre> edit etc/postfix/aliases</code>: - root: you + root: vmail</code></pre> edit etc/postfix/virtual</code>: - @uninsane.org root + @uninsane.org vmail</code></pre> update the database mappings and then restart the services: [root@postfix /]# newaliases [root@postfix /]# postmap /etc/postfix/virtual [root@postfix /]# postfix set-permissions [root@postfix /]# systemctl restart postfix</code></pre> the postfix</code> Arch package includes the /var/spool</code> files which are now owned by vmail</code>, and AFAICT Arch fixes package permissions on each boot. so for these changes to take permanent effect, you'll need to edit lib/systemd/system/postfix.service</code> to apply set-permissions</code> on each boot: - ExecStart=/usr/bin/postfix start + ExecStart=/usr/bin/bash -c '/usr/bin/postfix set-permissions \ + && /usr/bin/postfix start'</code></pre> because systemd limits postfix's ability to write outside of /var/spool</code>, you'll need to change which files postfix tries to enforce permissions on if you want this to succeed. in etc/postfix/postfix-files</code>, comment out every line which starts with one of: $config_directory</code></li> $daemon_directoy</code></li> $sample_directory</code></li> $readme_directory</code></li> $html_directory</code></li> $shlib_directory</code></li> $manpage_directory</code></li> </ul> since Arch manages these (correctly), you're not really losing anything. run that nc</code> command again: this time mail should show up in /var/spool/mail/vmail</code>, and that file should be owned by the vmail</code> user instead of root</code>. now we can work on the WAN side of things. to prevent spoofing & improve the likelihood that your messages will be accepted by other servers, you'll want to add some DNS records to your zone file: a SPF DNS record (instructs recipients to enforce that your message originates from a specific IP)</li> a DKIM DNS record (signs the message content with a key owned by your mailserver)</li> a DMARC DNS record (allows you to receive reports from recipient mailservers)</li> </ul> and of course you'll need a MX record so others know where to send mail. DKIM and DNS</h2> we installed opendkim during the earlier pacstrap</code> invocation: now we'll configure it to sign outgoing messages: [root@host /opt/postfix]$ cp usr/share/doc/opendkim/opendkim.conf.sample \ etc/opendkim/opendkim.conf</code></pre> open etc/opendkim/opendkim.conf</code> in an editor and: update the Domain</code> field</li> point the KeyFile</code> to /home/vmail/dkim/mx1.private</code> (created later)</li> set UserID</code> to vmail</code></li> make sure Socket</code> points to inet:8891@localhost</code></li> and consider changing Canonicalization from simple/simple</code> to relaxed/simple</code></li> </ul> then append this to etc/postfix/main.cf</code>: # For use by dkim milter smtpd_milters = inet:localhost:8891 non_smtpd_milters = $smtpd_milters milter_default_action = accept</code></pre> generate the keys (run this as the vmail</code> user): [vmail@postfix /home/vmail]$ mkdir dkim && cd dkim [vmail@postfix /home/vmail/dkim]$ opendkim-genkey -r -s mx1 -d uninsane.org</code></pre> start the service: [root@postfix /]# systemctl enable --now opendkim</code></pre> add the mx1._domainkey</code> TXT record (documented in /home/vmail/dkim/mx1.txt</code>) into your zone file. then run the nc</code> example again. you should get mail that has an Authentication-Results</code> header -- which fails, since we didn't sign our message. using the postfix sendmail</code> command we should be able to send something with a valid signature: [root@postfix /]# sendmail test@uninsane.org this message should be signed . [root@postfix /]# cat /var/mail/vmail [...] DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=uninsane.org; s=mx1; t=[...]; bh=[...] h=Date:From; b=[...] Message-Id: <YYYYMMDDTTTTTT.NNNNNNNNNNN@uninsane.org> Date: [...] From: root@uninsane.org this message should be signed </code></pre> then add a SPF DNS record and a DMARC record to receive delivery reports. if you're running a large mail server it would be good to install opendmarc</code> to send delivery reports to other servers (like mine!), but i'll skip that here. throw in the MX record, and your zone file should look like this: ; mailserver shares an IP with the rest of uninsane.org. @ MX 10 uninsane.org. ; Sender Policy Framework: ; +mx => mail passes if it originated from the MX ; +a => mail passes if it originated from the A address of this domain ; +ip4:.. => mail passes if it originated from this IP ; -all => mail fails if none of these conditions were met @ TXT "v=spf1 ip4:203.0.113.1 a mx -all" ; DKIM public key: mx1._domainkey TXT "v=DKIM1; k=rsa; s=email; p=<big long string>" ; DMARC fields <https://datatracker.ietf.org/doc/html/rfc7489>: ; p=none|quarantine|reject: what to do with failures ; sp = p but for subdomains ; rua = where to send aggregrate reports ; ruf = where to send individual failure reports ; fo=0|1|d|s controls WHEN to send failure reports ; (1=on bad alignment; d=on DKIM failure; s=on SPF failure); ; Additionally: ; adkim=r|s (is DKIM relaxed [default] or strict) ; aspf=r|s (is SPF relaxed [default] or strict) ; pct = sampling ratio for punishing failures (default 100 for 100%) ; rf = report format ; ri = report interval _dmarc TXT ( "v=DMARC1;p=quarantine;sp=reject;fo=1:d:s;" "rua=mailto:admin+mail@uninsane.org;ruf=mailto:admin+mail@uninsane.org" )</code></pre>Validation</h2> validate your DMARC record (and DKIM, SPF if you want): https://dmarcian.com/dmarc-inspector/</a>. confirm that your domain/IP isn't blacklisted: https://multirbl.valli.org/</a>. try sending/receiving mail: https://www.appmaildev.com/en/dkim</a>. if these fail, check journalctl -u postfix</code>. if there's no indication of traffic, it may be that your ISP blocks outbound port 25. you can check for that with nc -vz gmail.com 25</code> (will exit 0 if the port is open, hang if the port is blocked). less probably, your ISP might block inbound port 25. check for that here: https://canyouseeme.org/</a>. in my case, Centurylink blocks both directions, so i can't even use this setup to receive mail. for this case, i'll explore running postfix on a non-standard port and using a mail forwarder or transparent proxy in a subsequent blog post. but if your mail server is working, then instruct systemd to launch the container when the host boots. while the container's active, run: [root@host /]# ln -s /opt/postfix /var/lib/machines/postfix [root@host /]# machinectl enable postfix [root@host /]# systemctl enable machines.target</code></pre> alternatively, you could move the whole machine into /var/lib/machines/postfix</code> instead of symlinking it. populate /etc/systemd/nspawn/postfix.nspawn</code> (you may need to create the directory) with the settings we used earlier: [Network] VirtualEthernet=on Port=25:25</code></pre> then you can stop the machine, restart it, and administer it: [root@host /]# machinectl stop postfix [root@host /]# machinectl start postfix [root@host /]# machinectl login postfix</code></pre> you now have a postfix instance which starts on boot and can send/receive mail as long as port 25 is accessible. later on you may want to provide client access in a friendlier way than directly reading the spool or invoking sendmail</code>. that could be done by installing something like dovecot</a>. Self Hosting in More Detail Than You Asked For 2022-03-29T00:00:00+00:00 well i fell down the rabbit hole. this domain hosts 6 services and counting: nginx (serving you this page)</li> gitea (for git hosting/collaboration)</li> Pleroma (for federated shitposting</del> social networking)</li> Matrix (for chat/instant messaging)</li> Jellyfin (for A/V streaming)</li> Trust DNS (for serving the DNS records of all the above)</li> </ul> How It Started: A Brief History of Bitcoin</h2> i have a caffeine problem. first thing in the morning, i brew myself 3 double-shots of espresso. it's not even "make one, drink it, make the second, drink it, make the third": i brew three cups at once and then sip them on the couch while catching up on "The News". i wanted to break the habit. rather, i enjoy the benefits of caffeine, but i dislike the dependency. caffeine's not the only drug in its class, but you have to jump through hoops to obtain any of its cousins. pretty soon i got very accustomed to using Tor, PGP, etc. pretty soon i found myself caring much more about legal systems than before. and about social norms. and about the whole area of political philosophy. and pretty soon i noticed the weird chilling effects in my everyday life. the legal angle, obviously, but also just social inhibition: i don't want to be "that guy" who drags the whole room into a topic only i care about. and so when it comes to the bits of myself which are the most unusual, the most deviant, the most personal, i don't really have a space to explore those things openly. and that disappoints me. What's a "Fediverse"?</h2> there's this thing called Mastodon</a>, oft categorized as a "decentralized Twitter". i tried it a few years ago and had a positive experience, but didn't really have that strong a desire for "social media" at the time. after pandemic madness, or maybe just after experiencing that shrinking social circle that my older friends like to complain about, i'm a little more curious about the social internet than before. the novel thing about Mastodon is that it's "federated". anyone can host their own server and bridge it to the rest of the network. the main protocol it speaks is ActivityPub (AP), and there's a lot of software beyond Mastodon which speaks AP. i discovered Pleroma</a>, which claims support for hosting behind Tor, and i even found a few Tor-bridged instances out there. so i thought i'd set up my own and dive in. How Do I Host This Shit</h2> using Tor as a client is easy: just install the Tor browser and go. running a service behind Tor is slightly more complex, but still fairly easy to understand: run the Tor daemon. it exposes a SOCKS5 proxy service on port 9050. launch Pleroma and tell it to proxy all TCP traffic through that port. now you can make outbound requests to the Fediverse from behind Tor. but you have no public address yet, so you can't get incoming messages. configure Pleroma to listen on some local port. then configure Tor to run some Onion Service that's serviced by this local port. Tor will generate some <hash>.onion address which is now your publicly routable address. to cap: external actors send HTTP/TCP requests to <hash>.onion, these are serviced by Pleroma and the response is sent back through this tunnel. when Pleroma is the initiator of a request, it proxies that to the recipient by tunneling it through a separate Tor SOCKS5 proxy. with this setup you can send messages to anyone on the Fediverse (Tor or clearnet), but you can only receive messages from those who understand .onion addresses. this whole process is helpfully documented in the Pleroma docs</a>. Sounds Pretty Fragile</h2> too much complexity? AHAHA. let's add more. (there are millions of fridges out there running Linux as i write. i just saw somebody post a photo of their oven after its OS crashed. ponder that.) so the worry here is that Pleroma might be tricked or bugged into ignoring the proxy and communicating over the clearnet. we can take inspiration from Whonix</a> for this. set up two machines: the first machine (U) has two NICs. one NIC is connected to the WAN and the other NIC is connected directly to the second machine (D). U runs nothing but a Tor proxy, exposing only the proxy endpoint to D (and relaying traffic from its onion service to D).</li> the second machine (D) has only the one NIC, connected directly to U. there is no way for any traffic to escape the machine except by passing through the Tor proxy.</li> </ul> in actuality, we'll want to restrict D even further: it probably has hardware WiFi or Bluetooth, which is just another vector. so we package up all the application software and throw it inside a VM on D, exposing no IO except that relevant NIC to the VM. congrats, you've got a decently secure, anonymized computing setup. now you have to deal with the fact that even though Pleroma and Mastodon support federation over Tor, it's an optional configuration that pretty much nobody out there enables. plus, the Pleroma frontend requires Javascript, which just means you've shifted the security burden from the server onto the client. You're Telling Me It Was All for Naught?</h2> as if you didn't see it coming. but hey, i'm sure you'll find some way to use all that infrastructure for your... Bitcoin activities. so anyway, give up on your dream of perfect anonymity. you know first-hand now how difficult and restricting that actually is. meditate on why you're spending so much time fiddling with these logic gates and bits and reorient. Self-Hosting Is Fetch</h2> i think this whole Internet thing is maybe just a social playfield? something to do with exploration, connections, creativity, and self-discovery? an open environment wherein anyone with time/dedication can do these things? wait, is that where the Web went? i don't want to oversimplify or aggrandize it (i will anyway), but when i recount my favorite eras of the internet, they're like this: middle school: i built super amateur videogames with my buddies, hosted the downloads + discussion for these on a site we built by hand, and then distributed the binaries + web link by handing out CDs in the school hallway. it was stupidly successful (surely a function of the era).</li> high school: i encountered my first fandom. i wrote amateur music, internet friends made the song art, these things were shared on blogs and Skype and message boards. i attended cons and had the repeat experience of somebody discovering "oh, you're the guy who made that" 10 minutes into one of those late-night hotel-room conversations.</li> college: i maintained some open source projects and blogged about technical/academic topics. people from across the world emailed me private responses that must have taken hours to write. i'd video-chat with people to help them port/extend my software to larger purposes. a professor even assigned my work as reading material for their students.</li> </ul> and i never really got it. but i think it was just simple, social, creativity. and i want more of that in my life. Stripping It Down</h2> i don't really need anonymity for this project, in fact strict anonymity would detract from it. i just need whatever level of pseudonymity helps me to let my guard down (and to not worry about e.g. identity theft). that host machine (D) already has all the stuff we need for a secure-enough system if we strip out the anonymizing function of U. so do that, and use your Pleroma instance to explore the Fediverse. respectably insert yourself into conversations with everyday people and make connections. find some little bug, or missing feature, and create a fix for it. set up a Matrix (or xmpp) instance and reach out to the devs to coordinate. set up a gitea instance in which to host your improved version of the project and from which to initiate a merge request. give yourself your own personal homepage on the Web with a static site builder like Zola. throw all this behind nginx so that you can host these services on different subdomains on the same physical host. use certbot</code>/LetsEncrypt to secure the http traffic in all of 10 minutes. spin up different systemd-nspawn/LXC/Qemu instances to isolate each service, or ditch proper containerization and just embrace seperate, privilege-limited user accounts for each service. you make the call. just remember to take backups seriously, because things will go wrong as you're fiddling with all this stuff. once you're tired of updating DNS subdomain records through your registrar's portal, host your own nameserver. point your toplevel domain to afraid.org</a>'s free & friendly dynamic DNS service if you have an unstable residential IP. at some point, you'll have to deal with email. the state of email on the web is... pretty broken, so i'll forgive you if you settle on gmail/hosted Zoho/etc. really you can -- and maybe should -- skip as many of these components as you want if they don't align with your mission. but just remember that it's you who create the web. this was and can be a person to person network. and there are persons out there who want you in it. if you read this far and want a hand in any of it, reach out to another person. message me through one of the contacts listed on my about page</a>. i promise i'll respond, in all likelihood i'll be happy to share this space with you. Colin

Perfectly Sane

Kernel Hacking + Device Bringup on NixOS

Building a Kernel Module</h3> my kernel module has 3 files, which live in my nix repo inline:</p>

Deploying a Kernel Module</h3> add the following to your NixOS config:</p>

Wake-on-LAN and Push Notifications in Mobile Linux

How To Casually Contribute to NixOS/Nixpkgs

Bootstrapping a NixOS Pinephone Installation

The Jackson Pad: DIY Vibration Damping for Drumsets

Rescuing a Broken EXT4 System with Ext4Magic and dd

A Reasonably Secure Mailserver Installation

Self Hosting in More Detail Than You Asked For