Interesting change in Starlink: fewer reported Obstructions starting around April 29. But it looks like they just fixed a bug where outages from Reboots were being called Obstructions.
Outages
Here’s a graph of obstruction outages for the last 90 days. It’s a graph of the gRPC outage stats from Dishy, the same thing the mobile app shows.
Sure looks like an improvement! In the past week I’ve had 7.4s obstructions and about 6 minutes of total outages. In a week a month ago I had 176s of obstructions and 386s of total outages.
Unfortunately this improvement seems to just be a recategorization. One cause of Starlink outages is rebooting for firmware upgrades, around 30 seconds a few times a week at 3am. Last month those outages were sometimes categorized as “obstructions”. Now they’re properly marked as “booting”. Here’s a week from last month showing these 3am false “obstructions”: in the most recent graph these now show as “booting”.
Here’s a graph of all the outage types for 90 days all together. You can see the red booting is more common than before.
Packet loss
Unfortunately this lowering of obstructions doesn’t seem to be affecting overall packet loss. Below is a graph of Starlink’s own stats on “drops” for 90 days followed by my own IRTT monitoring which also shows dropped packets.
No clear change, so maybe things aren’t really better. No visible change in the graph and doing the math, last week I had 0.63% packet loss and a week a month ago had 0.50% packet loss. Packet loss might have even gotten worse but I wouldn’t read too much into that without closer look.
My Chromebook on the beta channel (v125) has been flaky, the Android and Linux subsystems kept crashing. So I decided to downgrade back to stable channel (v124). That all worked but was tricky enough I’m writing notes. See below for the dream of 1990s computing.
An important concept here is ChromeOS “powerwashing”. That’s basically a full software reset, wipe the whole system (including your local data) and reinstall. It’s a neat feature of Chromebooks, the easy full reset as an important part of the product.
Changing channels
There’s a “Channel” pane in settings with a “Change channel” button. If you use it to downgrade from beta to stable it tells you that you have to powerwash the system (fully resetting it) and reboot. But rebooting doesn’t do anything, neither does manually powerwashing. Turns out you have to click “check for updates” somewhere to get it to download the stable release, then follow that path to install it which results in the reboot and powerwash.
Note that powerwashing is the default way to downgrade ChromeOS and will avoid compatibility problems. But it’s not necessary. There’s a crosh shell command to downgrade without powerwashing. Enterprise-enrolled Chromebooks also have an alternate channel switching method where it doesn’t switch immediately but instead waits a few days / weeks for the stable release to catch up so you can smoothly move from beta to the newer stable without a reset.
But Chromebooks were designed to be safely powerwashable. In theory all your important stuff is stored in Google’s cloud already and a powerwash is painless. How does that work out?
Powerwash: the ChromeOS stuff
After a powerwash it is indeed easy to get your basic Chromebook back to the state it was in. Basically you log in once and Google syncs back over your web browser state, which in ChromeOS is basically everything on the computer. The big hassle is you have to log in again on every website, all your cookies are gone. But in general your Chromebook environment looks the same after a powerwash and login.
Google will force you to once again re-enable ad privacy in the Chrome configuration. Sleazy.
Two things special to me inside ChromeOS, unlikely most people would do this:
I had to re-enable the ComposeKey input method as a second keyboard. It was installed but I had to go into settings to make it accessible.
I will need to reinstall the Bypass Paywalls Clean Chrome extension. That has to be sideloaded from local files, Google won’t allow such useful stuff in their Chrome addon store.
However, I don’t just use ChromeOS stuff on my Chromebook. I am also a developer running Linux and I use a bunch of Android apps. That’s a little trickier.
Powerwash: Android
Android apps are almost first class citizens in ChromeOS these days. The powerwash restore app will offer to reinstall “your Play apps”, which means Android apps I guess. It reinstalled all the apps I had before on this Chromebook. But then they all have to be re-authenticated and configured. For me that means adding a new device to Tailscale and resyncing Obsidian with their sync service.
Powerwash: Linux subsystem
I also run a Linux container on my Chromebook. ChromeOS has a nice backup and restore system for that. I backed up to a USB drive, powerwashed, restored, and everything was back the same in Linux. Both the Debian image and my /home stuff. The Linux Signal flatpak app even stayed logged in. Couldn’t be easier. (the backup file is called .tini but is mostly just a .tar.gz of the filesystem with an extra metadata.yaml at the front.)
I had to manually re-share my ChromeOS Downloads folder to Linux so /mnt/chromeos/MyFiles/Downloads worked again.
Powerwash: Steam
ChromeOS now has experimental Steam support which basically amounts to a second Linux container. I just reinstalled that which was painless since Steam Cloud stores everything anyway.
The dream of the 90s is alive
There’s a couple of old ideas about computing from the 1990s that are finally sort of to fruition now.
One is the idea of a “thin client”, that the computer you sit in front of has nothing important on it and instead interfaces with what you want on the network. That’s how a lot of stuff works these days with hosted webapps. But we’ve never quite gotten free of your personal computer still being customized. The ChromeOS team put a lot of work into syncing that customization and it mostly is productive although then I messed it up again wanting a local stateful Linux environment. Also authentication is a huge PITA, no one wants to sync the actual login token to the cloud.
The other is the idea of “ubiquitous computing”, that I could access my personal computer environment anywhere I go seamlessly. Again we’re mostly there with modern Internet and cloud based stuff, but there’s still the tricky problem of local state. A simpler login system (like passkeys should be) would go a long way to making the re-authentication easier. And you’ll never completely avoid the reality that having local state on a local computer is faster than something over a network. That’s why I bother with the local Linux machine, for instance.
Just set up some long-overdue monitoring for small projects like my blog and linkblog.
updown.io makes HTTP requests to your server and alerts you if they fail or do not contain expected contents.
healthchecks.io alerts you if an HTTP URL on their server is not pinged in a configurable amount of time. It’s intended for cron jobs; if your job doesn’t load the URL in time you get a warning.
Both are free for initial usage, enough for at least a year of a small project (or more). And low cost beyond that. They’re both professionally made and have been around awhile. I’m impressed with how many integrations they have. Email notification is the default but there’s Slack, SMS, etc as well.
shout-out also to scrutineer.tech which runs a simple RSS feed warning you about upcoming SSL certificate expiration.
I was surprised in my last post about how I was getting 25 Mbps upload speeds. So that had me looking again at my bandwidth tests since I got Starlink. These are tested with the Ookla Speedtest CLI tool. They are a test of my house’s network connection, so occasionally speeds are slower than real because I’m using the network or I’ve temporarily failed over to my backup ISP (not Starlink). But overall these reflect Starlink performance.
Fine grained graph
The main trend that interest me is in 2023 and 2024, the steady improvement of speeds. Particularly upload speed. This is a great trend to see in your ISP!
It’s also interesting to see those halcyon days of 2021 before Starlink got heavily subscribed. That’s a suggestion of what the maximum radio throughput service is. Service was worse then: more outages, higher latency, more variability. But download bandwidth was same-ish for a lot of this period. Upload has definitely gone up.
Monthly average
The graph above is some fine-grained reporting interval that Grafana selected, probably a few hours per point. The graph below is an average of every 30 days. The same trend is more clearly visible but of course it doesn’t capture the variance.
This blog post contains screenshots of my monitoring dashboards for my residential Starlink node in Grass Valley, CA. Bottom line: the service is quite good these days.
Average speeds: 150Mbps down / 25Mbps up
Average latency: 36ms
Average packet loss: 0.6%
I do a lot of careful client-side monitoring. Used to pay more attention when the service was new and had some problems. But since January 2023 it’s worked very well for me, when their major over-capacity problem was solved at least for my area. And it got better in January 2024 when they started to improve latency. Mostly now I don’t pay attention to it very closely because it works fine.
The downsides are the median packet loss is high enough to matter. And performance is variable, it gets noticeably worse in evenings presumably because of congestion. But it’s pretty good always. FWIW I think Starlink’s behavior is more like a cellular ISP than wired or geosync satellite. Better quality in general than overloaded American cellular but similarly unpredictable packet loss and jitter.
I still have my dashboards and monitors running and they give very detailed client-side views of performance. Here’s screenshots of my 4 separate dashboards, all data for the last week (April 22 to 29). I think this week was mostly unremarkable and typical. First couple are basic ISP monitoring; the other two are more specific and unique to me or Starlink.
FWIW I have a gen 1 Dishy with ordinary residential service.
highly variable download and upload speeds with a mean of 150Mbps down and 25Mbps up.
few hours under 20 Mbps download speed
Compared to the old days, overall speeds are better. I used to be more like 100 / 15. Even better, the “hours under 20 Mbps” is greatly improved; that used to be 8 or more hours a day.
ICMP Ping tests
Using telegraf to send 5 pings every 15s to 8.8.8.8 and measure round trip time and packet loss. See below for the better IRTT tests showing similar data; starting here because pings are familiar to everyone.
average latency around 36ms
0.58% mean packet loss
fairly stable throughout the day but there’s some visible daily pattern
IRTT latency tests
These are much more detailed results using the IRTT tool to test latency and packet loss with UDP packets to a server I run in a datacenter at or very near the Starlink LA POP.
0.61% mean packet loss
36ms mean latency.
Several incidents a day of 20% or more packet loss. (You see this pattern in applications too; the network goes away for a few seconds).
Clear pattern of higher packet loss during evenings. 30 minute average gets as bad as 2% or so occasionally.
Starlink monitoring
Starlink’s own monitoring data, using their gRPC endpoint to pull data from the dish.
two 30s+ outages around 3am. these seem to be related to firmware upgrades
36ms mean latency
0.6% “obstruction fraction”. It’s unclear what the dish reports as obstructions; at least some of it is packet loss unrelated to my antenna. I have a very clear but not 100% perfect view where Dishy is pointed.
Starting to containerize some stuff that I originally migrated to a VM. My linkblog, for a start. I need a generic Linux environment that’s fairly beefy; libc, not musl. Ideally I’d bake all this into a reusable container but I’m rolling it by hand for now. Here’s my notes.
Container creation
Create a unprivileged container with hostname “blogs”
Add root password, my SSH key
Use the Ubuntu 22.04 (or latest) template. Ubuntu so I can get deadsnakes for Python versions. I’d like latest Ubuntu but can’t because Playwright (the screenshot library I need for the app) only supports LTS releases.
16GB of disk, 4 CPU cores, 16GB of RAM, 1GB of swap. Or more, these are all soft limits so it may be better to set these higher
Default networking but enable DHCP instead of static.
Consider adding access to vmbr1 for the NFS server.
Edit the container options to make it start on boot
Linux configuration
Start the container, launch a console, log in as root with password
apt update; apt upgrade
apt install joe sudo curl avahi-daemon git zip unzip rsync webp sqlite3
locale-gen en_US.UTF-8
Install tailscale curl -fsSL https://tailscale.com/install.sh | sh
Shut down container
On the Proxmox hypervisor server, modify /etc/pve/lxc/???.conf to give access to tailscale in the container
Start the container
Log in on console again
tailscale up, approve the URL to join the machine. Consider disabling tailscale key expiry in the web console.
Use ordinary ssh client to log in as root@ts.net tailnet name.
(not sure avahi-daemon is working here, I can’t use the .local DNS name I am expecting. I tend to use Tailscale to connect to things anyway.)
Linkblog configuration
Specific to my application, but some generic notes here also about creating a prod environment for anything.
User setup
As root, make a user for the project. adduser linkblog; adduser linkblog sudo;
Log in as linkblog@ with password
ssh linkblog@localhost, to create the ~/.ssh directory.
copy over the install from the old system. This gets credentials, config, and state. rsync -av --exclude=venv nelson@example.com:~/src/mastodon/linkblog-toot/ .
run a quick test, should show posting nothing if there are no new stories feed2toot -n --debug -c feed2toot.ini
Install crontab entry to run go.sh for user linkblog
TODO
Package linkblog as a proper Python package so deployment is better
Write linkblog code to manage crontab (or at least emit the line)
Looking for typing accommodations while my wrist is giving me trouble. I love how Google Voice recognition works on Android, you’d think that it would be even easier and better on a desktop PC. But the UI doesn’t quite seem to be there.
Update. I originally wrote this post using Windows 10. Turns out Windows 11 has an entire new thing called Voice Access and it’s quite good, just about what I need. I’ve edited the post to reflect all this, but it’s probably a little confusing.
I’m fortunate that I can use both hands still and mousing gives me no trouble, just trying to take some load off my left hand.
Windows 10
Windows 10 seems to have two different systems for voice recognition. There’s an older speech control system that is mostly focused on UI control like opening the start menu or closing windows. It has some speech to text dictation but the language model is terrible and not useful.
There’s also the newer windows dictation accessible via Win-H. It’s dictation only, not really system control, but reasonably good. However the UI’s awkward. Also the language model is not as good as what I get on my pixel eight. But it is usable.
The biggest problem is activating dictation mode. I have to click in the window I want to type in, then click this weird microphone icon that’s hovering in a fixed width window on the desktop. And even then it does not work reliably. I would dearly love to assign the “take dictation” function to something like a hot key or mouse button four. Haven’t found a way to script that. Quick searches make it look like it’s not possible.
I had thought Microsoft put more work into usability. The full speech control system does work, but it’s pretty awkward and is something you would only use if you had no alternative. And the fact it’s basic dictation speech model is no good makes it useless to me. Maybe it’s better in Windows 11? I think the full control Voice Access system is much better.
Windows 11
Microsoft did put more work into usability! Windows 11 is significantly better. The old speech control system that I didn’t like is basically gone. Windows Dictation has been extended to Windows Voice Access and gives you fairly reasonable control over the whole UI.
For my purposes, the most important thing is that the speech to text in the new Windows voice access is much better. The UI is also better. You can basically leave it running all the time, no need to press a button to activate the microphone every time you need it. It doesn’t do anything until you start talking. Then it will just insert what you say as if you typed it. That means I can leave the thing running all the time. And when I have something more extensive to say, I can talk instead of type. (Although it seems like it picks up audio from the speaker so that’s a little annoying if you are playing a video with someone talking.) The speech model is pretty good, other than the automatic punctuation system which I’ve had to turn off. Totally usable though, it’s closer to Google Board speech detection than not.
There’s a good little voice control demo that I ran through and I’m impressed that I could actually use this to control a PC with my voice if I’m patient enough. It works by you speaking voice commands. It has some generic controls for clicking named buttons or narrowing in on a region of the screen to click on. It also has some specific app integration. For instance, “close tab” seems to be something it understands explicitly to close tabs in a web browser. It doesn’t work in Chrome but it works great in Edge. Maybe Edge has extra code to interface with voice access? One drawback is there is some ambiguity. If I start a sentence with the word start or close it may interpret it as a command and not me trying to type those words. There is a dictation only mode but I haven’t figured out how to turn it on (saying “switch to dictation mode” should do it but does not for me.)
Voice In extension
Now I’m wondering why Chrome on desktop doesn’t have voice typing. It seems like it would be easy to build it into the browser. I’m finding some third party hacks like Voice In.
I gave Voice In a try and it seems to mostly work and be a lot simpler trigger than the windows built-in thing. You can bind it to any windows hotkey but in Chrome that means control or alt + a letter. I really want just a mouse button. I probably can use Autohotkey to make that happen. Being a Chrome extension makes it pretty limited but then also probably easier to integrate.
I gave it more of a try and it’s mixed results. The language model these things recognize to is so important. I was writing an email to a friend about gay stuff and it kept missing very basic words that would be common in a gay context but not in some other. Also it has a habit of capitalizing random words like extreme or City. Mostly it just made me wish I could use Google Voice typing model because I am very good at getting what I want to out of it.
Having the extension be Chrome only is a little irritating particularly when I switch to slack. I guess I could run slack in the browser. The free version works reasonably well but really you’re going to pay $60 a year if you use it all the time. It’s not a big problem not having a keystroke to activate, mostly you just leave it active all the time.
Google could absolutely build a good Windows desktop or Chrome product for voice typing. I suspect the market just isn’t big enough for them
My RSI is flaring up so I thought I’d fire up my ancient Kinesis Classic keyboard, the one that saved my career 20+ years ago when I first had problems.
The keyboard has an ancient AT keyboard cable. I found it with a PS/2 adapter already attached, those two ports are electrically identical so the adapter is easy. I don’t have a PS/2 to USB adapter around the house but fortunately my 2019 PC still has PS/2 ports on the back. Purple for keyboard! After plugging it in the keyboard lit up but I had to reboot to make it actually work. IIRC that’s a problem with the old Kinesis firmware. It works fine with a second keyboard on USB.
It mostly seems to still work fine. Sometimes keys seem to be repeating when I don’t mean it, like I’ll press x once and xxx is inserted. It’s not frequent though, a missed interrupt maybe? I mostly still remember how to type with it too, I suspect it’ll take a day to adjust (took a week or more the first time).
I can’t find any documentation of the custom remapping I set up long ago. It sounds complicated but the main effect is to move keys more to where they are on a normal keyboard. Here’s what I found I had:
Bottom row right hand: all 4 arrow keys in hjkl vi configuration.
[]{} remapped to bottom row left hand (where left and right arrow were)
`~ is moved to upper left (where += is on the Kinesis)
+= is moved to caps lock
Caps Lock is unmapped.
Escape is moved to the key marked “delete” on the thumb
Delete is on lower left (where ~` is on the Kinesis).
The Backspace key sends VK_BACK in windows, a Backspace.
The Insert key is default, VK_INSERT.
I also went and found my ancient notes on the keyboard firmware for programming.
Progrm-Shift-F6 Toggle shift as sticky modifier Progrm-Ctrl-F6 Toggle control as sticky modifier Progrm-Alt-F6 Toggle alt as sticky modifier
Progrm-F12 Enter/exit remapping mode
Progrm-F11 Enter/exit macro definition mode Progrm-F7 Make macro pause for data / End entering macro data Progrm-F8 Put a half-second delay into a macro Progrm-F10 Disable/Enable all macros
Shift-Shift-F12 Report firmware version (press both shift keys)
I’m working on my blog engine. I want to package it nicely as a Python package. I also want to be able to run my code from a command line using the click library, and write and run tests using pytest, and be able to do all this with the debugger in VS.Code. I’ve never made my peace with Python packaging, for many many years now I’ve avoided it. Because it’s a mess.
Anyway I just tried again here in April 2024 and more or less succeeded in just an hour or so. This is where I would write clear docs with a github repo with the code but.. nah. I’m not confident it’s right and I don’t understand it well enough to present coherently. But I’ll write up what I learned.
I mostly followed the Packaging Python Projects tutorial which explains the creation of this sample project. Following that gets you a basic Python project which you can install with pip install -e. The key thing here is everything’s driven by the pyproject.toml file. If you are reading any docs that talk about setup.py they are old and outdated; the TOML is the new way.
Some subtleties in the pyproject.toml:
My project has dependencies that I specified in the TOML. It depends on pytest and click.
I had to pick a build backend; I’m using setuptools as the old non-fancy default.
My project has console scripts. You can specify these in entry points or in a scripts section.
Getting click working was not hard. There’s docs for click + setuptools. Note these are in terms of the old setup.py style but translating to pyproject.toml isn’t too hard.
Here’s my source tree for a toy package called “whir”
pyproject.toml in the root directory
Python code in sys/whir/.
src/whir/__init__.py that imports all the public functions, particularly the click commands.
src/whir/lib.py is simple library code that other stuff imports.
src/whir/main.py contains a main_cli() function that’s a click command. __init__.py imports it so it can be used as a script in pyproject.toml. It also imports the lib code via import whir.lib, not a relative import.
tests/test_lib is a pytest function that tests the lib code. It imports it with from whir import lib.
Getting pytest working was not hard either. The default setup worked just fine from a shell. Getting it to work inside VS.Code’s extension for it required adding something to pyproject.toml to get pytest to use importlib, as described here. But now I can just press a button the IDE GUI and all my tests run with a nice display. Yay!
Getting the VS.Code debugger to work with click was tricky. AFAICT there’s no easy way to get VS.Code to invoke a click command. But there’s a hack where you call the click command with string command line arguments inside a if __name__ == '__main__'; then VS.Code can be used to just debug the file and it will work.
Edit: thanks to Claude I’ve now found two more ways to invoke a click command from a function. The intended way to just invoke it is to use main_cli().callback(args): the click decorator adds a callback function to the command function you can use. There’s also a whole library for testing invocation of Click commands, CliRunner, that will mimic a command line call. It sets up a special environment that captures stdout and stderr, can mock up a filesystem, etc. It’s for using in a unit test framework.
Ranting reflection
It all worked pretty straightforwardly! But I have no idea at all how it works. If something goes wrong I won’t understand what is happening or how to fi x it. And the tools are creating little files and shims that get between me and my code. It’s always like this with packaging and build systems and I hate it.
I’ve had an annoying problem on my phone for two months: notifications would be delayed by many minutes, mostly only showing up when I manually woke the phone. I think I found a fix.
Turn off Adaptive Connectivity. Details in this Reddit post the text of which I will duplicate below.
Update Apr 10: after a few days of trying this I’d say turned it off has helped but not fixed the problem entirely. Updates still don’t always come immediately but they are delayed by just a few minutes now, not 30+. I don’t have solid data on that, just observing.
Update Apr 24: notifications still seem better before but it’s not perfect. Also I’ve since learned about Android Doze, a feature that dates back to Android 6 where the phone goes into a low power Doze mode. It’d cause exactly the delay problem I’ve seen. Some more discussion here. I wonder if something changed recently with the implementation that made the delay much more noticeable?
But first, a rant. This is a supremely irritating kind of problem that shows how frustrating the modern tech support experience is. There was no realistic way to get helpful info from Google about what this bug might be or how to fix it. I don’t know if there’s even any real personalized support path at all, if there is one I assume it’s the usual “turn it off and on again” useless response. Web searches didn’t get me anywhere, in part because Google’s index is awash in spam. The Android docs are not good, Adaptive Connectivity doesn’t seem to be described adequately anywhere. I did finally find my way to Google Issue Tracker which has what looks like a relevant bug entry. But that’s write-only, barely any acknowledgment.
I’m a software engineer! I understand the tech, I’m careful, I’m willing to spend the time providing expert help to reproduce and debug this issue. But I have no way to get through to someone. I guess that’s inevitable given the economics of supporting millions of users. Still irritating for my $1000 flagship phone.
Not for the first time have I wished that shibboleet was a real thing. I did get some really helpful Google engineer feedback on Mastodon after a passkey problem I reported but that was just dumb luck and a particularly friendly Googler, the right person happened to see my complaints.
Are you getting delayed notifications on your phone? Try turning off the setting “Adaptive Connectivity” (under Network & internet in Android 14). I think that fixed it for me. Also check that “Adaptive Battery” and “Battery Saver” are off.
Two months ago I posted about getting delayed notifications; they’d only show up when I woke up the phone. A popular post, it keeps getting comments from people saying “me too”. Thanks to a hint in this comment I learned about the “Adaptive Connectivity” setting. I’d never heard of it before. But it was turned on; I turned it off and now 24 hours later my phone has been reliably getting notifications the moment they happen.
As a bonus, I think this may have also fixed my problem with the weather app showing me in the wrong location. Sometimes I’d get weather for Los Angeles where my IP address is rather than Grass Valley, CA where my phone is. I’m less certain of it but it’s plausible.
It’s not clear to me what Adaptive Connectivity is supposed to do, I can’t find clear docs about it. I think it may be a Google Pixel only thing? A lot of web pages talk about it only in terms of cellular, 4G vs 5G. I’m on WiFi almost always and still had notification problems. I don’t know what Adaptive Connectivity’s intent is but my experience is in Android 14 it means “break notifications”. My guess is it’s some power saving setting that is having the effect of taking the phone offline.
Note there’s other phone settings that might cause delayed notifications. Adaptive Battery is the big one (I’ve always had it off). Battery Saver mode may disrupt notifications.
Also individual apps all have battery settings. That used to be Unrestricted / Optimized / Restricted and my understanding is Optimized was supposed to let notifications through immediately. The UI for all that just changed in Android 14 QPR2 but I think it functions the same. Most of my apps are set to Optimized and are getting notifications on time.
Bottom line: turn off Adaptive Connectivity and Adaptive Battery if you are having trouble with delayed notifications.
Nelson's log 