Orange Sun

Declaring bankruptcy on Advent of Purescript 2023

2024-01-09T00:00:00+03:00

Advent of Code is a fun challenge and this year I decided to attempt solving it in Purescript. Today I'm declaring this attempt a failure. This post will serve as a postmortem.

Choosing Purescript

Last year I solved my first Advent of Code using Go. It was fun and I knew immediately that I will come back for the next AoC. I also knew that I won't be solving it using the same language - learning on the go (pun intended) contributed significantly to my enjoyment of AoC 2022.

In November 2023 I listened to an old podcast where Richard Feldman evangelises Elm programming language. I got interested and initially decided to use Elm for the upcoming Advent of Code. While I was learning the basics of the language I also learned about the ongoing old controversy in Elm community related to how its BDFL handles communication and development. I decided that that's too much drama for my liking and that it has too little hope of resolving anytime soon - and that it's better to stay away from Elm.

That's how I ended up with Purescript. It was probably the only other frontend language that offered functional programming with a strong type system and a nice UI framework. I hoped that using an unconventional language would introduce me to frontend development without dealing with unpleasant Javascript ecosystem.

Learning Purescript

My only prior experience with functional programming was Microsoft's Power Query M language which is rather narrowly focused on data processing and is not a general purpose language.

So, with effectively zero prior knowledge I started learning from The Purescript Book but quickly switched to Functional Programming Made Easier by Charles Scalfani - the former was too fast paced for me. Scalfani's book was an enjoyable read even if a little too verbose. I did not follow author's advice to type out and run all code snippets from the book - that may have contributed to my eventual failure but I don't think that it was a major factor.

Failing Purescript

I enjoyed solving small textbook problems in Purescript. It's a very nice language. Here are the things I liked most about it:

Separation of pure and effectful functions
Fearless refactoring
Clean and readable syntax
Pattern matching with exhaustiveness checking
Function currying, tail call optimisation and other FP niceties
Type system

I had no problems with understanding recursion, currying, pattern matching, etc. I had relatively little problems with understanding monads and related concepts.

My problem was that these nice abstract concepts just did not translate in my head to applicable programming techniques. Simple practical tasks tripped me up hard.

Parsing text was painful. For the first AoC puzzle I decided to tough it out with a bespoke char based parser even though I vaguely understood that there should be a monad based parser combinator library for that. That vague knowledge was not provided by any of these books, I've picked it up accidentally from some random blog post on the web.

For the second AoC day I used the proper library. It was better but still felt unnecessarily difficult. When third day's puzzle called for a parser not based on regular grammar my mind just blanked out. I was loaded up to the brim with pure theory and I lacked practical knowledge to apply it.

All this time while I was struggling with text parsing the actual tasks I picked up Purescript for (frontend experiments and puzzle solving) were deprioritized to background. I have to commend Purescript and Halogen on this because if it was anything less than straightforward none of UI and/or puzzle solutions would get done - I just barely devoted any time to that.

The project got stalled. I was reluctant to finish the last chapters of Scalfani's book because I was confident that they won't provide me with practical knowledge I was lacking. I was hesitant to go web diving for new, more practical learning materials because I wasn't sure they exist - Purescript community is rather small, and most learning resources are enumerated in multiple places. I evaluated those lists the first time I looked. I probably should've gone looking for more generic functional programming knowledge - and that would blow my free time budget.

So I'm just declaring bankruptcy on this project.

Conclusion

Even though I failed to cowboy my way into Purescript I still got the solutions to first two days of Advent of Code to show off - all logic executed client side, all UI generated on demand, all type safe and checked at compile time.

I could probably pour more effort, practise more, find new learning resources - and complete the remaining challenges. I'm stubborn enough to see this through. But my hobby time is not unlimited and there are other projects waiting - I'm certain I will enjoy some of those more.

Benchmarking ssh-agent performance

2023-07-19T00:00:00+03:00

I have an application idea that would require calling ssh-agent rather frequently - but how many requests per second can it handle?

To answer this question I wrote a small benchmark in Go. It runs a tight loop sending messages for ssh-agent to sign. Turns out the agent is pretty fast!

On a cheap cloud machine it was able to reliably sign more than 500 messages per second with an ED25519 key. RSA signing was about 4-5 times slower, as expected. For a personal heuristic I've decided to memorize that ED25519 signatures cost 2ms and RSA ones 8ms.

Message size had little effect on throughput because both in ED25519 and in RSA signatures inputs are hashed with a fast SHA algorithm prior to any other processing.

Just in case my random number generator was too slow I checked if it affects the benchmark results. Tests confirmed that it doesn't.

Of course, 500 rps does not sound web scale but for me it's more than enough. OpenSSH ssh-agent utilizes only a single CPU core, so there is some potential for performance improvement if you need - but that would mean doing the signatures in your software. I would rather trust OpenSSH team (who are known to be just the right amount of paranoid) than touch private key material with my clumsy hands.

You can run the same tests yourself: clone the repo and execute make from top-level directory. Benchmark names describe the key being used, message size and whether the message is unique or the same for each iteration. Here is a sample output:

BenchmarkSshAgent/key_ed25519/32B/unique-4          3553      1763363 ns/op
BenchmarkSshAgent/key_ed25519/32B/same-4            3568      1708270 ns/op
BenchmarkSshAgent/key_rsa4096/32B/unique-4           778      7824780 ns/op
BenchmarkSshAgent/key_rsa4096/32B/same-4             763      7657785 ns/op
BenchmarkSshAgent/key_ed25519/64B/unique-4          3456      1752457 ns/op
BenchmarkSshAgent/key_ed25519/64B/same-4            3598      1733750 ns/op
BenchmarkSshAgent/key_rsa4096/64B/unique-4           781      7639828 ns/op
BenchmarkSshAgent/key_rsa4096/64B/same-4             798      7720210 ns/op
BenchmarkSshAgent/key_ed25519/256B/unique-4         3549      1735906 ns/op
BenchmarkSshAgent/key_ed25519/256B/same-4           3417      1722301 ns/op
BenchmarkSshAgent/key_rsa4096/256B/unique-4          698      7738767 ns/op
BenchmarkSshAgent/key_rsa4096/256B/same-4            787      7625366 ns/op
BenchmarkSshAgent/key_ed25519/1024B/unique-4        3555      1703601 ns/op
BenchmarkSshAgent/key_ed25519/1024B/same-4          3651      1633226 ns/op
BenchmarkSshAgent/key_rsa4096/1024B/unique-4         805      7542115 ns/op
BenchmarkSshAgent/key_rsa4096/1024B/same-4           810      7437307 ns/op
BenchmarkSshAgent/key_ed25519/16384B/unique-4       3205      1935190 ns/op
BenchmarkSshAgent/key_ed25519/16384B/same-4         3296      1907921 ns/op
BenchmarkSshAgent/key_rsa4096/16384B/unique-4        783      7624776 ns/op
BenchmarkSshAgent/key_rsa4096/16384B/same-4          759      7620548 ns/op

Many-to-many relationships in Excel data model

2023-06-02T00:00:00+03:00

This is a quick hack to build many-to-many relationships in Excel data model even though they are not supported out of the box.

Create intermediate calculated table and use DAX to fill it with unique values from related columns on both sides of the relationship:

EVALUATE
  FILTER(
    DISTINCT(
      UNION(
        VALUES('TableA'[Field]),
        VALUES('TableB'[Field])
      )
    ),
    NOT(ISBLANK([Field]))
  )

Create two one-to-many relationships placing this intermediate table in between

For all intents and purposes you may now forget that this intermediate table exists: it will get updated automatically whenever you update the data model, and it will not consume much resources. Pivot tables and DAX formulas will work as if the two original tables were directly connected via many-to-many link.

A similar intermediate table may be created with Power Query, but that's a lot less elegant (unless you're already using Power Query elsewhere in the workbook) and takes significantly longer to recalculate.

A pull request 10 years in the making

2023-05-03T00:00:00+03:00

Once upon a time there was a bug in a free software project that annoyed me for long enough that I've learned C to fix it. And it felt good.

Now that we've got a TL;DR out of the way, here is the story.

There is a good torrent client called Transmission. It's lightweight, fast and reliable. I have compared it against alternatives several times and Transmission had always come out on top.

In 2011 I've found out that Transmission had open file limit essentially hardcoded to a value of 1024. Given that in Unix-like systems this limit is also spent on open network sockets the number seemed extremely low.

Back then FOSS culture was still very foreign to me, so I did what any consumer would naturally do: I contacted support. Helpful visitors of Transmission forums told me that open file limit is indeed hardcoded and pointed me to a bug tracker ticket which discussed this issue and which was closed several months prior without any real fix.

Bug tracker discussion pointed out that there was a good reason for setting such limit, a decades old limitation in glibc - important enough to be featured first in BUGS section of man 2 select. Since Transmission had hit this limitation via intermediate library (libcurl) there wasn't much else for Transmission developers to do at the time besides to hardcode a safe low value. Facebook had contributed an alternative libcurl API to work around that bug only a year later.

I did not handle the news well. In fact I did not understand most of it at the time, only that developers were aware of a problem and that they were not going to do anything about it. I switched to another torrent client and went on to moan on local forums about how silly Transmission is and how it's useful only for toy workloads. For several years every time someone mentioned Transmission in IRC/XMPP chats I would snarkily introduce them to this bug. Not my finest hour, I know.

In the mean time I was hitting the limitations of other torrent clients and remembered Transmission mostly fondly (if not for that one bug). Some time later I even returned to Transmission and was sharding workloads between multiple instances to avoid exhausting open files limit. I reviewed the alternatives from time to time and have not found any other client to be better enough to switch to.

As time went on other people got burned by the same bug. Some of them started badmouthing Transmission like I did. Forums posts and bug tracker tickets piled on, but no forward progress was made.

Unrelated to Transmission, I got introduced to FOSS scene. I've acquired a habit of checking if I could understand the source code upon encountering a nasty bug in a piece of software. I submitted a few small PRs to other projects I used, shared a few projects of my own and experienced being on the receiving end of an issue/PR. After many years I decided to look at that Transmission issue again.

Turned out that thanks to all commenters on Trac and on GitHub, the issue has been investigated to its root already. Libcurl was the culprit. Quick web search had introduced me to curl_multi_wait API, and after some introductory C tutorials I was able to replace all select() calls with the new API.

I submitted the pull request in April 2019 and the rest is history:

My PR got merged into Transmission in July 2020, providing closure to more than a dozen of bug reports
The first stable version of Transmission featuring my fix (v4.0.0) has become available in 2023

It felt good to finally remove this thorn instead of complaining about it. I probably should have done that much sooner.

Advent of Code 2022 was fun!

2023-03-09T00:00:00+03:00

This was the first year I participated in Advent of Code.

In case you're not familiar with it, AoC is a Christmas themed programming competition consisting of 25 challenges published daily (from December 1st to December 25th). The web site produces personalized puzzle inputs for each user and expects only the results to be submitted for validation, not full algorithms. Participants may use any programming language they like.

I did not compete for global or community leaderboards - that would be too high of a pressure to remain fun. Instead I took a self paced approach and solved puzzles whenever I felt like it - though I've never skipped ahead to start the next challenge until I was done with the current one. I completed the first 18 puzzles in December 2022 and finished the remaining ones in 2023.

It was very fun!

For me Advent of Code turned out to be the best computer game I played in years (though I'm not much of a gamer). Like many games it requires the player to develop and hone some arbitrary skills but in this case the skills are not useless outside of the game. In addition to programming (obviously) AoC tickled parts of my brain responsible for spatial thinking, math and creativity. I was reminded of how much I enjoyed similarly spirited math and physics puzzles when I was at school - it's a shame these experiences are so rare in adult life.

Roughly since Day 10 I've started taking notes about each puzzle and my thought process during solving it. I've intended to include them into this blog post, but I decided against it. There are enough AoC walkthroughs out there already. Here is a condensed list of bullet points from my notes:

Advent of Code is fun! Grid puzzles and mazes are very fun, especially 3D ones! Tetris... Yummy! Maze on the surface of a cube... Brilliant!
At some point I've grown tired. It was beginning to feel more like work and less like fun. Taking a (long) break here and there has helped to bring the joy back
Sometimes I got stuck. There is a large online community around Advent of Code, so there are a lot of ways to unblock oneself. I did not actively engage with any user group in particular, but on one occasion reading Reddit comments has helped to push me in the right direction, and on another one I've benefited from GitHub's social networking side.
AoC is a computer game you can continue playing while away from keyboard. A lot of good solution ideas have come to me while I was in shower or in a traffic jam.
A couple of times I felt very clever when I solved Part 2 of the puzzle before seeing the prompt.
Off-by-one errors are truly the bane of programmer's existence :-)

I solved all 2022's puzzles using Go language. I've picked it up less than a month before the start of Advent of Code, so I went in expecting to learn a lot and I was not disappointed. I've grown to appreciate the breadth of Golang standard library and to love type redefining. All of my solutions use only the standard library - this happened organically, I did not impose any restrictions in this regard.

All in all, Go turned out to be exactly what it has promised: a nice language with a fast compiler and strict type system. From now on I will choose it over Python for personal projects.

Working on these 25 puzzles I've gotten used to always having an extra thread of thought in background, completely unrelated to personal or work life. Even though I miss it now, I'm not yet sure if I should dive into AoC puzzles from previous years. Do they introduce enough variety to tickle my mind in some new ways or are they just more of the same thing? If I ever decide to try, I've heard that AoC 2019 IntCode puzzles are good - I'll probably start with those.

Negotiating down to 100Mbit between two 1Gbit devices

2022-07-29T00:00:00+03:00

Connecting two gigabit-capable devices via 4-wire UTP cable is an abomination, but sometimes we have to live with it (e.g. when a greedy ISP decides to save a few cents and pulls a cheap cable to your apartment).

The fun starts when the devices try to negotiate Ethernet connection speed. Autonegotiation pulses are essentially 10BASE-T and use only 4 wires with no checking of cable category, even CAT3 is enough. So autonegotiation will "succeed" and will settle on 1Gbit since it's supported by both ends, even though there is not enough physical conductors for it. The link will not work.

To negotiate a working connection instead we need to remove 1Gbit from advertised link modes at least on one end. In Linux it should be rather straightforward with ethtool:

$ ethtool -s $IFACE advertise 0x00f

0x00f is a sum of hex values for all 10Mbit and 100Mbit modes (0x001 + 0x002 + 0x004 + 0x008), explicitly excluding 1Gbit and everything above that. Tweaking advertised link modes is better than forcing a particular connection speed (ethtool -s $IFACE speed 100) because in absence of autonegotiation advertisement the other side may play it safe and switch to half-duplex.

That's the theory. In practise however, hardware is difficult. Drivers for NICs are tricky and sometimes incomplete. On D-Link DIR-825 changes made by ethtool command did not stick: after a short hiccup NIC would immediately return back to default settings, ignoring ethtool input completely.

But all is not lost. Through trial and error I found a sequence that worked. Don't ask me why, I'm not cool enough yet to understand kernel drivers code. Here is what worked for me:

# Excerpt from /etc/rc.local
# (order of commands and delays matter!)
WAN=eth1
ethtool -s $WAN autoneg off
sleep 1
ethtool -s $WAN speed 100
sleep 1
ethtool -s $WAN advertise 0x00f

Which results in a proper NIC configuration:

Supported ports: [ TP MII ]
Supported link modes:   10baseT/Half 10baseT/Full
                        100baseT/Half 100baseT/Full
                        1000baseT/Half 1000baseT/Full
Supported pause frame use: Symmetric Receive-only
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes:  10baseT/Half 10baseT/Full
                        100baseT/Half 100baseT/Full
Advertised pause frame use: No
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Link partner advertised link modes:  10baseT/Half 10baseT/Full
                                     100baseT/Half 100baseT/Full
Link partner advertised pause frame use: Symmetric
Link partner advertised auto-negotiation: Yes
Link partner advertised FEC modes: Not reported
Speed: 100Mb/s
Duplex: Full
Port: MII
PHYAD: 4
Transceiver: external
Auto-negotiation: on
Current message level: 0x000000ff (255)
                       drv probe link timer ifdown ifup rx_err tx_err
Link detected: yes

Let's hope some variation of this will work for your misbehaving device too! Good luck!

"No user exists for uid" when pushing to git repo

2022-07-21T00:00:00+03:00

Today I tried to automate pushing to a Git repository from a Docker container. And like many others I failed with an error:

$ git push
No user exists for uid 2918
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

Following best practices I was running the container under a random UID to drop root privileges, and of course there was no user with that UID in the system. I don't think that's egregious enough to warrant an error instead of a warning from git push, so I've started digging.

I was very surprised to learn where this error comes from:

/*
 * openssh/ssh.c: Main program for the ssh client.
 */
int main(int ac, char **av) {

    // ...lines omitted...

    /* Get user data. */
    pw = getpwuid(getuid());
    if (!pw) {
        logit("No user exists for uid %lu", (u_long)getuid());
        exit(255);
    }

getuid() is pretty self-explanatory, it returns UID of current user. Afterwards getpwuid() attempts to fetch data for the provided UID from /etc/passwd. It fails, of course, and returns NULL. OpenSSH client treats that as a show stopper and exits with an error.

I was hoping that finding the place where error is generated will help me to come up with a setup that avoids problematic code branch altogether, but no luck this time. It's straight in the main() function of ssh client, no conditional branching whatsoever.

I will be looking into generating a bogus /etc/passwd entry on-the-fly prior to launching my application in container. I would very much like to avoid hardcoding the UID at build time. (UPD: proper workaround would be to use libnss-wrapper like postgres does)

Meanwhile, here is a punchline for you:

When current UID is not in /etc/passwd OpenSSH client can not even print a usage message:

$ ssh
No user exists for uid 3432

$ ssh --help
No user exists for uid 3432

Pip-installable Pelican themes

2022-06-10T00:00:00+03:00

Installing Pelican themes the default way is not very pleasant:

You need to invoke a separate CLI tool
You may need to create some symlinks and ensure that they don't go stale the next time you use Pelican
Some people (me) have resorted to git submodules instead of official CLI

As a workaround I've been packaging my Pelican themes into simple Python packages with a sole __init__.py:

from pkg_resources import resource_filename
def path():
    '''
    Return path to theme templates and assets
    Use this value for THEME in Pelican settings
    '''
    return resource_filename(__name__, '')

This adds a dependency on setuptools but it's already present in most Python venvs anyways, so not a big deal. pkg_resources exposes full path to wherever pip installs the theme. It also handles unpacking to temporary directory if required (in case of wheels and zipped installs).

I have been using this trick for some time already, but only recently I noticed that Pelican plugins have officially transitioned to being pip-installable. They use a clever hack of adding extra packages to pelican.plugins namespace and I though it would be cool to use the same approach with themes.

Turns out it's not easy to do with setuptools, but is pretty straightforward with poetry. As a result I can now publish my themes to PyPI and provide easy invocation instructions:

# pelicanconf.py
from pelican.themes import smallweb
THEME = smallweb.path()

All the end users need to do is to add another line mentioning my theme to whichever file they use to create their Pelican venv.

On developer side we need to create pelican/themes/themename folder structure and point poetry at pelican for top-level package name. All theme files should be placed into pelican/themes/themename and one extra __init__.py file should be added there to provide path() method. See SmallWeb repository for an example.

Unexpected workaround for Libvirt VMs with cgroups v2 in Cirrus CI

2022-03-02T00:00:00+03:00

Today I wrote a commit message that was several screens long. I think it deserves to be a blog post of its own

Update: the commit linked above required some modification to remove flakiness, but the workaround still stands. Diff provided in this blog post was updated to reflect current state of affairs

Recently Cirrus CI builds that were using nested Libvirt VMs have started failing with the following error:

Call to virDomainCreateWithFlags failed: unable to open
'/sys/fs/cgroup/machine/qemu-2-defaultdebian11-server.libvirt-qemu/':
No such file or directory

First recorded failure occured on February 25th, 2022

Error message clearly indicates that the issue is related to Linux control groups (cgroups), and on a hunch I assumed that Cirrus CI (or Google Cloud) images were updated to use cgroups v2 by default. Unfortunately I haven't been recording debug information for cgroups before the failure, so I can not confirm my guess. Currently cgroups2 are in use, so the hypothesis stands.

Web search has led me to several useful pages:

RedHat bugzilla

Running Libvirt from inside a container (similar to how Cirrus does) produced the same error. The reason is that default cgroups mode was changed in podman when upgrading from cgroups v1 (host mode) to v2 (private mode).

I took note of this issue, but I moved on with my research since as a user I can not change the configuration of container runtime at Cirrus CI.

Libvirt documentation

Libvirt will not auto-create the cgroups directory to back this partition. In the future, libvirt / virsh will provide APIs / commands to create custom partitions, but currently this is left as an exercise for the administrator.

Based on the documentation quoted above I have tried to manually create the required cgroup with a simple mkdir -p $CGROUP. Please note that Libvirt uses different cgroup layouts when running on systems with and without systemd. Cirrus CI runners use a different (non-systemd) init in their containers, so the cgroup path in our case is $MOUNTPOINT/machine/qemu-$ID-$MACHINENAME.libvirt-qemu/

Manual creation of cgroup did not lead to any changes in Libvirt behavior. Error message stayed the same: no such file or directory.

Report author at RedHat bugzilla had mentioned that they had trouble with manually moving QEMU process into a different cgroup, so I've tried to do that and see if the error message will provide any further information.

I configured Cirrus CI to create a long-running background process and to migrate it to the newly created cgroup. I was expecting this to fail, so I did not comment out the code that later would launch a Libvirt VM on CI runner. Imagine my surprise when the pipeline turned green! Not only did the migration succeed, but its success had somehow lead to the success of Libvirt VM!

A few trial runs later I noticed that the name of cgroup used in the first step does not even have to match the cgroup (or cgroups) that will be used by Libvirt domains.

This is why I'm adding this meaningless cgroups burn-in to my pipelines. "It ain't stupid if it works", right?

Full commit diff

diff --git a/.cirrus.yml.j2 b/.cirrus.yml.j2
--- a/.cirrus.yml.j2
+++ b/.cirrus.yml.j2
@@ -21,6 +21,8 @@ task:
     # VENVDIR must be absolute path for 'cd && make' approach to work
     # VENVDIR should not be cached! Cirrus CI drops some binaries randomly

+    CGROUP_WORKAROUND: /sys/fs/cgroup/machine/qemu-cgroup_workaround.libvirt-qemu
+
     # Pass values from cirrus-run environment
     CLONE_URL: "{{ CI_REPOSITORY_URL }}"
     CLONE_SHA: "{{ CI_COMMIT_SHA }}"
@@ -61,6 +63,16 @@ task:
   libvirtd_background_script:
     - sleep 2 && /usr/sbin/libvirtd

+  # Workaround for cgroups v2
+  # I have no idea why or how this works (see commit message for a longer rant)
+  cgroups_workaround_background_script:
+    - mkdir -p "$CGROUP_WORKAROUND"
+    - ls -lF "$CGROUP_WORKAROUND"
+    - bash -c 'echo $$ > /tmp/cgroups_workaround.pid; sleep infinity' &
+    - sleep 1
+    - cat /tmp/cgroups_workaround.pid >> $CGROUP_WORKAROUND/cgroup.procs
+    - cat $CGROUP_WORKAROUND/cgroup.procs
+
   # Execute automated tests
   test_script:
     - cd ansible/tests
@@ -70,6 +82,9 @@ task:
   always:
     cache_debug_script:
       - find "$HOME/cache" -type f || echo "Exit code: $?"
+    cgroups_debug_script:
+      - fgrep cgroup /proc/mounts || echo "Exit code: $?"
+      - find /sys/fs/cgroup -exec ls -ldF {} \; || echo "Exit code: $?"
     kvm_debug_script:
       - free -h
       - pstree -alT

D-Link DIR-825 (rev.B1) throughput test

2022-01-31T00:00:00+03:00

So, the year is 2022 and I'm still using D-Link DIR-825, rev.B1 as my edge router at home.

Thanks to the power of opensource it is running a modern and secure OS (OpenWRT) long after the manufacturer has abandoned this product. Even though OpenWRT (and Linux in general) has increased the system requirements, DIR-825 still meets the minimal 8/64 criteria.

To put things into perspective: I bought this very device in November, 2011 for the equivalent of $120. Its WiFi standard is pretty outdated (802.11n), but that's enough since all my essential devices are hardwired.

Up until a few weeks ago it was pointless for me to think about router upgrade. The Ethernet cable coming into my apartment from ISP equipment was crimped to use only 2 twisted pairs which would never allow for speeds above 100Mbit. And DIR-825 routes 100Mbit just fine. Thankfully, a completely unrelated incident occurred and an ISP technician was on site - I seized the opportunity and asked them to recrimp their end (4 missing wires were just bent aside right before a connector, why would anyone do that?!)

Now that I have an option to upgrade above 100Mbit, the valid question is: Will my DIR-825 be able to handle that?

To test router throughput I used two laptops booted into Debian 11 live image, one of them running a DHCP server and connected to WAN port of the router, the other one connected to one of LAN ports. I put together all the required commands into a Makefile to avoid having to memorize them - you might find this repo useful if you ever need to perform a similar test on your router.

To establish a baseline I connected the laptops directly to each other without any routers or switches in between. I got the expected near-gigabit speeds: 932Mbps down, 941Mbps up - nothing unusual here.

Here are the test results with router in between (full logs: part one, part two)

Maximum throughput was achieved in upload tests (382Mbps). This is probably explained by having to process less firewall rules for outgoing traffic. Download speed even with a very basic iptables configuration was significantly less (256Mbps), and bidirectional tests with simultaneous traffic in both directions confirm that ~300Mbps is a hardware limit (bidirectional speed was 130Mbps down + 162Mbps up = 292Mbps total). top was showing 99% sirq load during these tests, but running a monitoring tool did not have any significant impact on the results (first 5 tests were executed before top was launched).

Conclusion

These tests show that while DIR-825 is a perfectly capable router for 100Mbps, it's completely out of its depth with faster connections. Even 300Mbps will frequently become underutilized if the traffic will happen to flow in both directions at once.

Looks like I finally need a new router after almost 11 years with DIR-825...

Ansible apt module fails to install python3-apt on Debian Testing

2020-12-09T00:00:00+03:00

I have encountered an unexpected Ansible failure today that turned out to be not a bug.

Ansible apt module had failed to auto install the required python3-apt package - only on Debian Testing. Same playbook worked fine with Debian Stable.

TASK [install some apt packages] *********************************************
[WARNING]: Updating cache and auto-installing missing dependency: python3-apt
ok: [debian10]
fatal: [debian11]: FAILED! => changed=false
  msg: 'Could not import python modules: apt, apt_pkg. Please install python3-apt package.'

After some troubleshooting I've been able to find the reason for this failure: Python interpreter was being automatically upgraded to the next minor version while installing python3-apt.

Since the apt_pkg module is distributed as compiled platform-specific binary (e.g. apt_pkg.cpython-37m-x86_64-linux-gnu.so), it is only compatible with Python version it's been built for. In my case the Python interpreter at the moment of Ansible module invocation was at version 3.8.6, but doing apt update; apt install python3-apt had upgraded it to 3.9.1 and installed apt_pkg was only compatible with new version of interpreter.

Ansible apt module was still running under the old version of interpreter and therefore was unable to import apt_pkg that it had just installed.

Such errors are a non-issue on Debian Stable where Python is never upgraded to the next upstream version, and even in Testing/Sid it's a rare occurence. More than that, I see no way to add a workaround to the Ansible module that could allow it to handle this edge case: the whole module is executed with one instance of Python interpreter and it can not accomodate such change in a single invocation.

The solution I see is to explicitly install python3-apt on Debian Testing/Sid systems before invoking apt module with Ansible. This can either be done with raw module or with the provisioning tools (machine template, preseed, terraform/packer/etc).

Pegatron Cape 7 nettop (thin client)

2020-04-04T00:00:00+03:00

Below are hardware details of an outdated compact computer that had since become available for low price on second-hand market. I bought mine for $15 (in March 2020).

This post is inspired by ParkyTowers Thin Client Database - many thanks to David Parkinson for gathering and sharing all that knowledge!

Pegatron Cape 7 was announced in early 2009 (marketing booklet is dated by 2009-04-13). It is based on Intel Atom 230 series CPU along with SiS 968/672 (models A, B, C, D) or nVidia Ion chipset (models E, F). The unit I have is model D, all further photos and description apply to that model.

The hardware is similar to Dell FX160 but is packaged into a significantly more compact case and uses external power supply. Pegatron is an ODM manufacturer which means the units were sold to end users under a variety of brand names. Mine was sold in Russia as Depo Sky 153. I've also seen mentions of it being sold as Pegasus CutePC in Indonesia, as unknown model under iClient brand in Brazil, and under some local brand in Poland.

Cape 7 is capable of running mainstream operating systems (mine came with Windows XP preinstalled), general purpose Linux distributions are also supported.

Specifications

Motherboard: Pegatron IPP71-CP with SiS 672 northbridge and SiS 968 southbridge
Processor: Intel Atom 230 (1.6GHz, single core, two threads)
RAM: DDR2 SO-DIMM, 1GB by default (mine came with 2GB preinstalled by seller)
Video: SiS Mirage 3
Storage: 2.5" SATA HDD
Ports:
- Network: Realtek 8111EL 10/100/1000
- USB: 6 USB 2.0 ports (2 front, 4 back)
- Video: 1 DVI output (other Cape 7 models may use D-Sub or HDMI)
- Audio: 3.5mm audio out, 3.5mm microphone in - Realtek ALC662
- Serial: none
- Parallel: none
- PS/2: none
- Other: 1 unknown port (next to DCIN), probably for Wi-Fi antenna
Power: External power supply - 19V DC, 2.1A, 5.5mm x 2.5mm connector (same as in many ASUS laptops)
Cooling: passive, completely silent. One removable aluminum heatsink for CPU, several thermal pads to transfer heat from northbridge/southbridge to metal case frame. My device runs pretty hot even after changing the thermal paste - I'll need to monitor how stable it is under workload.
Dimensions: approx. 173 x 154 x 20mm, the bottom of the case is slightly wider (26mm).

Processor

Click to view /proc/cpuinfo

processor   : 0
vendor_id   : GenuineIntel
cpu family  : 6
model       : 28
model name  : Intel(R) Atom(TM) CPU  230   @ 1.60GHz
stepping    : 2
microcode   : 0x218
cpu MHz     : 1599.527
cache size  : 512 KB
physical id : 0
siblings    : 2
core id     : 0
cpu cores   : 1
apicid      : 0
initial apicid  : 0
fpu     : yes
fpu_exception   : yes
cpuid level : 10
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc
arch_perfmon pebs bts nopl cpuid aperfmperf pni dtes64 monitor ds_cpl tm2
ssse3 cx16 xtpr pdcm movbe lahf_lm dtherm
bugs        :
bogomips    : 3199.05
clflush size    : 64
cache_alignment : 64
address sizes   : 32 bits physical, 48 bits virtual
power management:

processor   : 1
vendor_id   : GenuineIntel
cpu family  : 6
model       : 28
model name  : Intel(R) Atom(TM) CPU  230   @ 1.60GHz
stepping    : 2
microcode   : 0x218
cpu MHz     : 1599.365
cache size  : 512 KB
physical id : 0
siblings    : 2
core id     : 0
cpu cores   : 1
apicid      : 1
initial apicid  : 1
fpu     : yes
fpu_exception   : yes
cpuid level : 10
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc
arch_perfmon pebs bts nopl cpuid aperfmperf pni dtes64 monitor ds_cpl tm2
ssse3 cx16 xtpr pdcm movbe lahf_lm dtherm
bugs        :
bogomips    : 3199.05
clflush size    : 64
cache_alignment : 64
address sizes   : 32 bits physical, 48 bits virtual
power management:

PCI

Click to view lspci -nn

00:00.0 Host bridge [0600]: Silicon Integrated Systems [SiS] 671MX [1039:0671]
00:01.0 PCI bridge [0604]: Silicon Integrated Systems [SiS] AGP Port (virtual PCI-to-PCI bridge) [1039:0003]
00:02.0 ISA bridge [0601]: Silicon Integrated Systems [SiS] SiS968 [MuTIOL Media IO] [1039:0968] (rev 01)
00:02.5 IDE interface [0101]: Silicon Integrated Systems [SiS] 5513 IDE Controller [1039:5513] (rev 01)
00:03.0 USB controller [0c03]: Silicon Integrated Systems [SiS] USB 1.1 Controller [1039:7001] (rev 0f)
00:03.1 USB controller [0c03]: Silicon Integrated Systems [SiS] USB 1.1 Controller [1039:7001] (rev 0f)
00:03.3 USB controller [0c03]: Silicon Integrated Systems [SiS] USB 2.0 Controller [1039:7002]
00:05.0 IDE interface [0101]: Silicon Integrated Systems [SiS] SATA Controller / IDE mode [1039:1183] (rev 03)
00:06.0 PCI bridge [0604]: Silicon Integrated Systems [SiS] PCI-to-PCI bridge [1039:000a]
00:0f.0 Audio device [0403]: Silicon Integrated Systems [SiS] Azalia Audio Controller [1039:7502]
00:1f.0 PCI bridge [0604]: Silicon Integrated Systems [SiS] PCI-to-PCI bridge [1039:0004]
01:00.0 VGA compatible controller [0300]: Silicon Integrated Systems [SiS] 771/671 PCIE VGA Display Adapter [1039:6351] (rev 10)
02:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 03)

Disassembly

The case can be opened without any tools. All visible screws are holding internal components, not the cover. Bottom corners of the cover are easy to get a grip of - pull gently on those and expand the opening until plastic locks click open one by one.

Expansion

RAM: The unit accepts a single DDR2 SO-DIMM. Default module can be easily replaced with a larger one. I did not test with 4GB (DDR2 SO-DIMM of that size is expensive!), but I've seen multiple reports of 2GB working fine. My unit uses Kingston KVR800D2S6/2G module.

Storage: Cape 7 provides conventional SATA slot with enough space to fit a laptop HDD or SATA SSD. Thick 12.5mm drives fit only when HDD bracket is removed, but just barely - cover will be slightly bulged. Using SSD might not provide the expected performance benefit because SATA bus appears to be limited at 300 Mbps (37.6 MB/s) - numbers are from block diagram (page 15 of the booklet).

BIOS

BIOS is a standard AMI BIOS. Pressing Del at startup opens setup screen, F8 shows a shorter boot menu to select a device for one-off boot.

BIOS supports booting from USB devices and network boot. There are also several configurable options for power management:

Power On by PCI Devices
Power On by RTC Alarm
Restore on AC Power Loss

Pictures

See the gallery or download individual images in high resolution:

Running Libvirt (KVM) in Cirrus CI

2020-02-25T00:00:00+03:00

Up until the middle of 2019 it was very unusual to even expect that any CI service would allow nested virtualization. Those who required such functionality had to maintain their own CI runners on their own infrastructure. Things changed when Google Cloud introduced nested KVM support.

Cirrus CI was probably the first CI service to officially support nested virtualization in free tier. There are reports that Travis CI currently also provides such feature but no public announcement has been made yet.

It turns out I was the first person to try using Libvirt in Cirrus CI (I've even hit a previously unknown bug which was promptly fixed by their staff). Since the process has some subtle differences to the popular documented use cases I've decided to describe it here.

Hypervisor environment

Cirrus CI uses Docker images as environment for their runners. It significantly simplifies the setup and enables efficient caching between runs.

Since popular Docker images do not include any hypervisor packages we need to build our own image. I've decided to add the required packages to Debian base image. The whole Dockerfile is essentially one apt-get statement.

Keep in mind that libvirt package in Debian drops root privileges when launching qemu-kvm. You'll either need to disable that in /etc/libvirt/qemu.conf (as I did) or to change permissions for /dev/kvm to allow access by libvirt-qemu user.

Required system services

Default entry point for CI runner is not customizable in Cirrus CI - it's an agent process that communicates with CI service and sends progress reports you see in web interface. Because of that no systemd units are started automatically as it would have been the case on a normal system. More than that, starting systemd manually also looks impossible.

That means all the daemons required by libvirt must be started manually (see documentation on background_script syntax):

# .cirrus.yml
dbus_background_script:
  - mkdir -p /var/run/dbus
  - /usr/bin/dbus-daemon --system --nofork --nopidfile
virtlogd_background_script:
  - /usr/sbin/virtlogd
libvirtd_background_script:
  - sleep 2 && /usr/sbin/libvirtd

Firewall configuration

Hypervisor kernel is provided as is, and it currently runs legacy iptables firewall. Trying to use iptables-nft (which is the default in current Debian) produces a misconfigured guest network that is hard to debug.

That's why we need to tell Debian to use legacy iptables interface across the whole system:

# .cirrus.yml
iptables_legacy_script:
  - update-alternatives --set iptables /usr/sbin/iptables-legacy

That's it! Following these steps I was able to execute Libvirt (via Vagrant-Libvirt, via Molecule) in Cirrus CI environment. Full configuration is available here, it includes some extra caching steps and many debug statements that helped me to implement this process in the first place.

Cirrus CI integration for GitLab projects

2020-02-18T00:00:00+03:00

Cirrus CI is a relatively new hosted CI service that offers several unique features. It's probably the only CI provider to offer full virtualization (KVM) or FreeBSD runners for free. Currently their business model is centered around GitHub Marketplace and only the projects hosted at GitHub are supported.

Fortunately Cirrus CI provides a very capable GraphQL API. Thanks to that I was able to write a simple command line tool to trigger CI builds with custom configuration: cirrus-run. It reads a local YAML file and executes Cirrus CI build with that configuration. You can execute the build against any GitHub repo you're allowed to access.

Since build configuration is not provided by the repo the job may have no relation to it. You can execute any number of jobs for any number of projects against a single dummy holding repo at GitHub - which is the approach I suggest to use for setting up integration with other source code hosting platforms.

Here is how I've set it up with GitLab:

Project repo is hosted at GitLab
Each push triggers a new build in GitLab CI. The only purpose of that build is to execute cirrus-run
cirrus-run triggers the real build in Cirrus CI, waits for it to complete and reports the results.

Cirrus CI build is owned by a dummy GitHub repo that contains only one initial commit. Providing custom clone script allows to skip cloning that dummy repo.

That allows me to continue using GitLab to host my project and GitLab CI to run the jobs it's good at while delegating the more demanding jobs to Cirrus CI. All status reports are gathered by GitLab CI and failure notifications arrive uniformly to my inbox regardless of where the build was executed.

Configuration files that enable the workflow described above:

Cygwin CI journey

2020-01-28T00:00:00+03:00

Setting up Cygwin CI environment for testing one of my projects took more than fifty trial-and-error attempts - that's why I think it will be useful to leave some written notes on the issues I've encountered. Here is the end result - GitHub CI running some Makefile tests in Cygwin.

Cygwin gotchas

Cygwin installer likes to fail silently, providing only cryptic exit codes (127 or -1073741571)

%CYGWIN_ROOT%\setup.exe --quiet-mode --verbose --no-desktop --download --local-install --no-verify --site %CYGWIN_MIRROR% --local-package-dir %CYGWIN_PACKAGE_CACHE% --root %CYGWIN_ROOT%
Starting cygwin install, version 2.897
User has backup/restore rights
io_stream_cygfile: fopen(/etc/setup/setup.rc) failed 2 No such file or directory
Current Directory: cache\cygwin-packages
Could not open service McShield for query, start and stop. McAfee may not be installed, or we don't have access.
root: d:\a\Makefile.venv\Makefile.venv\cache\cygwin system
Selected local directory: cache\cygwin-packages
##[error]Process completed with exit code -1073741571.

At least some of those errors can be avoided by providing full absolute paths to the installer and any argument values instead of relative ones.

In the end I've switched to Chocolatey to handle the installation for me. It's certainly better but still requires calling setup.exe to install custom packages.

Since Cygwin installation and configuration are usually pretty slow (~3 min), CI setup benefits a lot from caching this step between runs. Unfortunately, default GitHub actions for caching can't help with Cygwin root, because Windows is pretty hostile to an average Joe trying to make symlinks.

Instead, I've added a separate step that calls tar --dereference explicitly and leaves only a simple archive for the caching action to handle. Please note that tar uses exit code 1 to indicate that everything was OK, but some warnings were printed to stderr. Also, order of command line arguments matters, at least for --exclude values. That was pretty unexpected for me.
You need to be careful when invoking apps under Cygwin - some environment variables may be inherited from cmd session. For example, $PATH will almost certainly point to binaries outside Cygwin root. I found that explicitly defining $PATH simplifies matters a lot.

PowerShell/cmd peculiarities

Since host OS is still Windows, you get to encounter all the usual PowerShell/cmd quirks. I tripped over the fact that PowerShell does not like double-dashed --GNU-style options, and over weird nested quotes required to properly combine environment variables and special characters in one value in cmd.

GitHub CI

Some of my attempts were just me trying to work around GitHub CI limitations. For example, dynamically calculating the value of environment variable based on values of other variables. This looks quite ugly:

- name: Use absolute path for CYGWIN_ROOT
  run: echo "::set-env name=CYGWIN_ROOT::${env:GITHUB_WORKSPACE}\${env:CYGWIN_ROOT}"

I've also failed at proper syntax for YAML multiline strings more times than I'm comfortable admitting :)

Don't blindly trust Docker for the selfhosted stuff

2020-01-27T00:00:00+03:00

It is my strong belief that you shouldn't go crazy with all-things-docker when deploying selfhosted services at home. Online forums, especially r/selfhosted, seem to foster an opinion that providing a Dockerfile or better yet a docker-compose.yml or even prebuilt public images on Docker Hub is an acceptable way to distribute software targeting the selfhosting crowd.

I agree it is very convenient to deploy complex multipart services via these tools. But the way many people appear to be doing that is a security nightmare! This is how we get to encounter Heartbleed in the wild four years after it should've been extinct.

There are many comprehensive writeups on Docker/Kubernetes security, I will highlight only a subset of problems below.

Shared libraries

Running each service in its separate container results in having a separate set of shared libraries for each one of those services. It is convenient when you need to provide multiple incompatible dependencies at once, but that way the burden of tracking the state of all those dependencies lies on the user. Host OS can not tell you that one of the containers still ships a vulnerable version of some critical library - it's up to you to monitor and fix that.
Container rebuilding

Fixing anything related to the container requires rebuilding the image. When you're using images from public registries you can not initiate image rebuild even when you know it's needed. Your best option is to contact the original uploader and to convince them to rebuild. That may take significant time during which the containers running that image remain vulnerable.
Images from untrusted sources

In addition to the points above you put enormous amount of trust into people who provide the container you're running. In containerless scenario you're required to trust the vendor who provides the base OS and the developer who provides the custom applications you run upon that OS. When containers come into play, you must extend your trust to the maintainer of the container image, to the vendors who provide the base image that image is built upon, to all the developers who provide any piece of code included into that container. It does not even require malicious intent to introduce a vulnerability into the resulting container, simple incompetence of any of the parties involved may be just enough.

This is why containerizing any workload comes with a significant extra cost of designing and automating security maintenance procedures. It is easy to skip this step when you're a hobbyist - but that's just burying your head in the sand and waiting for some script kiddie or botnet to hijack your network.

Here is a rough overview of the required overhead:

You need to run only containers that are based on the images you've built yourself. This is the only way you can ensure swift rebuilding in case one of the base images provides a security update. This step may include running your own image registry and build service.
You need to audit every Dockerfile you intend to build. This can only be done manually. And you need to check all the base images in the chain up to either a FROM scratch stanza or to a base image from trusted list.
You need to maintain a list of trusted base images that come from vendors with good reputation in regards to handling security issues.
You need to blacklist any image that does not come either from a trusted list or from a Dockerfile you've audited yourself.
You need to setup automated image rebuilds and container rollouts: a) on schedule b) on any update in the base image dependency chain
You need to setup automated vulnerability monitoring for the images you're running. This will require a lot more effort than subscribing to RSS feed of your distro security announcements - as it would've been the case with containerless deployment.

Add that on top of usual container orchestration chores - and bare metal suddenly becomes attractive. Docker and Kubernetes are great tools that solve real world problems but using them in a secure manner requires continuous dedicated effort. For enterprise deployments the benefits of containerization usually outweigh the extra maintenance cost, but for hobbyist use I'm not so sure.

Manage Python virtual environment from your Makefile

2019-10-01T00:00:00+03:00

I often use Makefiles not just as a build tool but as a handy way to execute sequences of commands. The commands I've found myself executing again and again lately are the ones to manage Python virtual environments:

Create new venv
Update pip to the latest version that enables all the cool new features
Install project requirements
Delete venv and redo it again to see if everything still works from clean slate

The process is tedious and begs to be automated. And Makefile is a good fit because in addition to basic scripting capabilities it offers proper dependency handling that simplifies the task quite a bit.

The outcome of my attempts at such automation is Makefile.venv - a Makefile that seamlessly handles all virtual environment routines without ever needing to be explicitly invoked. Instead, you write make targets that depend on venv and refer to all executables in virtual environment via $(VENV)/executable, e.g. $(VENV)/python or $(VENV)/pip.

Using Makefile.venv is easy:

.PHONY: test
test: venv
    $(VENV)/python -m unittest

include Makefile.venv  # All the magic happens here

Despite its apparent simplicity this Makefile will do very much when invoked (watch a screencast):

A virtual environment will be created in current directory
Pip will be automatically updated to the latest version
Project requirements will be installed (both requirements.txt and setup.py are supported)
If setup.py is present, the project will be installed in development mode into venv (pip install -e) - all changes to the source code will immediately affect the package in virtual environment.

All these steps will be repeated in case requirements.txt or setup.py is modified. That means you'll never have to worry about syncing venv with its description. Add new dependency to setup.py and consider it installed, because there is no way it'll be forgotten the next time you invoke make.

If you'll need to debug something interactively, there are make python and make ipython for REPL and make bash (or shell or zsh) for shell, but I rarely use those. Most of the time running make with my targets for executing the entry point or running unit tests is enough. In fact, I've noticed that after introducing Makefile.venv into my workflow I've completely stopped activating virtual environments manually.

I encourage you to try Makefile.venv and hope you'll find this approach useful. If you have some comments or would like to point out the faults of using Makefiles for venv, please shoot me an e-mail or create an issue at GitHub project's page.

PS: Makefile.venv was inspired by this StackOverflow thread and by this blog post from the authors of Bottle.py

Cygheap base mismatch in Git for Windows

2019-08-26T00:00:00+03:00

This error has haunted me for several months:

4 [main] head (6660) C:\...\usr\bin\head.exe:r
*** fatal error - cygheap base mismatch detected - 0x612C7410/0xAF7410.

This problem is probably due to using incompatible versions of the cygwin DLL.
Search for cygwin1.dll using the Windows Start->Find/Search facility
and delete all but the most recent version.  The most recent version *should*
reside in x:\cygwin\bin, where 'x' is the drive on which you have
installed the cygwin distribution.  Rebooting is also suggested if you
are unable to find another cygwin DLL.

The tricky part is I am not even using Cygwin. I'm running bash with Git for Windows as published at the official website.

I've tried suggested solution from the error message, googled around quite a bit, I've even dabbled with Windows Security settings (ASLR) - nothing helped and have almost made my peace with the fact that every fifth commandline action will fail loudly.

And yet after recent release I've decided to try one more time. This time I've downloaded the portable build for 64-bit architecture, and it worked! I don't know if it was the fact that I was previously running a 32-bit Git and bash on 64-bit Windows 7 or if Git maintainers have tweaked their build process for 2.23.

If you're experiencing segmentation faults while running bash from Git for Windows package, you should check for architecture mismatch with your OS and/or for newer (2.23+) Git version.

On dotfiles management

2019-07-30T00:00:00+03:00

This will be yet another description of dotfiles management by some random person on the Internet. I will try to explain what my setup is like and why it is that way.

If you're not yet using version control software for your configuration files I strongly encourage you to start doing so, whichever way you like. These pages are good places to start:

My chosen approach

After several attempts that have spanned across many years I've understood that neither tracking home directory directly with Git nor symlinking all of the dotfiles from a single directory tree is working for me. Both ways lead to a mess in the repository and made the whole endeavor of tracking changes cognitively expensive, so I inevitably started slacking off.

I've looked at the existing tools that are meant to automate some of the process and did not find one that would suit all my needs. I've ended up writing a small shell script that takes care of dotfiles installation but the main value for me is in the repo layout, not the script itself.

All configuration files are grouped into directories by topic. These directories are somewhat similar to packages in GNU stow. Topic directories recreate the directory structure for the target location, by default $HOME. Files that are meant to be installed into target location have to contain an appropriate suffix at the end of filename (any other files are ignored):

.copy - for files to be copied over to new location
.link - for files to be linked to from new location
.append - for files to be appended to the target file

Default behavior may be altered by a dotfiles.meta file placed into the topic directory. It is essentially a shell script that is being sourced during topic installation. Its main purpose is to provide alternative values for PREFIX and SCOPE variables:

PREFIX value determines target directory where the dotfiles will be placed. Also if PREFIX is set the dotfiles will not get an extra dot in front of their filename (which is the default behavior otherwise).
SCOPE variable may be used to indicate that a topic requires root privileges to be installed (SCOPE=system).

Multiple topics may be installed at once either by providing all of their names as command line arguments or by listing them all in a text file and providing path to that file as an argument to the installation script.

Examples

topic-foo/vimrc.link will be symlinked from ~/.vimrc
topic-bar/bashrc.copy will be copied over to ~/.bashrc
topic-baz/default/keyboard.copy with PREFIX=/etc will be copied to /etc/default/keyboard
topic-baz/file/without/valid/suffix will be ignored

More examples may be found in my dotfiles repo.

Comparison with existing tools

Strengths

Very small number of dependencies makes this script usable across all my Linux and Windows machines. It requires only the core GNU userland: bash, coreutils, find and grep.
Multiple install actions are supported (copy, link, append) unlike stow that only makes symlinks. More than that, my script detects if it's being executed on Windows machine and copies over any file that was meant to be symlinked - because symlinks on Windows are so tricky they're might as well be not supported at all.
Destination directory may be specified for each topic individually which makes it possible to install topics targeting different directories in one run.
Simple partial deployment. If machine requires only a subset of topics tracked in the repository it is easy to list them all in a plain text file or to provide them as command line arguments to the bootstrap script. yadm, for example does not provide such ability.
Dotfiles are not hidden in the repo by default. It makes no sense to have ~/.bashrc point to repo/bash/.bashrc instead of repo/bash/bashrc, so dots are added automatically for topics with default target PREFIX.
All operations are reversible because all overwritten files are backed up beforehand.

Weaknesses

Single pass execution. It means some topics may be left partially configured in case of errors. stow is a good example of cautious approach. This is an implementation detail and may be fixed in later versions of bootstrap script.
No support for tree folding/unfolding. I consider that an overkill for simple configuration management.
No automated reverse operation. In case you want to undo the changes made by this script you'll have to restore backups manually from $DOTFILES_BACKUP

Installing One by Wacom in Debian Stretch

2018-08-22T00:00:00+03:00

I believe there are many people who run Debian Stable as their main desktop OS. This article is a short how-to on enabling newer hardware in Debian Stable without switching to another version or distribution.

NOTE: This article was written in 2018, new Debian Stable (Buster) has been released since. The instructions below were written for Debian Stretch and have not been tested with other releases.

One by Wacom

Wacom has released their new graphics tablet, One by Wacom in the Fall of 2017 (judging by the dates of reviews in online shops). Linux drivers for this model (Small: CTL-472, Medium: CTL-672) were added to the git repo in December 2017 and were released in March 2018 (input-wacom-0.39.0).

That is several months after the current Debian Stable (Stretch) was released (June 2017). So there is exactly zero chance of it containing drivers for that hardware - "stable" means no new software except security updates gets introduced during the lifetime of the release (until at least 2020).

Fortunately, you don't need to switch distribution to use newer hardware. You don't even need to compile anything from source. Even more so, PLEASE DON'T INSTALL INPUT-WACOM FROM SOURCE, that will most likely lead to some undesired side effects.

Debian backports

Backports project was created exactly for cases like this. It allows users to install newer versions of some packages without breaking anything while keeping Debian overall at the same (stable) version.

To enable support for One by Wacom in Debian Stretch you need to:

Add stretch-backports to your sources.list.
Update (2019-11-15): Now that page contains instructions for Debian Buster. If you wish to enable backports in Debian Stretch you still can follow them while replacing every occurence of "buster-backports" with "stretch-backports"
Install newer kernel from backports: apt-get -t stretch-backports install linux-image-amd64. If you're running Debian on different CPU architecture, replace -amd64 with the corresponding suffix, like -686-pae for older 32-bit computers or -arm64 for ARMv8 CPUs).
Reboot your computer

That's it! Newer kernel will have updated drivers for your graphics tablet and it will be detected automatically. You can start using it right away or tweak some pressure options in your favorite graphics application (e.g. for Gimp it's in Edit -> Input Devices).

Timeline recap

Debian 9 (Stretch) was released in June 2017 and will be supported until at least 2020
One by Wacom (Small: CTL-472, Medium: CTL-672)
- Available in retail: Fall 2017 (judging by the dates of reviews in online shops)
- Driver for Linux: patch added in December 2017, drivers released in March 2018

Log messages (for reference)

Debian 9 (Stretch) without proper driver

# dmesg|grep -i wacom
usb 5-1: Manufacturer: Wacom Co.,Ltd.
input: Wacom Co.,Ltd. CTL-472 Pen as /devices/pci0000:00/0000:00:1d.0/usb5/5-1/5-1:1.0/0003:056A:037A.0001/input/input95
wacom 0003:056A:037A.0001: hidraw0: USB HID v1.10 Mouse [Wacom Co.,Ltd. CTL-472] on usb-0000:00:1d.0-1/input0
wacom 0003:056A:037A.0002: Unknown device_type for 'Wacom Co.,Ltd. CTL-472'. Ignoring.

Debian with updated kernel from backports

# dmesg|grep -i wacom
usb 6-1: Manufacturer: Wacom Co.,Ltd.
input: Wacom Co.,Ltd. CTL-472 as /devices/pci0000:00/0000:00:1d.0/usb6/6-1/6-1:1.0/0003:056A:037A.0001/input/input23
hid-generic 0003:056A:037A.0001: input,hiddev0,hidraw0: USB HID v1.10 Mouse [Wacom Co.,Ltd. CTL-472] on usb-0000:00:1d.0-1/input0
hid-generic 0003:056A:037A.0002: hiddev1,hidraw1: USB HID v1.10 Device [Wacom Co.,Ltd. CTL-472] on usb-0000:00:1d.0-1/input1
input: Wacom One by Wacom S Pen as /devices/pci0000:00/0000:00:1d.0/usb6/6-1/6-1:1.0/0003:056A:037A.0001/input/input24
wacom 0003:056A:037A.0001: hidraw0: USB HID v1.10 Mouse [Wacom Co.,Ltd. CTL-472] on usb-0000:00:1d.0-1/input0

Enhanced file path completion in bash (like in zsh)

2018-07-14T00:00:00+03:00

Zsh offers a lot of significant improvements over traditional shell experience. Some of those can also be implemented in bash, but others are not. For a long time I've thought that advanced file path expansion is something that can't be done in bash. Today I prove myself wrong.

Background

When the Tab key is pressed, zsh expands incomplete file path assuming any of its elements might be incomplete, while bash expands only the last piece. For example: cd /u/s/app<Tab> will produce nothing in bash, but will be expanded to cd /usr/share/applications in zsh.

This feature is a huge time saver, but it does not justify switching the shell completely (for me). So I've been looking for a way to enable the same behavior in bash.

The solution had to be:

Portable between Linux and Windows (msys);
Implemented in configuration files or short scripts that can be carried around easily. No third-party tools like fzf that would require system-wide installation or other tricks to enable.

All I've been able to find was this StackOverflow thread. Accepted answer suggests using a new bash function that is later bound to Tab keypress. The function provided is a quick hack that was not tested thoroughly and has problems with spaces in file path. The author recommends to use his function along with the default completer, not instead of it, so it was not what I needed.

Workaround

Typing cd /u*/s*/app*<Tab> is somewhat better but not as streamlined an experience as the one zsh offers.

This turned out to be an inspiration for the proper completer function though.

Better solution

I have coded a small function that adds wildcards to each element in the path and executes normal bash completion procedures with modified input. After a bit of documentation digging I've been able to inject this function into normal bash completion process. I'm pretty happy with path expansion now.

To enable the described behavior source this file from your ~/.bashrc and run _bcpp --defaults. Supported features are:

Special characters in completed path are automatically escaped if present
Tilde expressions are properly expanded (as per bash documentation)
If user had started writing the path in quotes, no character escaping is applied. Instead the quote is closed with a matching character after expanding the path.
If bash-completion package is already in use, this code will safely override its _filedir function. No extra configuration is required.

Watch a demo screencast to see this feature in action:

Liberating effect of Ansible

2018-06-26T00:00:00+03:00

Maintaining two or three Linux machines is not that hard of a task. For many years I have thought it was not worth the effort to automate - regular backups and version-controlled configuration files seemed to be just enough.

And then Ansible had blown my mind.

History

It all started with a web server and a series of subpar hosting providers. Setting that server up for the first time was an adventure. Repeating the setup for the first relocation has given me a chance to introduce some improvements but was otherwise uneventful. I started dreading the process when the need for the third iteration had arisen. I was willing to put up with sluggishness and several short downtimes just to delay the move to a new server. That was not OK.

Automating has clearly become a worthwhile task. I chose Ansible because it doesn't require any special software on the controlled machines and because it is mature enough to remain backwards compatible after updates. There are numerous reviews of pros and cons of different configuration management systems, but that's outside the scope of this article.

Unexpected outcome

Ansible has definitely succeeded at the task I've thrown it at. That was expected. What I couldn't foresee is how this experience would affect my mindset - it was like a breath of fresh air! I have suddenly started to understand why there exists all the buzz around the cloud and why cloud service providers are dominating the market of virtual servers.

Certainty

With my Ansible playbook I did not just automate the setup process, I created an enforceable specification of what my server has to be like. At any time in the future after executing the same playbook I can be certain that all configurable parameters will be at the values I've defined.

Idempotent behavior allows me to run the playbook again and again without fear of breaking anything. It should be noted though, that not all Ansible modules are idempotent and one should always check the documentation before incorporating a new module into the playbook.

Immutability

There is even an option to take it one step further to truly immutable infrastructure where any server can die at any point and no harm will be done. This approach is for a bigger scale than a humble hobby project, though.

I have limited the immutability effort to a gentle reminder in /etc/motd and a policy among administrators (me, myself and I) that no configuration changes are to be made via shell connection.

Disposability

Now it is incredibly easy for me to replicate that web server. It requires only one command and a short coffee break. That means I can switch the hosting provider at any moment I want. Choosing the hosting company has become a non-issue: I invest only a minimal payment and no labor at all. If I don't like what I get I'll be gone in no time.

Easy replication also means I can fire up another server for testing purposes, then destroy it in a couple of hours and it will cost me only a few cents because of hourly billing that most providers offer nowadays.

Emotional detachment

Setting up a new server used to be somewhat similar to moving into a new apartment. First, there was scrupulous comparing of offers, after that a dramatic moment of signing the lease and then the silent minutes alone in empty apartment, staring at the walls. Each server was important and loved, breaking or destroying it on purpose was unthinkable.

Now it's more like checking into a room at the hotel. And you get to be a celebrity and send the rider list in advance, so that everything is set up to your liking when you arrive. You can also check out at any moment without worrying about the lease. There is no need to lug your favorite vimrc along because you won't be editing anything there, in fact you'll hardly ever spend any time logged in at all.

Randy Bias has summed this attitude in the pets versus cattle analogy. I think it's quite on point even when you're managing a single machine, not the fleet of servers.

Infrastructure as code

And last, but not least, all you do with Ansible is automatically documented in the playbook. You can use any version control system to track when any particular change was introduced and why. That allows to revert the unwanted changes as easily as was introducing them in the first place.

Afterword

When you're outside the professional IT crowd it is not obvious that the cloud infrastructure and the corresponding tools are within the reach of hobbyist enthusiasts. But they are and they offer just as much value for personal projects as they do in production environment.

If you find yourself tinkering with Linux administration more than once a week it'll be worthwhile to try out Ansible or some other configuration management tool.

Accidental submersion into web development

2018-06-16T00:00:00+03:00

The Library Problem

I love reading books. My wife loves reading books. We enjoy shopping for books and we live a ten minute commute away from a huge used books store. That means we have a lot of books. Like, really a lot. A little more than one thousand.

We have lost count how many times we've bought a book that we already owned. Even more often we had foregone buying a book we liked because we were sure we have already bought it - only to find out we've been mistaken and have only considered buying that very book earlier. This has become a problem. The Library Problem.

We needed a way to catalog all our books. The catalog had to be accessible from mobile devices (to look up a book while at the book store) and to be easy to use. That is to add and edit book information, of which we've needed plenty: in addition to standard set of author, title and publishing year we wanted to be able to track book series and keep the list of missing books to look out for the next time.

I admit that my research into the subject matter was not scientifically thorough. I've dug up several comparisons of existing tools and have read several blog posts of people who have faced the same problem before. I particularly recommend this one. And I have decided that no pre-existing tool will meet our growing expectations.

Contents

The Library Problem
Naive foray into application architecture
The unknown unknowns
It works!

Naive foray into application architecture

I have never developed an application for any end user other than myself and I didn't even know I was about to start developing one.

Single spreadsheet approach

The initial idea was that of a "document". It had to contain some essential information about every book we own and (optionally) a second list of books we want to acquire. Excel spreadsheet seemed like a natural fit. We are both spending a better half of each workday juggling spreadsheets in Excel, so developing and maintaining such "document" appeared doable.

I have started drafting the column structure, applied some formatting tweaks and the skeleton of the "document" was ready. We have entered the test batch of books and were about to begin testing.

Obstacles arose when we were entering information for the first fifteen books. Manually typing in all the fields is tiresome and slow. Errors inevitably happen. What if I miss one letter in the author's name? That book will be lost when applying filter on author column. We need autocompletion! What if I accidentally switch the order of digits in ISBN? We won't be able to do a web lookup for that book later. We have to write a custom ISBN validator in VBA!

And the spreadsheet began to amass VBA code, data validation and conditional formatting rules. Full-blown spaghetti style. Version control has become a problem. What's the difference between versions 0.0.13 and 0.0.19? I had only a vague idea.

I stopped myself when I was about to sketch up a UserForm for data input. Excel road was leading me nowhere. It was difficult and even if all the difficulties were to be overcome it imposed some suboptimal compromises on us:

Single table storage limited the data structures we would be able to enter and view. If the book was written by two authors, which one should come first in the "author" string? How would we do column filtering in that case? What about sort order?
The local nature of storage meant there had to be one designated place for making changes (home laptop). Any changes made in other locations (smartphone, thumb drive) had to be agreed to be declared discardable.
The spreadsheet had to be exported to HTML to be accessible from smartphones. XML and XSLT made this possible but not very pleasant. Although, I am rather proud of the VBA code I wrote to save/load XML data automatically upon opening the workbook. The data was completely decoupled from the representation.

I'm glad I did not waste more time pursuing this path, but it was still hard to let go. It took quite some time for me to return to this project afterwards.

Local database driven application with web interface

A relational database was the logical solution to spreadsheet limitations. Store authors separate from books and manage how the former relate to the latter! Scientists who pioneered the relational model in 1970s were pure geniuses and now the whole world relies on their work.

I drafted the database schema on a piece of paper and have discussed it extensively with my wife. That's probably the point where book reviews were added to the requirements list. The database idea made me very enthusiastic and had swallowed a lot of free time, but that idea alone could not provide a complete solution for the problem at hand.

Data input and representation are what the Library project was all about. Yet relational database management systems provide only the storage solution, not the full package (if you don't take Microsoft Access into account). So I had to figure out how to implement the user interface on my own. I chose to write it in Python because I was somewhat familiar with the language and I enjoy how clean and readable Python code is. I'm not a programmer, so other options were either learning a new language or choosing between VBA and Bash, none of which could be considered enjoyable.

I have briefly considered building a conventional desktop GUI. The UserForm experience was still fresh and rather traumatic, so I was reluctant to dive into another UI toolkit. Even if I were to, which one should I have chosen? Qt seemed nice, but its Python support reports were contradictory. GTK? Installing it was rather tricky in Windows. Something non-crossplatform? And what about my Linux laptop?

I've had a bunch of HTML templates left from XML/XSLT operations and they could be trivially transformed to be used in conjunction with the database. While the catalog pages could be statically generated, the data input required some sort of server to interact with. And I've had zero experience with that.

Quick Google search has introduced me to the concept of web frameworks and I have semi-randomly chosen Bottle. At that moment I've had no intention to expose it to the open network, the app and the database were to be stored on the USB stick and launched locally when needed. Smartphone interaction was planned either via LAN or using saved HTML pages for read-only access when not at home. Bottle uses no dependencies other than standard library and the whole framework is packaged into a single file. It was perfect for the portable app scenario.

After the development started and I saw how difficult it is to make a web application, I've decided there is no point to confine the result of all that labor to a single computer. Why use static (maybe even outdated) HTML dumps on the smartphone when we could access full functionality of the application via web site?

Traditional web application

The development was already going on, powered by Python and Bottle. What I was missing was the server to host my application at. Turns out launching a web site can be very cheap these days! Even as cheap as free.

I've started with a free domain name from Freenom and a free hosting from Red Hat's OpenShift. Both of them have had upsides and downsides but no downsides were significant enough to turn down the price of free. Freenom domains are considered low quality from SEO standpoint, but that was completely irrelevant in my case. OpenShift server had to receive at least one http request every 24 hours to stay awake, and I've cheated that with a cron job on my router. No inconveniences whatsoever.

I have to digress to emphasize: OpenShift was great! It was easy to set up and very convenient to use. Its documentation was very thorough and up to date. OpenShift has introduced me to some concepts I would have not encountered otherwise, like automated deploying after git push and handling proper wsgi server. And all of that was for free. I'm very thankful to Red Hat for that.

Unfortunately, as Mr. Heinlein used to reiterate, there ain't no such thing as a free lunch. Red Hat could not pay the bills for all the computer enthusiasts forever and version 3 of OpenShift imposed much tighter restrictions on the free accounts. I've had to move to a cheap virtual private server and to learn to set up and maintain an Apache instance on my own. New VPS was sometimes too slow compared to what OpenShift has offered and I might need to migrate to another hosting provider later, but everything was fine for now.

I could concentrate on the development.

The unknown unknowns

Little did I know that while I was keeping myself busy with seemingly important problems such as whether to store images as blobs in database or as files on the file system a number of much more important problems were creeping up behind me. Donald Rumsfeld has called these things the unknown unknowns - the things that we don't know while remaining unaware about the very fact of not knowing.

ORM

SQLite to store data. Web framework to interact with user. Python to glue them together. The need in all these parts comes naturally, you could not "forget" to use any one of them. Yet they are not the only parts that are needed.

Nowhere in Python's database API documentation it says that composing each query individually is highly inefficient in a database-driven application. No one hints that there exist a whole other class of libraries called Object-Relational Mappers (ORM). And I was not clever enough to deduce that on my own.

But I wasn't too dumb either. I figured that repeating myself every time I needed to run a simple query was wrong. So I stashed that code away into SQL class. I figured that Book and Author objects require a lot of common methods to interact with database. So I separated that code into TableEntityWithID class. I have essentially implemented an ORM without knowing what ORM is. Of course, it is far inferior to SQLAlchemy and the likes. Of course, I will never use it in another project now that I know that there are industry standard solutions in this area. But I see no point in replacing my (suboptimal) implementation now. Because it works and nothing is broken and nothing is missing.

I consider my ORM to be a valuable educational experience even if it somewhat stalled the overall progress of the project.

Multi-threading

While I was developing the application I was using Bottle's builtin wsgiref web server. Running that server in production environment is not recommended because it was not built to withstand all the nastiness of open Internet. It is also a single threaded server which means it can not accept a request until it's done serving the previous one. The single-threadiness did not concern me - my app was intended to be used by two or three users tops and simultaneous requests would happen very seldom. Yet all the proper web servers (Apache, Nginx etc) are created to scale up to thousands and millions of users. Of course, they are multithreaded and multiprocessing. My application was not ready for that. Some would say SQLite is not ready for that either.

At first I considered crippling Apache to use only a single thread or switching to another server that can operate in a single thread mode. OpenShift seduced me with maintenance-free setup of Apache and I've abandoned that idea.

The code I've come up with to use a different instance of some objects for each thread is rather ugly. And it works only 90% of the time. So I just restart the Python workers every hour to avoid HTTP 500 errors when the database gets locked up. Not the best engineering decision of mine.

There probably exists a proper solution to multithreaded SQLite access, but so far I have not guessed what keywords to ask Google for it.

Database migrations

I was aware that changing database schema after the database was populated is hard and can lead to data loss. So I have put a lot of thought into designing it. I've even fired up GraphViz and created a nice chart that I've printed out and looked at on the commute. I wanted the database schema to be iron-clad and to require no changes in future.

I was very naive.

Of course it would require changes. Everything around us changes, we change, our expectations towards software change. At that point I understood that updating database manually was not a solution: there would be no trace left whether the schema was updated, which changes were introduced and which version does it correspond now to. So I've coded a simple transition module that would do that work for me. And later I've learned that the process I was automating is called database migration, and there are tools written by professionals for it.

JavaScript is a lot of work

I tried to avoid adding dependencies whenever I could. So I've decided to write what little client-side logic I needed in pure JavaScript. I have no experience with any JS framework so I thought that avoiding to learn one would compensate for the inconvenience. I'm not sure it did.

Most of my JS code is written intuitively, with no awareness of best practises and is filled with "code smells". Writing from scratch was sometimes quite educating, e.g with AJAX calls - now I understand and can explain them better than ever before. Overall conclusion is that JavaScript is hard. And that frameworks probably exist for a reason. If I'll do another web app project, I'll probably look into some lightweight framework to save time writing JS, which I find not very enjoyable.

Modular design

One might think that splitting code into modules comes naturally. At least I did. And I was wrong. When I was not specifically thinking about their size some modules tended to grow large. A thousand lines is pretty hard to do a quick overview on, and even harder to split after the fact. Some modularity has to be designed from the beginning.

I don't know how I could've avoided that. Guess it comes with the experience.

MVC

This is another example of a sensible principle that's hard to come by without someone teaching you. Model-View-Controller is a logical extension of modularity principle which I was totally unaware about. I have made some intuitive steps in the right direction, but overall my application is a soup of interleaved components. That complicates maintenance and further development and makes the code more difficult to understand for other developers.

If I would've gone with a bigger framework like Flask or Django, MVC mindset might have been forced down on me. For a newbie who doesn't know anything some dictatorship isn't that bad.

It works!

After all the difficulties and complications (both expected and unexpected) I can proudly say that the Library project is a success. The website is up and running for almost a year. There has been no significant downtimes and no data loss, most of our books have been catalogued. And most important, me and my wife do enjoy using it!

The application supports:

Creating and editing book entries.
Storing and displaying book metadata, cover thumbnail and arbitrary related files. Each book can be connected with any number of authors, series and/or tags.
Using ISBN to fetch book information from several third-party sources
Queuing ISBNs for input. This is helpful when you process a lot of books with barcode scanner and don't have the time to clean up automatic metadata on each one of them
Adding 1-to-5 star ratings and text reviews to any book in the library
Searching for books by exact metadata match and with wildcards

You can access the source code on GitHub and see the site in action at https://morebooks.ml (registration is for family members only, sorry). Of course, there are plenty of improvements to be made (you can see how long the TODO list is), but the maintenance itself requires almost zero attention now and I can happily switch from being a developer to becoming the end user.

Excel as a CSV editor (with VBA)

2018-06-01T00:00:00+03:00

One might think that Excel is a decent CSV editor as it is, but it's not. It is a very capable CSV reader, I do not dispute that. When it comes to writing though, Excel does not match what you'd expect from a mature application:

It might change the delimiter character arbitrarily;
It might write numbers in the regional format that does not map to a number anywhere outside Excel;
It might add quotes that are inconsistent with the rest of the file.

If you're collaborating on the CSV file with others, their Excel version might have different defaults and produce incompatible output. Even if you're the only one working on that CSV, you can forget about clean diffs and sensible atomic commits to your version control system.

The only solution is not to overwrite CSV files you've opened with Excel. Use another tool designed specifically for dealing with CSV or edit the file manually in the text editor of your choosing.

Append to CSV with VBA

I wrote a small helper utility to append data rows to the CSV files from Excel that ensures you won't mess up the existing data. This is a one-day hobby project, and Excel serves more as the UI toolkit and runtime environment than as the spreadsheet application, so you should be careful if you decide to rely on that code. The project is licensed under the Apache License, Version 2.0.

Here is the code:

Main VBA module
The resulting application, packaged in a workbook.

The application reads parameters from named ranges, opens the required file, parses CSV header and displays a submission form for a new data row. Upon submission it combines new values into a CSV string and appends it to the file. All data manipulation is done in VBA. This app could have and should have been written in any modern language - it would probably have cleaner code. Excel is super easy to draft a simple UI though :)

The code is pretty straightforward so I'll highlight only the most interesting parts.

Reading and writing Unicode with VBA

Visual Basic for Applications is a hopelessly outdated environment. Unicode support can be achieved only with the help of COM interoperability, namely the ADODB.Stream object. This object provides a very comfortable interface for reading and writing text files in a bytestream mode, and also handles character encoding nicely.

Appending to a file is done via combination of seeking to the end of stream and writing the new data.

CSV packing and unpacking

I'm not exactly proud of how CSV string manipulations are implemented in the code. If VBA would've provided some nicer regex capabilities or a CSV-aware library it would've been better. I know about VBScript.RegExp, but it's an overkill for a small task my app was created to accomplish.

Current implementation can not handle a quote symbol in the middle of the field value. This is a known bug.

Demo

This the main and the only UI my utility offers. Inputs and buttons are meant to be self explaining. No value conversion is done when saving - the value of the cell is written as is, quotes are added if delimiter character occurs within the value.

Screenshot below is produced after loading demo CSV file with the following header:

ID,Column1,Column2,Column3 with very long header,"Column4, with delimiter in the name"

The project is published for educational and archival purposes. I'll be glad if you'll find any use for it.

Why software translation is a waste of time

2018-05-24T00:00:00+03:00

Disclaimer: I am not a professional software developer, and my opinion might not be as authoritative as yours.

My native language is not English and since my first encounter with computers I have used multiple localized and non-localized computer programs. All these years of "user experience" have led me to believe that software localization is more often harmful than not.

Software translation is a waste of time. Generally.

I am not against localization as a whole. It has many positive aspects like supporting foreign date and currency formats, right-to-left writing or alphabetical sorting. But translating user interface, configuration files, error and log messages to other languages had destructive consequences most of the times I've seen it.

Documentation loss

The moment software is translated its documentation becomes fragmented and incomplete. Even if the developer translates 100% of official documentation they will still lose everything written by others (blog posts, forum questions, bug reports).

I was twelve when a friend of mine gave me a book on photo editing. The authors explained how images are stored on computers, what is the difference between raster and vector graphics, but the narrative was mostly centered on using Adobe Photoshop - version 7.0, if I recall correctly. And that was one large useless book. Because the authors used the English version of that editor and all we've had was the translated one.

You might think it was a mistake on the authors' part, but they were smart and experienced people. They knew it was pointless to reference a translated version because no professional user would have used Russian interface at that time. And they knew that the next version might come with totally different translation for the same UI elements.

Incomplete or wrong translation is worse than no translation at all

If you are not sure you can afford a good translation, don't do one. I can not stress enough how confusing it is to have a piece of software that uses your native language, and not to be able to understand the meaning of its messages without translating them back to English first. This happens all the time when software is translated by people who do not use it daily and do not understand all the usecases there are.

I took part in translation of an open source program once. I was a student and I've had a lot of free time, so I thought I could do some good and contribute back to the software I thought was worthy.

It was a social media plugin for a bigger application. We had a team of maybe a dozen volunteer translators and a coordinator with write access to the source control system. Usually he would email us a day before the next release with a file containing strings that needed to be translated. And then the farce started.

Those of us who were available at the moment started translating. We didn't know where in the application we would later see those strings. Even if we weren't lazy (guilty) and would've launched non-localized development version of the application, we would not have been able to match 100% of new strings to all the places they'd be used at. The coordinator was not any less blind than the rest of us. He knew a lot about the application code base, but he was not a superhuman - he could not possibly track all the developers and understand all their intentions. So we shipped some embarrassing translation errors... I'm glad no lives depended on that software!

Lost in translation

I concede that our team was lacking in terms of organizational skills, after all we were just part-time volunteers. But the translators hired by big corporations are merely human too, and they make mistakes. Especially when the headquarters is pressuring to ship a new product.

For more than ten years Microsoft Excel, a flagman spreadsheet application used by millions, has had two duplicate entries in row/column context menu: "Вставить" and "Вставить". The first one had a nice icon and meant "Paste (copied cells)" and the second one was iconless and meant "Insert (new row/column)". They've removed the text from the first one now, converting it to a button. Ambiguity still remains (the pop-up text for the button is the same) but is less confusing. Especially since users are already accustomed to it :)

Unsearchable error messages

Have you ever received a cryptic error message and had no idea what it meant? I'm sure you have. That message would only become more cryptic if it was translated. And if the error is not exactly common or the app is not popular in your country, Google will not be able to help you either.

So, for the sake of your users' sanity, please do not ever localize error messages and log files! Help people to help themselves.

Untranslatable abstractions

Some ideas are just so new, or the problem domain is so narrow that there is no point translating the terms. The concept of File was foreign to every person seeing the computer for the first time - but that knowledge is easily acquired. It would not have been any easier explaining that same concept and labeling it Файл (Russian translation), so why bother introducing two terms?

"File" ship has long sailed, but new abstractions are being introduced every day. Translating them to multiple languages just slows their adoption and hinders communication between users.

Afterword

I'm not hoping we will wake up one day and The Tower of Babel didn't happen. This rant is mostly useless, but if at any time because of it a software developer will decide that their users are educated enough to understand written English or a software user will decide to acquire entry-level English skills, I'll consider my time well spent.

This has been stewing for quite some time... At least since 2012, after I've read this.

Unit testing in Power Query M Language

2018-04-01T12:00:00+03:00

As your code base gets bigger, test automation becomes more and more important. This applies to any development platform, including Power Query / PowerBI. If you reuse your code and improve some low level function later, test automation allows you to make sure your changes did not break anything that depends on the part of code you've just modified.

As far as I know, there are no tools that allow us to perform automated testing of functions and queries written in Power Query M language. That's why I've built a simple unit testing framework into LibPQ.

LibPQ UnitTest framework

The UnitTest framework is modelled after the only other unit testing tool I have experience with: Python's unittest. It offers the following features:

Test suites to arbitrarily group individual test cases
Assertion functions to test simple statements
Subtests to execute the same test on a sequence of sample inputs
Test runner and test discovery functions to execute your test suites
Test results table that can be analyzed either manually or with any automation tool you create

Inner workings of the test framework are described in the documentation. This article will demonstrate how it works.

UnitTest demo

All modules described here are imported with LibPQ, so a basic familiarity with the library is assumed (readme, getting started).

Let's create a basic test suite and save it in the directory listed in LibPQPath:

/* DemoTests.pq - sample test suite */
[
    Assert = LibPQ("UnitTest.Assert"),
    testFirstTest = Assert[Equal](6*7, 42),
    testAlwaysFail = Assert[Equal]("foo", "bar")
] meta [LibPQ.TestSuite = 1]

The test suite is a record (note the square brackets surrounding the code) that contains:

Two test cases (values prefixed with "test")
- The first test will pass because 6 times 7 is 42
- The second test will always fail because "foo" and "bar" are different strings
And one related value: Assert is a helper for building test functions. Its use is not required, but makes writing tests much easier.

The last line contains metadata that marks the test suite as such and allows test discovery tools to distinguish it from just another record.

Here is what UnitTest.Discover function will do when invoked:

Search all locally available modules for valid test suites (hence the metadata)
Execute each located test suite with UnitTest.Run
Return the test results as a table, reporting as much data about the failed tests as possible

In the screenshot above we invoke UnitTest.Discover with compact_output = false but when you'll have dozens of test cases you'll probably prefer default behavior (group test results by status).

More about UnitTest in LibPQ

If you liked the idea of unit testing M language code, check out the main UnitTest documentation and a more extensive test sample that makes use of subtests.

Getting started with LibPQ

2018-04-01T00:00:00+03:00

This is a step by step guide to getting started with LibPQ, an illustrated version of "Installation and usage" section of the official documentation.

Installation

LibPQ source code

The source code of the library has to be present in each workbook that uses it.

Create a new blank query: Data > Get & Transform > From Other Sources > Blank Query
Go to "Advanced editor" and replace the query code with the contents of LibPQ.pq (switch to "Raw" view to make selecting easier)
Save new query under the name LibPQ

Specifying modules location

After the previous step LibPQ doesn't know yet where it should get the modules' source code from. You can specify an unlimited number of local and web locations where the modules are saved:

Create a new blank query and name it LibPQPath
Copy the contents of LibPQPath-sample.pq and modify it in Advanced editor.

LibPQ will search for your modules first in local directories (in order they are listed), then in web locations. If the module is found, no further locations are checked.

It helps with the name collisions:

Let's say you have a module FavoritePets.pq stored in your module collection at http://yoursite.com/PowerQueryModules/
At the same time you use some modules from a friend's module collection at http://friendname.com/PowerQueryModules/
If your friend adds a module with the same name to their collection, all you need to do to ignore it is to make sure that your collection address is higher in the LibPQPath than your friend's.
That works both ways: you and your friend can continue sharing your module collections while using personal modules with colliding names without any problems.

Reusable template

It is not necessary to repeat the installation steps every time you want to use LibPQ. You can add LibPQ to an empty workbook and save is as a template for future use.

Usage

Importing existing module

Import any available module with LibPQ("ModuleName") when writing your queries in Advanced editor. LibPQ will search for the file named ModuleName.pq in all locations that you've listed in LibPQPath. If the module is found, its source code will be evaluated and the result will be returned.

For example, let's import Date.Parse from standard LibPQ collection:

That works because LibPQPath contains reference to https://raw.githubusercontent.com/sio/LibPQ/master/Modules/, where the source code for Date.Parse.pq is located.

Creating a new module

You can save any reusable Power Query function or query to be imported by LibPQ later:

Copy the code of that module to any text editor (I recommend Notepad++) and save it with *.pq extension
Place the module into any location listed in LibPQPath and it will become available for importing

If you have any further questions about LibPQ please create an issue on GitHub or contact me via e-mail.

Roads and Bridges - sustaining modern digital infrastructure

2018-02-23T00:00:00+03:00

This week I have stumbled upon a very thorough review of existing problems and hidden costs of sustaining modern (open source) digital infrastructure. Here it is: Roads and Bridges - The Unseen Labor Behind Our Digital Infrastructure by Nadia Eghbal.

The essay was created with support from the Ford Foundation and is published on their website under a Creative Commons license. Unfortunately, that website denies access to users from certain countries (like Russia), so here is a mirror of PDF version.

The author discusses important and often overlooked topics of why open source software gets built and by whom, of who pays the costs of building and maintaining that software and of how to ensure that the software we all rely upon continues to be reliable. The essay poses more questions than it answers, but I still consider it the best read on the topic of sustaining open source development.

In my case Nadia Eghbal was "preaching to the converted" so this post is me trying to spread her word. Please read her essay and please do not take open source software for granted. The costs of building it are just payed by others, may be you can figure out how to help them?

Expanding Power Query standard library - introducing LibPQ

2018-01-03T00:00:00+03:00

Power Query formula language (also known as M language) is a very capable yet not very flexible tool. It lacks some features taken for granted by developers who are used to other programming languages such as compatibility with version control systems, extensibility by third-party libraries, etc.

That is why I have started LibPQ - an open-source M language library meant to expand the standard library and to make it easier for others to do so. Its main features are:

Importing source code from plain text files located on disk or on the web

LibPQ stores its modules as plain text files with *.pq extension. Detaching source code from the workbooks that execute it has a lot of advantages:

The source code can be managed by version control system such as git
Multiple workbooks referring to the same module will always use the same (latest) code
It encourages splitting your code into smaller reusable units
You can edit the source code with any editor you like (autocompletion and syntax highlighting are nice features even though Power Query's Advanced Editor does not have them)
Sharing your code and collaborating becomes much easier

Supporting several import locations ordered by priority

LibPQ does not dictate where you store your source code. Inspired by Python's sys.path it enables specifying unlimited number of local and/or remote sources (ordered by priority). When importing a module, LibPQ will check these sources one by one until the required module is found.

Unit testing framework

Having source code detached from the workbooks encourages you to improve and refactor existing modules. To make sure you do not introduce regressions you should cover your code with unit tests.

There are no unit testing tools in standard library, but LibPQ offers a basic unit testing framework that supports test discovery, grouping tests into test suites and comes with a bunch of handy assertion functions. To learn more read this help page.

A collection of general purpose functions and queries

And last, LibPQ contains some general purpose modules that you might find useful. If not - go write some new ones, you have the tools now!

LibPQ is built in such way that you do not need me (or anyone else) to approve of your work. Save your code to any convenient location, and LibPQ will help you to import it into your workbooks. You can even keep your modules private, no pressure here. Have fun and enjoy your coding!

Loops in Power Query M language

2017-10-31T00:00:00+03:00

Power Query Formula Language (also known as M language) is sometimes difficult to get your head around. This article explains how someone familiar with loops in other programming languages can approach the same concept in M language.

First of all let's look at the definition given by Microsoft:

The Power Query M formula language is optimized for building highly flexible data mashup queries. It's a functional, case sensitive language similar to F#, which can be used with Power BI Desktop, Power Query in Excel, and Get & Transform in Excel 2016.

"Functional" is the key word

Understanding (and accepting) that M is entirely different from most common programming languages has helped me as much as (maybe even more than) the exhaustive reference at MSDN. Functional language implies declarative programming paradigm: you describe what you want the computer to do instead of telling how to do it. If you're familiar with LISP or Erlang or Haskell, M might not look so foreign to you.

The code in M is not an explicit sequence of steps that will always be executed in the same order, it is just a bunch of ground rules that allow the computer to arrive to the solution. You can check that the order of lines within the let statement doesn't matter: as long as all necessary intermediate steps are described, Power Query will produce the same result even if you rearrange them randomly.

And that is the reason you don't get familiar control flow statements. If is kinda there, but it has its own quirks too. Loops are out of the question, unless you somehow manage to implement the function that does the looping for you. But...

There already is such a function! It is List.Generate!

List.Generate

This function takes 3 or 4 parameters, all of them functions. (You should always treat the each statement as a function because it is a shortcut for function definition.)

The parameters are:

start: a function that takes zero arguments and returns the first loop item.
condition: a function that takes one argument (loop item) and returns boolean value. If this function returns false the iteration stops, otherwise the loop item is added to the list of results. This function will be called at the end of each iteration.
next: a function that takes one argument (loop item) and returns the next loop item. This is the worker body of the loop. Be careful to return the next item as the same data type with the same structure, because the returned value will be fed to condition() and next() functions later. This function will be called at the beginning of each iteration.
transform: optional argument. A function that takes one argument - the item from the list of results and transforms it into something else. This function gets called once per each item in the list of results, and the list of values it returns becomes the return value of List.Generate. If transform() function is not specified, List.Generate will return the list of items at the moment when condition() returns false.

List.Generate might be easier to understand with the following pseudocode:

def List.Generate(start, condition, next, transform=None):
    results = list()
    item = start()
    while condition(item) == True:
        results.append(item)
        item = next(item)
    if transform is not None:
        output = list()
        for item in results:
            output.append(transform(item))
    else:
        output = results
    return output

A simple example

We will generate a table of data points for plotting a parabola. Internally we will be storing each item as the record with x and y fields. After that we will transform that data into a Power Query table for output.

let
    data = List.Generate(
        () => [x=-10, y=100],
        each [x]<=10,
        each [x=[x]+1, y=x*x]
    ),
    output = Table.FromRecords(data)
in output

In this example start() is an anonymous function that always returns the first data point, condition() and next() are also functions even though they are written using each shortcut. There is no transform() function because it is an optional parameter.

An example from real world

In the real world you will not need the List.Generate magic for such simple tasks, but you will still need it. Here is how I've used it recently.

Assume you have a list of tables that contain the data in the same format but for different time periods or for different locations. You have a separate list of locations (in the correct order), but each individual table does not contain that information. That's why combining all these tables into one would create a mess: you have to know which row comes from what table.

This can be done with List.Generate:

NamedTables = List.Generate(
    () => [i=-1, table=#table({},{})],  // initialize loop variables
    each [i] < List.Count(Tables),
    each [
        i=[i]+1,
        table=Table.AddColumn(Tables{i}, "TableName", each Names{i})
    ],
    each [table]
),

This code snippet assumes you have the list of tables in the Tables variable and the list of their respective names in the Names variable. The loop starts with index of -1 and an empty table, and adds a "TableName" column to each of the tables. After this modification the tables can be safely combined with Table.Combine(NamedTables) - no data loss will occur.

Conclusion

Using List.Generate should be considered a last-ditch attempt to looping. M has dedicated iterative functions for most common looping tasks, so please check the standard library reference before creating such C-style loops manually. They are rather hard to read, and readability counts!

I hope this article will help you to understand the Power Query Formula Language a little more. It is a powerful tool and even though it is not perfect, I hope you will find a lot of uses for it in your data crunching tasks.

An afterthought

Also, please keep in mind that the dot symbol in List.Generate does not have the same meaning as in other languages either. There are no object methods in M, and there are no namespaces, so the dot is just another character without any special meaning. It could have been a dash or an underscore - it wouldn't have mattered.

Temporary virtual environment for Python

2017-10-05T16:50:00+03:00

Using Python on Windows does not come as naturally as on Unix-like systems, so any help is appreciated.

I wrote a batch script to automate creation, setup and deletion of Python virtual environment. This can come in handy when you want to test something in a clean env, or to play with pip install and get acquainted with a new package from PyPI.

venv-temp.bat

You can download the script from https://gist.github.com/sio/...

The code is licensed under a permissive opensource license (Apache License, Version 2.0) so feel free to use it for your hobby and work projects.

Report any bugs, ideas, feature requests via GitHub issues/comments - all feedback is welcome!

Installation

Downloaded script does not require any installation.

If python is not available from your %PATH%, you have to specify the location of python.exe in the script (change the value of PYTHON variable).

Usage

Launch the script from cmd.exe to read all error output (if any) or by double-clicking if you're confident it works on your system.

After you're done experimenting and are ready to discard the venv, just end shell session with exit - the script will take care of cleanup.

If you close the terminal window without typing exit, the script will be terminated before it performs cleanup. This has no harmful consequences except taking 20-50MB of disk space. Old venv directory will be purged before reusing, so no changes you've made will affect the environment you'll get next time.

NOTE: If you have no internet connection, the script remains usable, but pip will print a lot of error messages while trying to update itself. Don't worry, that's ok.

Execute the same git subcommand in all local repositories

2017-10-05T15:40:00+03:00

If you work with more than one git project simultaneously, you often need to do the same maintenance tasks in each cloned repository:

check if there are some changes waiting to be pushed,
check remote URLs for all repos (e.g. when considering to switch from HTTPS authentication with GitHub to using SSH keys),
view last commit messages to refresh your memory.

Doing so with standard tools would involve a lot of cd-ing, and the inconvenience would deter you from checking all repos frequently.

That's why I wrote a simple bash script that helps to automate the boring stuff. The script is well-documented, so I won't discuss implementation details here.

git-projects.sh

You can download the script from https://gist.github.com/sio/...

The code is licensed under a permissive opensource license (Apache License, Version 2.0) so feel free to use it for your hobby and work projects.

Report any bugs, ideas, feature requests via GitHub issues/comments - all feedback is welcome!

Installation

Download the script from GitHub, add execution permissions
List the paths to the local clones of your git repos in a text file (one path per line). If you're using relative paths they must be valid relative to the location of the script
Update the value of PROJECT_LIST variable with the path of the file you've just created

Usage

All command-line parameters are passed on to the git command. When the script is launched without parameters, git-projects.sh checks the status of each repo.

Repositories are processed in alphabetical order sorted by paths listed in PROJECT_LIST.

Examples

Refreshing your memory

$ ./git-projects.sh log --oneline -3 --no-decorate

HomeLibraryCatalog
b5808f6 Always check the db before showing first run page
72d2481 Remove /quit route
75c707b Clean up destructors for WebUI and CatalogueDB

OpenShiftApp
b260276 Deploy from GitHub
05e0206 Deploy from GitHub
54e5cf1 Deploy from GitHub

server_common
bc33836 Indentation rule for Makefiles
72fb92a Use proper syntax for TODO in GitHub Flavored Markdown
a24e4f2 More familiar Home and Backspace behavior

View latest tag (if any)

$ ./git-projects.sh describe --tags --always

HomeLibraryCatalog
v0.1.0-71-gb5808f6

OpenShiftApp
b260276

server_common
bc33836

Checking project status

$ ./git-projects.sh

HomeLibraryCatalog
On branch master
Your branch is up-to-date with 'origin/master'.

nothing to commit, working tree clean

OpenShiftApp
On branch master
Your branch is up-to-date with 'origin/master'.

nothing to commit, working tree clean

server_common
On branch master
Your branch is up-to-date with 'origin/master'.

nothing to commit, working tree clean

Portable development setup for Python on Windows

2017-09-20T00:00:00+03:00

WinPython

https://winpython.github.io/

All-in-one distribution which comes with many difficult-to-build packages preinstalled. And their ...-Zero version is great for thumb drives!

Pip works just fine, but installing packages that require C compiler is always a pain on Windows. May be I should look into conda and see if it offers a portable variant.

NOTE: there are unofficial binary wheels for most common Python packages at http://www.lfd.uci.edu/~gohlke/pythonlibs/ The site's hosting is a little unreliable, so it might take a few trys to fetch a package.

Git Portable

https://git-scm.com/download/win

Git for Windows is now recommended by official Git website, and there always is a portable version.

This package provides not only Git but also bash and a basic MSYS environment (coreutils, sed, grep, awk, etc) which make life on Windows so much easier! Also, it comes with VIM preinstalled, which is a damn good editor and is preferred by many developers.

GNU Make

http://www.equation.com/servlet/equation.cmd?fa=make

Unfortunately Git for Windows does not come with GNU make preinstalled, so we have to download it manually. Great guys at Equation Solution are regularly building standalone versions of GNU Make for 32-bit and 64-bit Windows.

Downloaded file has to be placed somewhere in PATH.

GitHub with SSH keys

https://help.github.com/articles/connecting-to-github-with-ssh/

I don't know if it is even possible to setup HTTPS authentication without installing GitHub Desktop, and SSH key authentication works with GitHub same as everywhere.

I keep the keys on my laptop and the rest of the environment is on a thumb drive. That way I can develop anywhere I want, Windows comes as a given (sadly), and I don't have to worry about keys security, because they are not exposed to random computers.

Official documentation recommends using HTTPS just because it's easier for newcomers (https://stackoverflow.com/questions/11041729)

It does not require generating public/private keys and uploading the correct one to GitHub
HTTPS is allowed everywhere and SSH might be blocked by a firewall