Posts on tkainrad

Why Keyboard Shortcuts don't work on non-US Layouts and how Devs could fix it

Thu, 08 Apr 2021 00:00:00 +0000

Introduction

During the past 15 months, I have been thinking a lot about keyboard shortcuts and about how different applications handle them. I shouldn’t complain; I knew what I was getting into when I started to build KeyCombiner, an app for learning and looking up shortcuts and text snippets. Still, I didn’t quite know how much of a mess keyboard shortcut handling on the web is.

Do you use an international keyboard layout? Then you already know what I am talking about. You probably have had some issues typing keyboard shortcuts such as alt+/, or cmd+[. This post will help you to understand why this annoyance exists. However, the people who should most urgently read this post are web developers. Are you a web developer? Great! I will explain how you are currently letting down non-US users and how you can get us out of this mess.

I will focus on web applications. The behavior on desktop apps is often different. However, with the rise of Electron apps, web technologies are ever-present on the desktop as well. So, the arguments in this article probably apply to half of your installed apps as well.

Examples of Broken Apps

The problem I am talking about is so incredibly omnipresent that most international users are just used to the fact that they cannot type a good portion of a given web application’s keyboard shortcuts on their keyboards. However, web applications are becoming increasingly complex, and they are rapidly replacing desktop applications, making this problem bigger by the day.

This is most annoying when the most important keyboard shortcuts are inaccessible. A very common shortcut is / for accessing search functionality. Unfortunately, there is no /-key on most international layouts. Adding modifiers to produce this key with your layout rarely helps. For example, on my German layout, / is produced via Shift+7. Most web applications will ignore this. Similarly painful is when Electron apps use [ and ] for navigating backwards and forwards.

If you use a US layout, you might be surprised to hear about these problems. But rest assured, they are not new and I am not the only one who is affected. We are at a point where it is easy to find users complaining about this for almost any popular web application.

An interesting example are Google’s apps. For some reason, Google manages to handle international layouts correctly in Gmail, but fails to do so in any of its other applications. The web is full of complaints, e.g. for Google Sheets, Google Drive, and Google Docs.

Update: International Layouts now work in Google Chat. Interestingly, the fix was applied shortly after this article reached the front page of Hacker News. I am not saying that these events are correlated, but it is a nice thought.
Unfortunately, the problem persists for other Google Apps, such as GDrive.

Another, almost funny examply is Figma, the popular web-based vector-graphics editor. So many people complained that there is now a dedicated site with workarounds for different layouts.

This complaint in the Asana forums is funny too. They made some specific adjustments for some layouts, only to give up in the end and say that they will still not support more exotic layouts.

There are also plenty of complaints for Notion, see here or here. The competition doesn’t do much better, though, here is an open issue for Roam Research.

Finding all these applications with broken shortcut handling was very easy. I used KeyCombiner’s shortcut search to find all applications that use the US /-key in their shortcuts. Then I googled and found complaints for most of these apps. In the case of Notion and Google Drive/Docs/Sheets, I was already aware simply because I use these applications myself.

I want to be clear, broken keyboard shortcuts are not a law of nature. It is possible to implement web application shortcuts so that they can be typed with any layout. It shouldn’t be an excuse that even some of the most popular apps do not get it right.

Why is this happening?

What baffles me is that the problem exists in apps used by millions of people every day, developed by gigantic corporations who pride themselves with having the best engineers in the world. Why is this baffling? Because the underlying technical issues are rather trivial!

Web applications use JavaScript to process keyboard shortcuts. To do that, they listen to keyboard events that are emitted by your browser whenever your press a key. That’s where it gets a little messy. These keyboard events have several properties that can be used to identify which keys have been pressed. To make matters worse, three different events are associated with a single button press: keydown keypress keyup. Most applications use a Javascript library that handles all these things for them, primarily HotKeys.js or Mousetrap. Let’s look at what Hotkeys.js does:

let key = event.keyCode || event.which || event.charCode;

It uses keyCode, which, or charCode to find out which key was pressed. Why is this a problem? For starters, all three properties are deprecated. That isn’t even the main problem though. These event properties are simply not suited for finding out which character was pressed, at least when used with keydown or keyup. Still a vast majority of web applications use the keydown evente with these properties. It is very clearly documented that this should not be done and that it is not compatible across layouts, operating systems, and even browsers. To cite the MDN web docs:

Web developers shouldn’t use the keyCode attribute for printable characters when handling keydown and keyup events. As described above, the keyCode attribute is not useful for printable characters, especially those input with the Shift or Alt key pressed. When implementing a shortcut key handler, the keypress event is usually better.

Using Chrome on Linux with a german keyboard, my # key will produce the required keyCode for triggering / shortcuts. However, this does not work on Firefox, and neither does it work on Chrome on macOS. With many browser and OS combinations, there is no way to trigger this shortcut on a german layout. The same goes for many other international keyboard layouts that do not have a dedicated /-key.

If you want to see which button presses produce which event properties, you can use this neat little web tool: https://w3c.github.io/uievents/tools/key-event-viewer.html
Hint for developers: Switch your computer’s keyboard layout to something other than US to debug your application’s shortcut handling on different layouts.

Using this event viewer, you will see that the keypress event usually produces the correct code on any layout. Unfortunately, this event is deprecated as a whole and comes with its own problems. Still, the Mousetrap library can use the keypress event with fairly good results on international layouts.

As you can see, the situation is messy. However, developers are usually pretty good at dealing with complex problems. In the next section, I will point out some straightforward ways to do better than Google and other tech giants.

How you can do better

The good news is that this problem is easy to fix. Depending on your resources and the technical and semantical requirements of your application, there are a few things that you could do:

Simple Workarounds

If your web application does not have a huge amount of shortcuts, or if you don’t have the resources to spend a meaningful amount of engineering work on this problem, here are some trivial ways to work around the described issues.

Do not use special characters for your keyboard shortcuts.
That’s all you would need to do to make 99% of your international users happy. If you want to go one step further, you can try to stay on the home row or choose otherwise easy-to-type characters. In case you have an existing application with shortcuts that cannot be changed because your American users already learned them, consider the suggestion below.

Provide Alternatives
If you really do need to use special characters, provide alternatives that can be typed on all keyboard layouts. Both HotKeys.js and Mousetrap support multiple triggers when creating your bindings. For example if you have / as you shortcut for focusing the search field because your US users are accustomed to this behavior, just add alt+s as an alternative for international users. Some US users might welcome this change as well, because alt+s can be typed with just the left hand while / reqiures people to take their hand of the mouse.

Technical Fixes

Actually make the shortcuts work with any keyboard layout.
This means when the key combination is alt+/, the shortcut should be triggered when a german layout user presses alt+shift+7. This is surprisingly easy with some of the popular shortcut processing libraries. With Mousetrap, all you need to do is listen to the keypress events instead of keydown. In the next section of this post, I will mention some applications that work this way. Unfortunately, there is a minor downside to this approach. It can make some shortcuts tedious to type on some layouts, e.g. requiring two or three modifiers. It also has some potential for shortcut conflicts. Ctrl+/ is suddenly the same as ctrl+shift+7. You should be aware of that and not assign these things to two different bindings. The easiest way to avoid such conflicts is by staying away from the Shift modifier when defining your bindings.

If you want to go all the way, you could write your own shortcut processing code, and possibly make it into an open-source library. The current alternatives do not care too much about supporting different layouts. There are a lot of things one could do to solve all the issues described in this article. For example, you could use the key attribute, or even play around with the experimental keyboard.getLayoutMap API. KeyCombiner uses this API to display its on-screen virtual keyboards with the user’s layout. Sometimes, I think about making such a library myself. KeyCombiner already uses a fork of Mousetrap that has been customized quite a bit. However, I already have a demanding full-time job and a large side-project, so I would be happy if you beat me to it.

Allow users to change the default key bindings.
This way, users of international keyboard layouts, can change invalid bindings to something that works for them. As a bonus, you would make any shortcut enthusiast happy, and your marketing people could point out how all your sorry-ass competitors do not allow custom keyboard shortcuts.

Interestingly, I am not aware of a single web application that lets you customize keyboard shortcuts, even though it wouldn’t be that difficult to implement, especially with modern SPAs that would have to pull the custom bindings from the database only once. You could store them in your database, pull them out when a user logs in, and pass them to your shortcut processing library.

Wall of Fame

Fortunately, some applications already do it right. Three apps that I am using where my german layout works flawlessly are Gmail, Linear, and Discourse.

Gmail

It’s hard to believe that Gmail is created by the same company as Google Drive, Google Chat, and many other apps that do not handle shortcuts correctly on international layouts. The Gmail web interface works great on all layouts that I tested. On my german layout, I can even use AltGr+8 to archive a conversation ([). Similarly, search can be triggered via shift+7 (/). Note that this is the exact same shortcut that does not work in all the other Google apps. So, if you work at Google, maybe ask your colleagues on the Gmail team for a little help.

Discourse

Discourse is a great example because it is an open-source application, meaning that you can go to their GitHub repository and see how they are doing it. The answer is very simple: They use the Mousetrap library with keypress. As explained above, keyCode/which/charCode will contain the right code even on international layouts when used with keypress.

Linear

Linear is a relatively new and increasingly popular issue tracking software. It takes pride in having a keyboard first design, and being optimized for efficiency with extensive keyboard shortcuts. Their recent success story and my personal experience suggest that they deliver on this promise. Of course, many other web apps have a lot of keyboard shortcuts, too. However, with Linear, they actually work.

Do you know other applications that work with all keyboard layouts? Please let me know in the comments below, and I will add them here.

Conclusion

Next time you use an app that dangles its keyboard shortcut definitions in front of you, just so you can find out that they don’t work with your layout, please remember this post. You will know that there are ways to fix this.

If you are in the mood, you can write a nice message to the developers, explaining how they are probably misusing the keydown event and its keyCode property. We shouldn’t have to live like this.

An Interactive Virtual Keyboard to Visualize any Collection of Shortcuts

Fri, 22 Jan 2021 00:00:00 +0000

Introduction

An important part of KeyCombiner is displaying collections of keyboard shortcuts. Therefore, I have invested a lot of time to design searching and filtering features that help to browse even large collections.

Unfortunately, these features are not sufficient when you want to understand a collection of hundreds of shortcuts at a glance. I have been thinking about this problem since I started working on KeyCombiner almost precisely one year ago. Today, I am happy to announce that KeyCombiner offers a solution:
The Shortcut Collection Visualizer

Collection Visualizer for XCode, one of very few applications that use all 4 modifier keys at the same time.

It is heavily inspired by Waldo Bronchart’s open-source Application Shortcut Mapper. However, it is a new VueJS-based implementation, adding several additional features that work together with the rest of KeyCombiner. Most importantly, it can efficiently process KeyCombiner’s collection tables and hence works for any shortcut collection on KeyCombiner, even search results.

If you want to play around with it right away, go to any public KeyCombiner collection, e.g. for VSCode, IntelliJ IDEA, XCode, Chrome or one of the other 60+ public collections. If you want to fully understand its potential, please read on.

Features

Overview

The virtual keyboard packs a lot of data into a relatively small space. Each button of the keyboard consists of the following elements:

Elements on each key of the virtual keyboard.

Grouping by Modifier Combination

A proper keyboard shortcut consists of 0 or more modifier keys and exactly one non-modifier key. There are 4 modifier keys:

Ctrl
Shift
Alt
Cmd (macOS) / Super (Windows and Linux)

This order of modifiers is not random. KeyCombiner always shows keyboard shortcuts with precisely this order. There are good reasons for this.

This means that we have four boolean variables, resulting in $2^4$ possible modifier combinations. For each of these 16 states, the Collection Visualizer uses a different background color, or background gradient if there are multiple active modifiers.

To toggle modifiers, click on the virtual buttons with your mouse, or press the respective modifier key on your physical keyboard. The entire virtual keyboard will then update according to the active combination of modifiers.

It is very rare that a key has a shortcut for every modifier combination. However, it can happen, especially when combining shortcuts of multiple applications in personal collections.
(Please don’t tell me I forgot one of the 16 modifier combinations - it took me way too long to create this animation.)

Filter Collection Table

One of my favorite things about KeyCombiner’s shortcut collections is that I can filter them by context, category, or modifier combination with a single click using the panes on the side.

The collection visualizer expands on this concept. If you click on any non-modifier key, the collection table will show all shortcuts that use this particular key. To show all shortcuts containing the key F click on the F button on the virtual keyboard.

Filtering the collection table for all shortcuts that contain the F key.

Real-time updates on changes

Building personal collections of keyboard shortcuts and text snippets is the foundational concept behind KeyCombiner. You can then practice these collections with its interactive trainer, relying on spaced repetition techniques and advanced statistics to guide your learning progress. You can also use KeyCombiner Desktop to instantly look up all combinations in your collections without leaving your current context.

Oh wait, I am getting side-tracked. I meant to say that the collection visualizer updates immediately whenever you make a change to one of your collections. A change could be adding new shortcuts, editing existing entries, or re(moving) entries. This works in the blink of an eye, even if you remove hundreds of combinations at once.

Additional Features

I am getting the sense that this post will be too long for the average person’s interest in keyboard shortcuts. So, I will list some additional features in shorter form:

There are three levels of opacity:
1. Keys without any mapped combinations
2. Keys with mapped shortcuts, but none that use the current modifiers
3. Keys that have a combination with the currently activated modifiers

Different levels of opacity carry information.

If, for any modifier combination, there are two or more shortcuts bound to a key, the number of combinations in the top right of the button is marked red.
If there are two or more combinations on a key for the current modifier state, the shortcut description for this key says Conflict.
There is a small text below the virtual keyboard saying how many shortcuts are mapped onto the virtual keyboard, and how many combinations had to be skipped. (See Current Limitations)
The keyboard must be in focus if you want to activate modifiers by pressing the respective buttons on your physical keyboard. This is so that you can still use Ctrl and Shift for table selection operations without affecting the visualizer. Buttons below the virtual keyboard allow toggling the focus.

Use Cases

Quickly Grasp a Set of Shortcuts

Perhaps the most obvious use case is exploring a collection of shortcuts. The visual keyboard helps immensely in this process. Within seconds, you can get a feeling of which modifiers are used by a specific application and whether it uses Vim-like home row navigation or something else entirely.

The different layers of opacity aid this use case. Without activating any modifiers, you can already understand where the most shortcuts are located. This is supported further by the combination count in each virtual keyboard button’s top right. Filtering the collection table by clicking on a specific key lets you see all shortcuts for that key and understand how they are related.

See Conflicts and Free Combinations

At the moment, this is my favorite use case, as I have used it plenty of times already with great success.

I recently learned all VSCode shortcuts with KeyCombiner’s interactive trainer. However, since then, I have started to experiment with Foam and picked up some other extensions. All of these come with their own set of commands. So, I frequently have to find an available key combination for a new command I want to use efficiently. VSCode itself is not much help with that. It tells you after setting a combination that it is already taken:

Different levels of opacity carry information.

I guess it’s better than nothing, but trying multiple combinations and manually checking what other combination is already using that binding and whether you might be able to remove that other binding is not much fun. The collection visualizer made it trivial to see that there are actually plenty of free combinations in VSCode, only Ctrl and Shift are quite busy by default. Things start happening if you mix in Alt:

All modifier combinations with Alt are wide open for your own assignments in VSCode.

You can then go one step further and find free combinations that are easy to type. For me, these are combinations that I can type with just my left hand. If the non-modifier key is on the home row, that’s another big plus. In any case, a convenient shortcut should have a maximum of two modifiers.

Design a Coherent Set of Shortcuts

The collection visualizer helps design a coherent set of shortcuts, either for yourself or for an application you are developing.

Unfortunately, many application designers do not think very hard about keyboard shortcuts. Often, you end up with a set that is neither intuitive nor easy to type. Heck, even KeyCombiner’s own shortcuts are all over the place with sequences and different modifier combinations. Given that I work more or less alone on the project and try to be very efficient with my time, I didn’t think about these bindings enough. The collection visualizer makes this painfully obvious, and I will soon come up with new shortcuts. However, it will be very hard not to annoy users who have already memorized these shortcuts. So, I recommend that you be smarter than me and start to design a coherent set of shortcuts for your application right away. The collection visualizer is here to help you with that.

If you are not an application designer, you might still want to design a coherent set of key bindings for your personal use. Without any tools to assist you, this is a surprisingly hard tasks, especially when you try to find a coherent set for or multiple applications. You have to keep in mind which commands are available in these different apps, what the defaults are, and how to resolve these constraints into a set that works everywhere. The collection visualizer, along with KeyCombiner’s other collection management features, can help you get there.

Current Limitations

Above, I have written that a proper keyboard shortcut consists of 0 or more modifier keys and exactly one non-modifier key. However, KeyCombiner also allows sequences, such as the Go To shortcuts used by Gmail. I have been thinking a lot about how to visualize those on a virtual keyboard, but have not found a good solution yet.

Furthermore, KeyCombiner collections can also hold short text snippets, such as commands and programming language syntax. Many people use these snippets with the Desktop Apps' instant lookup. It turns your collections into an instant, context-aware, searchable cheatsheet. However, I struggle to find a way to visualize them on a keyboard.

Conclusion

In its first days, the collection visualizer has already helped me plenty of times. I improved my VSCode bindings, realized that KeyCombiner’s own default bindings are not intuitive, and found better ways to reuse my VSCode bindings in PyCharm and Eclipse. I’d be thrilled to hear about your experiences in the comments below or via mail.

I will write about the collection visualizer’s implementation in a future blog post. Spoiler: Vue and (S)CSS do the heavy lifting.

An app to show the shortcuts of the current application for Windows, Linux, and macOS

Sat, 12 Dec 2020 00:00:00 +0000

Introduction

Looking up keyboard shortcuts on the web takes you out of the current context and breaks your workflow. That’s why, as a Linux user, I have always been a bit envious of macOS users, who had access to tools that could instantly show the current application’s shortcuts, such as CheatSheet, and Pretzel.

On Windows and Linux, there was no such thing. You had to find shortcut information in the software’s documentation and hope that it was searchable. Or even worse, suffer a massive context switch and google for the shortcuts, until now. KeyCombiner Desktop is free to use and can show the active application’s shortcuts on Windows, Linux, and macOS.

Suppose you are on macOS and already use one of the existing solutions. In that case, you might still want to read further because KeyCombiner goes far beyond looking up the active application’s shortcuts.

How does it look like?

If a picture says more than 1000 words, an animation will speak for itself:

Instant shortcut lookup for three example applications.

Upon pressing the global trigger, the lookup will appear with a search field in focus, where you can start to type right away. The default trigger is super+shift+c on Windows and Linux, and cmd+shift+k on macOS. This can be configured, I use super+c on Linux.

The search terms will be applied to the shortcut description, the key binding, and some hidden fields like the shortcut category. Do you want to search for all file-related bindings that use Ctrl? Type something like file ctrl and you have your answer.

The lookup window disappears whenever you click outside of it, when you press esc, or when you switch to another application via cmd/ctrl+tab. If you close via esc, the search term will be cleared. If you use cmd/ctrl+tab, you can switch back to the active application without losing the cursor position.

What else can it do?

If you carefully watched the animation above, you might have noticed that the rows are grouped. The first group has the prefix Active: and is highlighted with a blue background. This group shows the shortcuts for the active application. Everything that is shown below this group is from the user’s personal shortcut and command collections.

To explain what these are, I have to provide some background. KeyCombiner is an application to organize, learn, and practice keyboard shortcuts. A core idea is to learn precisely the shortcuts you need. The only way to achieve this is to choose them yourself, which is done by creating personal collections of shortcuts and text snippets. I like to compare this concept to how you build playlists in music software. Instead of browsing your favorite artists’ albums, KeyCombiner allows you to browse shortcut collections of your favorite applications. Instead of adding songs to your playlists, you can add keyboard shortcuts and text snippets to your personal collections.

The instant lookup of KeyCombiner Desktop profits immensely from these collection building features, and vice-versa. Having instant access to your collections makes them a lot more valuable. In short, these two features complete each other.

The instant lookup works offline. Your personal collections are downloaded only on application startup. If you modified your collections while KeyCombiner Desktop was running, you may reload the lookup via (Ctrl/Cmd+R).

I will try to illustrate this with my personal experience. Currently, I have 10 personal KeyCombiner collections. My largest one contains more or less all shortcuts that I already knew when I started to use KeyCombiner. This collection is publicly accessible via this shareable link. Then, I did a blog post challenge to learn 50 new keyboard shortcuts as fast as possible. More recently, I learned all VSCode keyboard shortcuts, which evolved my developing habits and added another collection with 151 shortcuts to my repertoire. Add to that a few smaller collections I used for learning the shortcuts of Chrome DevTools, Nautilus, Spotify, and Vimium. Having instant access to all of these shortcuts without leaving my current context feels like a superpower, even though I admit that some of those superpowers you see in movies look even more powerful.

My most recent new use case is to look up regular expressions. There is an article on how to automatically add HTML <kbd>-tags to any text via a single regex replace operation. I use this for all of my blog posts, all <kbd> tags you see in this post are added this way. However, it wouldn’t be so convenient if I couldn’t instantly look up and copy-paste the required regular expression:

The lookup also shows shortcuts and text snippets from your personal collections.

KeyCombiner’s instant lookup is a cheatsheet of sorts. But it is context-aware, searchable, and you can expand it quickly by importing from public or shared collections. It is also friendlier to the environment than printing out PDFs and hanging them next to your desk.

How does it know the active application’s shortcuts?

There is a very boring answer to this question: By maintaining an extensive public database of keyboard shortcuts. You might have noticed from reading this post that I am quite excited about Keycombiner. That’s why I spend considerable time to parse the documentation pages of software tools for their keyboard shortcuts. It’s not as dull as it sounds like; creative use of regular expressions makes it quite fun. Don’t judge me; everyone has a weird hobby.

You can browse the database at https://keycombiner.com/collections/.
The instant lookup works for all desktop-based applications that are listed there. In particular, it should work for all apps that an average software developer uses (VSCode, JetBrains IDEs, Vim, Eclipse, Chrome, Firefox, Safari, Nautilus, Finder, Notion, Slack, Explorer, Terminal, iTerm2, Obsidian,…). This list is growing fast, and I am happy to take suggestions.

One downside of this approach is that KeyCombiner can only show the default bindings for the active application. If you have changed these bindings, that’s not ideal. Fortunately, If you went through the trouble of changing a binding, you usually won’t need to look it up. In any case, you can add the customized binding to a personal collection, and the lookup will show it as explained above.

Existing tools for macOS use a different approach. They rely on a macOS API to retrieve menu keyboard shortcuts for the current application. This has its advantages and disadvantages. The great thing about it is that it works automatically for all applications that have a menu. On the other hand, applications don’t necessarily register all of their shortcuts as menu shortcuts.

How can I use it?

Simply go to https://keycombiner.com/desktop/ and grab the installer for your system. On macOS, you need to grant some permissions for the instant lookup to work. Please refer to the bottom of the linked page for instructions on how to do that.

KeyCombiner is a SaaS with a generous free tier. All features described in this article are entirely free to use. If you use KeyCombiner a lot, please consider upgrading to a Pro subscription to get access to additional features and to support hosting and development efforts.

Are you a developer? You might be interested in how I implemented the checkout process for the Pro subscription. I recently published a blog post describing the implementation in detail.

Conclusion

KeyCombiner can do a lot more things than what’s described in this post. However, the instant lookup has quickly become one of my favorite features. Admittedly, I don’t use KeyCombiner every day for learning shortcuts. Usually, after I have mastered all shortcuts of a new collection, it takes a couple of days or even weeks until I get back to frequent practice.

In contrast, I use the instant lookup all the time. If I am on a different computer without KeyCombiner installed, I notice that something is missing after a couple of minutes.

Enough about me, though. I’d be thrilled to hear about your experience with the software.

Automatically add `-tags with a Single Regex`

Fri, 04 Dec 2020 00:00:00 +0000

Introduction

Until about a year ago, I had the same approach to regular expressions as most developers: Doing trial and error until the damn thing was working.

Then came KeyCombiner and with it some very specific challenges around parsing text that contains keyboard shortcuts. One of these challenges is writing blog posts about KeyCombiner:

These posts contain a lot of keyboard shortcuts that should be highlighted via <kbd> tags. By now, these tags are supported by most static site generators and blogging platforms, such as DEV.to. However, typing them in manually is extremely tedious. Initially, I felt quite smart when I added a user snippet in VSCode:

"kbd": {
    "scope": "markdown",
    "prefix": "kbd",
    "body": [
        "<kbd>$0</kbd>"
    ],
    "description": "Insert HTML kbd tags"
},

This lets me type kbd, press ctrl+space and hit enter to put my cursor in-between an opening and a closing <kbd> tag. However, it does not help with copy-pasted text, which is an essential use case for me. I write the first draft of my posts in Notion, as described in my post about personal knowledge management. Even if this were not an issue, it’s still quite a lot of keys to hit for just adding two tags. Another solution would be to write a script in some interpreted language and do all kinds of fancy things to this text. The problem is, this script will likely end up like many single-purpose scripts do: Abandoned and buggy.

So, I needed a better, more timeless solution: Regular Expressions.
Fear not, I won’t try to teach you general regex syntax. Instead, I will show how to use it for this particular use case. I will explain in detail how and why this works, but you can also just copy-paste the final regex and apply it for yourself. One of the best tools for putting the regex to work is Visual Studio Code, but any editor that supports regex-based replace operations will do.

How-To

Suppose we have the following text:

Modifier keys, such as ctrl, shift, alt, and cmd do nothing by themselves but can be used in combination with other keys. To paste without the original formatting, use ctrl+shift+v. Debugging is controlled via f5, f6, f10, and f11.

What we want to have is this:

Modifier keys, such as ctrl, shift, alt, and cmd do nothing by themselves but can be used in combination with other keys. To paste without the original formatting, use ctrl+shift+v. Debugging is controlled via f5, f6, f10, and f11.

The next sections will describe how we get there.

Modifiers and F-keys

We can already go pretty far with this regex that uses or operators (|) to match a predefined set of strings:

(f1|f2|f3|f4|f5|f6|f7|f8|f9|f10|f11|f12|backspace|tab|enter|shift|ctrl|cmd|alt|capslock|pageup|pagedown|ins|del|option|meta)

Putting the expression in brackets () means that it is a capture group. Capture groups are an essential concept when working with regular expressions. One of their greatest features is that we get to reference them by number in the replacement string:

<kbd>$1</kbd>

This means we will replace a match of the first capture group with itself ($1) surround by the desired tags.

The above text will become:

Modifier keys, such as ctrl, shift, alt, and cmd do nothing by themselves but can be used in combination with other keys. To paste without the original formatting, use ctrl+shift+v. Debugging is controlled via f5, f6, f10, and f11.

This is a good start, but we are not quite there yet. Non-modifier keys that are part of key combinations do not yet have the proper tags. We will deal with them in the next section. But before, we can shorten our expression a bit, by switching out the f-keys with a single sub-expression: f([2-9]|1[0-2]?). This will match any f-character followed by either 2-9 or a sequence of 1 and 0-2. It is equal to typing out the f-keys, but a little more fun. So we end up with this expression:

(f([2-9]|1[0-2]?)|backspace|tab|enter|shift|ctrl|cmd|alt|capslock|pageup|pagedown|ins|del|option|meta)

Regular keys in combinations

Matching regular keys is a bit more tricky. We cannot just match any v characters in our text. One approach is to match any single alphanumeric character that is preceded by a +-sign. To do this, we need to use something called Lookarounds, more more specifically, a Lookbehind: (?<=\+) The nice thing about lookarounds is that they do not increase the matched character sequence. Instead, they specify what must (or must not) come before or after the matched part, which is very useful when applying replacement operations. Adding this to our expression results in the following:

((?<=\+)[\w\d]{1}|f([2-9]|1[0-2]?)|backspace|tab|enter|shift|ctrl|cmd|alt|capslock|pageup|pagedown|ins|del|option|meta)

Using this expression, we end up with the desired text already:

Modifier keys, such as ctrl, shift, alt, and cmd do nothing by themselves but can be used in combination with other keys. To paste without the original formatting, use ctrl+shift+v. Debugging is controlled via f5, f6, f10, and f11.

However, we shouldn’t celebrate too early. There are still some issues and special cases to take care of:

What happens if a modifier appears within another word, such as ins in insert?
What happens if we apply this to text that already has some <kbd> tags?

The answer to both questions is clear: Terrible things will happen. We will have to go a bit deeper.

Avoiding False Positives

Now that we already know the Lookaround concept, this is relatively easy. We will ensure that the expression only matches occurrences surrounded by one of these characters: [\+ .,\n].

(?<=[\+ .,\n])(backspace|tab|enter|shift|ctrl|cmd|alt|capslock|pageup|pagedown|ins|del|option|meta|f([2-9]|1[0-2]?)|(?<=\+)[\w\d]{1})(?=[\+ .,\n])

Achieving Idempotence

An idempotent operation is one that can be applied more than once without changing the outcome. To achieve idempotence, we will make use of another advantage of our beloved lookarounds: They can be stacked one after another. All we have to do is make sure that our matches are not already preceded by a <kbd> tag, using a negative Lookbehind:

(?<!<kbd>)(?<=[\+ .,\n])(backspace|tab|enter|shift|ctrl|cmd|alt|capslock|pageup|pagedown|ins|del|option|meta|f([2-9]|1[0-2]?)|(?<=\+)[\w\d]{1})(?=[\+ .,\n])

We could add a negative Lookahead for the closing tag, too, but it is not needed.

Final regex

That’s it. We have a regular expression that we can use in VSCode and other editors to automatically surround all occurences of keys with <kbd> tags. To use it, search for this expression with the Use Regular Expression (alt+r)-toggle enabled:

(?<!<kbd>)(?<=[\+ .,\n])(backspace|tab|enter|shift|ctrl|cmd|alt|capslock|pageup|pagedown|ins|del|option|meta|f([2-9]|1[0-2]?)|(?<=\+)[\w\d]{1})(?=[\+ .,\n])

Then, replace all occurrences string:

<kbd>$1</kbd>

If you want to go one step further, you could even apply this in a build pipeline, e.g., after pushing to GitHub Pages or GitLab Pages. Personally, I like to do it myself in VSCode. These are very satisfying 30 seconds of work before publishing a new blog post.

Conclusion

Hopefully, I could help a bit to improve the bad reputation of regular expressions. But I will let you in on a little secret. Just like everyone else, I have a hard time remembering the syntax. The good thing is, I have a superpower to overcome this: KeyCombiner’s instant shortcut lookup. Whenever I press super+c, I have instant access to the shortcuts of the active application, all other shortcuts that I like to use, and even regex syntax:

Instant shortcut lookup of KeyCombiner’s free to use desktop app.

Learning all VSCode shortcuts evolved my developing habits

Sun, 15 Nov 2020 00:00:00 +0000

Preface

I spent a few hours spread over the last couple of weeks to learn every single one of VSCode’s keyboard shortcuts, specifically the 149 shortcuts from the official keyboard shortcuts reference. These are also present in KeyCombiner’s searchable table of VSCode shortcuts.

Admittedly, I started this challenge to write a blog post about it afterward and as a way to promote KeyCombiner via content marketing. The blog post I had in mind would be similar to my previous one, documenting my process of learning 50 new web application shortcuts in 42 minutes.

But, as it usually happens only in romantic comedies, my intentions changed over time. This article turned out much different. It is not about the learning process but about how this challenge changed some of the developing habits I had for many years. Workflows that I grew used to during many projects, different IDEs, and even varying programming languages suddenly evolved.

In case you do want to see how detailed statistics about how KeyCombiner helped me to learn all VSCode shortcuts, have a look at this overview. It shows a timeline of how my knowledge evolved. It took 205 practice runs (=205 minutes ~ 3.5 hours) until I mastered the complete collection.

Habits are Hard to Break

As software engineers, we like to think that we are continually learning and improving our skills. This is certainly true when it comes to the technologies we are using. However, our development habits, such as which features of our IDEs we use, are much more rigid. I believe most of our core workflows are formed early on and very hard to change. I don’t have data to back this up, but when I look at myself, my co-workers, and other developers I know, this certainly seems true. If you think about it, it is not surprising, the same thing happens in other areas of life. For this very reason, you will find that developers who have studied or worked together will often use similar shortcuts, similar IDE features, and have similar limitations in their workflows.

The usual way to combat this is reading books, blog posts, attending conferences, well, everything that is used in our industry to extend our knowledge. However, with fundamental developing habits, I have found that this works poorly. You might read about a neat feature of your IDE, or see someone demo their workflow, but when you are eventually sitting in front of your screen and writing code, it is easy to fall back to what we already know, what is already set up, and proven to work.

After this experiment, I am convinced that keyboard shortcuts are the main factor in how we form our developing habits, and they can also stop us from evolving our habits. Even before this challenge, I felt that I was good with shortcuts. I knew quite a few refactoring shortcuts, knew how to pause and stop program executing during debugging, was decent at jumping around between files, searching through my project, and all the other usual stuff. When I saw a co-worker using a shortcut that I didn’t know, I was quick to incorporate it into my workflow, too. After all, I was enough into shortcuts that I built a whole side-project around learning them. The thing is, all new shortcuts that I learned were the ones that would augment my existing habits.

This is a hen and egg problem; if you don’t know the shortcuts, you will not start to change your habits and use new IDE features because, without shortcuts, they are too tedious to use or not at all usable. But if you don’t form new habits, you will not learn the shortcuts.

Without intending it, I think I found a way to break out of this cycle: Learning all IDE shortcuts very well before getting back to work. By learning all shortcuts, you are not limited to what somebody shows you and might not be compatible with your situation. You can naturally include what makes sense, in addition to your existing skills.

The next section shows how this worked for me.

Evolving my Habits

I started the learning process as it is usually done with KeyCombiner: By importing some shortcuts from the public collection of VSCode shortcuts into a personal collection. Intuitively, I started with the ones I already knew and some additional that seemed most useful to me. Back then I was still trying to show that I could learn all shortcuts very quickly with KeyCombiner. I didn’t expect that shortcuts that had nothing to do with my existing workflows will be the real deal.

Some of the first combinations that I picked and that I did not know previously were alt+w/r/c for toggling find options (match whole word, regex, case sensitivity) because I was already using those features. This went on for a few days. I did a couple practice runs every day and added new shortcuts whenever KeyCombiner was saying that I mastered all of the previous ones.

Eventually, I knew all shortcuts that seemed useful for how I was using VSCode back then. So I had to start adding shortcuts that had nothing to do with my existing habits. This brought up an interesting misconception. Often keyboard shortcuts are seen as a way to do things faster, compared to using the mouse. While that is true, I think there is an even more important aspect to shortcut usage in IDEs: Many features are not used at all, if you don’t know the shortcut!

As I was learning new VSCode shortcuts every day, I started to use features that I previously didn’t even know of. One of the first things that suddenly made a lot of sense was Select all Occurences of Find Match with Alt+Enter as default binding, especially when combined with regex search. I was so excited about it, that I even tweeted a Gif showing how I use it to modify the CSV files that are the source for KeyCombiner’s public collection tables.

The process worked both ways: Sometimes, while I was practicing a shortcut on KeyCombiner, it seemed apparent that this was useful and that I should include it into my workflow. Sometimes it hit me while coding that I had just learned a shortcut that can make the current task more efficient.

In the following, I describe the most significant changes in my daily development habits. The list is far from complete though; there are many more small things that I picked up and frequently use now.

Editor and Panel Management
Since knowing all the shortcuts to switch focus between panels and move editors around, I am using VSCode’s powerful layouting capabilities much more. I have 3-5 editor panels open at pretty much all times, seeing many related pieces of code at once. Working this way by using the mouse is very impractical. I switch between editor panels with keyboard shortcuts and move editors from one group to the other.

There is one negative side-effect, though: I am starting to get annoyed when I use other IDEs (PyCharm, Eclipse), because they lack far behind VSCode when it comes to using multiple editors.

Multi-cursor Editing
If you don’t know the shortcut to add additional cursors, I doubt you will ever use this incredibly powerful feature of VSCode. It is a typical feature that only blends into your habits once you know the shortcuts. The basic commands are Ctrl+Alt+Up/Down for adding cursors above/below. However, I found the commands to Insert cursor at end of each line selected (Shift+Alt+i), and the above-mentioned Select all Occurences of Find Match (Alt+Enter) even more powerful. These commands are why I frequently paste text snippets into VSCode, apply a few operations, and then copy-paste it back into whatever application I was using.

Opening views by shortcut
I sometimes tried to do this before, but it never stuck. Now I finally switch to all views (Explorer, Debug, Search, Problems, etc.) with dedicated keyboard shortcuts. These shortcuts translate very nicely to other IDEs. After learning the VSCode default bindings, which are Ctrl+Shift+<char>, I set the same bindings also for Eclipse and PyCharm. Especially Ctrl+Shift+e for switching to the explorer view and Ctrl+Shift+d for switching to the debug view are among my most used shortcuts now.

Making Selections
I hate to admit it, but I did not use shortcuts to expand and shrink AST selections before. AST stands for abstract syntax tree, which means that the IDE will take the current language’s syntax into account for modifying the selection. Finally, I picked up Ctrl+Shift+left/right to shrink and expand selections. Not all IDEs handle this in the same way, which can be a bit annoying, but it works well for the most frequent tasks, such as expanding the selection to enclose the current string, method, class, or HTML tag.

Folding
I have started to use folding much more. For years, I have rarely used this feature that all IDEs support to a great extent. Now, I do it frequently. Folding is one of these things that are just not feasible with the mouse. Of course, if you need to fold code, chances are that you need to structure your code better. Many organizations even apply automatic code linters that will complain when methods or classes are too long. However, I have never seen someone recommend splitting up blog posts into multiple files. The same goes for other non-code file types, such as JSON. All we have there is folding.

I am not saying these are the things you should start to learn now, far from it. These are the things I personally integrated deep into my workflows after doing this challenge, without choosing them deliberately, but naturally. Who knows what you will pick up.

Of course, some of the shortcuts I learned still seem a bit, let’s say nonessential, but a little useless knowledge never killed nobody.

More Unexpected Benefits

In addition to finally breaking out of old habits and experiencing a real evolution of my IDE usage, there were some other, less significant, but still delightful second-order effects.

As a shortcut enthusiast, I often had the problem that I needed to decide which binding to use for a particular operation and application pair. KeyCombiner’s public shortcut search helps to find out which binding is the convention for a particular operation, but it can still be tricky. For example, take the frequently used Step Over operation used for debugging. KeyCombiner shows that VSCode, Eclipse, and JetBrains IDEs all use a different default binding(F6,F8,F10). Learning every single VSCode shortcut has freed me from this mental load of deciding what the best binding for a particular IDE operation is. I simply use VSCode’s bindings everywhere. There are good arguments for why VSCode is a good reference. With the rise of GitHub CodeSpaces and its thriving extensions marketplace, I think it is likely that it will emerge as the industry standard of IDEs within a few years. That’s the topic for a future post, though.

Another effect is that I rarely use the mouse anymore when working in VSCode. This is not ideological for me. I hold no grudges against mouse usage. It just so happens that the need to touch it rarely comes up anymore. I guess most people that can do this use Vim or some Vim extension for their IDE. I can say confidently that it also works to learn all the default shortcuts. However, if you aim to eliminate the mouse completely while using VSCode, you should probably still go for one of the popular Vim emulators.

And finally, maybe most importantly, VSCode feels like home now. As someone who also uses other IDEs daily, it’s hard to develop a feeling towards a particular one that this is your professional tool: The place where the magic happens. Now that I have a complete picture of VSCode’s feature set, I feel that I have truly made it my own.

Conclusions

Of course, you don’t have to learn all shortcuts of your IDE. I never thought I would either. You can pick up a new shortcut here or there. Read blog posts where developers show off their selection of favorite shortcuts or exchange favorites with your co-workers. That’s a perfectly fine approach, and it is how a large majority of developers tackle this topic. It is even a core idea of my own project KeyCombiner to build collections of selected keyboard shortcuts that can be learned, shared, and organized.

But, you can also invest a few hours spread over a couple of weeks, stop making compromises, and learn all the shortcuts. There is no way of knowing beforehand how this will influence your habits, how it will amend your existing workflows, and which entirely new possibilities will open up for you.

Implementing Paddle Payments for my Django SaaS

Thu, 08 Oct 2020 00:00:00 +0000

Introduction

A couple of months ago, I published my SaaS side-project: https://keycombiner.com.
It is a Django web application to organize, learn, and practice keyboard shortcuts! You can read more about the concept in some of my other blog posts. This post describes how I implemented payment processing for KeyCombiner with Paddle.

Everybody loves Stripe. Their docs and tooling are excellent. However, they do not deal with the whole billing stack. Most of all, they do not handle taxes. Trying to figure out on your own which VAT rules apply for each of your customers and how to pay the respective sum to the correct authority is not feasible for a side-project.

Paddle tackles this problem head-on by acting as a Merchant of Record (MOR). If you buy a subscription for KeyCombiner, you will not buy it from me, but from Paddle.

The paddle documentation is quite good, but there is not that much third-party content going into more detail and providing step-by-step tutorials. This post describes, in great detail, one possible approach for accepting payments with Paddle in a Django app.

Prerequisites

A Paddle account. It can be created in a matter of minutes.
A Django app that you want to monetize. This will probably take you a little longer.

Setting up your Plan in Paddle

KeyCombiner has a basic pricing model. There is only one Pro subscription plan. Before integrating such a plan into your Django web app, you need to create it. In your Paddle account, go to Catalog→Subscription Plans→New Plan, configure it according to your needs, and hit Save. We will later need the ID of the plan we just created. For this guide, I will use 487302. If you have an existing Paddle plan already, you don’t need to create a new one, just note its ID.

Installing and Configuring dj-paddle

As Django developers, we naturally shy away from implementing anything ourselves. There has to be an existing package for this, right?

Fortunately, there is. It is called dj-paddle, and I am grateful that Florian Pruchess put in the effort to create it. It is a relatively new project, though, and does not provide everything out of the box, so we will get to do a little bit of coding ourselves, too.

We will use dj-paddle for a couple of things:

To provide the database models
To provide service endpoints for Paddle’s webhooks
To provide some convenience templates

Setup

Start by installing the package:

pip install dj-paddle

Then, add it to your INSTALLED_APPS :

INSTALLED_APPS =(
    ...
    "djpaddle",
    ...
)

If you have your settings split into production and development like a pro, I would recommend adding djpaddle to your base settings file, as we can do some local testing with it.

Then, add the following URL configuration to your urls.py:

path("paddle/", include("djpaddle.urls", namespace="djpaddle"))

This is for the service endpoints that Paddle’s webhooks will contact whenever a subscription is created or modified. For this to work, we need to tell Paddle about our service URLs. In your Paddle account, head to Developer Tools→Alerts/Webhooks and look for the Receiving alerts section. There, you can enter your service URL that we just configured above. If you put the above line into your root [urls.py](http://urls.py) it will be <base-url/paddle/webhook/>. It’s also a good idea to specify an email address for receiving alerts. You don’t want to miss those sweet notifications about users purchasing subscriptions.

Now it is time to add dj-paddle’s models to our database, run the migrations that come with the package:

python manage.py migrate

The final thing we have to do before we can implement the actual checkout process is synching our Paddle Plans with our database. Fortunately, dj-paddle provides a management command that does just that by contacting Paddle’s web services. However, before we have to set up credentials so that our app can communicate with Paddle:

# can be found at https://vendors.paddle.com/authentication
DJPADDLE_VENDOR_ID = '<your-vendor-id>'

# create one at https://vendors.paddle.com/authentication
DJPADDLE_API_KEY = '<your-api-key>'

# can be found at https://vendors.paddle.com/public-key
DJPADDLE_PUBLIC_KEY = '<your-public-key>'

# ID of the plan we created before
PADDLE_PLAN_ID = 487302

Then, we are ready to run the management command:

python manage.py djpaddle_sync_plans_from_paddle

Implementing the Checkout

Now, we can get to work. To implement our checkout process, we first need a checkout page. If you want to see mine, you are very welcome to create an account on https://keycombiner.com and check it out here. Spoiler: It is loosely based on this basic Bootstrap pricing page template.

We don’t need much, though. Start by including PaddleJS, which is conveniently provided by dj-paddle:

{% include "djpaddle_paddlejs.html" %}

And, of course, a button to trigger the checkout process:

<a href="#!" class="btn btn-primary paddle_button" data-theme="none"
    data-product="{{ paddle_plan.id }}"
    data-email="{{ user.email }}">Purchase Subscription</a>

The corresponding class-based Django view looks like this:

class Checkout(TemplateView):
    template_name = 'checkout.html'

    def get_context_data(self, **kwargs):
        context = super().get_context_data(**kwargs)
        context['paddle_plan'] = Plan.objects.get(pk=settings.PADDLE_PLAN_ID)
        # If you have not added 'djpaddle.context_processors.vendor_id' as a template context processors
        context['DJPADDLE_VENDOR_ID'] = settings.DJPADDLE_VENDOR_ID
        return context

The crazy thing is, that is already everything you need for a basic checkout process. If you pair this checkout process with a method in your User object, such as the following, you are ready to accept payments and restrict features for paying customers:

def has_subscription(self):
    return self.subscriptions.filter(status = 'active').exists()

Then, in your templates, do

{% if user.has_subscription %}
{% endif %}

to restrict features for users with active subscriptions.

Believe it or not, your customers can purchase subscriptions now, and Paddle will send you the money in recurring intervals.

Unfortunately, as is often the case in software engineering, the final 10% of usability is 90% of the work. To make this convenient to use for your customers, read the Polishing the Checkout section covering some frequent pain points with payment processing.

Testing the Checkout

The Paddle docs suggest a couple of different methods for testing your checkout process. In my experience, the easiest method is to create a 100% off coupon. To do this, go to Catalog->Coupons->+ New Coupon in the Paddle web interface.

Then, you can use the created coupon code to purchase a subscription without having to spend any money.

Testing the KeyCombiner checkout.

This way, you can step through the whole checkout process and, most importantly, see what happens after you successfully purchased a subscription. With your current setup, you will probably need to wait for a moment and refresh the page until user.has_subscription returns True.

Polishing the Checkout:

This section covers everything I did to make the KeyCombiner checkout experience more convenient. If you have more complex problems, have a look at dj-paddle’s documentation, the Paddle Client project, or, if things get really serious, Paddle’s API reference documentation.

What to do directly after checkout?

With our current setup, we will only be aware of a new subscription once the webhook is triggered and the subscription object is added to our database. This can take a few moments. Letting users wait for this time and hope that they refresh the page is not a good look. A user that just purchased a subscription has every right to feel like a Pro right away.

Fortunately, the creators of dj-paddle came up with a solution. There is a template that will add data to the Checkout model upon a successful completion of the checkout process. We only need to include it:

{% include "djpaddle_post_checkout.html" %}

We can also redirect the user to a specific page after checkout:

context['djpaddle_checkout_success_redirect'] = reverse('users:checkout')

To make use of the Checkout object that will now be in the database immediately after a successful purchase, we need to adapt our has_subscription method. In addition to an active subscription, a recently completed checkout is enough to be considered a Pro user on KeyCombiner:

def has_subscription(self):
    return self.subscriptions.filter(status='active').exists() or Checkout.objects.filter(completed=True, email=self.email, created_at__day=now.day).exists()

Let a User Manage their Subscription

Unfortunately, some users don’t know what is good for them. So we need to allow them to cancel their subscription. Also, they should be able to update their payment details so they can ensure that the next payment is handled properly.

I implemented this in the simplest form possible. By displaying Paddle’s update payment and cancellation URLs. Conveniently, these are present in dj-paddle’s subscription model. To pass the user’s subscription to the template, modify the class-based Checkout view:

def get_context_data(self, **kwargs):
    ...
    if self.request.user.has_subscription:
        active_subscription = self.request.user.subscriptions.get(status='active')
        context['subscription'] = active_subscription
    ...

If you implement a grace period as described below, you will need to update the code to retrieve the subscription from the database accordingly. Also make sure to show appropriate information to the user when they have no subscription but a recently completed checkout, as described above.

Then, you can access the subscription’s update_url and cancel_url in your template:

<a href="{{ subscription.update_url }}" class="btn btn-lg btn-primary">
    Update Payment Method
</a>
<a href="{{ subscription.cancel_url }}" class="btn btn-lg btn-danger">
    Cancel Subscription
</a>

These URLs link to simple views provided and hosted by Paddle that offer the respective functionality. We can only rely on this simple approach because we listen to the subscription-related webhooks that keep our database models updated with the correct URLs.

Handle Grace Period

After a user cancels their subscription, they should still have access to the service until the end of their subscription period. I would have thought that Paddle has some API for this, but it appears that there is none. So, I copied the approach from Laravel Cashier, which is a Paddle integration project for the Laravel PHP framework. We simply use the next billing date field that remains the same even after a subscription is canceled:

def has_subscription(self):
    return self.subscriptions.filter(
        Q(status='active') | Q(status='deleted', next_bill_date__gte=now)).exists()
            or Checkout.objects.filter(completed=True, email=self.email, created_at__day=now.day).exists()

Conclusion

Stripe is often seen as the de-facto standard for accepting SaaS payments. Their documentation is universally praised, and integration is as simple as it gets. However, this post shows that integrating Paddle into your Django app isn’t rocket science either.

The Paddle web interface lets you create coupon codes. I hope you feel inspired by this classic comic. (Source: xkcd.com)

If you want to see the described approach in practice, you are very welcome to create an account on KeyCombiner and purchase a Pro subscription ;)

How I learned 50 new keyboard shortcuts in 42 minutes

Tue, 21 Jul 2020 00:00:00 +0000

Introduction

As developers, we frequently use keyboard shortcuts. Some enthusiasts know hundreds, others are contempt with the essential ones. But every developer does know some. It is unthinkable to browse the Web, searching for a solution to your CSS alignment problem without using Ctrl+f. Debugging would be tedious if we couldn’t pause and resume a program’s execution with the keyboard.

In recent weeks, I have been able to significantly expand my keyboard shortcut knowledge with my new side-project web app KeyCombiner.
In particular, I knew only a few shortcuts for the web-based tools I am using in my daily work. This post describes how it took me less than 1 hour to learn 50 new key combinations. Fortunately, KeyCombiner keeps a detailed history of a user’s learning progress, so that I could write this post retrospectively.

Admittedly, the 42 minutes of learning time was interrupted by breaks, and the process involved some other tasks, such as creating the collection of shortcuts I wanted to learn. However, I did, in fact, spend only 42 minutes practicing the shortcuts and have had similar results with other shortcut collections.

Setup

The first step to learning new keyboard shortcuts is to define which. I don’t think it is efficient to try and learn all shortcuts for a particular application. You will end up with many that you do not use in your daily work and that you will soon forget again.

Creating custom collections of keyboard shortcuts is perhaps the greatest strength of KeyCombiner and sets it apart from any other tool. Within minutes or less, you can have a personal collection by importing shortcuts from popular apps. I like to compare its approach to how you build playlists in music software. Instead of browsing your favorite artists’ albums, you browse categories of your favorite applications. Instead of adding songs to your playlists, you can add keyboard shortcuts to your collections.

For this challenge, I created a new collection named “50 to learn”. Then, I browsed KeyCombiner’s public collections and started to import everything I wanted to learn. As my goal was to get better with web application shortcuts, I focused on the public collections for Gmail, GitHub, GDrive, Docs, Slack, and Twitter. Importing the combinations into my new shortcut collection looked like this:

Collecting shortcuts from KeyCombiner’s public Gmail collection.

Additionally, I added a few shortcuts for Smartgit manually. It is a graphical Git client that I am using extensively. For some reason, I never learned its shortcuts. Unfortunately, KeyCombiner does not yet have a public shortcut collection for Smartgit that I could rely on. However, manual input does the job:

Manually adding keyboard shortcuts to my new collection.

You can browse the resulting collection with 50 keyboard shortcuts here: 50 to learn.

I am afraid you will have to trust me that I did not know these shortcuts already before this experiment. However, as reassurance, you can check my previous blog post covering all the shortcuts I was using until a few weeks ago. The list there does not include these new ones.

To validate the success of this experiment, I will simply use KeyCombiner’s confidence metric. It will analyze my performance and tell me which shortcuts are already etched into my muscle memory during practice. So, the challenge will be completed successfully once this card says 50/50:

Overview of my new collection with 50 shortcuts to learn.

Learning the shortcuts

Learning new shortcuts with KeyCombiner is dead simple. You click on the practice button for a particular collection, and the software does the rest. It will create 60-seconds training exercises where you are supposed to type the shortcuts of a collection as fast and as correct as possible.

With every input, KeyCombiner remembers if it was correct and how long you took. It will use this information along with some machine learning to calculate a so-called confidence value for each key combination in your collections. A high confidence value means that you mastered a combination. Key combinations with a low value will occur more often in practice sessions, so you are not stuck repeating what you know already. There are a few additional aspects to it, but the good thing is that users do not have to bother with the learning algorithm’s internal workings. Repeatedly doing practice runs is enough.

Starting to remember the keys

The very beginning of learning a new collection is the hardest part. While I do not yet know any of the shortcuts, I have to rely on KeyCombiner’s delayed key hints to type any. Sometimes, I just skip combinations immediately to see what the keys are.

The very first practice runs with a new collection are quite slow.

This is how results for such a practice run look like:

Typical practice statistics for a collection you do not know yet.

Surprisingly, even in the first practice run, I got a couple of combinations right without key hints. This means that they occurred multiple times already or simply that I was able to guess them correctly. The latter is often the case for go to shortcuts, which have been made popular by Gmail and are now used in many web applications.

After a few practice runs, I started to disable key hints and only relied on skipping combinations where I couldn’t remember the keys. KeyCombiner will then display the skipped combination’s keys, so I have a shot at remembering them next time.

The statistics for the 10th practice run looked as follows:

After just 10 minutes of practice, I already know some of the shortcuts.

KeyCombiner recognizes 9 shortcuts as mastered. So far, so good.

Building muscle memory

After another 10 practice runs, statistics showed that I knew already most of the shortcuts:

Statistics for my 20th practice run.

However, KeyCombiner punishes errors quite heavily and is hesitant to show keys as mastered that I have just recently mistyped. At this stage, it starts to get interesting to check the collection and see which specific shortcuts have very low confidence values. Usually, it is obvious to see some patterns:

The combinations with lowest confidence values after 20 practice runs.

The figure shows that I had problems knowing which modifiers to use for GSuite shortcuts (Gmail, GDrive and Docs). Some use Ctrl + Shift , some Ctrl + Alt and some don’t have a modifier at all (e.g. forwarding a mail in Gmail). I found that having a good look at such problems between practice runs can speed my progress, but it is not required.

All that is needed is to keep hitting the practice button. At this stage, muscle memory starts to build, and the average time I need for each combination goes down quickly. KeyCombiner considers the average time metric for its confidence values.

Completing the challenge

After 35 practice runs, things started to look very good:

Only few shortcuts with low confidence values remain.

A confidence value of 3 means that I mastered a particular combination.

Let’s have a look at the combination with the lowest confidence value. Surprisingly, it is the relatively simple shift+a:

Aggregated statistics for a specific combination.

A combination is not guaranteed to appear in every practice. Furthermore, it can appear multiple times, especially when it has a low confidence value compared to the other shortcuts of a collection. Consequently, the above figure shows only 11 practice runs but 15 correct occurrences and 3 errors.

I made quite a few mistakes with this combination in the beginning. Therefore KeyCombiner’s algorithm is skill skeptical about my skills. Errors usually mean that I thought I knew the shortcut but then typed a wrong combination. In this case, I suspect it is because I was hitting Ctrl+a instead or missing the a key. shift+a is a bit tricky to type if you are used to hit both shift and a with your little finger. However, my performance has been much better recently. Just a couple more practice runs to go and KeyCombiner will forgive my earlier mistakes.

Then, finally, after another 7 practice runs, I had reached my goal:

Collection overview card after I have mastered all 50 shortcuts.

Practicing on this collection looks much different now than in the beginning:

Pratice run after completing the challenge.

Tips

It is straightforward to do a similar challenge. Simply go to https://keycombiner.com, create an account, import some shortcuts from public collections, and start practicing. However, I would like to share some additional tips that make learning new shortcuts more pleasant for me:

It is a bit more fun in the beginning if you throw in some shortcuts that you already know. Even better, just extend one of your existing collections with new shortcuts that you want to learn. KeyCombiner will automatically show the new ones more frequently during practice.
I sometimes activate key hints without delay, which brings some variety to the learning experience and can help build muscle memory. If you are a good typist, you will be able to type a lot of shortcuts within the 60 seconds time limit.
KeyCombiner will try to filter out shortcuts that cannot be captured during training. Those are marked with a warning icon in your collections. However, sometimes an invalid combination may still make it into your practice session, for example, if a browser extension does not pass it on to KeyCombiner. Simply append an underscore to any of the keys, e.g. shift_+a, so that KeyCombiner will recognize it as invalid.
Depending on how frequently you use a shortcut in your daily work, your memory may fade over time. To counter this, I suggest practicing also mastered collections from time-to-time. A couple of one-minute practice sessions can be a nice break from work.

Results & Conclusion

I have already learned many new keyboard shortcuts with KeyCombiner before this challenge. However, I never kept track of how long it took exactly until I knew them. I am thrilled with finding out that I could learn 50 new shortcuts in just 42 minutes of practice time.

Believe it or not, without thinking much about it, I have started to use every single one of the 50 learned shortcuts in my daily work. I guess that’s the benefit of learning, not just any shortcuts, but precisely the ones you pick.

Clearly, the experience will vary between people and also depend on the complexity of the shortcuts. Given that I spent much of my free time working with the keyboard shortcuts domain during the last months, I can probably pick up new ones faster than an average user. Still, I am confident that you could achieve similar results.

I think this post makes a good argument that 42 minutes spent with https://keycombiner.com are an excellent investment in your productivity.

If you are interested in following KeyCombiner’s evolution, please consider signing up for its newsletter using the form below.

Why I built a new app for practicing keyboard shortcuts

Wed, 08 Jul 2020 00:00:00 +0000

Introduction

I have always wanted to build a software product as a side project. Something that I could gradually improve and work on for years without deadlines and technology restrictions. Something that was my side-project. Fortunately, I was never short of ideas. In my post on managing my personal knowledge base, I have mentioned that I keep a Notion database of project ideas. It has plenty of entries.

In the same post, I also described my shortcut database use case. It was a spreadsheet to keep track of all the keyboard shortcuts I was using. Something about this motivated me much more than all the Slack bots and browser extensions lurking around in my ideas list. So, I started to work and expand on the concept in the form of a new web application. Initially, I thought It would take me until the end of 2020 to have anything that I could share with the world. Then, the Corona-lockdown came, and suddenly most of my other free time activities were no longer possible.

Long story short, keycombiner.com is now available for everyone to use. This post covers how it compares to existing tools, what it tries to do, how it does it, and the road ahead.

KeyCombiner is completely free to use, with no strings attached. For a demo of the Practice mode or browsing public shortcut collections, you don’t even need to create an account.

Existing Tools

There are a lot of typing practice tools available. However, KeyCombiner is quite different from all of them.

The vast majority of existing tools focus on text typing. These applications will show you automatically generated pieces of text that you are supposed to type as fast and as correct as possible. Usually, you will get a report at the end about your speed and accuracy. I do like this a lot and often test my typing speed. It is a fun thing to do once in a while, and if you realize that your typing speed is low, you might want to work on it. Of course, these tools have nothing to do with keyboard shortcuts. They will not help with learning them and also do not include them in their typing practice.

When looking at practicing keyboard shortcuts, there are far fewer alternatives to choose from. But still, there are a couple of existing apps. To my knowledge, all of them work with pre-defined lessons, which is very unintuitive for me. I don’t want to learn all keyboard shortcuts of a particular app. Also, they are focused solely on learning keyboard shortcuts, not with your typing skills per se. Therefore, they do not record typing speed, accuracy, or any other such metrics. For me, this alone removes most of the motivation for using such a tool. I don’t want to only learn shortcuts, I also want to become faster and more accurate at using them.

KeyCombiner aims to bridge this gap between typing training software and shortcut learning apps. Besides, it goes to great lengths to make it easy to learn and practice not just any shortcuts but precisely the ones you want to use. If you like to look at Venn diagrams that probably shouldn’t be Venn diagrams, I have just the thing for you:

Goals

Developing KeyCombiner, I had and still have the following goals:

Enable efficient creation of keyboard shortcut collections

The first goal is, of course, to cover the original use case described in my knowledge management post. Having an overview of all the keyboard shortcuts you are using and intend to learn is already useful in itself. However, it is clear that most people don’t want to spend a lot of time with this task, so it has to be possible in a couple of minutes to build meaningful collections.

Facilitate learning of keyboard shortcuts

Generally, I want to provide as much value on top of a user’s shortcut collections as possible. Learning new keyboard shortcuts is probably the most powerful thing that can be done. It is therefore the second fundamental goal of KeyCombiner.

Allow to practice also text snippets

As developers, we have many short text snippets that we need to remember. Think of git commands or language syntax. These feel similar to keyboard shortcuts, and I would like to be able to practice them together.

Help to improve typing speed and accuracy

Going beyond mere memorization of keyboard shortcuts is an important aspect and one thing that sets KeyCombiner apart from other software. The goal is not just to learn keyboard shortcuts, but to be able to type them fast and accurately, too.

How it works

This section covers KeyCombiner’s three main areas. However, those are very much connected, as illustrated in the following figure:

Create shortcut collections

A core idea of KeyCombiner is to learn and practice exactly the shortcuts you need or want to use. The only way to achieve this is if you choose them yourself.

I thought a lot about how to make this collection building process as efficient as possible. From the start, it was apparent that it would need to be possible to import keyboard shortcuts and text snippets from a public database of popular application shortcuts. This mechanic can be used to build the bulk of your collections quickly. They can then be completed by manually adding entries.

I like to compare my approach to how you build playlists in music software. Instead of browsing your favorite artists' albums, KeyCombiner allows you to browse categories of your favorite applications. Instead of adding songs to your playlists, you can add keyboard shortcuts and text snippets to your collections:

KeyCombiner’s combination tables support all popular multi-selection patterns, i.e. drag selecting, Shift-selection and maintaining selection via Ctrl

KeyCombiner’s public collections already contain thousands of keyboard shortcuts. Each collection can be filtered and searched quickly.

Creating collections of keyboard shortcuts is already a use case in itself. It can help to answer a variety of questions:

How many shortcuts am I using?
Are my key bindings logical and consistent or am I using completely different combinations for similar things?
For which applications am I using my shortcuts?
I am setting up a new machine and want to set up my key bindings. What were those exactly?

When combining personal shortcut collections with the other two main areas of KeyCombiner, we can answer even more interesting questions:

How well do I actually know the shortcuts in my collections?
How fast can I type them?
How often do I make an error while typing a specific shortcut?

As shown in the illustration above, collections are annotated with information gathered from KeyCombiner’s statistics. Most importantly, the confidence value shows how good you are with a combination in your collections:

Some keyboard shortcuts that I have practice often and am therefore very confident with.

Learn shortcuts flashcard-style

Before going deeper into confidence values and other statistical measures, we need to take a step back and look at how data is gathered. It is done while you practice your shortcut collections.

In principle, KeyCombiner’s interactive training is similar to other applications. You are shown what a shortcut does and, ideally, you know the keys and type it in correctly:

KeyCombiner’s interactive trainer has a visual keyboard and options for displaying the keys to type.

What makes KeyCombiner unique in this regard is that you are practicing your very own collections, hence exactly the shortcuts you want to learn! Furthermore, KeyCombiner gathers detailed statistics that help to analyze your performance. You might know how many words you can type by minute (WPM), but do you also know how many keyboard shortcuts you can execute during this time?

There are many more things to be said about the Practice mode. It does all kinds of things to make learning as efficient as possible. For example, it uses ideas from spaced repetition and shows keyboard shortcuts with low confidence value more often than others. Then, there is the option to display the actual keys of a combination after a delay that gives you some time to think. This is a tricky thing to do because key combinations typed with hints should not influence the confidence value. But that’s a topic for another blog post.

What I like to do personally is to gradually expand my collections. For example, I set a goal of learning 10 new shortcuts a given week. Then, at the beginning of the week, I browse the public collections for 10 new shortcuts and add them to one of my collections. Throughout the week, I do a 60-seconds practice session from time to time. Because the new shortcuts have a low confidence value at the beginning, KeyCombiner will show them often during practice, and after just a couple of practice runs, I usually know them well.

Statistics to improve accuracy & speed

One of the defined goals is to help users improve their typing skills beyond just remembering shortcuts. It is of similar importance to be able to type them fast and accurately. To help identify weaknesses and bad habits, KeyCombiner gathers detailed statistics during practice.

The below figure shows an example bar chart for one of my early practice runs.

Interactive charts show which combinations need some further practice.

It is immediately obvious that I had problems typing Shift+a, Ctrl+Alt+m and Ctrl+Shift + c. Using such statistics, I actually found that I was occasionaly making mistakes with some of the most essential shortcuts: Ctrl+x/c/v. This was because I used my pointer finger for each of them. After realizing that this was a problem, I started to use the middle finger for Ctrl+x and am no longer making mistakes.

When taking the average time into account, there are many more observations to make. If you find out which combinations are the fastest to type for you, you can use this knowledge and set such convenient combinations wherever possible.

KeyCombiner presents statistics per practice run and per key combination.

Roadmap

The most pressing issue right now is user onboarding. Unfortunately, many people sign up and then never create any meaningful collections. Some import all combinations of popular apps into a collection and quickly lose interest. This is not surprising because learning with all combinations of a public collection is not a good idea. It has all the problems of apps based on pre-defined drills but misses out on some of their benefits. Therefore, I am exploring possibilities to make the process of building your own collections even more obvious and seamless.

After that, I will publish a desktop app. This will be nice because it will allow to train also shortcuts that are otherwise reserved for the browser, such as Ctrl+(Shift)+N/T/W.

Another important milestone in the relatively near future will be searching for shortcuts across all public collections. I am quite excited about this as I don’t think there is anything similar in any other app. How would you currently find out which applications offer Ctrl+Alt+R as a keyboard shortcut? The question is relevant e.g., for developers thinking about which shortcuts to include in their applications.

Update from the future: All three items are now done.

User onboarding was improved by showing hints for new users, and by improving the spaced repetition algorithm to facilitate learning with large, unknown collections.
The desktop app is available! In addition to enabling the practice of browser-reserved shortcuts, it comes with an instant look up for all shortcuts that are in your combined collections plus those of the currently active application.
The shortcut search is live and kicking! It works via URL parameters, so you can, for example, use this link to see all shortcuts in KeyCombiner’s database that use ctrl+enter on Windows/Linux, and cmd+enter on macOS.

Conclusion

I am very happy with the way my side project progressed, but there is a long road ahead. Keyboard shortcuts are a surprisingly rich domain.

If the described concepts sound interesting to you, please go ahead and have a look at https://keycombiner.com. There is a demo of the interactive trainer directly on the landing page, and you can sign up for free with just an email address or log in with your Google account.

I would love to hear what you think of it.

Finally, if you are interested in following KeyCombiner’s evolution, please consider signing up to its newsletter using the form below.

A Collection of all Keyboard Shortcuts I use

Tue, 09 Jun 2020 00:00:00 +0000

Introduction

I believe that typing proficiency is important for a software engineer. It is easier to stay in flow if you can type fast and navigate around your IDE and tools with shortcuts. In fact, I love that a part of our job has to do with manual skills. Improving these physical skills is a nice change from learning new technologies and tools.

A good start for getting better at something is to find out where you stand. At first, I wanted to get an overview of how many shortcuts I know and use. So, last year, I created a database of my shortcuts in Notion. I already described this in my post on personal knowledge management. In the process, I found some inconsistencies and tried to use the same key combination in as many applications as possible.

Since then, I have thought a lot more about this whole thing. I should probably not tell you that I had a few dreams where keyboard shortcuts played a prominent role. Anyway, I spent much of my social distancing free time on my first published solo project: KeyCombiner. It has made creating and maintaining my shortcut database much simpler.

This collection is also shared and available via KeyCombiner, which allows for more complex filtering and searching and enables you to import selected shortcuts into your personal collections.

The tables in this post are generated by using KeyCombiner’s CSV export and converting it to Markdown with this handy web tool. The collection is split by context. By context, I mean the application or type of applications in which the shortcut can be used.

Essential

The context of these shortcuts is not further defined because I expect them to work anywhere. As the name suggests, I think these shortcuts are so ubiquitous that everyone should know them.

Description	Context	Category	Keys
New File	Essential	File Management	`ctrl` + `n`
Open File…	Essential	File Management	`ctrl` + `o`
Save	Essential	File Management	`ctrl` + `s`
Save As…	Essential	File Management	`ctrl` + `shift` + `s`
Close	Essential	File Management	`ctrl` + `w`
Print	Essential	General	`ctrl` + `p`
Refresh	Essential	General	`f5`
Go Back	Essential	Navigation	`alt` + `left`
Go Forward	Essential	Navigation	`alt` + `right`
Search	Essential	Navigation	`ctrl` + `f`
Cut selection	Essential	Basic Editing	`ctrl` + `x`
Copy selection	Essential	Basic Editing	`ctrl` + `c`
Paste	Essential	Basic Editing	`ctrl` + `v`
Paste without formatting	Essential	Basic Editing	`ctrl` + `shift` + `v`
Undo	Essential	Basic Editing	`ctrl` + `z`
Redo	Essential	Basic Editing	`ctrl` + `y`
Go to Beginning of Line	Essential	Basic Editing	`home`
Go to End of Line	Essential	Basic Editing	`end`
Open a new tab, and jump to it	Essential	Tab and window	`ctrl` + `t`
Jump to the next open tab	Essential	Tab and window	`ctrl` + `tab`
Open a new window	Essential	Tab and window	`ctrl` + `n`

Editor

The next set of shortcuts is applicable in any kind of editing software. Except the first one (Replace), these work in pretty much all text fields independent of application.

Description	Context	Category	Keys
Replace	Editor	Basic Editing	`ctrl` + `h`
Left with Selection	Editor	Basic Editing	`shift` + `left`
Right with Selection	Editor	Basic Editing	`shift` + `right`
Move to Line End with Selection	Editor	Basic Editing	`shift` + `end`
Move to Line Start with Selection	Editor	Basic Editing	`shift` + `home`
Move to Next Word	Editor	Basic Editing	`ctrl` + `right`
Move to Next Word with Selection	Editor	Basic Editing	`ctrl` + `shift` + `right`
Move to Previous Word	Editor	Basic Editing	`ctrl` + `left`
Move to Previous Word with Selection	Editor	Basic Editing	`ctrl` + `shift` + `left`
Indent Selection	Editor	Basic Editing	`tab`
Unindent Selection	Editor	Basic Editing	`shift` + `tab`

IDE

As a software engineer, this context is very important for me. In my experience, many complex IDE features, especially those related to refactoring, are not used at all if one does not know the shortcut. So in this case, knowing additional shortcuts does not only make you faster, it gives access to additional features.

Those features can sometimes even have an impact on code quality. For example, if you know a shortcut to extract a selected code block into a method, you might be less prone to writing large blocks of code that are hard to read. Knowing the common debugging shortcuts can make you more comfortable with the process and less inclined to use print statements.

Currently, I use VSCode, PyCharm, and Eclipse as IDEs for different languages and project sizes. I have configured the listed shortcuts to work in all of them, except for some that I only use in PyCharm.

Description	Context	Category	Keys
Go to File…, Quick Open	IDE	Navigation	`ctrl` + `p`
Show command prompt	IDE	Navigation	`ctrl` + `shift` + `p`
Open Resource	IDE	Navigation	`ctrl` + `shift` + `r`
Switch to recent file	IDE	Navigation	`ctrl` + `e`
Go to symbol in current file (outline)	IDE	Navigation	`ctrl` + `o`
Go to Symbol in Workspace	IDE	Navigation	`ctrl` + `shift` + `t`
Go to line number…	IDE	Navigation	`ctrl` + `l`
Format Document	IDE	Rich Languages Editing	`ctrl` + `shift` + `f`
Go to Definition	IDE	Rich Languages Editing	`f3`
Go to References	IDE	Rich Languages Editing	`alt` + `ctrl` + `r`
Rename Symbol	IDE	Rich Languages Editing	`shift` + `alt` + `r`
Search in all files/workspace	IDE	Search	`alt` + `ctrl` + `f`
Start Debugging	IDE	Debug	`f11`
Continue	IDE	Debug	`f5`
Mute Breakpoints	IDE	Debug	`f6`
Step Into	IDE	Debug	`f7`
Step Over	IDE	Debug	`f8`
Stop Run/Debug	IDE	Debug	`f9`
Move Line Down	IDE	Basic Editing	`alt` + `down`
Move Line Up	IDE	Basic Editing	`alt` + `up`
Copy Line Down	IDE	Basic Editing	`ctrl` + `alt` + `down`
Copy Line Up	IDE	Basic Editing	`ctrl` + `alt` + `up`
Delete Line	IDE	Basic Editing	`ctrl` + `d`
Split editor horizontally	IDE	Tab and window	`alt` + `t`
Split editor vertically	IDE	Tab and window	`alt` + `d`
Extract Method	PyCharm	Refactoring	`ctrl` + `alt` + `m`
Introduce Variable	PyCharm	Refactoring	`ctrl` + `alt` + `v`
Introduce Field	PyCharm	Refactoring	`ctrl` + `alt` + `f`
Introduce Constant	PyCharm	Refactoring	`ctrl` + `alt` + `c`
Cut line (empty selection)	PyCharm	Basic Editing	`ctrl` + `x`
Copy line (empty selection)	PyCharm	Basic Editing	`ctrl` + `c`

OS

The shortcuts in this section are among my most used ones. For anyone of these, I can say with certainty that I use it many times a day.

You might notice the Media shortcuts with keys, such as num2. I repurposed my Numpad for this because I don’t use it for typing numbers. Reassigning keys works especially well with my keyboard with blank keycaps The labeling can never be wrong because there is none ;)

The first entry in this context’s table represents 9 separate shortcuts. Switching between applications via super + 1-9 has been a noticeable productivity boost for me. I rarely have to start an application by other means now.

Description	Context	Category	Keys
Open application 1-9	OS	Navigation	`super` + `1-9`
Go to previous application	OS	Navigation	`alt` + `Tab`
Move window to screen on the left	OS	Navigation	`super` + `shift` + `Left`
Move window to screen on the right	OS	Navigation	`super` + `shift` + `Right`
Cycle through windows of current application	OS	Navigation	`alt` + `^`
Play (Music etc.)	OS	Media	`num2`
Previous Track	OS	Media	`num1`
Next Track	OS	Media	`num3`
Mute Sound	OS	Audio	`num5`
Mute Mic	OS	Audio	`num7`
Increase Volume (louder)	OS	Audio	`num6`
Decrease Volume	OS	Audio	`num9`
Put window to left half of monitor	OS	Window Management	`super` + `left`
Put window to right half of monitor	OS	Window Management	`super` + `right`
Show Albert launcher	OS	Navigation	`super` + `shift` + `p`

Chrome

Many of the shortcuts that I use most frequently while browsing the internet are already covered in the sections above. However, there are some keyboard combinations unique to web browsing.

Description	Context	Category	Keys
Open the History page in a new tab	Chrome	Google Chrome features	`ctrl` + `h`
Open the Downloads page in a new tab	Chrome	Google Chrome features	`ctrl` + `j`
Jump to the address bar	Chrome	Address bar	`ctrl` + `l`
Search with default search engine	Chrome	Address bar	`ctrl` + `k`
Return everything on the page to default size	Chrome	Webpages	`ctrl` + `0`
Go to the top of the page	Chrome	Webpages	`home`
Go to the bottom of the page	Chrome	Webpages	`end`
Open a new window in Incognito mode	Chrome	Tab and window	`ctrl` + `shift` + `n`

Web Apps

Until recently, I didn’t use a lot of web application keyboard shortcuts. I feel like the applications are partly to blame for this. Why is it not common to show keyboard shortcuts in button tooltips? Or even in a small notification when a user does an action via mouse? Anyway, working on KeyCombiner motivated me to expand my shortcut knowledge in this area.

I have some learning ahead of me, but I am not quite comfortable with my most used apps. Knowing web application shortcuts is quite rewarding because the respective action would often take multiple mouse clicks. Sometimes it is even hard to find the right button for it, if you don’t know an action’s shortcut.

Description	Context	Category	Keys
Jump to a conversation	Slack	Navigation	`ctrl` + `k`
Next unread channel or DM	Slack	Navigation	`alt` + `shift` + `down`
Go to the project home page (Project > Details).	GitLab	Project	`g` > `p`
Go to the project issue boards list (Issues > Boards).	GitLab	Project	`g` > `b`
Go to the project file search page.	GitLab	Project	`t`
Go to the project commits list (Repository > Commits).	GitLab	Project	`g` > `c`
Go to the project merge requests list (Merge Requests).	GitLab	Project	`g` > `m`
Go to the CI/CD jobs list (CI/CD > Jobs).	GitLab	Project	`g` > `j`
Go to the project wiki (Wiki), if enabled.	GitLab	Project	`g` > `w`
Edit description.	GitLab	Issues and Merge Requests	`e`
Change assignee.	GitLab	Issues and Merge Requests	`a`
Change label.	GitLab	Issues and Merge Requests	`l`
Edit wiki page.	GitLab	Wiki pages	`e`
Rename selected item	GDrive	Take action on selected items	`n`
Go to personal collections	KeyCombiner	Global Navigation	`g` > `c`
Go to public collections	KeyCombiner	Global Navigation	`g` > `p`
Go to personal collection 1	KeyCombiner	Global Navigation	`c` > `1`
Learn personal collection 1	KeyCombiner	Global Navigation	`l` > `1`
Test your skills with personal collection 1	KeyCombiner	Global Navigation	`t` > `1`
Skip current combination	KeyCombiner	During Learn or Test	`esc`
Learn current collection	KeyCombiner	Collections	`shift` + `l`
Copy selected combinations to personal collection 1	KeyCombiner	Collections	`shift` + `c` > `1`
Search or jump to a recently viewed page.	Notion	Most popular	`ctrl` + `p`
Create a line break within a block of text	Notion	Create & style your content	`shift` + `enter`
Bold selected text	Notion	Create & style your content	`ctrl` + `b`
Italicize selected text	Notion	Create & style your content	`ctrl` + `i`
Turn selected text into inline code	Notion	Create & style your content	`ctrl` + `e`
Duplicate selected blocks	Notion	Edit & move blocks	`ctrl` + `d`

Conclusion

It is probably not a good idea to copy my collection, but seeing what kind of shortcuts are used by another software engineer might give you some ideas. Maybe I could even inspire you to create a similar list of your own shortcuts. If so, you might want to try out KeyCombiner, as I am sure it’s the most efficient way of accomplishing this.

After you created such a collection of all the shortcuts you use, you might realize that a mere spreadsheet cannot give a full picture of your shortcuts skills. Many questions remain:

How well do you actually know them?
How fast can you type them?
How often do you make an error while typing a shortcut?

KeyCombiner is built to answer precisely these questions with its interactive training features.

Am I missing some important shortcuts? I would be very interested to hear your suggestions.

Using VueJS alongside Django

Tue, 14 Apr 2020 00:00:00 +0000

Introduction

I am currently working on a very exciting project. The whole thing is quite challenging because the project’s scope is significant, and I am doing it alone in my spare time while working full time. So I have to be very efficient. Fortunately, I am using Django with its batteries-included approach.

The mentioned project is now in Open Beta! You can have a look at https://keycombiner.com.
It is a web application to organize the keyboard shortcuts you use, get better at using them, and to learn new ones.

I use all kinds of Django features that speed up my development, and I wouldn’t want to miss its template engine. Therefore, using Django only in the backend and building a JavaScript SPA for the frontend is not an option for me.
However, even the most avid backend developer has to admit that some things warrant a client-side implementation. Small user actions should not require page reloads. Also, some parts of the web application I am building require rather sophisticated user interaction.

Traditionally, one would mix Django with some jQuery to achieve the desired behavior. But there are newer JavaScript technologies now: React and Vue.
Since our goal is to find a framework that we can use alongside Django without rethinking everything, we will go for Vue as the more lightweight alternative. In this post, I will show that you can start to use Vue alongside Django’s template language with minimal effort.

Installation and Setup

One of the reasons to use Vue is its excellent documentation. It includes many examples, has a decent search, and a reasonably clear table of contents.

This post aims to show that you can start to use Vue with your Django projects immediately without any sophisticated setup that will take hours to complete. Therefore, we will use the simplest method to use Vue.js: Including it via a <script> tag.

<!-- development version, includes helpful console warnings -->
<script src="https://cdn.jsdelivr.net/npm/vue/dist/vue.js"></script>

That is it, we are now ready to create our first Vue.js instance:

<div id="app">
  {{ message }}
</div>

var app = new Vue({
  delimiters: ["[[", "]]"],
  el: '#app',
  data: {
    message: 'Hello Vue!'
  }
})

This example is taken from the official Getting Started Guide. However, there is one addition. Per default, Django and Vue use the same template tags. Hence, we need to set the Vue delimiters explicitly to avoid conflicts with Django’s template engine.

Access Django Data from Vue

The simplest way to do so is the built-in Django jscon_script filter. This way you can immediately start using your Django models as data for your Vue instances.

In HTML:

{{ django_template_variable|json_script:"djangoData" }}

Then, in JavaScript, we load this data into a variable:

let jsVariable = JSON.parse(document.getElementById('djangoData').textContent);

And it is ready to use with Vue:

new Vue({
  delimiters: ["[[", "]]"],
  el: '#app',
  data: jsVariable
})

Make async Backend Requests

One of the most frequent tasks of a Vue frontend is to make requests to a backend server application. With a full-stack Django application, we don’t have to do this for every user interaction. In some cases, a full page reload might be perfectly fine, and Django’s templating system provides all kinds of advantages. But to enhance user experience and to reap the full benefits of using Vue, we may still want to make backend requests in some places.

Vue itself cannot handle requests. In this post, I will use axios, because it is also recommended in the official Vue Docs. Other alternatives are perfectly fine too.

To pass Django’s CSRF protection mechanism, axios needs to include the respective cookie in its requests. The easiest way to accomplish this is to set global axios defaults:

axios.defaults.xsrfCookieName = 'csrftoken';
axios.defaults.xsrfHeaderName = "X-CSRFTOKEN";

Alternatively, we could also create an axios instance with the required settings:

var instance = axios.create({
    xsrfCookieName: 'csrftoken',
    xsrfHeaderName: "X-CSRFTOKEN",
});

Your Django template needs to contain the tag {% csrf_token %} or, alternatively, the respective view must use the decorator ensure_csrf_cookie().

The rest of Django’s default session backend for authentication will work out of the box, meaning that you can annotate your backend services with things like loginRequired and it will just work. To make the request, we can use axios as usual:

axios.post('/django/backend/endpoint', {
    data: jsVariable 
})
    .then(function (response) {
        // handle response
    })
    .catch(function (error) {
        // handle error
    });

This call can be done within a Vue instance’s mounted hook or any other place where you can put JavaScript code.

If you activated CSRF_USE_SESSIONS or CSRF_COOKIE_HTTPONLY in your Django settings, you need to read the CSRF token from the DOM. For more details, see the official Django docs.

Conclusion

When you google for Django + Vue, most results will be focused on using Django for your backend and Vue for a separate frontend project. Having two independent projects increases complexity, and you lose access to Django’s template system, which is a very powerful timesaver. On the other hand, access to a frontend framework such as Vue can do wonders for web applications that go beyond CRUD functionality.

Fortunately, we do not need to decide between the two. This guide shows that you can have the cake and eat it too!

Add CSV Export to Wagtail's Modeladmin

Thu, 06 Feb 2020 00:00:00 +0000

Introduction

Wagtail is a modern open source CMS written in Python and based on Django. It is easy to integrate with existing Django projects. Apart from traditional CMS features, it provides a nice UI for managing any Django database model via the modeladmin module.

The modeladmin IndexView lists entries for a specific model in tabular form. It is easy to define which columns should be included. Starting from here, there are buttons for editing, creating and deleting entries.

One feature is missing though: Data Export
As the data is already presented in a table, CSV is an obvious export format.

We will add an additional button to the modeladmin IndexView

This idea is not entirely original. This blog post and this StackOverflow question discuss the same thing and my code is heavily influenced by them. However, the solutions given at these sources are not quite ready to copy and paste, as they require some customization of the CSV exporting code for each model you want to export.

The code given in this blog post can be used with any Django model. Per default, all columns are exported, but this can easily be customized on a per-model basis.

Implementation

I will cover the different implementation parts in detail. If you just want to copy-paste and get on with your life, that’s fine too. Just make sure you copy all the given code snippets. It is fine to put everything into wagtail_hooks.py, except the HTML template.

ButtonHelper

The first thing you need whenever you want to add custom functionality to Wagtail’s modeladmin is usually a ButtonHelper:

class ExportButtonHelper(ButtonHelper):
    export_button_classnames = ['icon', 'icon-download']

    def export_button(self, classnames_add=None, classnames_exclude=None):
        if classnames_add is None:
            classnames_add = []
        if classnames_exclude is None:
            classnames_exclude = []

        classnames = self.export_button_classnames + classnames_add
        cn = self.finalise_classname(classnames, classnames_exclude)
        text = _('Export {} to CSV'.format(self.verbose_name_plural.title()))

        return {
            'url': self.url_helper.get_action_url('export',
                                            query_params=self.request.GET),
            'label': text,
            'classname': cn,
            'title': text,
        }

Most of this code is just to get the CSS classes for the button right. The CSS classes icon and icon-download will ensure a simple button with a download icon and some text.

AdminURLHelper

Next, we need an AdminURLHelper that helps with generation, naming, and referencing of our new export URL:

class ExportAdminURLHelper(AdminURLHelper):
    non_object_specific_actions = ('create', 'choose_parent', 'index',
                                    'export')

    def get_action_url(self, action, *args, **kwargs):
        query_params = kwargs.pop('query_params', None)

        url_name = self.get_action_url_name(action)
        if action in self.non_object_specific_actions:
            url = reverse(url_name)
        else:
            url = reverse(url_name, args=args, kwargs=kwargs)

        if query_params:
            url += '?{params}'.format(params=query_params.urlencode())

        return url

        def get_action_url_pattern(self, action):
        if action in self.non_object_specific_actions:
            return self._get_action_url_pattern(action)

        return self._get_object_specific_action_url_pattern(action)

Once again, this looks a little more complicated than it is. We just need to add the export action to the non_object_specific_actions, because Wagtail treats actions as object-specific per default and will attempt to add the an object’s PK to the URL. Additionally, the URL helper appends the modeladmin filters to the action so that only the filtered data is exported.

ExportView

Finally, we need an ExportView that implements the CSV export. For this, we will use some help from django-queryset-csv.

Install via

pip install django-queryset-csv

Using this, our view is very simple:

class ExportView(IndexView):
    model_admin = None
    
    def export_csv(self):
        if (self.model_admin is None) or not hasattr(self.model_admin,
                                                        'csv_export_fields'):
            data = self.queryset.all().values()
        else:
            data = self.queryset.all().values(
                *self.model_admin.csv_export_fields)
        return render_to_csv_response(data)

    @method_decorator(login_required)
    def dispatch(self, request, *args, **kwargs):
        super().dispatch(request, *args, **kwargs)
        return self.export_csv()

It is worth to note the model_admin field. We will use this for specifying a custom list of exported fields. Lines 5 and 6 make sure that whenever model_admin is set and the csv_export_fields attribute is given, the custom field list is used instead of the default behavior that just exports all fields.

Mixin

Making use of Python’s Multiple-Inheritance system, we can create a Mixin that we will later use to override button_helper_class, url_helper_class, export_view_class and get_admin_urls_for_registration all at once:

class ExportModelAdminMixin(object):
    button_helper_class = ExportButtonHelper
    url_helper_class = ExportAdminURLHelper
    export_view_class = ExportView

    def get_admin_urls_for_registration(self):
        urls = super().get_admin_urls_for_registration()
        urls += (url(self.url_helper.get_action_url_pattern('export'),
                        self.export_view,
                        name=self.url_helper.get_action_url_name('export')), )
        return urls

    def export_view(self, request):
        kwargs = {'model_admin': self}
        view_class = self.export_view_class
        return view_class.as_view(**kwargs)(request)

HTML Template

Now, we need to create an HTML template that includes our new button plus the original modeladmin buttons:

{% extends "modeladmin/index.html" %}

{% block header_extra %}
    {% include 'modeladmin/includes/button.html' with button=view.button_helper.export_button %}
    {{ block.super }}{% comment %}Display the original buttons {% endcomment %}
{% endblock %}

Enabling CSV Export for Models

To make a modeladmin table exportable, just add the mixin to your ModelAdmin definitions in wagtail_hooks.py and set the index_template_name:

class FooModelAdmin(ExportModelAdminMixin, ModelAdmin):
    index_template_name = "wagtailadmin/export_csv.html"

If you want to customize the CSV columns, you can use our new optionl csv_export_fields attribute. It even allows to export attributes of related tables using regular Django ORM syntax (__):

class FooModelAdmin(ExportModelAdminMixin, ModelAdmin):
    index_template_name = "wagtailadmin/export_admin.html"
        csv_export_fields = [
        'bar', 'foobar', 'other_model__attribute'
    ]

Conclusion

In 2020, Wagtail is almost certainly the best option to add CMS functionality to a Django project. This post illustrates its extensibility. In a couple of minutes, you can enable CSV export for all your Django models.

Managing my personal knowledge base

Sun, 05 Jan 2020 00:00:00 +0000

Introduction

It is hard to imagine any other field where lifelong learning is more important than in software engineering. Another unique characteristic is the degree to which learning material is available for free on the internet. On top of that, we create various resources ourselves by documenting issues, submitting bug reports, writing notes, creating documentation, and many others. The sum of all these resources can be called a knowledge base. You could argue that every developer has a system to manage their personal knowledge base, whether they know it or not. In this post, I explain my knowledge management practices.

A software engineer’s personal knowledge base is likely to overlap with the knowledge of their employers and project partners. Be sure to carefully study your data protection obligations. A good basis is to keep your personal knowledge base strictly technical and to never include data that is related to customers, or people in general.

I chose this topic mainly because of three reasons:

Fun:
I enjoy thinking about my workflows and trying to improve them. Maybe a little too much, some friends have started to roll their eyes when I try to recommend a new tool to them.
Improving my system and workflows:
Writing a post about my Linux setup has been very rewarding for me. While documenting my configuration, especially my command-line workflows, I identified some shortcomings that I have since eliminated. After writing this article, I can already say that my knowledge management workflows have improved similarly.
Relevance:
Knowledge management in all its forms is integral to software engineering.

Knowledge management tasks are done frequently. It pays off to do them efficiently. (Source: xkcd.com)

Much of this post comes down to describing my usage of software tools. The stars will be

Copying all of my practices will likely not serve you well. I invite you to try out things at your own pace and revisit this post whenever you are looking for new ideas.

Bookmarking

Even though everybody loves books, or at least says so, myself included, the reality is that we rely on digital resources most of the time. The challenge is to organize this knowledge efficiently without creating too much overhead.

There are some naive approaches that I do not deem sufficient. Until a few years ago, I relied mainly on browser bookmarks, but for some reason, browsers do not provide proper organizing features. Another way is to just copy the URLs into notes, project issues, documentation and all other kinds of manually created content. This is viable and we all do it regularly, but it has nothing to do with what I think is bookmark management.

My current setup is a little more complex, but in the end, it does not require more work than keeping bookmarks solely in the browser. I organize my bookmarks in multiple layers, depending on how often I want to access them and how long they should persist.

Maybe you are thinking of caching now, but fortunately, bookmarking layers are not caching levels. This would be hard to manage because as we all know, there are two hard problems in computer science: cache invalidation, naming things, and off-by-1 errors. In contrast to cache entries, a bookmark is not supposed to move between layers. Instead, it gets inserted at the right place and stays there.

The following figure illustrates my layers:

Bookmarking layers

1. Chrome Bookmarks

Chrome bookmarks have some nice benefits. They are synchronized between devices, new ones can be added easily, and organizing them in folders brings at least some structure. Another advantage for me is the Albert Chromium extension that lets me access and open my Chrome bookmarks from Albert:

Searching through Chrome bookmarks directly from Albert. Pressing enter opens the selected entry in Chrome.

Because of these advantages, Chrome bookmarks are ideal for things that I need often and want to have accessible somewhat independent of context. They are, however, not well suited for keeping extensive bookmark libraries. There is no tagging, it is not possible to add comments, and browsers do not even save a timestamp for when a page was bookmarked. It is further not possible to filter and search Chrome bookmarks using complex queries. In my experience, browser bookmark libraries are hard to manage if they grow beyond a certain size.

Some examples of well-suited Chrome bookmarks:

AWS EC2 dashboard
Google cloud platform console
Slack workspaces
GitLab boards
Content sites, such as Hacker News, InfoQ, and DZone

Examples of things that should not be Chrome bookmarks:

Blog posts to be read later
Links to Open Source Projects
Stack Overflow questions

2. Workona

Workona is a relatively new addition to my toolbox. I have mixed feelings about it, mainly because it can feel a little slow at times. There are some things it does very well though.

Workona lets you create browser workspaces that come with their own list of bookmarks. This works nicely when you have multiple projects that you work on for longer periods. It will also remember which tabs you had open in a specific workspace and synchronize this information across devices. It is, therefore, a nice place for things that I need often dependent on context.

Screenshot of a Workona Workspace (Source)

Workona further allows you to add apps and will then collect links and even features for those apps to be quickly accessible. For example, I use it to access my different GitLab and GitHub projects, Slack Workspaces, and sometimes even StackOverflow questions. These entries are automatically created by Workona and therefore I do not see them as bookmarks. It is also possible to access features of these apps directly from Workona, mainly to create new resources, such as GitLab/GitHub projects, Google Docs, DropBox Files and so on. It feels a little bit like Station built directly into the browser.

The selling point for me is that Workona lets me do all of the described things via shortcuts and it is even possible to customize these shortcuts at chrome://extensions/shortcuts.

3. Notion

Finally, we arrive at the layer that is suited for maintaining a large library of bookmarks. For this, and many other use cases, I use Notion. It has become the core of my personal knowledge base.

Notion’s free tier comes with a hard limit on the number of blocks you can create. This means that you will have to switch to the paid version if you use it seriously for a while. It is free for students and teachers.
All links to Notion in this post are affiliate links. If you sign up for an account through these links, you will get 10$ of free Notion credit and I will get 5$. This works even if you create a free account.

The killer feature for me is the database. It is amazingly flexible and is single-handedly able to replace multiple other tools for me. For my bookmark collection, I use a single large database. Adding new items can be done via the Notion Web Clipper. I am a little annoyed that the web clipper doesn’t let me add properties and tags directly, but other than that, it works well. Previously, I used Trello, which was also quite good at keeping bookmarks. However, to limit the number of different tools I use, I replaced it with Notion.

My bookmarks database filtered to show only entries related to this post.

The goal with this database is that I can find something years from now with only a vague memory that it should be there. This is possible because Notion automatically stores metadata, such as a creation timestamp. Even more important, I can add tags and arbitrary properties.

Things get really exciting once you use relations between different Notion databases. As an example, I have a database of my blog posts that has a relation to my bookmarks. I can now filter my bookmarks by blog posts and quickly see which sources influenced a particular post.

I cannot cover all Notion database features in this post. I think this is something you have to see for yourself. The Notion docs page about the topic might give you some further impressions though.

The downside of this rich feature palette is that it requires some discipline. I tag and annotate new entries, that I added via the web clipper, about twice a month. Often, I just delete new items because they do not seem important anymore. If I can not think of appropriate tags or relations for an item, this often means that it is not very relevant to me.

Another important lesson I had to learn the hard way:
Do not use too many databases in parallel. Notion provides excellent methods for filtering, and searching tables. You can even define different views onto the same database that only shows a specific part of the data. It is therefore not required to separate all kinds of things into different databases. For example, at first, I had three different databases for Python resources, Django resources, and Wagtail resources. This was a bad solution. Now, these all live in the same database with appropriate tags.

I hope that someday I will be able to search through my Notion databases from Albert. Maybe I will build an extension myself once the Notion API is finally released.

To illustrate what I have described in probably too many words, you can have a look at a public fraction of my bookmarks database containing all the resources that were helpful for writing this post. The post-column in this public database appears empty because the related table is not public.

4. Native Bookmark Sources

The last layer does not require any work at all. It only means to be aware of native bookmark sources when trying to find something, and also when thinking about adding new bookmarks. For example, it is quite easy to search through questions you have answered on Stack Overflow. It is also not a problem to go back to your Hacker News posts or search through projects you starred on GitHub. Keeping those things in your own bookmark sources adds redundancy and noise.
These are some of my most commonly used native bookmark sources:

Summary

I am aware that this sounds immensely complicated. However, I stick to my statement that it does not actually require more work than keeping all bookmarks in the browser. You still have to add every bookmark only once. If you internalize your system, you will not have problems searching for bookmarks as it will be quite obvious where a specific resource could be. Admittedly, it does require some maintenance to keep a large bookmark collection in Notion, but it pays off soon and the pay-off accumulates over time.

Organizing Self-Written Resources

Having a proper system for external resources is great, but it does not help much if everything you write yourself is a mess. Therefore, it is essential to organize self-written resources.

Blog Posts

My most successful blog post is about how I run and host this website for free. However, the post does not cover the most work-intensive part about running this site: Writing and maintaining blog posts. For this, I heavily rely on Notion. I have a database with all my past blog posts and ideas for future posts. It is simple to add tags like done, doing, idea, and to view the database as a kanban board with lanes based on those tags.

Drafting Posts

All posts that are published on this blog have been drafted in Notion. When they are more or less finished, I use the markdown export and copy the content into a new text file in my Hugo project. There are some minor issues with this. Usually, I have to adjust code formattings and links, but overall it works very well and has some nice benefits:

Edit on all devices
Using notion, I can easily edit my drafts on all devices, even on my phone when I am moving.
Use existing notion content
I can quickly add code snippets, links, and other stuff that I keep in Notion. This can be done via database relations or just by dragging the respective blocks into the draft page.
Lightweight version control
As developers, we love GIT. However, for adding a couple of lines to a blog post, it is an overkill. Notion has versioning features build-in and keeps track of all changes automatically. Naturally, it does not offer sophisticated merging options, but it’s easy to stay clear of conflicts when you are writing a personal blog.

Once a post is released, I move the post from doing to released.

Keeping Track of Update Ideas

You may have noticed that some of my blog posts are quite long. Keeping them up-to-date is a little challenging. Fortunately, I have Notion to assist me with this. I have a database where I add proposals for updating my existing posts. Using a relation, I link these ideas to the respective posts in my posts database:

It is hard to know when you are finished setting up the perfect Linux workstation.

Tracking Sources

To keep track of third-party resources that were helpful in creating a post, I add them to my bookmarks database and link them to my posts database.

My bookmarks database filtered to show only entries related to this post.

You can also have a look at the full list of sources for this post via a public fraction of my bookmarks database.

Taking Notes

I suppose every software engineer takes notes to some extent, even if they do not have a system and just casually write stuff into text files. There is also a large amount of free and paid software for taking and organizing notes. Some people even like to use physical paper with the BuJo system.

No matter which method you prefer, you have to think about when, why, and how to take notes.

When to take Notes

I thought quite a bit about this question, and previously I did not adhere to any predefined rules regarding this matter. To gain a better understanding of my habits and to decide what worked best in the past, I looked through my notes from the last years that were spread over multiple applications and lots of files. I tried to find common patterns:

Capturing information from audio sources
I am a visual learner and prefer written content over audio. However, there are many types of audio sources that everybody encounters:
- Meetings
- Presentations
- Meetups
- Conferences
- Informal Discussions
- Videos
- Podcasts

Preserving information that is often needed but doesn’t fit in any proper documentation project
In my case, these are things like the following:
- Command-lines with multiple parameters, complex options, and paths. I keep them in my notes so that I can quickly copy-paste them in the potentially distant future.
- Instructions for tasks that will likely have to be repeated in the future, for example
  - setting up a development environment for a specific tech stack
  - deploying a project on a specific hosting service.
Questions
Sometimes I think of a question that I would like to ask someone specific, who is not available at the moment. The question and its eventual answer are a nice use case for a note.

Thanks to a more sophisticated command-line setup with Zsh and various plugins for auto-completion and history-search, preserving command-lines has become much less important for me. If you are still using Bash with its default Ctl - r search, I think you would profit from my post on setting up a Linux workstation.

On the other hand, it is important to make clear also what should not be a note in this system:

Project-specific knowledge
Everything that is directly related to a specific software project, should not be a note in your personal knowledge base. Instead, it should go into whichever system is used to keep track of issues, merge requests, documentation, and so on. This post includes a separate section on this below.
Information that is needed for a very short time Sometimes I need to take notes during a conversation that I will need only immediately afterward. For this, I just open VSCode and type ahead. Once the information is no longer needed, the file can be deleted completely.

In general, don’t overdo and don’t underdo it. If you take notes all the time, you create overhead and redundancy. If you never take notes, well, you will not have notes.

Why Take Notes

The obvious answer is to remember things. This does not mean that note-taking should replace your memory. To the contrary, taking notes provides structure and context and therefore helps your brain build up a map of your knowledge. I believe that taking notes increases the amount of data that you can recall from memory.

Additionally, some things are just very hard to remember, such as complex tech stacks presented at meetups, command-lines with several opaque options, numbers, and so on. Writing these things down can extend your knowledge base significantly. Notes are also very helpful when writing documentation in the future, or for writing blog posts like this one ;-)

How to Take Notes

It can be tempting to just type ahead during a talk. Noting down whatever seems interesting at the moment. However, you should meet basic formal requirements and write down some context. Otherwise, you will have trouble extracting useful information from your notes in the future. If you take notes regularly, it is also important to organize them by assigning tags and properties, having a creation date, being able to filter and search through them, and so on. Software can help with these things.

Until about a year ago I kept notes mainly on my local machines, syncing them with Dropbox. Applications that focus almost exclusively on note-taking, such as Evernote, never had that much appeal to me, even though I used Google Keep/Notes. It is not so bad, supports e.g. labeling and works with little friction on mobile. However, there is not even basic formatting support, let alone markdown formatting or code syntax highlighting.

Then, I discovered Notion. At first, It wasn’t really about note-taking for me. I liked it mainly because of its database concept and for organizing third-party resources as described in the bookmarking section. However, by now, I use it heavily for almost all of my notes and other types of self-written content. I think Notion’s mix of markdown syntax support, slash commands, and WYSIWYG makes for a great writing experience.

I strongly recommend organizing all your notes in a single database with appropriate tags, such as meeting, presentation, tc, and question. Similar to the bookmarks database described above, Notion will automatically provide a column with the creation time of the note. If you have frequent meetings with the same people, you can think about adding a Participants column.

A notes database is the perfect use case for another great Notion feature: Templates. This feature will allow you to created templates for your different types of notes and then, for example, add a new meeting note with one click. Depending on the template, the new note comes with a pre-defined layout and might include fields for the meeting’s participants and agenda. This is what opens when I click to add new meeting notes in my database:

My basic meeting notes template.

Project-specific Knowledge

In my experience, it is best to keep project information as closely together as possible. For me, this means that issue tracking, merge requests (i.e. pull requests), documentation, and everything else that comes with a software project should live alongside the source code.

This practice guarantees that anyone who works on the code now, or in the future, has access to all the information. My solution for this is to use GitLab for everything related to a specific project. This includes pretty much everything except some quick notes that I sometimes write in Notion because I do not want anyone else to see them.

To do this, GitLab offers a lot of project management and documentation features, such as issue management, a project wiki, a Code snippet space for all projects and accounts, and more. A project wiki is a great place for all documentation that is not specific to an issue. This can include the following:

Instructions for setting up the development environment
Instructions for manual deployment
Architecture descriptions and diagrams. I am a big fan of mermaid, which allows creating UML diagrams with Markdown and is supported by GitLab.
Guidelines regarding naming conventions, design paradigms, etc.

If you have read my post on running and hosting this website, you know that I use GitLab also for this personal blog project. Nevertheless, I use Notion for much of the project documentation and management tasks, because it comes with some unique benefits for this use case, as described above.

Of course, there are alternatives. GitHub has caught up in the past and offers a very similar feature set. I have only limited experience with Atlassian products, but I suppose you can achieve the same thing by using a mix of BitBucket, Confluence, and Jira.
On a side note, writing in Confluence feels a lot like writing in Notion. Both recognize markdown syntax and offer slash commands to quickly insert complex content types.

Managing Files

This one is relatively straightforward. I think the most important thing is to use some form of cloud storage. Personally, I use Dropbox.
Next, you should think about a proper folder structure. Finally, you should be able to search for your files quickly.

I use Linux and rely on Albert for searching through files. If this doesn’t work I use the locate command and, my last resort are command-line tools, such as find, and grep. They are still king for complex search requirements, such as regex matching and including also file content.

If you can tick the three boxes

availability on all devices
structure
searchable

you should be all set.

Notion databases support file columns. This means you can upload a different file to each cell. I have not yet found a use case related directly to software engineering for this. However, it is great for managing invoices.

Additional Use Cases

In this section, I want to present some special kinds of knowledge management use cases that are unique to software engineering. Most of them I have picked up properly only after starting to use Notion. By no means, I want to say that you should copy all of them, but you are very welcome to try out what you think looks interesting.

Code Snippets

Currently, I have two ways of collecting code snippets. I hope I can eventually integrate them somehow.

VSCode User Snippets

The first one are VSCode User Snippets. VSCode does many things right and snippet management is one of them. It is very easy to define new snippets and use them by typing a predefined string. Simply press Ctrl-Shift-p and choose Preferences: Configure User Snippets. Then, you can split your snippets into multiple files or just put all of them together in the global-snipptes.code-snippets file. These are some of my snippets for writing blog posts with markdown and Hugo:

"figure": {
    "scope": "markdown",
    "prefix": "fig",
    "body": [
        "{{< figure src=\"/images/\"  class=\"center-figure\"  caption=\"\" >}}"
    ],
    "description": "Insert hugo figure shortcode"
},
"callout": {
    "scope": "markdown",
    "prefix": "callout",
    "body": [
        "{{% callout \"warning info tip note\" %}}",
        "",
        "{{%/callout %}}"
    ],
    "description": "Insert hugo callout shortcode"
},
"code": {
    "scope": "markdown",
    "prefix": "code",
    "body": [
        "{{< highlight bash \"linenos=table\">}}",
        "",
        "{{< / highlight >}}"
    ],
    "description": "Insert hugo code shortcode"
}

The snippets can then be used by typing the prefix and pressing Ctrl+Space:

Notion Snippet database

The second use case concerns small pieces of code that I have written myself and that I would like to remember. This is not necessarily for productivity reasons, but rather for nostalgia. It is ideal for small algorithms, for example when you manage to substantially increase the performance of something by applying some clever technique.

I keep those snippets in a Notion database, which works great because I can tag them and add arbitrary properties. A nice benefit of this habit is that it’s not a problem if you don’t do it for a while. Your database will not look messy all of a sudden.

An exemplary entry of my code snippets database in Notion.

Shortcut Database

I believe that typing proficiency is important for a software engineer. It is easier to stay in flow if you can type fast and navigate around your IDE and tools with shortcuts. As my set of tools grew, I realized that I have a hard time to keep track of which shortcuts I am using, which of them work in which application, and whether I can change them so that they work across as many tools as possible.

Therefore, I started to keep track of some shortcuts I would like to use everywhere. These are things like creating new tabs, creating new windows, opening the respective tool’s search feature and more.

This database helps me identify overlaps and motivates me to think about which task could fit for a specific key-combination that I use often. For example, I use Ctrl - Shift - T in IDEs to search for type definitions and Ctrl - Shift - R to search for resources in general. Since I have these key combinations memorized for years and can use it very quickly, I want to make use of it wherever possible. Therefore, I try to set up my applications so that some kind of search opens whenever I use one of these combinations. My Notion database makes it easy to see which applications offer search features and which of them I can modify.

For example, my Browser now forwards Ctrl - Shift - T / R to Workona and allows me to switch/search workspaces or use the general search, respectively.

Screenshot of my Notion shortcut database

Notion’s filtering and search capabilities are great for identifying problems with your setup. For me, a problem is when two applications use different shortcuts for a very similar operation. Unfortunately, Notion is negative example of an application that doesn’t let you customize its shortcuts and therefore reduces the usefulness of my database. If someone from the Notion team reads this at some point, I kindly ask you to lobby on my behalf to prioritize personalized keyboard shortcuts.

I am excited about this use case and will talk about it in much more detail in a future blog post.

As of June 3rd, 2020, there is still no dedicated blog post for this. However, there is something more valuable:
This post gave me the idea for https://keycombiner.com.
It is a web application to track the keyboard shortcuts you use, get better at using them, and to learn new ones. It is in open beta and I would love to hear your feedback!

Tools Database

I also have a Notion database to keep track of small web tools that I use. This serves little purpose for things that are used frequently as you will remember them anyway. However, for minor things that are only needed once in a while, such as color palette generators, detecting handwriting as LaTeX symbols, and checking SSL certificates this is quite handy. Usually, you can find these tools via google, but sometimes there are many tools for a task and it is nice to keep track of the ones that work best.

Screenshot of my software tools database in Notion.

Again, the important thing here is to not overdo it. There are extensive publicly available lists of such tools. Your list should not be a copy of those but rather a small collection of the things you like and need from time to time.

Collecting Ideas for new Projects

Like many software engineers in these golden times, I sometimes think about starting new software projects, maybe even bootstrapping a small business.
You probably guessed it already, I keep ideas for possible projects in a Notion database. The main benefit is, once again, that it allows me to easily link it with other databases, such as my bookmarks.

Conclusion

Similar posts often conclude with warnings that you should be very careful not to spend too much time on organizing and maintaining your knowledge and workflows. In principle, I agree with this sentiment. After all, you want to increase your productivity.

I do believe, however, that it is worth it to spend some time thinking about your knowledge management system. The most important part about conceptualizing a system is to decide exactly which types of information you want to maintain in your knowledge base. If you get this right you will benefit for the rest of your career. Even if the tools might change in the future, the system will stay.

Please don’t hesitate to share your own experiences in the comments below. I am sure that my workflows are not perfect, but they might get closer with your tips.

This post is released together with a new post in my Reading List series, which lists related resources. If you want to dive even deeper into knowledge management concepts, you might want to have a look.

Reading List (Organizing Knowledge)

Sun, 05 Jan 2020 00:00:00 +0000

Introduction

Knowledge management is an important topic for a software engineer. I have always been interested in material on the topic. When I decided to write an extensive post on my system and workflows, I started to collect sources and similar articles. To avoid further blowing up the original post, I will cover these sources in this separate post.

There is a vast amount of resources covering specific tools and usage examples. However, I was a little surprised to see that there are relatively few resources elaborating on a complete software engineering knowledge management system. Still, I found a couple of very interesting sources.

All resources linked in this post are also listed in this public Notion database.

Hacker News Discussions

Type: Ask HN: posts on Hacker News
Author: Various authors
Date: Various dates
Links:

Hacker News gets quite a bit of hate for its discussions and comments. While it is true that people can be a little ignorant, I think it is one of the better places to have a civil discussion online. The linked posts give a nice overview of what software engineers are looking for in the field of note-taking and knowledge management.

GitLab Handbook

Type: Company Handbook
Author: GitLab
Date: Frequently updated
Link: https://about.gitlab.com/handbook/

I guess everybody knows the GitLab Handbook by now. It is a public document documenting how GitLab is run as a company, and therefore can be seen as a knowledge base. GitLab has a lot of success with it and the concept now gets picked up by more and more companies. It is actually another use case for Notion: Blendle’s employee handbook is hosted as a public Notion document.

How I organize my knowledge as a Software Engineer

Type: Post on DEV Community
Author: Bruno Paz
Date: Jan 16, 2019
Link: https://dev.to/brpaz/how-do-i-organize-my-knowledge-as-a-software-engineer-4387

This post is similar to my own approach of documenting my knowledge management workflows An engineer describes his personal strategies for organizing software engineering knowledge. I think the post is excellent and it gave me some ideas.

How I’ve Been Using Notion Personally and Professionally

Type: Post on CSS-Tricks
Author: Chris Coyier
Date: Jan 16, 2019
Link: https://css-tricks.com/how-ive-been-using-notion-personally-and-professionally/

This is an extensive post from the co-found of CodePen about how he uses Notion for all kinds of things.

Content by the creator of Dnote

Type: blog posts and company website
Author: Sung Cho
Date: Feb 5, 2018 & Sep 9, 2018
Links:

Sung Cho is someone who has thought even more about personal knowledge management than me. In his articles, he describes the usual problems, such as organization overhead, and limited searchability. Because of these issues, he decided to build his own solution that takes a radical approach to the issue. As outlined in my own post, I think that a fundamental part of a knowledge base is to incorporate existing public web resources, which is something that Dnote does not seem to address. Still, I learned a lot from his reasoning and Dnote is an interesting tool, possibly in addition to some of the tools I have presented.

GeistMap: A personal knowledge base with a focus on connections

Type: Post on Hacker Noon and GitHub repository
Author: Bryan Haakman Date: Jan 16, 2019
Links:

Similar to the posts above, Chris Coyier has created his own knowledge base tool. His focus lies on visualizing connections between notes and information. The tool is open source and published on https://github.com/bryanph/GeistMap. His blog post defines excellent criteria that any such software should adhere to.

My Text Data: Public text file repository

Type: GitLab repository
Author: Miloslav Číž
Date: Frequent updates
Link: https://gitlab.com/drummyfish/my_text_data

This is a unique take on the personal knowledge base concept, a public GIT repository with text files about all kinds of topics. I do not think that it’s a good idea to share this kind of personal data publicly, but it is interesting to see someone doing it.

Keep Productive Blog

Type: Blog and Youtube Channel
Author: Mainly Francesco D’Alessio
Date: Frequent updates
Links:

Francesco D’Alessio is a professional content creator focusing on productivity software. He often interviews guests about their Notion usage, which is a nice way for me to encounter fresh ideas.

Conclusion

I hope you found something that is of interest to you. If you would like to point out other resources covering knowledge management, you are very welcome to share them in the comments below or privately via email.

People to follow on Twitter who actually tweet about Software Engineering

Sun, 03 Nov 2019 00:00:00 +0000

Introduction

I admit it, the topic of this post is not very original. If you google for its title, you will find countless lists, some of them by large publications. I should know, because I tried to incorporate recommendations from these resources multiple times throughout my 8-year long Twitter journey. However, it always turned out that many of the recommendations are of little use for me due to one or multiple of the following reasons:

The recommended person is famous, but tweets very rarely or not at all.
The person is famous, but does not tweet about software engineering.
The person is CEO of a software company and their tweets are primarily advertisements or announcements related to the company.
The person tweets the same things as everyone else. In principle, there is nothing wrong with that, but if you follow a hundred people who tweet variations of the same 240 characters take, it can become a bit dull.

To avoid these pitfalls, I define the following criteria:

Must tweet somewhat regularly, at the very least more than once a month.
Must actually tweet about software engineering or at least closely related topics, such as the software startup scene.
I (subjectively) find the tweets interesting.

With my list, you don’t have to rely on Twitter’s verification system (Source: xkcd.com)

Obviously, any such list is highly opinionated and will always represent only a very small subset of all the amazing twitter accounts out there. Nevertheless, I am confident that my list contains many accounts that most people working in software engineering will find interesting.

I try to fit the people into different categories for better readability. Please note that some people could easily fit into multiple categories.

The listed information is taken directly from the respective person’s Twitter account. Only if their description does not give information about their role, I give an additional line stating their position in italics.

I created the first version of this post manually, but as the list grew, this was no longer feasible. So I googled for a suitable Twitter API library and wrote a few lines of hacky Python code, that can generate the required Markdown:

import twitter
import re

# You have to apply for a developer account to get API credentials
api = twitter.Api(consumer_key='',
                  consumer_secret='',
                  access_token_key='',
                  access_token_secret='')

# In practice, the code below is in a loop
user = api.GetUser(screen_name=current_screen_name, return_json=True)

screen_name = user['screen_name']
name = user['name']
description = re.sub(r'\n+',  ' - ', user['description'])
# convert twitter handles to Markdown links
description = re.sub('(@[\w\d]+)', '[\\1](https://twitter.com/\\1)', description)
description = description.replace("twitter.com/@", "twitter.com/")
location = user['location']

print("[**%s @%s**](https://twitter.com/%s)  " %(name,screen_name,screen_name))
if description:
    print(description + "  ")
if location:
    print(location + "  ")
if('url' in user['entities']):
    print(user['entities']['url']['urls'][0]['expanded_url'])
print("\n---\n")

Software Engineers

Daniel Stenberg @bagder
Internet protocol geek. I write curl. I play with HTTP and QUIC within IETF. On team @wolfSSL. HTTP/3 is coming. I don’t know anything. Stockholm Sweden
https://daniel.haxx.se

Dan Abramov @dan_abramov
he/him
https://overreacted.io
Co-author of Redux and Create React App.

Armin Ronacher @mitsuhiko
Creator of Flask; Building stuff at @getsentry — writing and talking about system arch, API design and lots of Python / Rust. https://www.twitch.tv/themitsuhiko
Vienna, Austria
https://lucumr.pocoo.org

Nick Craver @Nick_Craver
Dad, Architecture Lead, Developer, Site Reliability Engineer & DBA. @StackOverflow. MS MVP. I build very fast things to improve life for millions of developers.
North Carolina
https://nickcraver.com

Scott Hanselman @shanselman
Code, Race, OSS, Dad, Open Web, Black Hair, STEM, Beyoncé, @TheOfficialACM, MSFT, #T1D. Umuntu ngumuntu ngabantu @Hanselminutes podcast=inclusive tech talk!
Portland, Oregon
https://hanselman.com

David Fowler @davidfowl
Partner Software Architect at Microsoft on the http://ASP.NET team, Creator of SignalR, Barbadian, Tennis Player, Father, and Husband of @symonefowler
Redmond
https://davidfowl.com

Chris Smith @aChrisSmith
Author of Programming F# and all-around nice guy. Employee #3 at @PulumiCorp. Ex-Googler, Microsoftie too, in case that matters. Opinions are my own, obvs.
Seattle, WA

Safia Abdalla @captainsafia
I help maintain the @nteractio ecosystem, make open source software at @microsoft, and write books and blogs. Dream big and follow through even bigger.
Seattle, WA
https://safia.rocks

Jeff Triplett @webology
Python, Django, Docker, Engineer, Partner at @revsys - Affiliations @ThePSF @defnado @djangocon @djangoproject - Interests 🏀 ✨ 💪 🏃‍♂️ 🤖 (he/him)
Lawrence, KS
https://jefftriplett.com

Sarah Lim @sliminality
I want to talk to you about the affect and aesthetics of computing.
she/her/她
http://slim.computer

Max Stoiber @mxstbr
Maker https://t.co/lDhfGoK9Ke https://t.co/86uNp5ZSzr Δ - Eng @gatsbyjs, prev @github (acq @withspectrum) 💼 - Created styled-components, react-boilerplate… 💅 - https://t.co/565oxi5nr2
Wien, Österreich
https://mxstbr.com

Jonathan Carter @LostInTangent
I build dev tools and services @ Microsoft (Codespaces, Live Share, IntelliCode) and maintain some OSS VS Code extensions (GistPad, CodeTour, GitDoc)
Ballard, Seattle, WA

Alex Russell @slightlylate
Chrome Project 🐡 & Web Standards TL; Blink API OWNER - Named PWAs w/ @phae; probably making her ☕ - Tweets are my own and do not represent my employer’s views.
San Francisco, The Internet
https://infrequently.org/

Natalie Marleny @NatalieMarleny
✏️ Application Engineer (Independent)
London, England
http://github.com/nataliemarleny

Indie Hackers

The people in this section are mostly software engineers as well, and many could also be described as founders. However, I feel like the theme of their tweets is quite different from those other categories and deserving of its own section.

Monica Lent @monicalent
Latin major turned software engineer 💖 JS, FP, U2F, Linux, and embarrassing myself in German on a daily basis. Building @affilimateio 👩🏻‍🚀
Berlin, Germany
https://monicalent.com

Nikita Voloboev @nikitavoloboev
Make @LearnAnything_. Love animals. Vegan. http://github.com/nikitavoloboev http://instagram.com/nikitavoloboev
London
https://nikitavoloboev.xyz

Caleb Porzio @calebporzio
Launching Soon: http://MakeVSCodeAwesome.com. Co-Host of @noplanstomerge | Creator of @LaravelLivewire & @Alpine_JS
Buffalo, NY
https://calebporzio.com

Umar Hansa @umaar
• I tweet developer tips and tricks • Made http://ModernDevtools.com • @GoogleDevExpert member • Developer tips http://umaar.com/dev-tips/ • Previously web developer @shazam
London
https://numaar.com

Francesco Di Lorenzo @frankdilo
Founder. Hacker. Building @mailbrew to fix how we stay informed online.
Milan, Italy
https://mailbrew.com

Chris Achard @chrisachard
Building @meetingplace_io. Teaching everything I know about: React, Rails, JS, Node, at https://t.co/F6LJmRvN4t; online instructor @eggheadio
Carmel, IN
http://chrisachard.com

Daniel Vassallo @dvassallo
Becoming self-employed is easy, staying so is hard. I tweet about how I’m doing it. Building https://t.co/CP5YrWTk2Y. Creator on https://t.co/Ua6JqF8zb7.
Seattle, WA
https://danielvassallo.com

InfoSec

Troy Hunt @troyhunt
Pluralsight author. Microsoft Regional Director and MVP for Developer Security. Online security, technology and “The Cloud”. Creator of @haveibeenpwned.
Australia
https://troyhunt.com

SwiftOnSecurity @SwiftOnSecurity
Systems security, industrial safety, author http://DecentSecurity.com + http://GotPhish.com , write SciFi, sysadmin, & use Oxford commas. they/them/tay
Cyber, USA
https://decentsecurity.com
Taylor Swift parody account tweeting about InfoSec topics

Matthew Green @matthew_d_green
I teach cryptography at Johns Hopkins.
Baltimore, MD
https://blog.cryptographyengineering.com
Professor for cryptography at Johns Hopkins

Initially, I wanted to do a section with Academics. However, it turned out that I do not actually follow a lot of university researchers. Dou you know of any that would fit the criteria given in the introduction?

Blockchain

Vitalik Buterin @VitalikButerin
See https://about.me/vitalik_buterin Not giving away ETH. For inquiries about me advising your ICO, please email: noreply@buterin.com
Earth
https://vitalik.ca
Co-founder of Ethereum

Emin Gün Sirer @el33th4xor
CEO @avalabsofficial, Prof @Cornell, co-director @initc3org, currently on leave from the last two
Williamsburg, NY
https://hackingdistributed.com

Neeraj K. Agrawal @NeerajKA
comms @coincenter—the cryptocurrency policy think tank | neeraj@coincenter.org https://t.co/SASIdi6UWM
Washington, DC
http://coincenter.org

Elizabeth Stark 😷 @starkness
big fan of the internet. like building things. cofounder @lightning labs, fellow @coincenter. taught @stanford + @yalelawtech.
brooklyn + sf

Startup Scene

Paul Graham @paulg
https://paulgraham.com
Co-founder of Y Combinator

Garry Tan @garrytan
Managing Partner, @Initialized. Earliest investor in @coinbase @instacart @flexport & 100+ more—$36B in startup market cap so far. Forbes Midas List 2019 #21 🚀
San Francisco, CA
https://youtube.com/garrytan

Kate Clark @KateClarkTweets
Writing about venture capital & startups @theinformation | Tips to kate@theinformation.com | Signal & Telegram: 415-409-9095 San Francisco
https://techcrunch.com/newsletter

Andrew Wilkinson @awilkinson
Co-founder of Tiny, owner of @Dribbble, @MetaLab, and many others. Buying, starting, and investing in wonderful internet businesses since 2007.
Victoria, Canada
http://www.tinycapital.com

Sahil Lavingia @shl
Founder @gumroad, impending investor looking for QPs 👀
Portland, OR
http://sahillavingia.com

Suhail Doshi @Suhail
New: https://t.co/YYa4AbttbR, Founder https://t.co/EFNC6mZg4B - Pizzatarian, programmer, & music maker
Remote
https://mightyapp.com

CEOs and Founders

David Cramer @zeeg
building @getsentry
San Francisco
https://sentry.io
CEO of Sentry

Sid Sijbrandij @sytses
Legal first name: Sytse. Co-founder & CEO of GitLab. I love innovative projects, accessible education, all remote work, new cities, and macro economics. he/him
San Francisco, California
https://about.gitlab.com

Sam Altman @sama
ready player sama
Former president of Y Combinator and current CEO of OpenAI

John Collison @collision
Co-founder of @stripe.
San Francisco
https://johncollison.ie

Jeff Atwood @codinghorror
Indoor enthusiast. Co-founder of http://stackoverflow.com and http://discourse.org . Abyss domain expert. Disclaimer: I have no idea what I’m talking about.
Bay Area, CA
https://blog.codinghorror.com

Jack Dorsey @jack
CEO of Twitter

Matthew Prince @eastdakota
A little bit geek, wonk, and nerd. Repeat entrepreneur, recovering lawyer, and former ski instructor. Co-founder & CEO of Cloudflare (NYSE: NET).
San Francisco, CA
https://cloudflare.com

David Heinemeier Hansson @dhh
Creator of Ruby on Rails, Founder & CTO at Basecamp, NYT best-selling author, and Le Mans 24h class-winning racing driver.
https://dhh.dk

Tobi Lutke @tobi
@Shopify CEO by day, Dad in the evening, hacker at night. - Rails alumni; Comprehensivist. I like video games. (All tweets auto delete after 1y)
Canada
http://tobi.lutke.com

Austen Allred @Austen
CEO @LambdaSchool: A CS education with no tuition until you get a job. I have seen the “Memes” and I think they’re very good.
The Oasis
http://austenallred.com

Conor White-Sullivan @Conaw
Co-founder of @RoamResearch. Believer in tools for thought.
San Francisco, CA
http://roamresearch.com

Other Industry Professionals

Gergely Orosz @GergelyOrosz
Writing http://newsletter.pragmaticengineer.com & @EngGuidebook. Advisor @mobile__dev. Uber & Skype alumni. Jobs with great eng culture: http://pragmaticurl.com/jobs
Amsterdam, The Netherlands
https://pragmaticengineer.com

Adrienne Porter Felt @apf
I like writing silly Tweets, but that doesn’t pay so I also make @googlechrome. 🇺🇸🇨🇷 she/her Mountain View, CA
https://adrienneporterfelt.com

Richard Minerich @rickasaurus
Unicorn Dad, Dinosaur Programmer, Fancy Types, Machine Learning, Physics. CTO at Safe Banking Systems (Part of Accuity). Tweets/Opinions are my own.
Hoboken, NJ
https://richardminerich.com

Amanda Silver @amandaksilver
CVP of Product for Developer Tools @ Microsoft. @code, @VisualStudio, @TypeScript, @dotnet, etc. Nerd of nerds. Momming hard. Opinions mine. She/her. 🌈 HIRING!
Seattle, WA
http://www.visualstudio.com

Reza Zadeh @Reza_Zadeh
Machine Learning @Matroid & @Stanford. Optimizing for Intelligence.
Palo Alto, CA
http://reza-zadeh.com

Conclusion

I believe that Twitter is a solid tool to stay aware of current discussions and trends in software engineering. Hopefully, this list helped you to improve your timeline. In case you are just starting with Twitter, consider following also people from other domains to create a more diversified Twitter experience.

Of course, suggestions are very much welcome, especially if they fit the above criteria.

Last but not least, you are welcome to follow me on Twitter, too. I tweet about all things software engineering, and post about working on my side-project KeyCombiner.

Copy-paste ready commands to set up SGE, PBS/TORQUE, or SLURM clusters

Fri, 18 Oct 2019 00:00:00 +0000

Introduction

Local HPC clusters continue to play a vital role in scientific research. Many universities, research institutions, and companies continue to maintain on-site clusters despite the efforts of the big cloud providers to expand their offerings into this area.

The software systems responsible for making these clusters of computers work together can be called Distributed Research Management System. The most commonly used ones are SGE, PBS/TORQUE, and SLURM. Unfortunately, setting those up can be very painful. There is a lot of outdated information on the internet. It is required to fight with several different configuration files and minor issues can mean that no command will run properly.

Operating principle of a traditional resource-manager based computing cluster.

If your task is to set up a large system that will be used for many years and you have got several days to do this perfectly, this might not be a problem. However, if you just want to play around with the different systems, spending several hours until you can submit your first job is a very bleak outlook.

Therefore, I present copy-paste-ready instructions for setting up SGE, PBS/TORQUE, or SLURM on a single machine, which will act as the master and a compute node at the same time. All guides are tested on Ubuntu 18.04, but should work with little modification on most recent Linux installations. These commands can also serve to create container images or other automated setup workflows.

Naturally, caution is warranted when executing commands as a super user. If you apply the listed commands on a production system, please make sure you understand everything that is done.

Slurm

Trying to set up SLURM on my development machine in order to run some test suites that need to interface with SLURM cost me quite a bit of time. There are plenty of extensive tutorials available. In contrast to SGE and PBS, there is even an up-to-date official documentation. However, those resources cover much more than is needed for a single-node cluster and it is easy to get lost in the many configuration options that you will not need at first.

Fortunately, I found this thread in the Ubuntu forums, which is the basis for my SLURM instructions.

Start by installing munge and slurm:

sudo apt install munge slurm-wlm

Then, create a new file at /etc/slurm-llnl/slurm.conf with the following content:

ControlMachine=<YOUR-HOST-NAME>

MpiDefault=none

ProctrackType=proctrack/pgid
ReturnToService=1
SlurmctldPidFile=/var/run/slurm-llnl/slurmctld.pid
SlurmdPidFile=/var/run/slurm-llnl/slurmd.pid
SlurmdSpoolDir=/var/lib/slurm-llnl/slurmd
SlurmUser=slurm
StateSaveLocation=/var/lib/slurm-llnl/slurmctld
SwitchType=switch/none
TaskPlugin=task/none
#
# SCHEDULING
FastSchedule=1
SchedulerType=sched/builtin
SelectType=select/linear
#
# LOGGING AND ACCOUNTING
AccountingStorageType=accounting_storage/none
ClusterName=<YOUR-HOST-NAME>
JobAcctGatherType=jobacct_gather/none
SlurmctldLogFile=/var/log/slurm-llnl/slurmctld.log
SlurmdLogFile=/var/log/slurm-llnl/slurmd.log
#
# COMPUTE NODES
NodeName=<YOUR-HOST-NAME> CPUs=4 Sockets=1 CoresPerSocket=4 ThreadsPerCore=2 State=UNKNOWN
PartitionName=long Nodes=<YOUR-HOST-NAME> Default=YES MaxTime=INFINITE State=UP

You only need to replace <YOUR-HOST-NAME> with your actual hostname. Simply execute the hostname command if you are not sure what it is. Of course, you can also modify other settings, such as the number of CPUs, but the basic configuration from above should be enough for our 1-node cluster.

We are almost finished, just enable and start the manager slurmctld:

sudo systemctl enable slurmctld
sudo systemctl start slurmctld

Finally, enable and start the agent slurmd:

sudo systemctl enable slurmd
sudo systemctl start slurmd

Congratulations, your Slurm system should be up an running! Use sinfo to check the status of the manager and the agent. The command scontrol show node will give you information about your node setup.

If Slurm did not start, fear not, there is likely only a small fix required. Start by looking for an error message with

systemctl status slurmd.service

Does it say that the slurmd.pid file coud not be opened? If so, your slurm.conf file probably has different values for SlurmctldPidFile and SlurmdPidFile than your slurmctld.service file. Check by printing the latter:

less /usr/lib/systemd/system/slurmctld.service

It might show PIDFile=/run/slurmctld.pid, which means that we need to change SlurmctldPidFile and SlurmdPidFile in our /etc/slurm-llnl/slurm.conf to the following:

SlurmctldPidFile=/run/slurmctld.pid
SlurmdPidFile=/run/slurmd.pid

After that, try again to start slurmctld and slurmd with the above commands.

SGE

This one is not so simple.

Some parts of the following setup are taken from a public Dockerfile created by Robert Syme.

First, gain root permissions. On Ubuntu you can type sudo -i. All commands have to be executed with root permissions.

Then, create a new folder, let’s say via mkdir /opt/sge/installfolder. Set the new folder’s location as an environment variable in the current shell via

export INSTALLFOLDER=/opt/sge/installfolder.

Then, execute the following commands:

cd $INSTALLFOLDER
wget https://arc.liv.ac.uk/downloads/SGE/releases/8.1.9/sge-common_8.1.9_all.deb .
wget https://arc.liv.ac.uk/downloads/SGE/releases/8.1.9/sge-doc_8.1.9_all.deb .
wget https://arc.liv.ac.uk/downloads/SGE/releases/8.1.9/sge_8.1.9_amd64.deb .
dpkg -i ./*.deb

Then, download the following 4 files and place them also into the created folder, in our case into /opt/sge/installfolder:

Most of the magic will happen through those scripts and configuration files. After the download, we need to set some environment variables in the current shell:

export SGE_ROOT=/opt/sge
export SGE_CELL=default

We also need to set a new profile.d config via

ln -s $SGE_ROOT/$SGE_CELL/common/settings.sh /etc/profile.d/sge_settings.sh

Then, execute the following to install SGE and perform setup operations:

cd $SGE_ROOT && ./inst_sge -m -x -s -auto $INSTALLFOLDER/sge_auto_install.conf \\
&& sleep 10 \\
&& /etc/init.d/sgemaster.sge-cluster restart \\
&& /etc/init.d/sgeexecd.sge-cluster restart \\
&& sed -i "s/HOSTNAME/`hostname`/" $INSTALLFOLDER/sge_exec_host.conf \\
&& sed -i "s/HOSTNAME/`hostname`/" $INSTALLFOLDER/sge_hostgrp.conf \\
&& /opt/sge/bin/lx-amd64/qconf -Me $INSTALLFOLDER/sge_exec_host.conf

Now, our new cluster is already up and running. However, we still need to add users to the sgeusers group, which was defined in the sge_hostgrp.conf file you just applied. Only users from this group are allowed to submit jobs. Therefore, we run the following:

/opt/sge/bin/lx-amd64/qconf -au <USER> sgeusers

Finally, run $INSTALLFOLDER/sge_init.sh.

You can now delete the complete $INSTALLFOLDER, or at least run

rm $INSTALLFOLDER/*.deb

After a restart to load the new profile.d settings, all users that were added to sgeusers should be able to submit jobs. Test this via:

echo "echo Running test from $HOSTNAME" | qsub

PBS / Torque

This is a little simpler again. Most of the setup is taken from another blog. Some inspiration is also from a public Dockerfile available at Docker Hub.

First, gain root permissions. On Ubuntu you can type sudo -i. All commands have to be executed with root permissions.

We start by installing the relevant packages:

apt-get install torque-server torque-client torque-mom torque-pam

Installing these packages will create a default setup. Unfortunately, this is complex and would require complex changes to get to a working cluster. Instead, we stop all torque services and create a clean setup:

/etc/init.d/torque-mom stop
/etc/init.d/torque-scheduler stop
/etc/init.d/torque-server stop
pbs_server -t create
killall pbs_server

We start by setting localhost as the server host and allowing root to change the database configuration:

echo localhost > /etc/torque/server_name
echo localhost > /var/spool/torque/server_priv/acl_svr/acl_hosts
echo root@localhost > /var/spool/torque/server_priv/acl_svr/operators
echo root@localhost > /var/spool/torque/server_priv/acl_svr/managers

With the following commands, we set localhost also as compute node with 4 cores available:

echo "SERVER.DOMAIN np=4" > /var/spool/torque/server_priv/nodes
echo localhost > /var/spool/torque/mom_priv/config

Now, we can already start the daemon processes again:

/etc/init.d/torque-server start
/etc/init.d/torque-scheduler start
/etc/init.d/torque-mom start

After this qmgr is ready to start the scheduler:

qmgr -c 'set server scheduling = true'
qmgr -c 'set server keep_completed = 300'
qmgr -c 'set server mom_job_sync = true'

The following commands are to create a default queue and to configure it:

qmgr -c 'create queue batch'
qmgr -c 'set queue batch queue_type = execution'
qmgr -c 'set queue batch started = true'
qmgr -c 'set queue batch enabled = true'
qmgr -c 'set queue batch resources_default.walltime = 3:00:00'
qmgr -c 'set queue batch resources_default.nodes = 1'
qmgr -c 'set server default_queue = batch'

Finally, we allow our machine to submit to the new cluster:

qmgr -c 'set server submit_hosts = <hostname>'
qmgr -c 'set server allow_node_submit = true'

To test the setup, you can use the following command:

echo "echo Running test from $HOSTNAME" | qsub

Conclusion

The target audience for this post is probably very small. I decided to do it anyway, as there might be someone that can benefit immensely. Personally, I would have saved a lot of time if I found such a blog post a year ago. Also, I might need it again myself at some point.

Reading List (August and September 2019)

Mon, 30 Sep 2019 00:00:00 +0000

Introduction

You and me have something in common: We both read blog posts about software engineering.

In my opinion, they are a great source of information and play an important role for the open-source ecosystem. Many projects almost depend on third-party blog posts to provide additional documentation and tutorials.

Starting with today, I will regularly present blog posts, articles and other online resources that I think are worth a read. My selection criteria will be somewhat arbitrary. In general, I will try to share content that I think contains valuable information for a large spectrum of software engineering professionals. Additionally, I will provide short summaries for all the articles I post.

Note from the future: I did not publish regular reading lists like I planned. When I started my side-project KeyCombiner, other topic ideas became more interesting for me. Sorry! Please let me know if you know another blogger that does this regularly.

So, without further ado, here are posts and other resources from the last two months that I found particularly interesting:

How I Built A $7K/Month WordPress Plugin Because I Needed The Product

Type: Starter Story Blog Post
Author: Laura Elizabeth
Date: August 30, 2019
Link: https://www.starterstory.com/build-profitable-wordpress-plugin-from-scratch

I guess almost every software engineer has thought at some point about developing an app or other kind of small project that could ideally generate a consistent revenue stream, maybe in parallel to a regular employment.

In this article Laura Elizabeth describes with impressive clarity how she managed to do just that. After reading it, you will have a very good idea about how she started her business with Client Portal, a tool that can help enterprises work on projects together with their customers.

The boring technology behind a one-person Internet company

Type: Medium Blog Post
Author: Wenbin Fang
Date: January 24, 2018
Link: https://broadcast.listennotes.com/the-boring-technology-behind-listen-notes-56697c2e347b

Similar to the previous post, this article talks about building a one-person company. However, the difference is that this one focuses on the technical implementation and gives tips how large software products can be built with very few engineers. The author of the article is Wenbin Fang, the founder of Listen Notes.

Fang fits a lot of information into a relatively short article, while managing to keep it very easy to follow.

Google Engineering Practices Documentation

Type: Static GitHub Pages Site
Author: Google
Date: September 05, 2019
Link: https://google.github.io/eng-practices/

In September, Google has published resources on their internal engineering practices. Currently, it is mainly focused on the code reviewing process from both the reviewer’s perspective and from the perspective of the code author. Much of the stuff in there is rather straightforward, but it is very nice to have all of this documented in a condensed fashion.

If your company struggles with code reviewing, this might be the resource to find ideas for improving your process.

Software Architecture Guide

Type: Personal Website
Author: Martin Fowler et al.
Date: August 01, 2019
Link: https://martinfowler.com/architecture/

Martin Fowler has published a guide to material on his blog about popular topics related to Software Architecture, such as Micro Frontends, Serverless, and Enterprise Architecture.

His introduction resonates a lot with me:

Like many in the software world, I’ve long been wary of the term “architecture” as it often suggests a separation from programming and an unhealthy dose of pomposity. But I resolve my concern by emphasizing that good architecture is something that supports its own evolution, and is deeply intertwined with programming.

I too have some issues with identifying as a Software Architect even though it is written on my company business cards. I believe a good software engineer is both an architect and a developer.

In my opinion, the resources collected in Fowler’s Guide are of very high quality and can serve as excellent references for the respective concepts.

Software Architecture is Overrated, Clear and Simple Design is Underrated

Type: Personal Website
Author: Gergely Orosz
Date: September 17, 2019
Link: https://blog.pragmaticengineer.com/software-architecture-is-overrated/

Gergely Orosz might not appreciate the lengthy abstract articles from the resource above. In his post, he argues that formal architecture practices are overrated and engineers should instead focus on simple designs.

I do agree with Orosz that teams should be careful not to get sidetracked while designing an overly complicated architecture. Of course, the idea is not to skip design steps, but to create a more friction-less process. I recommend the article especially to engineers working in large enterprises, where architecture documents often have to traverse several hierarchy levels before decisions are made.

Why our team cancelled our move to microservices

Type: Medium Blog Post
Author: Steven Lemon
Date: August 09, 2019
Link: https://medium.com/@steven.lemon182/why-our-team-cancelled-our-move-to-microservices-8fd87898d952

Chances are you already know this one. It caused a lot of discussion, especially on Reddit.

The post is about the efforts of an engineering team to break up a software monolith into microservices. As you can guess from the title, the endeavour turned out to be more difficult than expected and was ultimately cancelled. Even though many of the identified issues seem rather obvious, I think this post is an excellent read for everyone preparing to break up monoliths. You might want to take a step back and think about whether your organization is really in a position to benefit from various independent service applications.

Conclusion

I hope you found something that is of interest to you. If you would like to point out other recent articles, you are very welcome to share them in the comments below or privately via email.

Setting up a Linux Workstation for Software Development

Tue, 13 Aug 2019 00:00:00 +0000

Introduction

In my opinion, Linux is the best operating system for a software engineer, except they develop explicitly for Windows or macOS. Whenever I read about new features for the Windows Subsystem for Linux, I wonder why people settle for anything less than the real thing. Linux aids many of my workflows with its customizability, superior command-line experience, native support for containerization platforms, a dependable package management system, and many other features that I will describe in this blog post. macOS comes close, but, in my opinion, falls short due to exorbitant hardware costs and less room for customization.

Still, as with any other operating system, optimizing a Linux workstation for software development requires some configuration. In this post, I will present tools and concepts that I think are very helpful, but overlooked by many. I will try to show only things that are not specific to my workflows, but rather useful for almost any software engineer, DevOps specialist, or just regular productivity enthusiast. Naturally, many of the configuration options and tools covered in this post are subjective. Nevertheless, I am confident that my suggestions are a good place to start. They have served me well over the years and I continually try to optimize my setup. Hence, I very much appreciate any improvement you might suggest.

Prerequisites

A Linux machine
For this guide, I will use Ubuntu 20.04, which is the current LTS release. I am very happy with Ubuntu. Anything I don’t like can easily be changed. The biggest advantage over many other distributions is probably that the Ubuntu community is very large and it is usually easy to find people with similar problems. There is also a very active StackExchange community. However, most of the things covered in this post will apply to pretty much any Linux distribution and even macOS.

Some command-line experience
Much of this post is intended for people who feel right at home when they see a ~- character. If you are not comfortable working with command-lines and have no intention to change this, you can still read the other sections though.

Tweaking Ubuntu

Before we start any serious work, I will point out some changes to Ubuntu itself that I apply on all my machines.

Appearance

One of the first things I do after installing Ubuntu is to install the GNOME Tweak Tool:

sudo apt-get install gnome-tweak-tool

It enables complex modifications of the GNOME desktop environment. I recommend you play around with some of its options. For this guide, we will only consider the Appearance section. Specifically, we want to change the default Ubuntu theme to another one that provides better contrasts and an overall cleaner experience, namely the Adapta-gtk-theme, which is based on the Material Design guidelines.

sudo apt-get install adapta-gtk-theme

After installing the theme, you should restart the GNOME Tweak Tool, and you can then select Adapta-Eta as your new theme. The theme comes with some nice benefits that you wouldn’t think of at first. For example, it fixes the ridiculously small toolbar icons in the Eclipse IDE.

Adapta Theme elements (Source).

Unfortunately, the Adapta theme is no longer actively maintained. It is still a perfectly fine theme to use and if you notice any glitches, you can always switch themes with a couple clicks. A possible alternative is Materia, which also works well with the remainder of this blog post.

Nautilus Type-Ahead Search

The next thing I suggest is to change the Nautilus search behavior. With recent Ubuntu distributions, there is no type-ahead search in the standard file explorer. This means you can not start to type some letters and jump between files that start with these letters. Instead, Nautilus will enter a dedicated search mode that is confusing and quite slow for folders with a large number of files.

Fortunately, we can easily get the old behavior back:

sudo add-apt-repository ppa:lubomir-brindza/nautilus-typeahead
sudo apt dist-upgrade
killall nautilus # to force a restart

Searching through my /bin folder by typing ahead (Source).

There is a StackExchange answer that provides further details.

Hibernation

Another thing that is essential for me is hibernation. Usually, I will only shutdown my computer once a week or even less. However, I am not a big fan of suspending for long periods of time. Power outages could have unforeseen consequences and I simply feel a little better if my machines are off the grid altogether.

Unfortunately, setting up hibernation is tricky and can depend on your hardware. I updated this section multiple times already to contain the most robust solution, but you never quite know. If you don’t care about hibernation, you should probably skip this section and spare yourself the trouble.

In principle, hibernation is possible out of the box. Just type sudo systemctl hibernate. However, this is likely to fail with an error. My fix for this relies on a mixture of this and that blog post. First, we need to find out the current size of our swap file:

The below instructions assume that you use Ubuntu 20.04 with a swap file. This is the default for new installations. However, if you upgraded from a previous version, you might still have a swap partition. In this case, the process is similar, but not covered here.

sudo swapon --show # or `cat /proc/swaps`

If this is less than your main memory (RAM), you will need to increase it:

sudo swapoff -a
sudo fallocate -l <SIZE>G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
sudo swapon --show # to verify changed size

Next, we need to find out the UUID and offset of our now large enough swapfile:

sudo swap-offset /swapfile
> <OFFSET>
sudo findmnt -no SOURCE,UUID -T /swapfile 
> <UUID>

With this, we are almost done. Using the UUID and the file offset, we edit the following line in /etc/default/grub:

`GRUB_CMDLINE_LINUX_DEFAULT="quiet splash resume=UUID=<UUID> resume_offset=<OFFSET>"`

Then, we create a conf.d resume file. Simply do sudo nano /etc/initramfs-tools/conf.d/resume and add the content RESUME=UUID=<UUID> to the file. At last, we need to update grup and initramfs:

sudo update-grub
sudo update-initramfs -u -k all

And finally, you should be able to do sudo systemctl hibernate.

Please let me know if this approach doesn’t work on your system. I would like to include potential problems in this post, if there are any.

Boosting Command-Line Productivity

I believe that command-line proficiency can significantly boost developer productivity. This is especially true in times of DevOps and NoOps, where developers are supposed to handle deployment pipelines and cloud infrastructure. Fortunately, there are some tools that can help you become a command-line wizard.

Shortcuts

Before going forward, I would suggest to set up some general terminal shortcuts. Simply go to Edit -> Preferences -> Shortcuts. I like to replicate web browser shortcuts for opening new tabs and windows, closing tabs, and cycling between tabs.

Did you know that all common terminal emulators have an Undo shortcuts with Ctrl+Shift+-? Among many other things, my side-project KeyCombiner has a public collection of terminal shortcuts, many of which are barely known and still very useful.

However, using this interface, it is not possible to set up Ctrl+Tab and Ctrl+Shift+Tab as tab switching shortcuts. Gnome has some issues with using the Tab key in shortcut configurations. Thanks to this answer from the Ubuntu community, we can get around this. Simply execute the following in your terminal:

gsettings set org.gnome.Terminal.Legacy.Keybindings:/org/gnome/terminal/legacy/keybindings/ next-tab '<Primary>Tab'
gsettings set org.gnome.Terminal.Legacy.Keybindings:/org/gnome/terminal/legacy/keybindings/ prev-tab '<Primary><Shift>Tab'

Switching from Bash to Zsh

A couple of months ago, I made an important change in my life. I switched from Bash to Zsh.

There are a lot of blog posts, threads, and articles that compare Bash and Zsh, e.g.:

At the end it boils down to one thing: Plugin and theme support

Zsh has an amazing themes and plugins ecosystem. Theme support allows you to easily get a beautiful shell. Plugins range from simple auto-completion for all kinds of applications and tools(git, mvn, hugo, …) to complex stuff like improved search capabilities. Because of this fact alone, Zsh is superior to Bash.

However, chances are that you grew fond of Bash and that you get a bad feeling when somebody tells you that there is something better. I understand, it was the same for me. Still, I urge you to give it a shot. The transition is very painless since pretty much everything you like about Bash still works with Zsh. You will not have to learn new commands.

Installing Zsh

Installing Zsh on Ubuntu is very easy. We can use the regular package manager:

sudo apt install zsh

Then, we set Zsh as our new default shell:

chsh -s $(which zsh)

The chsh command updates the /etc/passwd file, but does not affect the current session. You need to log out and log in again to have zsh as your default shell.

Configuring Zsh History

For me there are few things worse than searching for an old shell command only to find out that the shell’s history size was set to a ridiculously low default number and that, because of this, the command was not saved.

Therefore, we extend the Zsh history size by putting this at the end of the ~/.zshrc configuration file:

HISTSIZE=10000000
SAVEHIST=10000000

For me, the command history is a tool to search for previous commands I might need now. I do not use it as a protocol for what I did when and in which exact order. Therefore, I use the following options to get rid of all duplicates in my history automatically:

## History command configuration
    
setopt HIST_IGNORE_ALL_DUPS # ignore duplicated commands history list

Also, I like to use multiple terminal sessions in parallel, potentially working on the same problem. For this, it helps a lot to share the same history among all sessions. It can be accomplished by

setopt SHARE_HISTORY # share command history data

If you are transitioning from Bash to Zsh, you probably want to keep your existing Bash history. In contrast to Bash, Zsh stores timestamps in its history file. Therefore, re-using the Bash history file does not work out of the box. Fortunately, all we need to do is to append some timestamp to all the commands from our previous Bash history and put them into the new Zsh history file. You could easily script this yourself or use one of the many good scripts that other people wrote, such as this one.

Zsh Features that are not in Bash

Once we have Zsh installed and finished our basic history configuration, we can start to take a look at some out-of-the-box features that make it stand out in comparison with Bash. If these do not impress you, keep in mind that the real big thing is plugin support, which I cover below.

Recursive Path Expansion

Zsh can do much more sophisticated expansions than Bash. This is documented in detail in the Zsh manual.

One feature that you can easily start to use is recursive path expansion. If you type cd /usr/local/b in Bash, you can then press Tab to expand this to cd /usr/local/bin. With Zsh, you can also type cd /u/lo/b and then press Tab. Zsh will recursively expand all path elements:

The above animation shows also that you can omit cd when changing directories.

Calculator: zcalc

Before switching to Zsh, I was often using the Python Shell as a calculator. However, this has a few downsides. Obviously, the Python Shell is not primarily developed to be a good calculator app. For example, simple operations, such as sqrt, require import statements, and there are no convenient ways to access results of previous lines.

The calculator script shipped with Zsh is a very simple and efficient solution. Before using it, it has to be loaded:

autoload -Uz zcalc

If you plan to use it often, you can add the autoload command to your ~/.zshrc.

Then, start the calculator with zcalc and do some calculations:

Results of previous operations can be accessed with the $-character.

Advanced Aliasing

Zsh has some unique aliasing capabilities.

Suffix Aliases
This means you can assign applications to file suffixes, e.g. alias -s txt=vscode will allow you to open a .txt file with Visual Studio Code by just typing file.txt.

Global Aliases
Global aliases are set with the -g flag and allow to specify a replacement for any part of a command. For example, if you set alias -g G=' | grep' you will be able to do the following:

To make aliases permanent, it is required to add them to your ~/.zshrc.

Installing ZSH plugins

Finally, it is time to explore the real strength of Zsh, which means installing plugins to boost our productivity above anything this anime character has ever seen before:

Package Manager: Oh my Zsh

A good place to start is to install Oh my Zsh, “a community-driven framework for managing your Zsh configuration.” Simply do:

wget https://github.com/robbyrussell/oh-my-zsh/raw/master/tools/install.sh -O - | zsh

Copy the template configuration file .zshrc.zsh-template in the .zshrc home directory and apply the configuration by executing the source command. Both can be done at once by executing the following line:

cp ~/.oh-my-zsh/templates/zshrc.zsh-template ~/.zshrc

Oh my Zsh comes with many themes and plugins that we only have to enable in the configuration file in order to use them. In the next section, we will start by selecting a pretty theme.

Theme: Agnoster

Personally, I like the agnoster theme. It works well with the Adapta-Eta Gnome theme we have set previously.

To enable a theme, set ZSH_THEME to the name of the theme in your ~/.zshrc, in my case this is ZSH_THEME=agnoster. If you have troubles with special characters not being shown properly, install the Powerline font package:

sudo apt-get install fonts-powerline

The Oh my Zsh GitHub wiki provides a well curated list of available themes along with screenshots.

All pictures and animations of terminals you see in this post show the Agnoster theme with slight modifications to the prompt. In particular, I removed the git-part of the prompt and added a newline part instead. If you would like to replicate this, edit your ~/.oh-my-zsh/themes/agnoster.zsh-theme and replace the ## Main prompt part close to the end with the following:

prompt_newline() {
  if [[ -n $CURRENT_BG ]]; then
    echo -n " %{%k%F{$CURRENT_BG}%}$SEGMENT_SEPARATOR
%{%k%F{$CURRENT_BG}%}$SEGMENT_SEPARATOR"
  else
    echo -n " %{%k%}"
  fi

  echo -n " %{%f%}"
  CURRENT_BG=''
}

## Main prompt
build_prompt() {
  RETVAL=$?
  prompt_status
  prompt_virtualenv
  prompt_aws
  prompt_context
  prompt_dir
  prompt_bzr
  prompt_hg
  prompt_newline
  prompt_end
}

Additionally, I recommend changing the font color of the directory part of the prompt from black to white. In my opinion, this provides better contrast with the dark blue background:

# Dir: current working directory
prompt_dir() {
  prompt_segment blue white '%~'
}

Basic Plugins

Next, you should enable some included plugins via adding them to your plugins setting in your ~/.zshrc file. After this step, it might look like this:

plugins=(colored-man-pages git python django mvn aws docker vscode)

You can browse all the plugins that come with Oh my Zsh in the official GitHub repository.

The above listed plugins are rather trivial, they provide command completion and define a set of aliases and shortcuts that come in handy when working with the respective tool. Nevertheless, this provides a real productivity boost. For example, finding out which git commands start with r becomes much simpler:

Jump to Frequently Used Directories

Most command-line work happens in just a handful of directories. For me, these are primarily the root folders of various software development projects. Of course, I could manually set aliases for these folders so that I can quickly switch between them, but this would be tedious.

Fortunately, there is a better solution with z. It tracks your most used directories and will take you to them via z <query>. The query can be any part of target directory’s path:

The software is already bundled with Oh my Zsh, simply add z to the list of plugins:

plugins=(z colored-man-pages git python django mvn aws docker vscode)

I could have probably included this one in the list of basic plugins above, but I wanted to highlight it because I find it particularly useful.

Supercharging History Search

The original Bash history search always bothered me. Repeatedly pressing Ctrl+r and hoping to get lucky at some point is very cumbersome if you have lots of similar commands containing the query. To be fair, there are also great additions for Bash that improve this situation, such as McFly.

However, this is about improving history search with Zsh. I believe the best solution is using peco-history. It is a Zsh plugin that relies on peco to interactively filter through your history file. Peco is a general-purpose interactive filtering tool that you can (and should!) also use as a stand-alone tool in your terminal. On Ubuntu, we can install it from the official repositories:

sudo apt-get install peco

Afterwards clone zsh-peco-history into the $ZSH_CUSTOM/plugins directory. Zsh will automatically resolve the pre-set shell variable:

git clone https://github.com/jimeh/zsh-peco-history.git $ZSH_CUSTOM/plugins/zsh-peco-history

Then, add zsh-peco-history to the list of plugins:

plugins=(zsh-peco-history z colored-man-pages git python django mvn aws docker vscode)

The plugin will be ready once you start a new terminal session.

The following animation shows that this improved history search was very helpful for writing this blog post. It allowed me to easily search for the exact commands I used when I set up my current machine:

Searching shell history with peco.

Autosuggestions

Our new history search is great. The bad news is that we will not use it very often, because there is something that is even better in most cases: zsh-autosuggestions
This plugin suggests completions for your commands as you type. By default, the suggested completion will be the first match in your history. This is how it looks like:

Using a suggested git command-line.

git clone https://github.com/zsh-users/zsh-autosuggestions ${ZSH_CUSTOM:-~/.oh-my-zsh/custom}/plugins/zsh-autosuggestions

Afterward, add zsh-autosuggestions to your plugins setting in your ~/.zshrc file:

plugins=(zsh-autosuggestions zsh-peco-history z colored-man-pages git python django mvn aws docker vscode)

Syntax Highlighting

Another great plugin is zsh-syntax-highlighting. It highlights commands while you are typing them. This is especially useful for catching syntax errors. Once again, install via git:

git clone https://github.com/zsh-users/zsh-syntax-highlighting.git ${ZSH_CUSTOM:-~/.oh-my-zsh/custom}/plugins/zsh-syntax-highlighting

Then, add zsh-syntax-highlighting to your plugins setting in your ~/.zshrc file:

plugins=(zsh-syntax-highlighting zsh-autosuggestions zsh-peco-history z colored-man-pages git python django mvn aws docker vscode)

Syntax highlighting also indicates whether a command is typed correctly.

Clipboard Tool

This one is not really a full-blown plugin. Oh my Zsh comes with a script called clipcopy out of the box. It is a simple tool that enables clipboard copy and past operations from the command-line:

<command> | clipcopy    # copies stdin to clipboard
clipcopy <file>         # copies the content of a file to clipboard

If this doesn’t work, you are probably missing xclip. Install it via

sudo apt-get install xclip

Somehow, I have a hard time remembering the name clipcopy, therefore I set a global alias as described above:

alias -g ctrlc=' | clipcopy'
echo "copythisinclipboard" ctrlc

If you don’t use Oh my Zsh, you can get the functionality of clipcopy via an actual plugin.

SSH Setup

One of the great advantages of using Linux instead of Windows is the out-of-the-box SSH support. Immediately after installing your distribution, you can start to access remote servers.

However, there is some room for streamlining your SSH tasks. You can avoid lots of repetitive typing if you modify your .ssh/config file and setup meaningful defaults for the machines you access regularly:

Host ec2-instance # you can freely chose a name here that will act as alias for this host
    HostName <ip-address or hostname>
    User tkainrad


Host cluster # you can freely chose a name here that will act as alias for this host
    HostName <ip-address or hostname>
    User tkainrad
    IdentityFile `/home/tkainrad/cluster_rsa.pub`

For more sophisticated settings, look up the ssh_config man page.

If you want to go one step further, you can set up public key authentication so you can login to your most-used remote hosts without having to re-enter your password each time. With ssh-copy-id, there is a simple tool that streamlines this process. If you have your hosts already configured via the .ssh/config file, just do

ssh-copy-id ec2-instance

By default, ssh-copy-id will take your public key in ~.ssh/id_rsa.pub. If you want to use different keys for each of your remote hosts, check out the man page page or this more extensive guide. If you use keys other than ~.ssh/id_rsa.pub you also have to add an IdentityFile entry to your .ssh/config settings. Afterwards, you will be able to access the machine by simply doing:

ssh ec2-instance

Additional Tools

A great terminal gets you a long way. However, some additional tools can help make your workstation more complete. In this section, I try to present some of my favorites that could be useful to almost anyone.

Text Editing: VSCode

For most of my more extensive software engineering projects, I use JetBrains products or Eclipse. I will not describe setting up those in this post, as this would easily mandate a separate article. However, I present a more lightweight editing environment that is often a better choice than to start up a full-blown IDE. For a software engineer, especially for one that deals also with DevOps workloads, it is the daily business to edit and view all kinds of text-based files, such as .txt, .csv, .md, xml, .yml, and .json. For this, it is essential to have a fast text editor which can deal with large files and still provides sophisticated editing features.

Previously, I used Sublime. For a couple of months now, I have switched to Visual Studio Code and I am still amazed by it. Comparing it to Sublime makes little sense in my opinion, because VSCode is just on a whole other level. Even for software projects with relatively simple technology stacks, such as this blog website, I now prefer VSCode over Eclipse or the JetBrains suite.

Screenshot of VSCode showing the default light theme (Light+) (Source).

The easiest way to install Visual Studio Code on Ubuntu is to visit the official website and to download the Debian package. You could have probably figured out how to do this yourself, but do you also know how to set VSCode as the default application for all text files? Simply do

xdg-mime default code.desktop text/plain

This will spare you the headache of doing Right-Click -> Properties -> Open With -> Set as default on every file type separately.

Switching to VSCode from Sublime (or any other editor) is a surprisingly pleasant experience. VSCode will recommend the most relevant plugins as you go. For example, if you open a .py file, some extensions for Python development will be recommended. Installing them is then just a click away and doesn’t even require a restart of the application. After some days of following VSCode’s recommendations and occasionally searching for some extensions manually, my setup was more complete and efficient than it ever was with Sublime.

VSCode Terminal with Agnoster

If you don’t intend to use the integrated VSCode terminal, you can safely skip this section.

If you followed this guide until here and you now open an integrated Terminal in VSCode, you will likely be a bit disappointed. There are several problems. With our current setup, we need a monospace font, and the coloring does not work nicely with VSCode’s light theme.

Fortunately, we can fix these things:

First, install the Menlo for Powerline font:

git clone https://github.com/abertsch/Menlo-for-Powerline.git
sudo mv "Menlo for Powerline.ttf" /usr/share/fonts/
sudo fc-cache -vf /usr/share/fonts/ # Refresh fonts cache

Then, we change our terminal prompt colors a little bit to be compatible with both light and dark themes. Open ~/.oh-my-zsh/themes/agnoster.zsh-theme and change prompt_context to the following:

# Context: user@hostname (who am I and where am I)
prompt_context() {
  if [[ "$USER" != "$DEFAULT_USER" || -n "$SSH_CLIENT" ]]; then
    prompt_segment cyan default "%(!.%{%F{yellow}%}.)%n@%m"
  fi
}

Finally, we apply the new font and the new colors in the VSCode settings. Press Ctrl+Shift+P and start to type Preferences: Open Settings (JSON). We can now add the following lines to settings.json:

    "terminal.integrated.fontFamily": "Menlo for Powerline", 
    "workbench.colorCustomizations": {
        "terminal.ansiBlue": "#01A0E4",
        "terminal.ansiCyan": "#B5E4F4",
    },

The integrated VSCode terminal now looks nice and has all the functionality of all of our Zsh plugins.

Sync Configuration Files: Mackup

So you just set up your desktop workstation, modified a bunch of configuration files, manually adjusted your VSCode keyboard shortcuts and now you want to do this also with your notebook?

The solution to this is one of my favorite tools of all: Mackup
It will use any software that can sync a folder, such as Dropbox, to keep your application settings in sync.

You can install it with pip:

pip install --upgrade mackup

If you have versioning troubles with pip or Python on Ubuntu 20.04, I recommend to do sudo apt install python-is-python3. It will symlink /usr/bin/python to python3 and hence make your transition to a world without Python 2 easier.

Once Mackup is installed, you can copy your application settings to Dropbox (per default) doing

mackup backup

To apply them at a different machine, just use

mackup restore

Behind the scenes, Mackup copies your configuration files to your Dropbox and then replaces the originals with symbolic links. Have a look at the GitHub Readme for more details.

Advanced Screenshots: Shutter

If you are coming from Windows, you probably use the Snipping Tool on a regular basis. With Screenshot, Ubuntu has a similar tool installed by default. However, there is another option that is superior to both the Windows Snipping Tool and the Ubuntu default: Shutter
Unfortunately, it is no longer included in the default package repositories since Ubuntu 18.10. So, we have to add a third-party PPA first:

sudo add-apt-repository ppa:linuxuprising/shutter
sudo apt-get install shutter

It is very simple to use and does not force you to become familiar with its advanced features. There are a few alternatives that are still actively developed, such as Flameshot. Unfortunately, those are missing a lot of Shutter’s more advanced features. So, I will stick with Shutter for as long as I can make it run on Ubuntu.

Pick Colors: Pick

If you do front-end development, chances are you need a color picking tool on your computer. The most commonly used tool on Ubuntu is GPick. However, it drives me crazy. Usually, I just want to pick a color and get the hex code into my clipboard. GPick makes this very hard. So instead, I recommend Pick, a simple and functional alternative.

Cheatsheet: KeyCombiner Desktop

Most people know KeyCombiner as a tool for browsing and learning shortcuts. However, its desktop app serves an additioanl purpose with its instant lookup feature: It is a context-aware cheatsheet that can show the shortcuts of the currently active application in addition to whatever key combinations and text snippets I have in my personal KeyCombiner collections.

To trigger it, I only have to press Super+Alt+C. The main advantage is that I don’t have to leave the current context and can resume work right after looking up the required piece of information:

KeyCombiner’s instant shortcut lookup for VSCode, Terminal, and Chrome.

Communication Hub: Ferdi

Did you ever see someone using their phone while sitting in front of their personal computer? Chances are, they were using a communication app that they could have used much more efficiently from their larger device with an actual keyboard. To get the benefit of receiving all notifications from all my communication tools, and to be able to respond easily, I use Ferdi. It is particularly useful if you have to monitor several Slack teams.

In contrast to most of its competitors (Rambox, Wavebox, Franz 5), Ferdi is completely free and open-source.

If you want Ferdi or any other software that is distributed via an AppImage to show up properly in the dock, have a look at appimaged. It is open-source software that monitors specific directories for AppImage files and installs them properly in the system.

Previous versions of this article recommended Station instead of Ferdi. However, Station pivoted to become a browser extension and is no longer an option for bundling communication channels outside of your regular browser.

Record Short Screencasts: Peek

Did you wonder how I created those fancy gifs showing my terminal? I thought so.

Peek is a very lightweight and simple tool to record short screencasts. To install it on Ubuntu do

sudo add-apt-repository ppa:peek-developers/stable
sudo apt update
sudo apt install peek

Peek in action. (Source).

During the creation of this article, I was made aware by a good friend and fellow software engineer of dedicated terminal recording tools, such as Terminalizer and asciinema. However, these require some initial setup work, especially if you want to show off a theme and require that your animated terminal looks exactly like your actual terminal.
Since I already recorded most of the animations, I decided to stay with Peek for this post. Nevertheless, if your use case is terminal recording, you should probably go with one of the linked projects instead of Peek.

Mail Client: Thunderbird

For mail, I use Thunderbird. I don’t particularly like it, but to my knowledge it is still the best desktop email client for Linux. Unfortunately, some minor design decisions make the program almost unusable for me in its default state. The reason I include this section about software that everyone knows already is mainly to highlight three changes I always make to the default setup:

By default, Thunderbird sorts messages by date ascending. I think that’s an unfortunate default. Who wants to scroll down constantly to see the newest messages? To change the sort order to descending (newest first), go to Preferences -> General -> Config Editor and set mailnews.default_sort_order to value 2. See this StackExchange answer for more details about default sorting orders and other options.
Install the No Message Pane Sort by Mouse add-on. With this extension, a click on the mail list table header will no longer change the defined sort order. Without it, misclicks while trying to select the first mail in the list are very annoying. Personally, I never want to change the sort order at all. However, it is still possible by holding Ctrl while clicking.
Finally, go to Preferences -> Composition and uncheck the Use Paragraph format instead of Body Text by default to prevent Thunderbird from adding absurd amounts of whitespace whenever you press the Enter key while composing a mail.

Are you already using Thunderbird on another computer? Fortunately, there is an easy way to migrate all your mail accounts and settings. Essentially, you can copy a hidden folder where Thunderbird stores all its data and move it to the new machine, as described in Mozilla’s official documentation.

If you know of a new fancy alternative to Thunderbird that handles parallel usage of several mail accounts well, please let me know.

Bonus Tip

If you have read this far, I am willing to share my most secret Linux hack with you. Joking aside, it is very helpful to know what to do when your Linux workstation freezes. For some reason, I did not know until a couple years ago. Maybe the following trick can help you to avoid some forced restarts in the future.

The most flexible way to re-gain control of a system that has frozen completely is pressing Ctrl+Alt+F3. Then, use top and other process monitoring tools to find the process IDs of the faulty processes. Then, simply kill them (e.g. type kill <pid>, killall, or kill -9). Afterward, you can get back to your work using Ctrl+Alt+F1. (Source).

Conclusion

I hope this post showed you some tools or practices that you didn’t know yet. I have had a good experience with all of them. Especially switching from Bash to Zsh and installing the described plugins came with a real productivity boost. If you feel that I left out something important, I’d be very interested to hear your suggestions.

If you are not mainly using Linux yet, the length of this post might give the impression that it is still a pain to set up a proper Linux workstation. However, you do not necessarily need all of the things I showed here. Also, everything covered in this post is very easy to install with minimal configuration overhead. I strongly believe that you would have a much harder time setting up a comparable Windows development workstation.

You can still spend your life setting up Linux. However, if you use a popular, well-supported distribution you definitely don’t have to. (Source: xkcd.com)

Using Hugo, GitLab Pages, and Cloudflare to create and run this Website

Sat, 01 Jun 2019 00:00:00 +0000

In this post, I outline how I develop and run this blog site. I briefly explain my technology choices and give detailed instructions about how to use the resulting stack to create such a project. If you stick with me until the end, you will know how to run a beautiful, easily customizable static website for free.

When I decided to start this blog for the reasons outlined in my first post, I was already familiar with GitLab Pages, and the MkDocs static site generator. I am using this combination for creating documentation websites, and it is working very well.

Therefore, it was an easy choice for me to also rely on GitLab Pages for this site. I will cover its features in a later section of this post. With hosting already covered, I just needed to create the actual site.

Creating the site with Hugo

Before starting this blog, the only static site generator I used in production environments was MkDocs, which is intended for documentation and clearly not a good choice for creating a personal blog site.

A good overview of more appropriate alternatives is given on staticgen.com. For my purpose, probably all the popular projects would have been good choices. I decided to go with Hugo for two reasons:

It is written in Go, a language with a promising future that I wanted to try out for a while already.
There is a very well-curated list of open-source themes directly on the official website.

So let’s get to work.

To get started on Ubuntu, we can execute just two commands to install hugo and create a new project:

# install hugo
sudo snap install hugo

# create a new project
hugo new site personal-blog

If you are not using Ubuntu, your system’s package manager likely provides a similarly easy experience. Refer to the official documentation for installation instructions for other operating systems.

Before we start to work on our site, we want to install a theme:

# install the theme
cd personal-blog
git init
git submodule add https://github.com/luizdepra/hugo-coder.git themes/hugo-coder

I like that Hugo is very forward about using GIT submodules to install themes. This makes it easy to apply changes to the theme while keeping the connection to the original theme project. In an upcoming blog post, I will detail which changes I made to the theme itself. For now, we are perfectly fine with the default Hugo-Coder theme.

We only need to apply some basic configuration. Simply replace the content of config.toml with the following:

baseurl = "http://www.example.com"
title = "johndoe"

theme = "hugo-coder"

[params]
    author = "John Doe"
    description = "John Doe's personal website"
    keywords = "blog,developer,personal"
    info = "Full Stack Magician"
    avatarurl = "images/avatar.jpg"
    footercontent = "Enter a text here."

[taxonomies]
    series = "series"

# Menu links
[[menu.main]]
    name = "Blog"
    weight = 1
    url  = "/posts/"

Then, we create the first post:

hugo new posts/my-first-post.md

For the theme to properly recognize the new blog post, we have to replace the content of the newly created /content/posts/my-first-post.md with the following:

+++
date = "2019-01-01"
title = "My first Post"
slug = "my-first-post" 
tags = []
categories = []
series = []
+++

Finally, we can get a first look at our new personal portfolio and blogging site, by starting up the Hugo development server. Simply type

hugo server

Then go to http://localhost:1313/ to see your new site:

Hugo coder default page.

Testing your site locally is great, but showing the whole world that you are John Doe, the Full Stack Magician is something else entirely. Luckily, we are very close to doing just that.

Hosting with GitLab Pages

GitLab Pages is similar to GitHub Pages. Both allow free hosting of static websites. GitLab Pages enables you to build the hosted static content via GitLab’s CI/CD features. This means that your site will automatically be rebuilt on every commit to the repository. Until the release of GitHub Actions, GitHub could not do this. Instead, you had to build the site locally and commit the static files to the repository. Nowadays, both GitLab and GitHub are perfectly fine choices to host your static site.

Naturally, we start by creating a new repository. Before pushing anything to this repository, you might want to perform some usual GIT maintenance tasks, such as creating a .gitignore file. If you do not know it yet, you should have a look at the gitignore.io service service. The site generates .gitignore files depending on your technology stack. For Hugo, it will for example exclude the /public/ folder, which is very much what we want, because we will build this folder via a GitLab CI pipeline on every push to the repository.

Once everything is in order, we can simply follow GitLab’s instructions for pushing our existing project folder:

cd personal-blog
git remote add origin git@gitlab.com:<path-to-your-repository>
git add .
git commit -m "Initial commit"
git push -u origin master

Finally, we get to the exciting part: Having our site built and published on every new commit to the repository. If you are not yet familiar with GitLab’s CI/CD features, check out their docs.

Fortunately, getting started is very simple. Simply put a file called .gitlab-ci.yml in the root of your project. This YAML configuration file holds all CI/CD configuration. For our Hugo-powered blog, the following file content is sufficient:

image: monachus/hugo

pages:
    script:
    - hugo
    artifacts:
        paths:
        - public
    only:
    - master

The first line specifies the docker base image to use for our pipeline. For this brief tutorial, we will not use any testing stages, instead, we have just one simple job called pages. It executes the hugo command, which will build the static site content in the public directory. Somewhat incidentally, this is also where GitLab Pages expects to find our static site content. The only thing left to do is publishing the public directory as a job artifact. The last two lines are optional and state that the site should only be built and published on commits to the master branch.

Once you push this file to your repository’s master branch, the static site will be built and published within a few minutes. Go to Settings->Pages to verify that everything is in order and to see the default URL of your new website. If you own a domain you would like to use rather than the default *.gitlab.io one, you can head to the docs or check out this more detailed tutorial.

However, before you do this, you may want to read the next section of this blog post, as the information given there could spare you some headaches.

Speeding up load times and configuring URL redirections with Cloudflare

Initially, I configured SSL for this site via Let’s Encrypt. In principle, this is a perfectly fine way to do it. There is a good tutorial in the official GitLab Docs.

However, it is a little bit tedious, as it requires frequent renewals, a process that can not be automated easily. Furthermore, I had problems redirecting from www. to non-www URLs with Namecheap, which is my domain name registrar. The issue was complicated by the fact that this site uses a .dev domain which is by default on the HSTS preload list. Namecheap seems to prefer selling their own SSL certificates rather than fully support and document working with Let’s Encrypt.

I solved all those issues at once by simply signing up to Cloudflare. The free version is perfectly fine for me. After signing up, Cloudflare provides very simple and precise instructions for getting started. Basically, you have to configure your domain to use their nameservers instead of the ones provided by your original domain name registrar.

Once your site is configured to run with Cloudflare, you can use their Page Rules feature to easily redirect traffic from the www. subdomain to your canonical non-www URL or the other way around.

Furthermore, Cloudflare provides SSL encryption out of the box from their servers to your clients. All your static sites hosted with GitLab Pages and served via Cloudflare will be accessible over HTTPS without any additional configuration. Of course, we also want to encrypt the traffic from Cloudflare to the origin GitLab Pages host. For this, Cloudflare offers Origin Certificates. I will not explain this further, as there is an excellent tutorial on this provided directly by GitLab.

Edit: With GitLab 12.1, released on July 22, 2019, it is possible to automatically get HTTPS certificates for GitLab Pages using Let’s Encrypt. Using Cloudflare Origin Certificates is still a perfectly fine approach, however, it is no longer more convenient than just using GitLab’s Let’s Encrypt integration.

Conclusion

The biggest advantage of this setup is its financial cost. Apart from buying the tkainrad.dev domain, creating this site did not cost any money. Furthermore, the page is now running at absolutely zero cost. This is not restricted to a limited timeframe, thanks to GitLab Pages.

Fortunately, we do not have to resort to chemistry puns to reduce our hosting bills. (Source: xkcd.com)

If you have basic programming skills, I highly recommend hosting your static sites with this free service instead of paying a monthly fee to a hosting provider.

High-performance Computing with AWS Parallelcluster

Tue, 07 May 2019 00:00:00 +0000

Overview

AWS ParallelCluster is a toolkit for automating the process of building, configuring, and managing clusters of virtual machines on the Amazon Elastic Compute Cloud (EC2) cloud. These clusters can be used similar to traditional HPC clusters, as illustrated below.

Operating principle of a traditional resource-manager based computing cluster.

The software is essentially a command-line tool written in Python that provides simple commands for creating, updating, stopping, starting, and deleting HPC clusters in the AWS EC2 cloud. Naturally, to provide these functionalities, the locally installed CLI has to use the web APIs of existing AWS services. The most important one of these services is AWS Cloudformation. It facilitates the creation and management of a collection of related AWS resources. A set of nodes belonging to a specific HPC cluster can be such a collection. Another essential service is Auto Scaling, providing dynamic elastic scaling of the number of compute nodes depending on the current workload. As a consequence, ParallelCluster can be configured to automatically add nodes when jobs are pending and not able to run due to the cluster being fully utilized.

Cloud elasticity allows automated addition and deletion of nodes depending on the current job queue size.

ParallelCluster is maintained directly by Amazon Web Services (AWS) and developed as an open source project accessible on GitHub. While the restriction to AWS is certainly disadvantageous, the official support from and for a specific cloud provider results in a stable and regularly updated product. The latter is especially important for any framework facilitating cluster computing in the cloud, because the tool has to be able to deal with new versions of operating systems, updates to cluster computing software and changes in the product palette of the supported cloud providers.

Performance Compared to On-Site Clusters

HPC in the cloud faces complex challenges that have to be overcome in order to compete with on-site clusters. Specifically, these include the interconnection networks between machines in the cloud datacenters, which are most often not built for supporting HPC applications, and secondly, a performance decrease due to virtualization.

Already in 2009, Napper et al. have used the LINPACK benchmark to assess the performance of HPC clusters allocated in the Amazon EC2 cloud. The sobering conclusion has been that, at least for communication intensive problems, HPC in the cloud can not yet make use of its potential strengths regarding scalability and cost saving. To the contrary, it has been shown that the achieved floating point operations per second (FLOPS) per dollar spent decrease exponentially as the allocated cluster grows. This is, as expected, due to the slow interconnection network and limited memory available on single nodes. One year later, in 2010, Jackson et al. have performed similar experiments and also encountered the familiar problem of slow network connections. They observed a clear correlation between the runtime of different distributed applications and the amount of time spent communicating. However, Jackson et al. have also shown that for applications requiring little communication between computing nodes, HPC cloud clusters can scale well.

The situation has since progressed further. Real world problems are often embarrassingly parallel or come close to it, enabling all benefits of cloud computing, even with sub-par interconnection networks. This development is illustrated by various case studies of different scientific fields, especially the computational life sciences. Novartis used the Amazon EC2 cloud with 87 000 on-demand CPU cores to vastly decrease computation time for a VS experiment, a scientific technique explained in detail in Section 2.5. HGST built a similarly strong cloud cluster of 70 000 cores to run a simulation for finding an optimal hard drive head design.

Getting Started

Perhaps the most important benefit of using a cloud cluster is the possibility to set it up in a couple of minutes. Provisioning a physical HPC cluster requires competent system administrators, air conditioned space for the machines, a large up-front hardware investement, etc. etc.

Getting started with AWS Parallelcluster only requires an AWS account and the ParallelCluster software. Detailed instructions for creating an account can be found at the official AWS website. The installation procedure for ParallelCluster is also well described at the official docs.

Before you can configure ParallelCluster, it is need to create an AWS Access Key. To do this, click on your Account name in the top right of the AWS management console browser application. There is an option My Security Credentials. Additionally, you should create an AWS EC2 Key Pair via Services -> EC2 -> Key Pairs -> Create Key Pair.

Finally, you can set up your local ParallelCluster installation. Simply type

pcluster configure

in a terminal window. ParallelCluster will ask for the access key you just created. This automated configuration process will further be able to retrieve default values for master_subnet_id and vpc_id. These are otherwise a little tedious to find out. In general, you can just press enter on all questions except aws_access_key_id and aws_secret_access_key. More sophisticated configuration can easily be applied later by directly modifying the file ~/.parallelcluster/config.

The following is a basic configuration for an elastic cluster with one to five compute nodes:

[aws]
aws_region_name = eu-central-1
aws_access_key_id = <AWS_ACCESS_KEY_ID>
aws_secret_access_key = <AWS_SECRET_ACCESS_KEY>

[cluster default]
vpc_settings = public
key_name = <AWS_KEY_NAME>
ebs_settings = custom
post_install = <POST_INSTALL_SCRIPT_DOWNLOAD_PATH>
base_os = ubuntu1604
maintain_initial_size = true
compute_instance_type = c4.2xlarge
initial_queue_size = 1
max_queue_size = 5

[vpc public]
master_subnet_id = <AWS_MASTER_SUBNET_ID>
vpc_id = <AWS_VPC_ID>

[global]
update_check = true
sanity_check = true
cluster_template = default

[ebs custom]
ebs_snapshot_id = <SNAPSHOT_ID>
volume_type = gp2

Once your configuration is finished, you can create your first cloud cluster with a simple:

pcluster create <cluster-name>

Naturally, for working with AWS and ParallelCluster, you will want to also look at the official documentation. In the following I list some of the most important resources:

Parts of this blog post have been taken from my thesis on Providing Transparent Remote Access to HPC Resources for Graphical Desktop Applications. If you are further interested in working with AWS Parallelcluster you can freely download it here. For a brief description of an in-production AWS Parallelcluster use case, check out the journal article that has emerged from the thesis.

Nitrite as embedded NoSQL Database for Java

Mon, 06 May 2019 00:00:00 +0000

Problem Description

Recently, I had the exciting challenge to redesign the database representation of a potentially very large list of elements, each with an arbitrary number of possibly different attributes. Previously, the list was internally stored in an embedded SQL database. In order to deal with the fixed schema requirements of SQL, the following schema was conceptualized:

ER-Diagram for handling elements with different attributes in relational databases.

This approach is perfectly fine for a few thousand elements. However, when performance becomes critical because you would like to support very large databases, this approach reaches its limitations. This is not surprising because the relational database needs to perform JOIN operations for almost all use cases. Simply displaying an element with all its properties requires a JOIN of all three tables.

Nevertheless, it is a very common approach with relational databases. I cannot think of many ways to improve this schema without introducing other pitfalls. I would be very interested to hear about improved schema suggestions in the comments below.

However, there is definitely one way to improve this situation:
Getting rid of the relational database completely.

Having a list of elements that can each have an arbitrary number of properties which might differ from each other is a perfect use case for a document-oriented NoSQL database. The only challenge remaining was to find an embeddable open-source project, as those were strict requirements.

Nitrite Overview

When I started to look for NoSQL alternatives to tackle the described problem, I immediately thought of MongoDB. I have used it already in some small projects and feel quite comfortable with it. Unfortunately, there is no embedded version of MongoDB, except for testing and for mobile app development.

So my search went on. At first, I stumbled upon some very interesting key-value store projects:

Chronicle Map: https://github.com/OpenHFT/Chronicle-Map
MapDB: https://github.com/jankotek/mapdb/

I am happy to have these on my radar now, they are certainly very interesting and might find some use in future projects. However, for the above described use case, a key-value store is not a good fit. Instead, a document-oriented approach similar to MongoDB would be ideal. With this requirement, I actually had trouble finding open source projects. In the end, I had two relatively unknown alternatives:

Jsondb: https://github.com/Jsondb/jsondb-core
Nitrite Database: https://github.com/dizitart/nitrite-database

The decision in favor of Nitrite was quite obvious, it has beautiful and extensive documentation and boasts a “Very fast and lightweight MongoDB like API”.

Nitrite Database logo.

Implementation

Implementing our use case in a document-oriented NoSQL database is actually very straightforward. We simply create a NitriteCollection and add all our elements as Document to this collection. Each document will contain exactly the attributes that the respective element actually has set. This requires much less effort than with fixed-schema databases.

Nitrite db = Nitrite.builder()
    .filePath("databaseFile.db").openOrCreate("user", "password");

NitriteCollection collection = db.getCollection("elementCollection");

int i = 0;
for (Element element : elements)
{
  Document document = Document.createDocument("tableId", i);
  for (Map.Entry<String, String> entry : map.entrySet()))
  {
    document.put(entry.getKey(), entry.getValue());
  }
  collection.insert(document);
  i++
}
collection.createIndex("tableId", IndexOptions.indexOptions(IndexType.Unique, true));

The tableId attribute is specific to my use case. It allows to index my elements for later use with a JTable.

Using Nitrite as JTable Backend

I did not (yet) create a proper, reproducible evaluation setup. However, it is quite easy to get a feeling for how fast Nitrite compares to embedded SQL solutions (Derby, H2, …). Simply create a a basic Java Swing GUI containing a JTable with the following TableModel:

public class NitriteTableModel extends AbstractTableModel
{
  private NitriteCollection nitriteCollection;
  private List<String> attributeNames;
  private final List<Document> filteredDocuments = new ArrayList<Document>();


  public NitriteTableModel(NitriteCollection nitriteCollection,
      List<String> attributeNames)
  {
    super();
    this.nitriteCollection = nitriteCollection;
    this.attributeNames = attributeNames;
  }

  @Override
  public int getColumnCount()
  {
    return attributeNames.size();
  }

  @Override
  public int getRowCount()
  {
    return filteredDocuments.size();
  }

  @Override
  public Object getValueAt(int rowIndex, int columnIndex)
  {
    int tableId = (int) filteredDocuments.get(rowIndex).get("tableId");

    Document document = nitriteCollection.find(Filters.eq("tableId", tableId))
        .firstOrDefault();

    return document.get(attributeNames.get(columnIndex));
  }

  public void setFilteredIndices(Cursor find)
  {
    filteredDocuments.clear();
    Document projection = Document.createDocument("tableId", null);
    RecordIterable<Document> documents = find.project(projection);

    for (Document document : documents)
    {
      filteredDocuments.add(document);
    }
  }
}

This allows to easily filter collections and display the results.

Sorting:

nitriteTableModel.setFilteredIndices(nitriteCollection.find(Filters.eq(attributeNameField.getText(), attributeValueField.getText())));
nitriteTableModel.fireTableDataChanged();

Filtering:

nitriteTableModel.setFilteredIndices(nitriteCollection.find(FindOptions
                .sort(attributeNameField.getText(), SortOrder.Ascending)));
nitriteTableModel.fireTableDataChanged();

attributeNameField and attributeValueField are basic JTextField GUI elements that allow to specify which column should be used for filtering or sorting as well as which value the filtered elements have to match. Nitrite provides all the obvious filtering (eq, gt, gte, lt, lte, in, …) and sorting (ascending, descending) options out of the box.

Conclusion

I have used Nitrite as JTable backend with several million elements. The table performs amazingly smooth. Filtering and sorting on indexed attributes is almost instant as compared to several seconds or even minutes with the SQL-based approach described in the beginning.

If you happen to know of a more extensive performance evaluation of Nitrite, ideally comparing it against embedded SQL-based systems, please post it in the comments or contact me directly. If there is none, I might do one myself in a future blog post.

About Writing this Software Engineering Blog

Sun, 05 May 2019 00:00:00 +0000

Motivation

At the beginning of 2018, I finished my Master’s degree at TU Wien and started to work full time as a Software Engineer. As I was already working part-time before, I knew what I was getting into. Still, I am continuously amazed about how dynamic our industry and the people working in it are. Especially now, with the all-around success of (F)OSS, the software technology landscape is changing incredibly fast. This blog should help both my readers and me to keep up with the latest trends. I also see it as a way to contribute to the international software community.

Another reason for starting this blog is the inspiration from others. It seems like all the cool kids are doing it. A friend and former study colleague has already started his blog a couple months ago. Other great blogs I read are from software engineering heavyweights Troy Hunt, Scott Hanselman, and Jeff Atwood.

Finally, I hope to further develop my writing skills. In general, I think writing in all forms is an underestimated skill in software engineering, and probably many other fields as well. However, since last year, when I finished my Master’s thesis and a subsequent journal article, my writing efforts have been more or less limited to communication and documentation tasks. This blog is a chance for me to change this.

Planned Topics

In general, I will keep this blog strictly on topics related to software engineering. Further, I will try to cover technologies that are (not yet) widely used and discussed elsewhere. My first two technical posts about AWS ParallelCluster and the Nitrite embedded NoSQL database are good examples of this approach.

At some point, I probably can’t restrain myself from writing about productivity tools, as trying out new ones is kind of my guilty pleasure. Thanks to a tweet from Austrian developer Armin Ronacher I am currently starting to use Notion. It’s great, check it out!

A Request to my Readers

Obviously, I am aware that the number of readers of this blog will be very limited. Nevertheless, I hope to receive some feedback. Please feel free to contact me via email or Twitter. You are welcome to point out any mistakes, recommend technologies, propose topics, or come forward with any other request you might have.

My naive idea of blogging is to achieve reader engagement through worthwhile content. (Source: xkcd.com)

Posts on tkainrad

Why Keyboard Shortcuts don't work on non-US Layouts and how Devs could fix it

Introduction

Examples of Broken Apps

Why is this happening?

How you can do better

Simple Workarounds

Technical Fixes

Wall of Fame

Gmail

Discourse

Linear

Conclusion

An Interactive Virtual Keyboard to Visualize any Collection of Shortcuts

Introduction

Features

Overview

Grouping by Modifier Combination

Filter Collection Table

Real-time updates on changes

Additional Features

Use Cases

Quickly Grasp a Set of Shortcuts

See Conflicts and Free Combinations

Design a Coherent Set of Shortcuts

Current Limitations

Conclusion

An app to show the shortcuts of the current application for Windows, Linux, and macOS

Introduction

How does it look like?

What else can it do?

How does it know the active application’s shortcuts?

How can I use it?

Conclusion

Automatically add -tags with a Single Regex

Introduction

How-To

Modifiers and F-keys

Regular keys in combinations

Avoiding False Positives

Achieving Idempotence

Final regex

Conclusion

Learning all VSCode shortcuts evolved my developing habits

Preface

Habits are Hard to Break

Evolving my Habits

More Unexpected Benefits

Conclusions

Implementing Paddle Payments for my Django SaaS

Introduction

Prerequisites

Setting up your Plan in Paddle

Installing and Configuring dj-paddle

Setup

Implementing the Checkout

Testing the Checkout

Polishing the Checkout:

What to do directly after checkout?

Let a User Manage their Subscription

Handle Grace Period

Conclusion

How I learned 50 new keyboard shortcuts in 42 minutes

Introduction

Setup

Learning the shortcuts

Starting to remember the keys

Building muscle memory

Completing the challenge

Tips

Results & Conclusion

Why I built a new app for practicing keyboard shortcuts

Introduction

Existing Tools

Goals

How it works

Create shortcut collections

Learn shortcuts flashcard-style

Statistics to improve accuracy & speed

Roadmap

Automatically add `-tags with a Single Regex`