linux on tkainrad

Managing my personal knowledge base

Sun, 05 Jan 2020 00:00:00 +0000

Introduction

It is hard to imagine any other field where lifelong learning is more important than in software engineering. Another unique characteristic is the degree to which learning material is available for free on the internet. On top of that, we create various resources ourselves by documenting issues, submitting bug reports, writing notes, creating documentation, and many others. The sum of all these resources can be called a knowledge base. You could argue that every developer has a system to manage their personal knowledge base, whether they know it or not. In this post, I explain my knowledge management practices.

A software engineer’s personal knowledge base is likely to overlap with the knowledge of their employers and project partners. Be sure to carefully study your data protection obligations. A good basis is to keep your personal knowledge base strictly technical and to never include data that is related to customers, or people in general.

I chose this topic mainly because of three reasons:

Fun:
I enjoy thinking about my workflows and trying to improve them. Maybe a little too much, some friends have started to roll their eyes when I try to recommend a new tool to them.
Improving my system and workflows:
Writing a post about my Linux setup has been very rewarding for me. While documenting my configuration, especially my command-line workflows, I identified some shortcomings that I have since eliminated. After writing this article, I can already say that my knowledge management workflows have improved similarly.
Relevance:
Knowledge management in all its forms is integral to software engineering.

Knowledge management tasks are done frequently. It pays off to do them efficiently. (Source: xkcd.com)

Much of this post comes down to describing my usage of software tools. The stars will be

Copying all of my practices will likely not serve you well. I invite you to try out things at your own pace and revisit this post whenever you are looking for new ideas.

Bookmarking

Even though everybody loves books, or at least says so, myself included, the reality is that we rely on digital resources most of the time. The challenge is to organize this knowledge efficiently without creating too much overhead.

There are some naive approaches that I do not deem sufficient. Until a few years ago, I relied mainly on browser bookmarks, but for some reason, browsers do not provide proper organizing features. Another way is to just copy the URLs into notes, project issues, documentation and all other kinds of manually created content. This is viable and we all do it regularly, but it has nothing to do with what I think is bookmark management.

My current setup is a little more complex, but in the end, it does not require more work than keeping bookmarks solely in the browser. I organize my bookmarks in multiple layers, depending on how often I want to access them and how long they should persist.

Maybe you are thinking of caching now, but fortunately, bookmarking layers are not caching levels. This would be hard to manage because as we all know, there are two hard problems in computer science: cache invalidation, naming things, and off-by-1 errors. In contrast to cache entries, a bookmark is not supposed to move between layers. Instead, it gets inserted at the right place and stays there.

The following figure illustrates my layers:

Bookmarking layers

1. Chrome Bookmarks

Chrome bookmarks have some nice benefits. They are synchronized between devices, new ones can be added easily, and organizing them in folders brings at least some structure. Another advantage for me is the Albert Chromium extension that lets me access and open my Chrome bookmarks from Albert:

Searching through Chrome bookmarks directly from Albert. Pressing enter opens the selected entry in Chrome.

Because of these advantages, Chrome bookmarks are ideal for things that I need often and want to have accessible somewhat independent of context. They are, however, not well suited for keeping extensive bookmark libraries. There is no tagging, it is not possible to add comments, and browsers do not even save a timestamp for when a page was bookmarked. It is further not possible to filter and search Chrome bookmarks using complex queries. In my experience, browser bookmark libraries are hard to manage if they grow beyond a certain size.

Some examples of well-suited Chrome bookmarks:

AWS EC2 dashboard
Google cloud platform console
Slack workspaces
GitLab boards
Content sites, such as Hacker News, InfoQ, and DZone

Examples of things that should not be Chrome bookmarks:

Blog posts to be read later
Links to Open Source Projects
Stack Overflow questions

2. Workona

Workona is a relatively new addition to my toolbox. I have mixed feelings about it, mainly because it can feel a little slow at times. There are some things it does very well though.

Workona lets you create browser workspaces that come with their own list of bookmarks. This works nicely when you have multiple projects that you work on for longer periods. It will also remember which tabs you had open in a specific workspace and synchronize this information across devices. It is, therefore, a nice place for things that I need often dependent on context.

Screenshot of a Workona Workspace (Source)

Workona further allows you to add apps and will then collect links and even features for those apps to be quickly accessible. For example, I use it to access my different GitLab and GitHub projects, Slack Workspaces, and sometimes even StackOverflow questions. These entries are automatically created by Workona and therefore I do not see them as bookmarks. It is also possible to access features of these apps directly from Workona, mainly to create new resources, such as GitLab/GitHub projects, Google Docs, DropBox Files and so on. It feels a little bit like Station built directly into the browser.

The selling point for me is that Workona lets me do all of the described things via shortcuts and it is even possible to customize these shortcuts at chrome://extensions/shortcuts.

3. Notion

Finally, we arrive at the layer that is suited for maintaining a large library of bookmarks. For this, and many other use cases, I use Notion. It has become the core of my personal knowledge base.

Notion’s free tier comes with a hard limit on the number of blocks you can create. This means that you will have to switch to the paid version if you use it seriously for a while. It is free for students and teachers.
All links to Notion in this post are affiliate links. If you sign up for an account through these links, you will get 10$ of free Notion credit and I will get 5$. This works even if you create a free account.

The killer feature for me is the database. It is amazingly flexible and is single-handedly able to replace multiple other tools for me. For my bookmark collection, I use a single large database. Adding new items can be done via the Notion Web Clipper. I am a little annoyed that the web clipper doesn’t let me add properties and tags directly, but other than that, it works well. Previously, I used Trello, which was also quite good at keeping bookmarks. However, to limit the number of different tools I use, I replaced it with Notion.

My bookmarks database filtered to show only entries related to this post.

The goal with this database is that I can find something years from now with only a vague memory that it should be there. This is possible because Notion automatically stores metadata, such as a creation timestamp. Even more important, I can add tags and arbitrary properties.

Things get really exciting once you use relations between different Notion databases. As an example, I have a database of my blog posts that has a relation to my bookmarks. I can now filter my bookmarks by blog posts and quickly see which sources influenced a particular post.

I cannot cover all Notion database features in this post. I think this is something you have to see for yourself. The Notion docs page about the topic might give you some further impressions though.

The downside of this rich feature palette is that it requires some discipline. I tag and annotate new entries, that I added via the web clipper, about twice a month. Often, I just delete new items because they do not seem important anymore. If I can not think of appropriate tags or relations for an item, this often means that it is not very relevant to me.

Another important lesson I had to learn the hard way:
Do not use too many databases in parallel. Notion provides excellent methods for filtering, and searching tables. You can even define different views onto the same database that only shows a specific part of the data. It is therefore not required to separate all kinds of things into different databases. For example, at first, I had three different databases for Python resources, Django resources, and Wagtail resources. This was a bad solution. Now, these all live in the same database with appropriate tags.

I hope that someday I will be able to search through my Notion databases from Albert. Maybe I will build an extension myself once the Notion API is finally released.

To illustrate what I have described in probably too many words, you can have a look at a public fraction of my bookmarks database containing all the resources that were helpful for writing this post. The post-column in this public database appears empty because the related table is not public.

4. Native Bookmark Sources

The last layer does not require any work at all. It only means to be aware of native bookmark sources when trying to find something, and also when thinking about adding new bookmarks. For example, it is quite easy to search through questions you have answered on Stack Overflow. It is also not a problem to go back to your Hacker News posts or search through projects you starred on GitHub. Keeping those things in your own bookmark sources adds redundancy and noise.
These are some of my most commonly used native bookmark sources:

Summary

I am aware that this sounds immensely complicated. However, I stick to my statement that it does not actually require more work than keeping all bookmarks in the browser. You still have to add every bookmark only once. If you internalize your system, you will not have problems searching for bookmarks as it will be quite obvious where a specific resource could be. Admittedly, it does require some maintenance to keep a large bookmark collection in Notion, but it pays off soon and the pay-off accumulates over time.

Organizing Self-Written Resources

Having a proper system for external resources is great, but it does not help much if everything you write yourself is a mess. Therefore, it is essential to organize self-written resources.

Blog Posts

My most successful blog post is about how I run and host this website for free. However, the post does not cover the most work-intensive part about running this site: Writing and maintaining blog posts. For this, I heavily rely on Notion. I have a database with all my past blog posts and ideas for future posts. It is simple to add tags like done, doing, idea, and to view the database as a kanban board with lanes based on those tags.

Drafting Posts

All posts that are published on this blog have been drafted in Notion. When they are more or less finished, I use the markdown export and copy the content into a new text file in my Hugo project. There are some minor issues with this. Usually, I have to adjust code formattings and links, but overall it works very well and has some nice benefits:

Edit on all devices
Using notion, I can easily edit my drafts on all devices, even on my phone when I am moving.
Use existing notion content
I can quickly add code snippets, links, and other stuff that I keep in Notion. This can be done via database relations or just by dragging the respective blocks into the draft page.
Lightweight version control
As developers, we love GIT. However, for adding a couple of lines to a blog post, it is an overkill. Notion has versioning features build-in and keeps track of all changes automatically. Naturally, it does not offer sophisticated merging options, but it’s easy to stay clear of conflicts when you are writing a personal blog.

Once a post is released, I move the post from doing to released.

Keeping Track of Update Ideas

You may have noticed that some of my blog posts are quite long. Keeping them up-to-date is a little challenging. Fortunately, I have Notion to assist me with this. I have a database where I add proposals for updating my existing posts. Using a relation, I link these ideas to the respective posts in my posts database:

It is hard to know when you are finished setting up the perfect Linux workstation.

Tracking Sources

To keep track of third-party resources that were helpful in creating a post, I add them to my bookmarks database and link them to my posts database.

My bookmarks database filtered to show only entries related to this post.

You can also have a look at the full list of sources for this post via a public fraction of my bookmarks database.

Taking Notes

I suppose every software engineer takes notes to some extent, even if they do not have a system and just casually write stuff into text files. There is also a large amount of free and paid software for taking and organizing notes. Some people even like to use physical paper with the BuJo system.

No matter which method you prefer, you have to think about when, why, and how to take notes.

When to take Notes

I thought quite a bit about this question, and previously I did not adhere to any predefined rules regarding this matter. To gain a better understanding of my habits and to decide what worked best in the past, I looked through my notes from the last years that were spread over multiple applications and lots of files. I tried to find common patterns:

Capturing information from audio sources
I am a visual learner and prefer written content over audio. However, there are many types of audio sources that everybody encounters:
- Meetings
- Presentations
- Meetups
- Conferences
- Informal Discussions
- Videos
- Podcasts

Preserving information that is often needed but doesn’t fit in any proper documentation project
In my case, these are things like the following:
- Command-lines with multiple parameters, complex options, and paths. I keep them in my notes so that I can quickly copy-paste them in the potentially distant future.
- Instructions for tasks that will likely have to be repeated in the future, for example
  - setting up a development environment for a specific tech stack
  - deploying a project on a specific hosting service.
Questions
Sometimes I think of a question that I would like to ask someone specific, who is not available at the moment. The question and its eventual answer are a nice use case for a note.

Thanks to a more sophisticated command-line setup with Zsh and various plugins for auto-completion and history-search, preserving command-lines has become much less important for me. If you are still using Bash with its default Ctl - r search, I think you would profit from my post on setting up a Linux workstation.

On the other hand, it is important to make clear also what should not be a note in this system:

Project-specific knowledge
Everything that is directly related to a specific software project, should not be a note in your personal knowledge base. Instead, it should go into whichever system is used to keep track of issues, merge requests, documentation, and so on. This post includes a separate section on this below.
Information that is needed for a very short time Sometimes I need to take notes during a conversation that I will need only immediately afterward. For this, I just open VSCode and type ahead. Once the information is no longer needed, the file can be deleted completely.

In general, don’t overdo and don’t underdo it. If you take notes all the time, you create overhead and redundancy. If you never take notes, well, you will not have notes.

Why Take Notes

The obvious answer is to remember things. This does not mean that note-taking should replace your memory. To the contrary, taking notes provides structure and context and therefore helps your brain build up a map of your knowledge. I believe that taking notes increases the amount of data that you can recall from memory.

Additionally, some things are just very hard to remember, such as complex tech stacks presented at meetups, command-lines with several opaque options, numbers, and so on. Writing these things down can extend your knowledge base significantly. Notes are also very helpful when writing documentation in the future, or for writing blog posts like this one ;-)

How to Take Notes

It can be tempting to just type ahead during a talk. Noting down whatever seems interesting at the moment. However, you should meet basic formal requirements and write down some context. Otherwise, you will have trouble extracting useful information from your notes in the future. If you take notes regularly, it is also important to organize them by assigning tags and properties, having a creation date, being able to filter and search through them, and so on. Software can help with these things.

Until about a year ago I kept notes mainly on my local machines, syncing them with Dropbox. Applications that focus almost exclusively on note-taking, such as Evernote, never had that much appeal to me, even though I used Google Keep/Notes. It is not so bad, supports e.g. labeling and works with little friction on mobile. However, there is not even basic formatting support, let alone markdown formatting or code syntax highlighting.

Then, I discovered Notion. At first, It wasn’t really about note-taking for me. I liked it mainly because of its database concept and for organizing third-party resources as described in the bookmarking section. However, by now, I use it heavily for almost all of my notes and other types of self-written content. I think Notion’s mix of markdown syntax support, slash commands, and WYSIWYG makes for a great writing experience.

I strongly recommend organizing all your notes in a single database with appropriate tags, such as meeting, presentation, tc, and question. Similar to the bookmarks database described above, Notion will automatically provide a column with the creation time of the note. If you have frequent meetings with the same people, you can think about adding a Participants column.

A notes database is the perfect use case for another great Notion feature: Templates. This feature will allow you to created templates for your different types of notes and then, for example, add a new meeting note with one click. Depending on the template, the new note comes with a pre-defined layout and might include fields for the meeting’s participants and agenda. This is what opens when I click to add new meeting notes in my database:

My basic meeting notes template.

Project-specific Knowledge

In my experience, it is best to keep project information as closely together as possible. For me, this means that issue tracking, merge requests (i.e. pull requests), documentation, and everything else that comes with a software project should live alongside the source code.

This practice guarantees that anyone who works on the code now, or in the future, has access to all the information. My solution for this is to use GitLab for everything related to a specific project. This includes pretty much everything except some quick notes that I sometimes write in Notion because I do not want anyone else to see them.

To do this, GitLab offers a lot of project management and documentation features, such as issue management, a project wiki, a Code snippet space for all projects and accounts, and more. A project wiki is a great place for all documentation that is not specific to an issue. This can include the following:

Instructions for setting up the development environment
Instructions for manual deployment
Architecture descriptions and diagrams. I am a big fan of mermaid, which allows creating UML diagrams with Markdown and is supported by GitLab.
Guidelines regarding naming conventions, design paradigms, etc.

If you have read my post on running and hosting this website, you know that I use GitLab also for this personal blog project. Nevertheless, I use Notion for much of the project documentation and management tasks, because it comes with some unique benefits for this use case, as described above.

Of course, there are alternatives. GitHub has caught up in the past and offers a very similar feature set. I have only limited experience with Atlassian products, but I suppose you can achieve the same thing by using a mix of BitBucket, Confluence, and Jira.
On a side note, writing in Confluence feels a lot like writing in Notion. Both recognize markdown syntax and offer slash commands to quickly insert complex content types.

Managing Files

This one is relatively straightforward. I think the most important thing is to use some form of cloud storage. Personally, I use Dropbox.
Next, you should think about a proper folder structure. Finally, you should be able to search for your files quickly.

I use Linux and rely on Albert for searching through files. If this doesn’t work I use the locate command and, my last resort are command-line tools, such as find, and grep. They are still king for complex search requirements, such as regex matching and including also file content.

If you can tick the three boxes

availability on all devices
structure
searchable

you should be all set.

Notion databases support file columns. This means you can upload a different file to each cell. I have not yet found a use case related directly to software engineering for this. However, it is great for managing invoices.

Additional Use Cases

In this section, I want to present some special kinds of knowledge management use cases that are unique to software engineering. Most of them I have picked up properly only after starting to use Notion. By no means, I want to say that you should copy all of them, but you are very welcome to try out what you think looks interesting.

Code Snippets

Currently, I have two ways of collecting code snippets. I hope I can eventually integrate them somehow.

VSCode User Snippets

The first one are VSCode User Snippets. VSCode does many things right and snippet management is one of them. It is very easy to define new snippets and use them by typing a predefined string. Simply press Ctrl-Shift-p and choose Preferences: Configure User Snippets. Then, you can split your snippets into multiple files or just put all of them together in the global-snipptes.code-snippets file. These are some of my snippets for writing blog posts with markdown and Hugo:

"figure": {
    "scope": "markdown",
    "prefix": "fig",
    "body": [
        "{{< figure src=\"/images/\"  class=\"center-figure\"  caption=\"\" >}}"
    ],
    "description": "Insert hugo figure shortcode"
},
"callout": {
    "scope": "markdown",
    "prefix": "callout",
    "body": [
        "{{% callout \"warning info tip note\" %}}",
        "",
        "{{%/callout %}}"
    ],
    "description": "Insert hugo callout shortcode"
},
"code": {
    "scope": "markdown",
    "prefix": "code",
    "body": [
        "{{< highlight bash \"linenos=table\">}}",
        "",
        "{{< / highlight >}}"
    ],
    "description": "Insert hugo code shortcode"
}

The snippets can then be used by typing the prefix and pressing Ctrl+Space:

Notion Snippet database

The second use case concerns small pieces of code that I have written myself and that I would like to remember. This is not necessarily for productivity reasons, but rather for nostalgia. It is ideal for small algorithms, for example when you manage to substantially increase the performance of something by applying some clever technique.

I keep those snippets in a Notion database, which works great because I can tag them and add arbitrary properties. A nice benefit of this habit is that it’s not a problem if you don’t do it for a while. Your database will not look messy all of a sudden.

An exemplary entry of my code snippets database in Notion.

Shortcut Database

I believe that typing proficiency is important for a software engineer. It is easier to stay in flow if you can type fast and navigate around your IDE and tools with shortcuts. As my set of tools grew, I realized that I have a hard time to keep track of which shortcuts I am using, which of them work in which application, and whether I can change them so that they work across as many tools as possible.

Therefore, I started to keep track of some shortcuts I would like to use everywhere. These are things like creating new tabs, creating new windows, opening the respective tool’s search feature and more.

This database helps me identify overlaps and motivates me to think about which task could fit for a specific key-combination that I use often. For example, I use Ctrl - Shift - T in IDEs to search for type definitions and Ctrl - Shift - R to search for resources in general. Since I have these key combinations memorized for years and can use it very quickly, I want to make use of it wherever possible. Therefore, I try to set up my applications so that some kind of search opens whenever I use one of these combinations. My Notion database makes it easy to see which applications offer search features and which of them I can modify.

For example, my Browser now forwards Ctrl - Shift - T / R to Workona and allows me to switch/search workspaces or use the general search, respectively.

Screenshot of my Notion shortcut database

Notion’s filtering and search capabilities are great for identifying problems with your setup. For me, a problem is when two applications use different shortcuts for a very similar operation. Unfortunately, Notion is negative example of an application that doesn’t let you customize its shortcuts and therefore reduces the usefulness of my database. If someone from the Notion team reads this at some point, I kindly ask you to lobby on my behalf to prioritize personalized keyboard shortcuts.

I am excited about this use case and will talk about it in much more detail in a future blog post.

As of June 3rd, 2020, there is still no dedicated blog post for this. However, there is something more valuable:
This post gave me the idea for https://keycombiner.com.
It is a web application to track the keyboard shortcuts you use, get better at using them, and to learn new ones. It is in open beta and I would love to hear your feedback!

Tools Database

I also have a Notion database to keep track of small web tools that I use. This serves little purpose for things that are used frequently as you will remember them anyway. However, for minor things that are only needed once in a while, such as color palette generators, detecting handwriting as LaTeX symbols, and checking SSL certificates this is quite handy. Usually, you can find these tools via google, but sometimes there are many tools for a task and it is nice to keep track of the ones that work best.

Screenshot of my software tools database in Notion.

Again, the important thing here is to not overdo it. There are extensive publicly available lists of such tools. Your list should not be a copy of those but rather a small collection of the things you like and need from time to time.

Collecting Ideas for new Projects

Like many software engineers in these golden times, I sometimes think about starting new software projects, maybe even bootstrapping a small business.
You probably guessed it already, I keep ideas for possible projects in a Notion database. The main benefit is, once again, that it allows me to easily link it with other databases, such as my bookmarks.

Conclusion

Similar posts often conclude with warnings that you should be very careful not to spend too much time on organizing and maintaining your knowledge and workflows. In principle, I agree with this sentiment. After all, you want to increase your productivity.

I do believe, however, that it is worth it to spend some time thinking about your knowledge management system. The most important part about conceptualizing a system is to decide exactly which types of information you want to maintain in your knowledge base. If you get this right you will benefit for the rest of your career. Even if the tools might change in the future, the system will stay.

Please don’t hesitate to share your own experiences in the comments below. I am sure that my workflows are not perfect, but they might get closer with your tips.

This post is released together with a new post in my Reading List series, which lists related resources. If you want to dive even deeper into knowledge management concepts, you might want to have a look.

Copy-paste ready commands to set up SGE, PBS/TORQUE, or SLURM clusters

Fri, 18 Oct 2019 00:00:00 +0000

Introduction

Local HPC clusters continue to play a vital role in scientific research. Many universities, research institutions, and companies continue to maintain on-site clusters despite the efforts of the big cloud providers to expand their offerings into this area.

The software systems responsible for making these clusters of computers work together can be called Distributed Research Management System. The most commonly used ones are SGE, PBS/TORQUE, and SLURM. Unfortunately, setting those up can be very painful. There is a lot of outdated information on the internet. It is required to fight with several different configuration files and minor issues can mean that no command will run properly.

Operating principle of a traditional resource-manager based computing cluster.

If your task is to set up a large system that will be used for many years and you have got several days to do this perfectly, this might not be a problem. However, if you just want to play around with the different systems, spending several hours until you can submit your first job is a very bleak outlook.

Therefore, I present copy-paste-ready instructions for setting up SGE, PBS/TORQUE, or SLURM on a single machine, which will act as the master and a compute node at the same time. All guides are tested on Ubuntu 18.04, but should work with little modification on most recent Linux installations. These commands can also serve to create container images or other automated setup workflows.

Naturally, caution is warranted when executing commands as a super user. If you apply the listed commands on a production system, please make sure you understand everything that is done.

Slurm

Trying to set up SLURM on my development machine in order to run some test suites that need to interface with SLURM cost me quite a bit of time. There are plenty of extensive tutorials available. In contrast to SGE and PBS, there is even an up-to-date official documentation. However, those resources cover much more than is needed for a single-node cluster and it is easy to get lost in the many configuration options that you will not need at first.

Fortunately, I found this thread in the Ubuntu forums, which is the basis for my SLURM instructions.

Start by installing munge and slurm:

sudo apt install munge slurm-wlm

Then, create a new file at /etc/slurm-llnl/slurm.conf with the following content:

ControlMachine=<YOUR-HOST-NAME>

MpiDefault=none

ProctrackType=proctrack/pgid
ReturnToService=1
SlurmctldPidFile=/var/run/slurm-llnl/slurmctld.pid
SlurmdPidFile=/var/run/slurm-llnl/slurmd.pid
SlurmdSpoolDir=/var/lib/slurm-llnl/slurmd
SlurmUser=slurm
StateSaveLocation=/var/lib/slurm-llnl/slurmctld
SwitchType=switch/none
TaskPlugin=task/none
#
# SCHEDULING
FastSchedule=1
SchedulerType=sched/builtin
SelectType=select/linear
#
# LOGGING AND ACCOUNTING
AccountingStorageType=accounting_storage/none
ClusterName=<YOUR-HOST-NAME>
JobAcctGatherType=jobacct_gather/none
SlurmctldLogFile=/var/log/slurm-llnl/slurmctld.log
SlurmdLogFile=/var/log/slurm-llnl/slurmd.log
#
# COMPUTE NODES
NodeName=<YOUR-HOST-NAME> CPUs=4 Sockets=1 CoresPerSocket=4 ThreadsPerCore=2 State=UNKNOWN
PartitionName=long Nodes=<YOUR-HOST-NAME> Default=YES MaxTime=INFINITE State=UP

You only need to replace <YOUR-HOST-NAME> with your actual hostname. Simply execute the hostname command if you are not sure what it is. Of course, you can also modify other settings, such as the number of CPUs, but the basic configuration from above should be enough for our 1-node cluster.

We are almost finished, just enable and start the manager slurmctld:

sudo systemctl enable slurmctld
sudo systemctl start slurmctld

Finally, enable and start the agent slurmd:

sudo systemctl enable slurmd
sudo systemctl start slurmd

Congratulations, your Slurm system should be up an running! Use sinfo to check the status of the manager and the agent. The command scontrol show node will give you information about your node setup.

If Slurm did not start, fear not, there is likely only a small fix required. Start by looking for an error message with

systemctl status slurmd.service

Does it say that the slurmd.pid file coud not be opened? If so, your slurm.conf file probably has different values for SlurmctldPidFile and SlurmdPidFile than your slurmctld.service file. Check by printing the latter:

less /usr/lib/systemd/system/slurmctld.service

It might show PIDFile=/run/slurmctld.pid, which means that we need to change SlurmctldPidFile and SlurmdPidFile in our /etc/slurm-llnl/slurm.conf to the following:

SlurmctldPidFile=/run/slurmctld.pid
SlurmdPidFile=/run/slurmd.pid

After that, try again to start slurmctld and slurmd with the above commands.

SGE

This one is not so simple.

Some parts of the following setup are taken from a public Dockerfile created by Robert Syme.

First, gain root permissions. On Ubuntu you can type sudo -i. All commands have to be executed with root permissions.

Then, create a new folder, let’s say via mkdir /opt/sge/installfolder. Set the new folder’s location as an environment variable in the current shell via

export INSTALLFOLDER=/opt/sge/installfolder.

Then, execute the following commands:

cd $INSTALLFOLDER
wget https://arc.liv.ac.uk/downloads/SGE/releases/8.1.9/sge-common_8.1.9_all.deb .
wget https://arc.liv.ac.uk/downloads/SGE/releases/8.1.9/sge-doc_8.1.9_all.deb .
wget https://arc.liv.ac.uk/downloads/SGE/releases/8.1.9/sge_8.1.9_amd64.deb .
dpkg -i ./*.deb

Then, download the following 4 files and place them also into the created folder, in our case into /opt/sge/installfolder:

Most of the magic will happen through those scripts and configuration files. After the download, we need to set some environment variables in the current shell:

export SGE_ROOT=/opt/sge
export SGE_CELL=default

We also need to set a new profile.d config via

ln -s $SGE_ROOT/$SGE_CELL/common/settings.sh /etc/profile.d/sge_settings.sh

Then, execute the following to install SGE and perform setup operations:

cd $SGE_ROOT && ./inst_sge -m -x -s -auto $INSTALLFOLDER/sge_auto_install.conf \\
&& sleep 10 \\
&& /etc/init.d/sgemaster.sge-cluster restart \\
&& /etc/init.d/sgeexecd.sge-cluster restart \\
&& sed -i "s/HOSTNAME/`hostname`/" $INSTALLFOLDER/sge_exec_host.conf \\
&& sed -i "s/HOSTNAME/`hostname`/" $INSTALLFOLDER/sge_hostgrp.conf \\
&& /opt/sge/bin/lx-amd64/qconf -Me $INSTALLFOLDER/sge_exec_host.conf

Now, our new cluster is already up and running. However, we still need to add users to the sgeusers group, which was defined in the sge_hostgrp.conf file you just applied. Only users from this group are allowed to submit jobs. Therefore, we run the following:

/opt/sge/bin/lx-amd64/qconf -au <USER> sgeusers

Finally, run $INSTALLFOLDER/sge_init.sh.

You can now delete the complete $INSTALLFOLDER, or at least run

rm $INSTALLFOLDER/*.deb

After a restart to load the new profile.d settings, all users that were added to sgeusers should be able to submit jobs. Test this via:

echo "echo Running test from $HOSTNAME" | qsub

PBS / Torque

This is a little simpler again. Most of the setup is taken from another blog. Some inspiration is also from a public Dockerfile available at Docker Hub.

First, gain root permissions. On Ubuntu you can type sudo -i. All commands have to be executed with root permissions.

We start by installing the relevant packages:

apt-get install torque-server torque-client torque-mom torque-pam

Installing these packages will create a default setup. Unfortunately, this is complex and would require complex changes to get to a working cluster. Instead, we stop all torque services and create a clean setup:

/etc/init.d/torque-mom stop
/etc/init.d/torque-scheduler stop
/etc/init.d/torque-server stop
pbs_server -t create
killall pbs_server

We start by setting localhost as the server host and allowing root to change the database configuration:

echo localhost > /etc/torque/server_name
echo localhost > /var/spool/torque/server_priv/acl_svr/acl_hosts
echo root@localhost > /var/spool/torque/server_priv/acl_svr/operators
echo root@localhost > /var/spool/torque/server_priv/acl_svr/managers

With the following commands, we set localhost also as compute node with 4 cores available:

echo "SERVER.DOMAIN np=4" > /var/spool/torque/server_priv/nodes
echo localhost > /var/spool/torque/mom_priv/config

Now, we can already start the daemon processes again:

/etc/init.d/torque-server start
/etc/init.d/torque-scheduler start
/etc/init.d/torque-mom start

After this qmgr is ready to start the scheduler:

qmgr -c 'set server scheduling = true'
qmgr -c 'set server keep_completed = 300'
qmgr -c 'set server mom_job_sync = true'

The following commands are to create a default queue and to configure it:

qmgr -c 'create queue batch'
qmgr -c 'set queue batch queue_type = execution'
qmgr -c 'set queue batch started = true'
qmgr -c 'set queue batch enabled = true'
qmgr -c 'set queue batch resources_default.walltime = 3:00:00'
qmgr -c 'set queue batch resources_default.nodes = 1'
qmgr -c 'set server default_queue = batch'

Finally, we allow our machine to submit to the new cluster:

qmgr -c 'set server submit_hosts = <hostname>'
qmgr -c 'set server allow_node_submit = true'

To test the setup, you can use the following command:

echo "echo Running test from $HOSTNAME" | qsub

Conclusion

The target audience for this post is probably very small. I decided to do it anyway, as there might be someone that can benefit immensely. Personally, I would have saved a lot of time if I found such a blog post a year ago. Also, I might need it again myself at some point.

Setting up a Linux Workstation for Software Development

Tue, 13 Aug 2019 00:00:00 +0000

Introduction

In my opinion, Linux is the best operating system for a software engineer, except they develop explicitly for Windows or macOS. Whenever I read about new features for the Windows Subsystem for Linux, I wonder why people settle for anything less than the real thing. Linux aids many of my workflows with its customizability, superior command-line experience, native support for containerization platforms, a dependable package management system, and many other features that I will describe in this blog post. macOS comes close, but, in my opinion, falls short due to exorbitant hardware costs and less room for customization.

Still, as with any other operating system, optimizing a Linux workstation for software development requires some configuration. In this post, I will present tools and concepts that I think are very helpful, but overlooked by many. I will try to show only things that are not specific to my workflows, but rather useful for almost any software engineer, DevOps specialist, or just regular productivity enthusiast. Naturally, many of the configuration options and tools covered in this post are subjective. Nevertheless, I am confident that my suggestions are a good place to start. They have served me well over the years and I continually try to optimize my setup. Hence, I very much appreciate any improvement you might suggest.

Prerequisites

A Linux machine
For this guide, I will use Ubuntu 20.04, which is the current LTS release. I am very happy with Ubuntu. Anything I don’t like can easily be changed. The biggest advantage over many other distributions is probably that the Ubuntu community is very large and it is usually easy to find people with similar problems. There is also a very active StackExchange community. However, most of the things covered in this post will apply to pretty much any Linux distribution and even macOS.

Some command-line experience
Much of this post is intended for people who feel right at home when they see a ~- character. If you are not comfortable working with command-lines and have no intention to change this, you can still read the other sections though.

Tweaking Ubuntu

Before we start any serious work, I will point out some changes to Ubuntu itself that I apply on all my machines.

Appearance

One of the first things I do after installing Ubuntu is to install the GNOME Tweak Tool:

sudo apt-get install gnome-tweak-tool

It enables complex modifications of the GNOME desktop environment. I recommend you play around with some of its options. For this guide, we will only consider the Appearance section. Specifically, we want to change the default Ubuntu theme to another one that provides better contrasts and an overall cleaner experience, namely the Adapta-gtk-theme, which is based on the Material Design guidelines.

sudo apt-get install adapta-gtk-theme

After installing the theme, you should restart the GNOME Tweak Tool, and you can then select Adapta-Eta as your new theme. The theme comes with some nice benefits that you wouldn’t think of at first. For example, it fixes the ridiculously small toolbar icons in the Eclipse IDE.

Adapta Theme elements (Source).

Unfortunately, the Adapta theme is no longer actively maintained. It is still a perfectly fine theme to use and if you notice any glitches, you can always switch themes with a couple clicks. A possible alternative is Materia, which also works well with the remainder of this blog post.

Nautilus Type-Ahead Search

The next thing I suggest is to change the Nautilus search behavior. With recent Ubuntu distributions, there is no type-ahead search in the standard file explorer. This means you can not start to type some letters and jump between files that start with these letters. Instead, Nautilus will enter a dedicated search mode that is confusing and quite slow for folders with a large number of files.

Fortunately, we can easily get the old behavior back:

sudo add-apt-repository ppa:lubomir-brindza/nautilus-typeahead
sudo apt dist-upgrade
killall nautilus # to force a restart

Searching through my /bin folder by typing ahead (Source).

There is a StackExchange answer that provides further details.

Hibernation

Another thing that is essential for me is hibernation. Usually, I will only shutdown my computer once a week or even less. However, I am not a big fan of suspending for long periods of time. Power outages could have unforeseen consequences and I simply feel a little better if my machines are off the grid altogether.

Unfortunately, setting up hibernation is tricky and can depend on your hardware. I updated this section multiple times already to contain the most robust solution, but you never quite know. If you don’t care about hibernation, you should probably skip this section and spare yourself the trouble.

In principle, hibernation is possible out of the box. Just type sudo systemctl hibernate. However, this is likely to fail with an error. My fix for this relies on a mixture of this and that blog post. First, we need to find out the current size of our swap file:

The below instructions assume that you use Ubuntu 20.04 with a swap file. This is the default for new installations. However, if you upgraded from a previous version, you might still have a swap partition. In this case, the process is similar, but not covered here.

sudo swapon --show # or `cat /proc/swaps`

If this is less than your main memory (RAM), you will need to increase it:

sudo swapoff -a
sudo fallocate -l <SIZE>G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
sudo swapon --show # to verify changed size

Next, we need to find out the UUID and offset of our now large enough swapfile:

sudo swap-offset /swapfile
> <OFFSET>
sudo findmnt -no SOURCE,UUID -T /swapfile 
> <UUID>

With this, we are almost done. Using the UUID and the file offset, we edit the following line in /etc/default/grub:

`GRUB_CMDLINE_LINUX_DEFAULT="quiet splash resume=UUID=<UUID> resume_offset=<OFFSET>"`

Then, we create a conf.d resume file. Simply do sudo nano /etc/initramfs-tools/conf.d/resume and add the content RESUME=UUID=<UUID> to the file. At last, we need to update grup and initramfs:

sudo update-grub
sudo update-initramfs -u -k all

And finally, you should be able to do sudo systemctl hibernate.

Please let me know if this approach doesn’t work on your system. I would like to include potential problems in this post, if there are any.

Boosting Command-Line Productivity

I believe that command-line proficiency can significantly boost developer productivity. This is especially true in times of DevOps and NoOps, where developers are supposed to handle deployment pipelines and cloud infrastructure. Fortunately, there are some tools that can help you become a command-line wizard.

Shortcuts

Before going forward, I would suggest to set up some general terminal shortcuts. Simply go to Edit -> Preferences -> Shortcuts. I like to replicate web browser shortcuts for opening new tabs and windows, closing tabs, and cycling between tabs.

Did you know that all common terminal emulators have an Undo shortcuts with Ctrl+Shift+-? Among many other things, my side-project KeyCombiner has a public collection of terminal shortcuts, many of which are barely known and still very useful.

However, using this interface, it is not possible to set up Ctrl+Tab and Ctrl+Shift+Tab as tab switching shortcuts. Gnome has some issues with using the Tab key in shortcut configurations. Thanks to this answer from the Ubuntu community, we can get around this. Simply execute the following in your terminal:

gsettings set org.gnome.Terminal.Legacy.Keybindings:/org/gnome/terminal/legacy/keybindings/ next-tab '<Primary>Tab'
gsettings set org.gnome.Terminal.Legacy.Keybindings:/org/gnome/terminal/legacy/keybindings/ prev-tab '<Primary><Shift>Tab'

Switching from Bash to Zsh

A couple of months ago, I made an important change in my life. I switched from Bash to Zsh.

There are a lot of blog posts, threads, and articles that compare Bash and Zsh, e.g.:

At the end it boils down to one thing: Plugin and theme support

Zsh has an amazing themes and plugins ecosystem. Theme support allows you to easily get a beautiful shell. Plugins range from simple auto-completion for all kinds of applications and tools(git, mvn, hugo, …) to complex stuff like improved search capabilities. Because of this fact alone, Zsh is superior to Bash.

However, chances are that you grew fond of Bash and that you get a bad feeling when somebody tells you that there is something better. I understand, it was the same for me. Still, I urge you to give it a shot. The transition is very painless since pretty much everything you like about Bash still works with Zsh. You will not have to learn new commands.

Installing Zsh

Installing Zsh on Ubuntu is very easy. We can use the regular package manager:

sudo apt install zsh

Then, we set Zsh as our new default shell:

chsh -s $(which zsh)

The chsh command updates the /etc/passwd file, but does not affect the current session. You need to log out and log in again to have zsh as your default shell.

Configuring Zsh History

For me there are few things worse than searching for an old shell command only to find out that the shell’s history size was set to a ridiculously low default number and that, because of this, the command was not saved.

Therefore, we extend the Zsh history size by putting this at the end of the ~/.zshrc configuration file:

HISTSIZE=10000000
SAVEHIST=10000000

For me, the command history is a tool to search for previous commands I might need now. I do not use it as a protocol for what I did when and in which exact order. Therefore, I use the following options to get rid of all duplicates in my history automatically:

## History command configuration
    
setopt HIST_IGNORE_ALL_DUPS # ignore duplicated commands history list

Also, I like to use multiple terminal sessions in parallel, potentially working on the same problem. For this, it helps a lot to share the same history among all sessions. It can be accomplished by

setopt SHARE_HISTORY # share command history data

If you are transitioning from Bash to Zsh, you probably want to keep your existing Bash history. In contrast to Bash, Zsh stores timestamps in its history file. Therefore, re-using the Bash history file does not work out of the box. Fortunately, all we need to do is to append some timestamp to all the commands from our previous Bash history and put them into the new Zsh history file. You could easily script this yourself or use one of the many good scripts that other people wrote, such as this one.

Zsh Features that are not in Bash

Once we have Zsh installed and finished our basic history configuration, we can start to take a look at some out-of-the-box features that make it stand out in comparison with Bash. If these do not impress you, keep in mind that the real big thing is plugin support, which I cover below.

Recursive Path Expansion

Zsh can do much more sophisticated expansions than Bash. This is documented in detail in the Zsh manual.

One feature that you can easily start to use is recursive path expansion. If you type cd /usr/local/b in Bash, you can then press Tab to expand this to cd /usr/local/bin. With Zsh, you can also type cd /u/lo/b and then press Tab. Zsh will recursively expand all path elements:

The above animation shows also that you can omit cd when changing directories.

Calculator: zcalc

Before switching to Zsh, I was often using the Python Shell as a calculator. However, this has a few downsides. Obviously, the Python Shell is not primarily developed to be a good calculator app. For example, simple operations, such as sqrt, require import statements, and there are no convenient ways to access results of previous lines.

The calculator script shipped with Zsh is a very simple and efficient solution. Before using it, it has to be loaded:

autoload -Uz zcalc

If you plan to use it often, you can add the autoload command to your ~/.zshrc.

Then, start the calculator with zcalc and do some calculations:

Results of previous operations can be accessed with the $-character.

Advanced Aliasing

Zsh has some unique aliasing capabilities.

Suffix Aliases
This means you can assign applications to file suffixes, e.g. alias -s txt=vscode will allow you to open a .txt file with Visual Studio Code by just typing file.txt.

Global Aliases
Global aliases are set with the -g flag and allow to specify a replacement for any part of a command. For example, if you set alias -g G=' | grep' you will be able to do the following:

To make aliases permanent, it is required to add them to your ~/.zshrc.

Installing ZSH plugins

Finally, it is time to explore the real strength of Zsh, which means installing plugins to boost our productivity above anything this anime character has ever seen before:

Package Manager: Oh my Zsh

A good place to start is to install Oh my Zsh, “a community-driven framework for managing your Zsh configuration.” Simply do:

wget https://github.com/robbyrussell/oh-my-zsh/raw/master/tools/install.sh -O - | zsh

Copy the template configuration file .zshrc.zsh-template in the .zshrc home directory and apply the configuration by executing the source command. Both can be done at once by executing the following line:

cp ~/.oh-my-zsh/templates/zshrc.zsh-template ~/.zshrc

Oh my Zsh comes with many themes and plugins that we only have to enable in the configuration file in order to use them. In the next section, we will start by selecting a pretty theme.

Theme: Agnoster

Personally, I like the agnoster theme. It works well with the Adapta-Eta Gnome theme we have set previously.

To enable a theme, set ZSH_THEME to the name of the theme in your ~/.zshrc, in my case this is ZSH_THEME=agnoster. If you have troubles with special characters not being shown properly, install the Powerline font package:

sudo apt-get install fonts-powerline

The Oh my Zsh GitHub wiki provides a well curated list of available themes along with screenshots.

All pictures and animations of terminals you see in this post show the Agnoster theme with slight modifications to the prompt. In particular, I removed the git-part of the prompt and added a newline part instead. If you would like to replicate this, edit your ~/.oh-my-zsh/themes/agnoster.zsh-theme and replace the ## Main prompt part close to the end with the following:

prompt_newline() {
  if [[ -n $CURRENT_BG ]]; then
    echo -n " %{%k%F{$CURRENT_BG}%}$SEGMENT_SEPARATOR
%{%k%F{$CURRENT_BG}%}$SEGMENT_SEPARATOR"
  else
    echo -n " %{%k%}"
  fi

  echo -n " %{%f%}"
  CURRENT_BG=''
}

## Main prompt
build_prompt() {
  RETVAL=$?
  prompt_status
  prompt_virtualenv
  prompt_aws
  prompt_context
  prompt_dir
  prompt_bzr
  prompt_hg
  prompt_newline
  prompt_end
}

Additionally, I recommend changing the font color of the directory part of the prompt from black to white. In my opinion, this provides better contrast with the dark blue background:

# Dir: current working directory
prompt_dir() {
  prompt_segment blue white '%~'
}

Basic Plugins

Next, you should enable some included plugins via adding them to your plugins setting in your ~/.zshrc file. After this step, it might look like this:

plugins=(colored-man-pages git python django mvn aws docker vscode)

You can browse all the plugins that come with Oh my Zsh in the official GitHub repository.

The above listed plugins are rather trivial, they provide command completion and define a set of aliases and shortcuts that come in handy when working with the respective tool. Nevertheless, this provides a real productivity boost. For example, finding out which git commands start with r becomes much simpler:

Jump to Frequently Used Directories

Most command-line work happens in just a handful of directories. For me, these are primarily the root folders of various software development projects. Of course, I could manually set aliases for these folders so that I can quickly switch between them, but this would be tedious.

Fortunately, there is a better solution with z. It tracks your most used directories and will take you to them via z <query>. The query can be any part of target directory’s path:

The software is already bundled with Oh my Zsh, simply add z to the list of plugins:

plugins=(z colored-man-pages git python django mvn aws docker vscode)

I could have probably included this one in the list of basic plugins above, but I wanted to highlight it because I find it particularly useful.

Supercharging History Search

The original Bash history search always bothered me. Repeatedly pressing Ctrl+r and hoping to get lucky at some point is very cumbersome if you have lots of similar commands containing the query. To be fair, there are also great additions for Bash that improve this situation, such as McFly.

However, this is about improving history search with Zsh. I believe the best solution is using peco-history. It is a Zsh plugin that relies on peco to interactively filter through your history file. Peco is a general-purpose interactive filtering tool that you can (and should!) also use as a stand-alone tool in your terminal. On Ubuntu, we can install it from the official repositories:

sudo apt-get install peco

Afterwards clone zsh-peco-history into the $ZSH_CUSTOM/plugins directory. Zsh will automatically resolve the pre-set shell variable:

git clone https://github.com/jimeh/zsh-peco-history.git $ZSH_CUSTOM/plugins/zsh-peco-history

Then, add zsh-peco-history to the list of plugins:

plugins=(zsh-peco-history z colored-man-pages git python django mvn aws docker vscode)

The plugin will be ready once you start a new terminal session.

The following animation shows that this improved history search was very helpful for writing this blog post. It allowed me to easily search for the exact commands I used when I set up my current machine:

Searching shell history with peco.

Autosuggestions

Our new history search is great. The bad news is that we will not use it very often, because there is something that is even better in most cases: zsh-autosuggestions
This plugin suggests completions for your commands as you type. By default, the suggested completion will be the first match in your history. This is how it looks like:

Using a suggested git command-line.

git clone https://github.com/zsh-users/zsh-autosuggestions ${ZSH_CUSTOM:-~/.oh-my-zsh/custom}/plugins/zsh-autosuggestions

Afterward, add zsh-autosuggestions to your plugins setting in your ~/.zshrc file:

plugins=(zsh-autosuggestions zsh-peco-history z colored-man-pages git python django mvn aws docker vscode)

Syntax Highlighting

Another great plugin is zsh-syntax-highlighting. It highlights commands while you are typing them. This is especially useful for catching syntax errors. Once again, install via git:

git clone https://github.com/zsh-users/zsh-syntax-highlighting.git ${ZSH_CUSTOM:-~/.oh-my-zsh/custom}/plugins/zsh-syntax-highlighting

Then, add zsh-syntax-highlighting to your plugins setting in your ~/.zshrc file:

plugins=(zsh-syntax-highlighting zsh-autosuggestions zsh-peco-history z colored-man-pages git python django mvn aws docker vscode)

Syntax highlighting also indicates whether a command is typed correctly.

Clipboard Tool

This one is not really a full-blown plugin. Oh my Zsh comes with a script called clipcopy out of the box. It is a simple tool that enables clipboard copy and past operations from the command-line:

<command> | clipcopy    # copies stdin to clipboard
clipcopy <file>         # copies the content of a file to clipboard

If this doesn’t work, you are probably missing xclip. Install it via

sudo apt-get install xclip

Somehow, I have a hard time remembering the name clipcopy, therefore I set a global alias as described above:

alias -g ctrlc=' | clipcopy'
echo "copythisinclipboard" ctrlc

If you don’t use Oh my Zsh, you can get the functionality of clipcopy via an actual plugin.

SSH Setup

One of the great advantages of using Linux instead of Windows is the out-of-the-box SSH support. Immediately after installing your distribution, you can start to access remote servers.

However, there is some room for streamlining your SSH tasks. You can avoid lots of repetitive typing if you modify your .ssh/config file and setup meaningful defaults for the machines you access regularly:

Host ec2-instance # you can freely chose a name here that will act as alias for this host
    HostName <ip-address or hostname>
    User tkainrad


Host cluster # you can freely chose a name here that will act as alias for this host
    HostName <ip-address or hostname>
    User tkainrad
    IdentityFile `/home/tkainrad/cluster_rsa.pub`

For more sophisticated settings, look up the ssh_config man page.

If you want to go one step further, you can set up public key authentication so you can login to your most-used remote hosts without having to re-enter your password each time. With ssh-copy-id, there is a simple tool that streamlines this process. If you have your hosts already configured via the .ssh/config file, just do

ssh-copy-id ec2-instance

By default, ssh-copy-id will take your public key in ~.ssh/id_rsa.pub. If you want to use different keys for each of your remote hosts, check out the man page page or this more extensive guide. If you use keys other than ~.ssh/id_rsa.pub you also have to add an IdentityFile entry to your .ssh/config settings. Afterwards, you will be able to access the machine by simply doing:

ssh ec2-instance

Additional Tools

A great terminal gets you a long way. However, some additional tools can help make your workstation more complete. In this section, I try to present some of my favorites that could be useful to almost anyone.

Text Editing: VSCode

For most of my more extensive software engineering projects, I use JetBrains products or Eclipse. I will not describe setting up those in this post, as this would easily mandate a separate article. However, I present a more lightweight editing environment that is often a better choice than to start up a full-blown IDE. For a software engineer, especially for one that deals also with DevOps workloads, it is the daily business to edit and view all kinds of text-based files, such as .txt, .csv, .md, xml, .yml, and .json. For this, it is essential to have a fast text editor which can deal with large files and still provides sophisticated editing features.

Previously, I used Sublime. For a couple of months now, I have switched to Visual Studio Code and I am still amazed by it. Comparing it to Sublime makes little sense in my opinion, because VSCode is just on a whole other level. Even for software projects with relatively simple technology stacks, such as this blog website, I now prefer VSCode over Eclipse or the JetBrains suite.

Screenshot of VSCode showing the default light theme (Light+) (Source).

The easiest way to install Visual Studio Code on Ubuntu is to visit the official website and to download the Debian package. You could have probably figured out how to do this yourself, but do you also know how to set VSCode as the default application for all text files? Simply do

xdg-mime default code.desktop text/plain

This will spare you the headache of doing Right-Click -> Properties -> Open With -> Set as default on every file type separately.

Switching to VSCode from Sublime (or any other editor) is a surprisingly pleasant experience. VSCode will recommend the most relevant plugins as you go. For example, if you open a .py file, some extensions for Python development will be recommended. Installing them is then just a click away and doesn’t even require a restart of the application. After some days of following VSCode’s recommendations and occasionally searching for some extensions manually, my setup was more complete and efficient than it ever was with Sublime.

VSCode Terminal with Agnoster

If you don’t intend to use the integrated VSCode terminal, you can safely skip this section.

If you followed this guide until here and you now open an integrated Terminal in VSCode, you will likely be a bit disappointed. There are several problems. With our current setup, we need a monospace font, and the coloring does not work nicely with VSCode’s light theme.

Fortunately, we can fix these things:

First, install the Menlo for Powerline font:

git clone https://github.com/abertsch/Menlo-for-Powerline.git
sudo mv "Menlo for Powerline.ttf" /usr/share/fonts/
sudo fc-cache -vf /usr/share/fonts/ # Refresh fonts cache

Then, we change our terminal prompt colors a little bit to be compatible with both light and dark themes. Open ~/.oh-my-zsh/themes/agnoster.zsh-theme and change prompt_context to the following:

# Context: user@hostname (who am I and where am I)
prompt_context() {
  if [[ "$USER" != "$DEFAULT_USER" || -n "$SSH_CLIENT" ]]; then
    prompt_segment cyan default "%(!.%{%F{yellow}%}.)%n@%m"
  fi
}

Finally, we apply the new font and the new colors in the VSCode settings. Press Ctrl+Shift+P and start to type Preferences: Open Settings (JSON). We can now add the following lines to settings.json:

    "terminal.integrated.fontFamily": "Menlo for Powerline", 
    "workbench.colorCustomizations": {
        "terminal.ansiBlue": "#01A0E4",
        "terminal.ansiCyan": "#B5E4F4",
    },

The integrated VSCode terminal now looks nice and has all the functionality of all of our Zsh plugins.

Sync Configuration Files: Mackup

So you just set up your desktop workstation, modified a bunch of configuration files, manually adjusted your VSCode keyboard shortcuts and now you want to do this also with your notebook?

The solution to this is one of my favorite tools of all: Mackup
It will use any software that can sync a folder, such as Dropbox, to keep your application settings in sync.

You can install it with pip:

pip install --upgrade mackup

If you have versioning troubles with pip or Python on Ubuntu 20.04, I recommend to do sudo apt install python-is-python3. It will symlink /usr/bin/python to python3 and hence make your transition to a world without Python 2 easier.

Once Mackup is installed, you can copy your application settings to Dropbox (per default) doing

mackup backup

To apply them at a different machine, just use

mackup restore

Behind the scenes, Mackup copies your configuration files to your Dropbox and then replaces the originals with symbolic links. Have a look at the GitHub Readme for more details.

Advanced Screenshots: Shutter

If you are coming from Windows, you probably use the Snipping Tool on a regular basis. With Screenshot, Ubuntu has a similar tool installed by default. However, there is another option that is superior to both the Windows Snipping Tool and the Ubuntu default: Shutter
Unfortunately, it is no longer included in the default package repositories since Ubuntu 18.10. So, we have to add a third-party PPA first:

sudo add-apt-repository ppa:linuxuprising/shutter
sudo apt-get install shutter

It is very simple to use and does not force you to become familiar with its advanced features. There are a few alternatives that are still actively developed, such as Flameshot. Unfortunately, those are missing a lot of Shutter’s more advanced features. So, I will stick with Shutter for as long as I can make it run on Ubuntu.

Pick Colors: Pick

If you do front-end development, chances are you need a color picking tool on your computer. The most commonly used tool on Ubuntu is GPick. However, it drives me crazy. Usually, I just want to pick a color and get the hex code into my clipboard. GPick makes this very hard. So instead, I recommend Pick, a simple and functional alternative.

Cheatsheet: KeyCombiner Desktop

Most people know KeyCombiner as a tool for browsing and learning shortcuts. However, its desktop app serves an additioanl purpose with its instant lookup feature: It is a context-aware cheatsheet that can show the shortcuts of the currently active application in addition to whatever key combinations and text snippets I have in my personal KeyCombiner collections.

To trigger it, I only have to press Super+Alt+C. The main advantage is that I don’t have to leave the current context and can resume work right after looking up the required piece of information:

KeyCombiner’s instant shortcut lookup for VSCode, Terminal, and Chrome.

Communication Hub: Ferdi

Did you ever see someone using their phone while sitting in front of their personal computer? Chances are, they were using a communication app that they could have used much more efficiently from their larger device with an actual keyboard. To get the benefit of receiving all notifications from all my communication tools, and to be able to respond easily, I use Ferdi. It is particularly useful if you have to monitor several Slack teams.

In contrast to most of its competitors (Rambox, Wavebox, Franz 5), Ferdi is completely free and open-source.

If you want Ferdi or any other software that is distributed via an AppImage to show up properly in the dock, have a look at appimaged. It is open-source software that monitors specific directories for AppImage files and installs them properly in the system.

Previous versions of this article recommended Station instead of Ferdi. However, Station pivoted to become a browser extension and is no longer an option for bundling communication channels outside of your regular browser.

Record Short Screencasts: Peek

Did you wonder how I created those fancy gifs showing my terminal? I thought so.

Peek is a very lightweight and simple tool to record short screencasts. To install it on Ubuntu do

sudo add-apt-repository ppa:peek-developers/stable
sudo apt update
sudo apt install peek

Peek in action. (Source).

During the creation of this article, I was made aware by a good friend and fellow software engineer of dedicated terminal recording tools, such as Terminalizer and asciinema. However, these require some initial setup work, especially if you want to show off a theme and require that your animated terminal looks exactly like your actual terminal.
Since I already recorded most of the animations, I decided to stay with Peek for this post. Nevertheless, if your use case is terminal recording, you should probably go with one of the linked projects instead of Peek.

Mail Client: Thunderbird

For mail, I use Thunderbird. I don’t particularly like it, but to my knowledge it is still the best desktop email client for Linux. Unfortunately, some minor design decisions make the program almost unusable for me in its default state. The reason I include this section about software that everyone knows already is mainly to highlight three changes I always make to the default setup:

By default, Thunderbird sorts messages by date ascending. I think that’s an unfortunate default. Who wants to scroll down constantly to see the newest messages? To change the sort order to descending (newest first), go to Preferences -> General -> Config Editor and set mailnews.default_sort_order to value 2. See this StackExchange answer for more details about default sorting orders and other options.
Install the No Message Pane Sort by Mouse add-on. With this extension, a click on the mail list table header will no longer change the defined sort order. Without it, misclicks while trying to select the first mail in the list are very annoying. Personally, I never want to change the sort order at all. However, it is still possible by holding Ctrl while clicking.
Finally, go to Preferences -> Composition and uncheck the Use Paragraph format instead of Body Text by default to prevent Thunderbird from adding absurd amounts of whitespace whenever you press the Enter key while composing a mail.

Are you already using Thunderbird on another computer? Fortunately, there is an easy way to migrate all your mail accounts and settings. Essentially, you can copy a hidden folder where Thunderbird stores all its data and move it to the new machine, as described in Mozilla’s official documentation.

If you know of a new fancy alternative to Thunderbird that handles parallel usage of several mail accounts well, please let me know.

Bonus Tip

If you have read this far, I am willing to share my most secret Linux hack with you. Joking aside, it is very helpful to know what to do when your Linux workstation freezes. For some reason, I did not know until a couple years ago. Maybe the following trick can help you to avoid some forced restarts in the future.

The most flexible way to re-gain control of a system that has frozen completely is pressing Ctrl+Alt+F3. Then, use top and other process monitoring tools to find the process IDs of the faulty processes. Then, simply kill them (e.g. type kill <pid>, killall, or kill -9). Afterward, you can get back to your work using Ctrl+Alt+F1. (Source).

Conclusion

I hope this post showed you some tools or practices that you didn’t know yet. I have had a good experience with all of them. Especially switching from Bash to Zsh and installing the described plugins came with a real productivity boost. If you feel that I left out something important, I’d be very interested to hear your suggestions.

If you are not mainly using Linux yet, the length of this post might give the impression that it is still a pain to set up a proper Linux workstation. However, you do not necessarily need all of the things I showed here. Also, everything covered in this post is very easy to install with minimal configuration overhead. I strongly believe that you would have a much harder time setting up a comparable Windows development workstation.

You can still spend your life setting up Linux. However, if you use a popular, well-supported distribution you definitely don’t have to. (Source: xkcd.com)