Christian Aichinger's posts

Separate development feedback and performance evaluation

2022-10-24T00:00:00+02:00

Development feedback—intended to help team members learn and improve—needs an environment where the recipient can absorb information. This requires psychological safety: no blame, no pressure, no personal attacks, no consequences, no reason to get defensive.

Performance evaluation—informing employees how happy the company is with their output—is a high-stakes and high-stress situation that precludes learning.

While both are important, they're distinct types of communication:

	Development Feedback	Performance Evaluation
Goal	Learning and information gain	Assessment and rating
Scope	Feedback on a specific action, close in time to the actual performance	Feedback on performance over a longer duration
Frequency	Daily-weekly: whenever reviewing a report's work output	Quarterly/yearly or on-demand when performance problems arise
Level	Often about details, such as: missing error handling code, a better way to do X, better wording for customer communication	Higher-level, such as: progress toward yearly goals, or work output over the last weeks
Stakes	Low stakes: "you are not your code"	High stakes: career progress or negative consequences
Formality	Informal and usually verbal	More formal and usually documented in writing
Consistency	Individual feedback for each team member	Fairness requires consistent success criteria for all team members
Preparation	Little or no preparation necessary: a casual discussion	Fair evaluation requires research and preparation

It's easy to slip a performance remark into a feedback discussion, but it's a mistake. The employee will focus on the performance aspect and not on learning.

Off-hand, negative performance evaluations are especially damaging. They encourage a defensive, argumentative mindset which precludes learning. They also risk escalating into a larger performance discussion you're not prepared for.

Here are some examples of what to avoid:

BAD: Your code is missing error handling. You seem distracted recently.
BAD: Your presentation was hard to follow. This way you'll never reach your sales quota.
BAD: Your customer email sounded rude. I'm fairly unhappy with your work at the moment.

If general performance concerns arise during a feedback session, wrap up as usual, schedule a separate performance talk, and prepare accordingly.

An example of a better approach:

GOOD:
- Feedback session: Here are some points you could improve in your project proposal: «detailed explanation».
- Performance discussion a day later in private: I had the impression that your recent work is less polished then it used to be. Is everything alright?

Decide up-front which discussion you will have: feedback to help your reports grow or performance evaluation. Avoid mixing the two and pick the right approach and setting for each.

Git Best Practices

2022-10-04T00:00:00+02:00

Here's a list of Git version control best practices I recommend.

Staging changes: `git add -p` and `git diff --cached`

Use git add -p to stage changes to commit. It displays each change individually and asks whether to include this part in the commit or not. It thus gives control of the contents of the commit.

Similarly, use git diff --cached to review the staged changes before committing.

I'm a big fan of both methods. They force me to code review my own commits and make sure there is no commented out code and no spurious whitespace or formatting change. More importantly, this approach finds countless bugs! Re-reading the diff is a great opportunity to ask questions like "is this really correct," or "does it cover all corner cases."

Rewriting history: `git commit --amend`

When discovering a problem with the most recent commit, make your changes, stage them, and use git commit --amend to fix the problem directly in the previous commit. This helps to keep a clean history.

A note on changing history in git: Altering commits that were already pushed to others is considered a bad practice, because it breaks work that others might have based on top of the now-obsolete commit. In most situations, I agree with this assessment. However, the risk is manageable when you're working with a small team. Especially for feature branches that are unlikely to be the basis for further work, I tend to advocate rebasing / amending pushed commits to keep a clean history. Just make sure that it's OK with the team.

Rewriting history: `git rebase -i HEAD~10`

Interactive rebasing allows reordering or fixing past commits and generally reworking recent commit history. See the Git documentation on rewriting history for in-depth explanation.

The HEAD~10 part tells git to look at the 10 previous commits, which often is a sensible default.

Also note the "changing history" section above.

Atomic commits

Commits should contain one logical change. Avoid having multiple, independent changes in a single commit. Why? It's easier to review and understand multiple smaller commits than one big commit. It's easier to merge commits together than to split one big commit. If bisecting found that a bug originated from a commit, it's easier to identify and fix the issue in a small commit than in a big commit.

git add -p is great for creating atomic commits from a source tree that already has several changes.

Commit messages

Three key points:

Form: Write a proper subject line, separated by a blank line from the rest of your commit message.
References: Include a link/ID of the relevant task in your ticket system.
Content: Write for your future self. One year down the line questions like "why did we want this / why did we choose this particular approach / what alternatives did we consider" are going to come up. Between your commit message and your ticket system task description, you should be able to answer those questions.

Git branching patterns

The branch structure of a git repository should adapt to the development activity:

If several people work on the same repository at the same time, you'll need feature branches.
If you have to support several releases in parallel, you'll need release branches.
If you need frequent updates of your releases, you might need hotfix branches as well.

The large number of branches created with these schemes is unavoidable in heavily used repositories. However, the added complexity makes the commit history hard to understand.

For less active repositories, such as small tools or legacy software, simpler approaches are better suited. Often a single main or master branch together with occasional feature branches is enough.

If you have other Git best practices to include, please comment below.

Covid variant competition and a longer-term strategy

2022-01-16T00:00:00+01:00

How do new Covid strains replace older ones? A simple model provides some answers:

The model consists of three rules:

At the beginning only the base variant exists, until on day 200 the Omicron variant appears.
Each base variant infection causes 1.25 follow-up infections. Each Omicron infection causes 1.40 follow-up infections.
A lockdown starts when the number of sick people reaches 20000. This results in a reduction of new infections by 36%. The lockdown ends when the number of sick people falls below 6000 again.

This might be the world's worst Covid model, but it gets one thing right: the competition between variants. It shows the dynamic:

Earlier variants vanish due to the stronger human interventions required to keep the newer variants in check. We decide on preventive measures based on total numbers, not per-variant details. More infectious strains require longer, more frequent, or more stringent lockdowns. These measures hit older variants harder, which makes them recede.¹

At some point we might not be able to enact strict enough countermeasures to prevent newly evolved strains from spreading. Arguably, this already applies to Omicron. If not, new variants will continue to emerge. Our lukewarm vaccination efforts exacerbate this issue by applying evolutionary pressure on the virus. Vaccination rates are high enough to stop earlier strains, but low enough to allow the spread of more infectious, more vaccination-resistant variants. Thus, we should expect to see more problematic strains in the future.

Where does this leave us? Covid is on track to become endemic, so everyone will get infected eventually. The big question is how to manage this process without overwhelmed hospitals, and ideally without endless lockdowns. Vaccination is a key ingredient, since it reduces hospitalization and death rates tenfold. A 100%-vaccinated society could support ten times more infections at the same time.² During Corona peaks, masks, social distancing, and lockdowns help to remove pressure from hospitals. Outside of peak seasons, a relaxed approach is probably best.

¹ Does this argument still hold despite the lower hospitalization rates of Omicron patients? It does: avoiding overwhelmed hospitals still requires strict Omicron countermeasures, just at a higher number of infected people.
² Compared to an unvaccinated society. Countries with a 65–75% vaccination rate can afford 3–4 times the infections as before vaccination campaigns. The comparison of Covid numbers of winter 2020/2021 and winter 2021/2022 supports this.

How to create pixel-perfect PDF reports in 2021

2021-09-07T00:00:00+02:00

Web development projects often require printable reports or invoices, but relying on the browser's built-in print function is a frustrating experience. Results vary with operating systems and render engines, leading to frequent bugs and broken layouts. Worse, fixing them risks degrading other platforms.

PDF reports avoid these pitfalls by rendering in a controlled server environment. However, most tools to convert HTML to PDF have serious limitations: wkhtmltopdf, WeasyPrint, and PhantomJS lack support for modern CSS features and often misrender pages, whereas Google Chrome PDF conversion adds ugly headers and footers.

Puppetteer, the successor to PhantomJS, solves these issues by bundling its own Chromium browser. It supports pixel-perfect PDF generation and respects CSS page margins, allowing full control over the PDF page layout.

My htmlpdf.js command-line tool automates HTML to PDF conversion:

npm install puppetteer
./htmlpdf.js input.html output.pdf
./htmlpdf.js https://google.com output.pdf

Give it a try and let me know what you think.

Capture Bluetooth Traffic from Android

2020-11-08T00:00:00+01:00

Bluetooth devices have become ubiquitous, whether it's headphones, heart rate monitors, smart watches, or light bulbs. While many of them come with mobile apps for control and configuration, sometimes it's convenient to access them from a computer instead. If this is not supported out of the box, recording the Bluetooth traffic is useful in order to create a custom tool.

Helpfully, newer versions of Wireshark include an androiddump utility to capture Bluetooth traffic directly from Android phones. The setup couldn't be simpler: connect the phone to the computer via USB, enable USB debugging, select the "Android Bluetooth" capture source in Wireshark, and start recording!

Android also has a function to log all traffic into a file, which can later be copied to a computer for analysis in Wireshark. This is similar to creating a tcpdump of TCP/IP traffic, just for Bluetooth. Unfortunately, the location of this log file changes between Android versions, and access seems to require root in recent versions. One reliable method I found was:

Enable "Bluetooth HCI snoop log" in the Android developer options.
Connect to the phone via USB debugging with adb
Run adb root to gain root-level access to the phone
Run adb bugreport bugreport.out to create a bug report ZIP file. Either this contains the Bluetooth log file directly (search for a btsnoop_hci.log), or the full path of the log file (grep the bug report for btsnoop_hci.log)
If the log was not contained in the bug report, get it via adb pull <full_path_to_logfile>

Force enable dark mode in Chrome, Safari, Firefox

2020-05-16T00:00:00+02:00

The last years have seen a rapid rise in the number of web sites that support a "dark mode". Some pages offer an explicit light/dark switch. But typically the selection is based on the browsers "prefers-color-scheme" CSS selector. It is surprisingly difficult to change this browser default without switching the whole operating system.

Follow the instructions below to switch to dark mode.

The color scheme preference of your browser is not setlightdark.

Chrome

For Chrome, the instructions depend on the system it is running on.

On Android

In Chrome, open the top-right "..." menu and go to "Settings"
Open "Themes"
Select "Dark"

On iOS

There is no direct way to enable dark mode only for Chrome on iOS. You have to change the whole device to iOS via "Settings" → "Display & Brightness".

On Windows

Chrome switches into dark mode when it is started with the --force-dark-mode command line flag.

Close all Chrome instances
Shift-Right click the Chrome shortcut in the taskbar or on the desktop
Select "Properties"
In the "Shortcut" tab, append --force-dark-mode to the "Target" field
Close the dialog with "OK"
Restart Chrome with that shortcut

Safari

Safari doesn't have a separate setting for dark mode. It always follows the operating system setting.

Changing the system setting on iOS

Open Settings
Open "Display & Brightness"
Select "Dark"

Changing the system setting on MacOS

Open the system settings in the Apple menu
Open the "General" dialog
Select the "Dark" appearance

Firefox

Firefox has hidden configuration option that enables dark mode:

Type about:config into the address bar and press Enter
Type ui.systemUsesDarkTheme into the search bar
The search will not find anything, but allow you to add a new preference with that name
Set the property type to "Number" and click the "+" button to create:
Enter the value "1" to enable dark mode and click the check mark to save:

LDAP basics for users

2019-02-06T00:00:00+01:00

Administrating an LDAP server may be hard — using it doesn't have to be.

LDAP servers commonly provide auth services to web servers, mail servers, web apps, and so on. To do this, the LDAP database stores user and group membership information. The combination of these two datasets allows both authentication (is the user who they claim to be?) and authorization (is the user in a group that has permission to perform a specific action?). Thus, LDAP enables central management of user, group, and permission information for any number of services.

Concepts

So what does an LDAP database consist of?

An LDAP database contains a hierarchical data structure, similar to a directory tree.
Each tree node is called an entity.
- LDAP doesn't distinguish between files and directories. Entities often contain both child entities — like a directory — as well as attributes, which are similar to a file's content.
Each entity has a distinguished name (DN), which is the entity's absolute pathname in LDAP tree. The elements of the pathname are called relative distinguished names (RDNs).
These concepts are pretty similar to filesystem directory trees. The key differences are:
- Directory separators: LDAP uses , instead of /
- RDN format: RDNs are typically key-value-pairs, instead of simple strings: uid=ca instead of Desktop. Commonly used keys are dc, o, ou, uid.
- Parent nodes are on the right: so it's dc=child,dc=parent instead of /parent/child
  Consequently, DNs usually look like this: uid=ca,ou=people,dc=caichinger,dc=com
Entities have attributes, which store the entity's data, similar to a file's contents. Each attribute has a type that describes the attribute's data structure, as well as one or more values containing the attribute's information. Additionally, each attribute can have options — a rarely used feature for distinguishing different versions (e.g. English, German) of the same attribute.
Entities also have associated object classes, which are conceptually similar to attribute types. But whereas types describe attributes, object classes describe which attributes must be found within the entity.
Both attribute types and entity object classes are metadata: they describe the database's schema. Each of these metadata objects has an OID. Aside from the schema definition, OIDs are also used for other database-specific metadata, such as identifying extended requests and responses. OIDs are denoted as dot-separated numbers, e.g. 1.2.840.1234567890, but often have human-readable names assigned as well.

What actions can be performed over the LDAP protocol?

Binding: authenticating to the LDAP server — essentially "logging in". Since most servers don't allow un-authenticated querying, this is required before performing any other actions. Many servers also support re-authentication as a different user over existing connections: this is known as re-binding.
Searching: querying the existing LDAP directory tree, and listing its information.
Add, modify, and delete: altering the LDAP directory tree.
Many others, often including custom commands.

Querying

Querying an LDAP server is straight-forward with the ldapsearch tool:

ldapsearch
  -h ldap.caichinger.com                       # LDAP server name
  -D "uid=ca,ou=people,dc=caichinger,dc=com"   # Authenticate as uid=caichinger
  -W                                           # Ask for password for uid=caichinger
  -b "ou=people,dc=caichinger,dc=com"          # Base search path
  (uid=caichinger)                             # Search expression

The -D and -W switches tell ldapsearch as which user to authenticate as. The -b switch defines the "base directory" where the search should start. The search expression is then applied to all entities under this directory tree.

The server response will then contain all matching users, as well as their associated attributes.

If you have any questions after this whirlwind-tour of LDAP, please leave a comment!

Converting Unix timestamps

2018-10-15T00:00:00+02:00

Sometimes, we need to convert Unix timestamps (seconds since January 1st, 1970) to human-readable dates. For example, we might transform 1539561600 to 2018-10-15 00:00 UTC.

There are multiple online services that do this, I like unixtimestamp.com.

Every now and then we need to batch-convert timestamps. The date command shipped on Linux distributions does this nicely:

date "+%c" --date=@1539561600

I recently ran into a similar problem when logfiles contained Unix timestamps instead of human-readable dates. Using date seemed a bit clumsy here. Fortunately, Superuser.com had a nice solution involving Vim. The following sequence converts the timestamp under the cursor and records a macro q to facilitate future conversions:

qq                             " start recording
"mciw                          " put time in register m and replace it…
<C-r>=strftime("%c", @m)<CR>   " …with localized datetime
<Esc>                          " exit insert mode
q                              " stop recording

Quick and convenient — and easily incorporated into a macro to convert timestamps across the entire file.

The Kelly Criterion: Comparison with Expected Values

2018-04-19T00:00:00+02:00

This is the final post in a four-part series exploring the Kelly criterion:

The Kelly Criterion: Introduction

The Kelly Criterion: Multiple Investment Opportunities

The Kelly Criterion: Where does the logarithm come from?

The Kelly Criterion: Comparison with Expected Values (this post)

In a previous post, we looked at the Kelly formula, which maximizes earnings in a series of gambles where winnings are constantly re-invested. Is this equivalent to maximizing the expected return in each game? It turns out that the answer is "no". In this post we'll look into the reasons for this and discover the pitfalls of expected values.

We will look at the same game as in the previous post:

\frac{V_1}{V_0} = (1+lr_W)^W(1+lr_L)^{(1-W)}

with the variables:

$V_0, V_1$ : the available money before and after the first round
$l$ : fraction of available money to bet in each round (the variable to optimize)
$r_W, r_L$ : return on win and loss, 0.4 and -1 in our example (i.e. 40% of wager awarded on win, otherwise 100% of wager lost)
$W$ : Random variable describing our chances to win; valued $1$ with $p=0.8$ , $0$ with $p=0.2$

The Kelly formula obtained from maximizing $\log V_1/V_0$ tells us to invest 30% of our capital in such a gamble. Let's see what the result is if we maximize the expected value $E[V_1/V_0]$ instead.

This is trivial by hand, but we'll use SymPy, because we can:

import sympy as sp
import sympy.stats as ss
sp.init_printing()

l = sp.symbols('l')             # Define the symbol/variable l
W = ss.Bernoulli('W', 0.8)      # Random variable, 1 with p=0.8, else 0
def f1(W):                      # Define f1 = V_1/V_0
    return (1 + 0.4*l)**W * (1 - l)**(1-W)
ss.E(f1(W))                     # Calculate the expected value

Evaluating this gives:

E[\frac{V_1}{V_0}] = 1 + 0.12l

Uh-hum... so the expected return has no maximum, but grows linearly with increasing $l$ . Essentially, this approach advises that you should bet all your money, and more if you can borrow it for negligible interest rates.

Could the problem be that we only look at a single round? Let's examine the expected return after playing 10 rounds:

# Note: We can not do f10 = f1(W)**10, since we need independent samples
W_list = [ss.Bernoulli('W_%d' % i, 0.8) for i in range(10)]
f10 = sp.prod([f1(Wi) for Wi in W_list])
sp.expand(ss.E(f10))

E[\frac{V_{10}}{V_0}] = 6 \cdot 10^{-10} l^{10} + 5 \cdot 10^{-8} l^9 + ...

All the coefficients of the polynomial are positive, there are no maxima for $l >= 0$ . What's going on?

Time to dig deeper. Let's say we bet all of our money each round. If we lose just once, all of our money is gone. After 10 rounds playing this strategy, the probability of total loss is:

p({\text{one loss in 10 games}}) = 1 - 0.8^{10} = 0.89

So 89% of the time we would lose all our money. However, the expected return after 10 rounds at $l=1$ is:

E[\frac{V_{10}}{V_0}]_{l=1} = 3.1

So on average we'd have $3.10 after 10 rounds for every Dollar initially bet, but 90% of the time we'd lose everything. Strange.

The big reveal

Things become clearer when we look at in more detail at the calculation of $E[V_{10}/V_0]$ :

\begin{aligned} E[\frac{V_{10}}{V_0}]_{l=1} && = &&& 1.4^0 0^{10} && \times && 0.8^0 0.2^{10} && + \\ && &&& 1.4^1 0^9 && \times && 0.8^1 0.2^{9} && + \\ && &&& 1.4^2 0^8 && \times && 0.8^2 0.2^{8} && + \\ && &&& ... && && \\ && &&& 1.4^9 0^1 && \times && 0.8^9 0.2^{1} && + \\ && &&& 1.4^{10} 0^0 && \times && 0.8^{10} 0.2^{0} \\ \end{aligned}

The expected value is the sum of probability-weighted outcomes( $1.4$ and $0$ are the per-round outcomes for win and loss). Since a single loss results in loss of all money, the only non-zero term in the sum is the starred one, that occurs with about 11% probability, at a value gain of $1.4^{10} = 28$ . This high gain is enough to drag the expected return up to 3.1. When more then 10 rounds are played, these numbers become more extreme: the winning probability plummets, but the winning payoff skyrockets, dragging the expected return further upwards.

This is a bit reminiscent of the St. Petersburg paradox, in that arbitrarily small probabilities can drag the expected return to completely different (read "unrealistic") values.

Different kinds of playing

The Kelly approach builds on the assumption that you play with all your available wealth as base capital, and tells you what fraction of that amount to invest. It requires reinvestment of your winnings. Obviously, investing everything in one game ( $l=1$ ) is insane, since a single loss would brankrupt you. However, following Kelly's strategy is the fastest way to grow total wealth.

The expected-value approach of "invest everything you have" is applicable in a different kind of situation. Let's say you can play only one game per day, have a fixed gambling budget each day, and thus are barred from reinvesting your wins. If you invest your full daily gambling budet, you may win or lose, but over the long run you will average a daily return of $1.4 \cdot 0.8 = 1.12$ for every dollar invested. The more you can invest per day, the higher your wins, thus the pressure towards large $l$ values.

In a way, Kelly optimizes for the highest probability of large returns when re-investing winnings, while the expected value strategy optimizes for large returns, even if the probability is very low.

A dubious game

Should you play a game where the winning probability $p$ is $10^{-6}$ , but the winning return $r_W$ is $2\cdot 10^6$ ? Mathematically it seems like a solid bet with a 100% return on investment in the long run. The question is whether you can reach "the long run". Can you afford to play the game a million times? If not, you'll most likely lose money. If you can afford to play a few million times, it becomes a nice investment indeed.

Kelly would tell you to invest only a very small fraction of your total wealth into such a game. The expected-value formalism advises to invest as much as possible, which for most people is bad advice even when playing with a fixed daily budget and no reinvestment (i.e. the expected-value play style).

This is an interesting example for two reasons:

It demonstrates one of the ways how "rich become richer" - the game has high returns, but also a significant barrier to entry.
It demonstrates a downside of both the Kelly- and the expected-value approach. The two strategies are optimal in their use cases in the limit of infinitely many games, however for finitely many games they may give bad advice, especially regarding to very low-probability winning scenarios.

Conclusion

So, a brief discussion of the relationship between the Kelly strategy and the expected return. For me, it was striking how two seemingly similar approaches ("maximize the moneys") lead to so different results and how unintuitively the expectation value can be in the face of outliers.

If you're interested in interactive plots that really helped me understand this material, you can find them in this Jupyter Notebook.

The Kelly criterion

Over the course of this blog post series, we looked at the classical Kelly criterion in the first post, and how it can be extended to situations such as stock buying, with multiple parallel investment opportunities, in the second post. Next, we investigated the origin of the logarithm in the Kelly formula in the third post, before finishing up with the current discussion about expected values.

Surely, there's more to say about the Kelly criterion. If you want to leave your thoughts, please do so in the comments below!

The Kelly Criterion: Where does the logarithm come from?

2018-04-18T00:00:00+02:00

This is the third post in a four-part series exploring the Kelly criterion:

The Kelly Criterion: Introduction

The Kelly Criterion: Multiple Investment Opportunities

The Kelly Criterion: Where does the logarithm come from? (this post)

The Kelly Criterion: Comparison with Expected Values

The neat thing about the derivations in the last two posts is that they give a motivation for "optimizing the logarithm of wealth". The logarithm is not put in by decree, but is a mathematical technicality that arises from the repeated betting process! Kelly mentions this in his original paper:

At every bet he maximizes the expected value of the logarithm of his capital. The reason has nothing to do with the value function which he attached to his money, but merely with the fact that it is the logarithm which is additive in repeated bets and to which the law of large numbers applies.

This argument is very general. Let's say we model the wealth $V_n$ after $n$ rounds of betting based on the initial wealth $V_0$ in terms of a function $f(R, l) = V_{n+1} / V_n$ as

V_n = V_0 f(R, l)^n \tag 1

where $R$ is a random variable describing the possible outcomes of the game, and $l$ is the fraction of available money to invest in each round. Then we can derive the following formula for the best $l$ value:

\begin{aligned} l_{opt} & = \operatorname*{argmax}_l \prod_j f(r_j, l)^{p_j n} \\ & = \operatorname*{argmax}_l \sum_j p_j \log f(r_j, l) \\ & = \operatorname*{argmax}_l E \left[ \log \frac{V_{n+1}}{V_n} \right] \tag 2 \end{aligned}

where $r_j$ are the possible investment outcomes and $p_j$ are the associated probabilities. The crucial change from random variable $R$ to outcomes and probabilities $r_j$ and $p_j$ is justified by the law of large numbers. Based on the exponential nature of the formula, switching to a logarithmic view feels very natural.

Consequently, neither the details of the game — represented by the random variable $R$ — nor the exact form of the per-round return $f$ matter. Any iterative scheme with reinvestment of profits should be representable in the form of equation (1), leading to the logarithm in solution (2). Beyond the origin of the logarithm, this analysis also shows the universality of the Kelly derivation.

Unfortunately, this argument is mostly skipped in online discussions. Often the logarithm is not justified at all, or it is treated as "genius from the 1950s says: use $\log$ ". Sometimes the result is also linked to utility theory, which posits that having twice the money is not twice as useful. While utility theory may be true, reasonable people can disagree on their utility function — exactly how useful more or less money is to them. However, Kelly's result is not grounded in utility, and the $\log$ does not represent logarithmic utility of money. Consequently, even people who disagree on their utility function should agree that the Kelly criterion is the fastest way to gain wealth.

I hope this post shed some light on the reasoning behind the Kelly decision scheme. If you're interested in interactive plots that really helped me understand this material, you can find them in this Jupyter Notebook.

In the next post, we'll take a closer look at the relation of the Kelly criterion to expected values.

The Kelly Criterion: Multiple Investment Opportunities

2018-04-17T00:00:00+02:00

This is the second post in a four-part series exploring the Kelly criterion:

The Kelly Criterion: Introduction

The Kelly Criterion: Multiple Investment Opportunities (this post)

The Kelly Criterion: Where does the logarithm come from?

The Kelly Criterion: Comparison with Expected Values

After the last post introduced the Kelly criterion and its application in deciding how much to invest in a single gamble, we'll investigate whether Kelly can help us choosing between multiple investment opportunities.

We'll start with a mathematical model:

V_1 = V_0 \left( (1+r_0)(1 - \sum_i l_i) + \sum_i l_i(1 + r_i) \right)

Simple, right? :-)

$V_0, V_1$ are the available money before and after the first round, as before.
$r_0$ is the risk-free return rate. This allows us to model opportunity costs (could put the money into a bank account instead of investing, $r_0 = 0.004$ ), or inflation (non-invested money loses value over time $r_0 = -0.02$ ).
$l_i$ is the fraction of the available money invested in stock $i$ .
$r_i$ is a random variable describing returns for stock $i$ .

The first term $(1+r_0) (1 - \sum_i l_i)$ represents all the money not invested in any stock, being invested at the risk-free return rate $r_0$ . The second term $\sum_i l_i(1 + r_i)$ represents the outcomes of the investments in individual stocks $i$ .

We will re-formulate a bit to simplify the expression:

\begin{aligned} \frac{V_1}{V_0} & = (1+r_0) - \sum_i l_i (1+r_0) + \sum_i l_i(1 + r_i) \\ & = (1+r_0) + \sum_i l_i(r_i - r_0) \end{aligned}

We can further simplify by merging the $l_i$ and $r_i$ into the vectors $\vec l$ and $\vec r$ , with $\vec 1$ being a vector where all elements are $1$ :

\frac{V_1}{V_0} = (1+r_0) + \vec l \cdot (\vec r - r_0 \vec 1)

Let's move from a single game round to $n$ rounds:

\frac{V_n}{V_0} = \left( (1+r_0) + \vec l \cdot (\vec r - r_0 \vec 1) \right)^n

Now we are again in a position where we can follow Kelly's prescription: invoking the law of large numbers and finding the $l$ which maximizes the gain $V_n/V_0$ .

For large $n$ , we can approximate:

\frac{V_n}{V_0} \approx \prod_j \left( 1+r_0 + \vec l \cdot (\vec r_j - r_0 \vec 1) \right)^{p_j n}

In this step we switched from the random variable $\vec r$ to its outcomes $\vec r_j$ . Each possible outcome $\vec r_j$ occurs with probability $p_j$ . We justify this switch with the law of large numbers: the outcome $\vec r_j$ will be observed proportionally to its probability, $p_j n$ times (for large $n$ ). Consequently, we will have $p_j n$ factors involving $\vec r_j$ in the overall product, with $j$ iterating over all potential outcomes of $\vec r$ .

Note that since $\vec r_j$ is vector-valued, $p_j$ is a joint probability distribution.

At this point we are nearly finished. We are looking for the vector $\vec l_{opt}$ that maximizes ${V_n}/{V_0}$ . In analogy to last post, we arrive at:

\begin{aligned} \vec l_{opt} & = \operatorname*{argmax}_{\vec l} \frac{V_n}{V_0} = \operatorname*{argmax}_{\vec l} \log \frac{V_n}{V_0} \\ & = \operatorname*{argmax}_{\vec l} \sum_j p_j \log \left( 1+r_0 + \vec l \cdot (\vec r_j - r_0 \vec 1) \right) \end{aligned}

This equation is not analytically solvable, but may be approximated as a quadratic programming problem as described in a paper by Vasily Nekrasov.

Conclusion

It should be obvious that the Kelly criterion is applicable in a wide range of scenarios, from gambling over investment decisions to whether to buy insurance. If you're interested in interactive plots that really helped me understand this material, you can find them in this Jupyter Notebook.

In the next post we'll discuss the origins of the logarithm in the Kelly formula.

The Kelly Criterion: Introduction

2018-04-16T00:00:00+02:00

This is the first post in a four-part series exploring the Kelly criterion:

The Kelly Criterion: Introduction (this post)

The Kelly Criterion: Multiple Investment Opportunities

The Kelly Criterion: Where does the logarithm come from?

The Kelly Criterion: Comparison with Expected Values

The Kelly criterion is a formula used to determine the optimal size of a series of bets in order to maximize wealth. It is often described as optimizing the logarithm of wealth, and will do better than any other strategy in the long run.

Let's look at a simple gamble: when you play, you win with probability $p$ and lose with probability $q = 1-p$ . If you win, you get back your initial bet plus a fraction $r$ of that bet. If you lose, your bet is forfeited. The Kelly formula calculating the optimal fraction of your available wealth to bet is:

l_{opt} = \frac{pr - q}{r} = \frac{pr - (1-p)}{r} = p - \frac{1-p}{r}

Here is a small demo to explore the formula:

Kelly calculation demo

p		0.80
r		0.40

Probability of winning: 80 %
Profit on win, as percentage of placed bet: 40 %
Optimal fraction of wealth bet in this game: 30 %

The main takeaways are:

In each game you invest a fraction $l$ of the total amount of money you have. If you play the game from the demo ( $p=0.8, r=0.4$ ) and you have $100 available, you should invest $30 (since $l=0.3$ ). If you win, you will have $112 ( $100 + 30 \times 0.4$ ) and invest $33.60 ( $112 \times 0.3$ ) in the second round. If you lost the first round you'd have $70, and invest $21 in the second round. If you can not reinvest your earnings, Kelly does not apply.
This formula holds when a large number of games are played. If you play only few rounds of a game this may not apply.
When it applies, the Kelly strategy will generate the most wealth out of your initial starting capital.

Derivation of the Kelly criterion

We will take a short dive into mathematics to understand how Kelly came up with this formula. Let's compute the wealth $V_1$ after the first round of gaming, when starting from an initial wealth $V_0$ :

V_{1} = V_{0} (1 + Wlr_W + (1-W)lr_L)

Here, $l$ is the fraction of money bet, $r_W$ is the profit on win, $r_L$ is the fraction of the bet forfeited on loss (in the above scenario, $r_L=-1$ ). $W$ is a random variable that's $1$ with probability $p$ and $0$ with $1-p$ . So with probability $p$ we'll have $W=1$ , giving $V_{1,win} = V_0 (1+lr_W)$ , with probability $1-p$ we'll get $V_{1,loss} = V_0 (1+lr_L)$ .

We can reformulate the expression in a way that will show an important generalization:

V_1 = V_0 (1+lr_W)^W(1+lr_L)^{(1-W)}

The two expressions look very different, but they are equivalent since $W$ can only take on the values $0$ and $1$ . If $W=1$ the second term will vanish since $x^0 = 1$ , and we will get the same result for $V_{1,win}$ . The same consideration holds for $V_{1,loss}$ .

If we repeat this game $n$ times, we arrive at

V_n = V_0 (1+lr_W)^{W_n}(1+lr_L)^{(n-W_n)}

with $W_n$ being the number of wins in those $n$ games. Essentially, each won round adds a factor $(1+lr_W)$ to the expression, each lost round $(1+lr_L)$ .

Now comes a crucial step: how many wins do we expect after playing many games? Asked mathematically, what value will $W_n$ have for large $n$ ? Because of the law of large numbers we expect $W = pn$ . So for $p=0.8$ and $n=10000$ we would expect to win 8000 out of the 10000 games.

Let's plug this into our formula:

V_n = V_0 (1+lr_W)^{pn}(1+lr_L)^{(1-p)n}

The law of large numbers allowed us to eliminate the random variables $W$ , $W_n$ and instead obtain an explicit formula in terms of $p, n, r_W, r_L$ . Since we are interested in maximizing our profit, we are looking for $l_opt$ — the $l$ value that maximizes ${V_n}/{V_0}$ .

\begin{aligned} l_{opt} & = \operatorname*{argmax}_l \frac{v_n}{v_0} \\ & = \operatorname*{argmax}_l \log \frac{v_n}{v_0} \\ & = \operatorname*{argmax}_l p \log(1 + l r_W) + (1-p) \log(1 + l r_L) \end{aligned}

We are free to introduce the logarithm and drop the $n$ factor because this does not change the value of $l_{opt}$ .¹ Differentiating the logarithmic expression and solving for $l$ gives:

l_{opt} = -\frac{p}{r_L} - \frac{1-p}{r_W}

This is identical with the formula given in the introduction after substituting $r_L = -1$ .

Extension to games with more than two possible outcomes

We can extend the Kelly argument to games that have a greater number of possible outcomes. Such games could be:

A game with 6 different rewards selected by roll of a 6-sided die, where rewards may be negative (money lost)
An investment in a stock, here we have a continuous range of outcomes.

If we have multiple outcomes $r_i$ , each happening with probability $p_i$ , we obtain:

l_{opt} = \operatorname*{argmax}_l \sum_i p_i \log(1 + l r_i)

While this equation is not easily solvable analytically, it is trivial to solve numerically. This allows us to extend the Kelly approach to a new kind of game with some interesting practical applications.

If you're interested in interactive plots that really helped me understand this material, you can find them in this Jupyter Notebook.

The Kelly formalism can be generalized further still, taking into account multiple parallel investment opportunities. We'll look at this in the next post.

¹ The logarithm is a strictly monotonically increasing function, so $\alpha \gt \beta \Leftrightarrow \log \alpha \gt \log \beta$ . Thus it doesn't change the location of any maxima: $\operatorname*{argmax} f(x) = \operatorname*{argmax} \log f(x)$ . See Mark Schmidt's Argmax and Max Calculus paper for more information about properties of $\operatorname*{argmax}$ .

Exploring GDB's Python API with Jupyter

2018-03-30T00:00:00+02:00

GDB — the most common console debugger on Linux systems — has a Python API for adding new debugger commands and pretty-printers for complex data structures, or to automate debugging tasks.

While Python scripts can be loaded from files, it is nice to interactively explore the API or the debugged program. IPython would be perfect for the job, but starting it directly inside GDB doesn't work well. Fortunately, it's easy to launch an IPython kernel and connect with an external Jupyter console.

Launching the kernel from the gdb prompt:

(gdb) python
>import IPython; IPython.embed_kernel()
>end

Gives the following message:

To connect another client to this kernel, use:
    --existing kernel-12688.json

We can start Jupyter on a separate terminal and connect to this kernel:

$ jupyter console --existing kernel-12688.json
In [1]: gdb.newest_frame().name()
Out[1]: 'main'

The GDB Python API is then available from the gdb module within this Python session. To get started, I'd suggest the API documentation or this series of tutorials.

Currently, only the console client can connect to existing kernels. Support in the Notebook or in Jupyter Lab is tackled in this Github issue. Even with the limited capabilities of the console client, it's a great way to explore the API and to tackle more complicated debugging problems that require automation to solve.

PyDays 2017

2017-06-20T00:00:00+02:00

PyDays 2017 was Austria's first conference dedicated to the Python programming language. It took place on May 5 and May 6 and was graciously hosted by the Linuxwochen Wien at FH Technikum in Vienna. It was great on many levels: meeting new people excited about Python and where it's headed, over 20 talks, interesting hallway conversations, etc.

I helped out with the organization of the conference, mostly taking care of catering. We (the organization team) are very happy with how everything went. The atmosphere was welcoming and open, the quality of the talks and workshops was very good, attendance was great, and people seemed to have a really good time. From an organizational perspective, the talks mostly stayed on-time, the audio/video hardware worked fine, and the PyDays booth was well-received. Giving out snacks, coffee, soda, cake during breaks worked out nicely and was received very well. The feedback, both at and after the conference, was very positive.

In addition to the co-organization of the Conference, I also held a talk about Dask, a Python library for parallel computing. It provides data structures similar to NumPy Arrays or Python Dataframes that process operations in parallel and scale to out-of-memory datasets. Dask is exciting since it provides the analytical power and familiar mental model of Dataframes, but handles hundreds of gigabytes of data and allows seamless scaling from a single laptop to a computing cluster. The talk was well-received, and there was a lot of interest at the end and in the hallway afterwards. The slides are online, if you're interested as well.

Finally, I'd like to thank my co-organizers, Claus, Kay, Helmut and Sebastian, as well as the people who helped out during the conference, for all their great work to make the PyDays a success. I'd also like to thank our sponsors: the Python Software Foundation, the Python Software Verband e.V., T-Mobile, UBIMET, and Jetbrains.

Overall it was great to see so much community interest in Python. If you're passionate about Python as well, join us on the Python Austria Slack and our meetups!

GCC compatibility: inline namespaces and ABI tags

2016-10-29T00:00:00+02:00

Keeping libraries binary-compatible with old versions is hard. Recently, GCC was in the unenviable situation of having to switch its std::string implementation.¹ GCC used inline namespaces and ABI tags to minimize the extent of breakage and to ensure that old and new versions could only be combined in a safe way. Here, we'll have a look at those mitigation techniques: what they are and how they work.

First, some background: GCC used to have a copy-on-write implementation for std::string. However, C++11 does not allow this anymore because of new iterator and reference invalidation rules. So, GCC 5.1 introduced a new implementation of std::string. The new version was not binary-compatible with the old one: exchanging strings between old (pre-5.1) and new code would crash. We say the application binary interface (ABI) changed.

To understand the consequences of this, let's look at several scenarios:

A program uses only code compiled with GCC < 5.1: only old strings, works
A program uses only code compiled with GCC >= 5.1: only new strings, works
A program mixes code compiled with different GCC versions: both types of strings exist in the program, it could crash.

Let's dig into the last bullet point. When would it crash? Whenever "new" code accesses an "old" string or vice versa. Here are some examples where f() is compiled with an older GCC version and called from "new" code:

void f(int i);                 // (a) safe, no std::string involved
void f(const std::string& s);  // (b) will crash when f accesses s
std::string f();               // (c) will crash when the returned string is used

Given how common std::string is, such crashes would happen frequently if "old" and "new" code were combined. Unfortunately, it's surprisingly easy to end up with a program with some pre-5.1 parts and some newer ones. It's sufficient to link a "new" executable to an "old" library. Given the bad consequences, the GCC developers needed to solve this.

The solution is to change the symbol names of the GCC 5.1 std::string and all functions using it. Because the linker uses symbol names to resolve function calls into libraries, this would cause link-time errors for cases (b) and (c) while (a) would still work. Exactly the intended behavior.

How could this be done? The mechanism that converts C++ names into symbol names is called name mangling. The generated symbol names contain information about namespaces, function names, argument types, etc. Putting the new std::string into a separate namespace would change the symbol names of all functions accepting std::string as argument. But that's crazy—std::string needs to be in namespace std, right?

The solution for this is inline namespaces. All classes, functions, and templates declared in an inline namespace are automatically imported into the parent namespace. Their mangled name still references the original location, though.

Here's what GCC 5.1 does:

namespace std {
  inline namespace __cxx11 {
    template<typename _CharT, ...>  basic_string;
    typedef basic_string<char>      string;
  }
}

Looking at f(const std::string& s), what would the symbol names be when compiled with older and newer GCC versions?

	Older GCC	GCC 5.1
Symbol name	_Z1fRKSs	_Z1fRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE
Decoded symbol name²	`f(std::basic_string<char, ...> const&)`	`f(std::__cxx11::basic_string<char, ...> const&)`

Indeed, the symbol names are different and the linker would give an error if GCC 5.1 code called f(const std::string &s) from a library compiled with an older GCC version. This solves the compatibility problem for all functions taking std::string as argument.

One problem remains: case (c) from above, std::string f(). The return value type of a function is not part of its mangled name.³ Thus, the symbol name wouldn't change for functions that return std::string, which could still lead to runtime crashes.

GCC 5.1 solves this with the use of ABI tags. From the documentation:

The abi_tag attribute can be applied to a function, variable, or class declaration. It modifies the mangled name of the entity to incorporate the tag name, in order to distinguish the function or class from an earlier version with a different ABI

[...]

When a type involving an ABI tag is used as the type of a variable or return type of a function where that tag is not already present in the signature of the function, the tag is automatically applied to the variable or function.

GCC applies the attribute __abi_tag__ ("cxx11")) to the std::__cxx11 namespace. This affects all classes therein, including the new version of std::string. The ABI tag propagates to all functions that return a string and changes their symbol name.

Let's look at the symbol names for std::string f() for different compiler versions:

	Older GCC	GCC 5.1
Symbol name	_Z1fv	_Z1fB5cxx11v
Decoded symbol name²	`f()`	`f[abi:cxx11]()`

Again, the symbol name is different, and the "cxx11" ABI tag is applied to std::string f() with GCC 5.1. This completes the second part of GCC's solution for the std::string ABI change.

These two methods to induce symbol name changes for anything using std::string make the migration path to GCC 5.1 much easier. If the program links correctly it should work, and be safe from difficult-to-debug runtime errors. There would be more to discuss about the GCC 5.1 changes, such as how libstdc++.so still exports the old std::string implementation for compatibility. But this post has gone on too long already, so let's leave it at that :-)

¹ GCC 5.1 also introduced a new version of std::list. It is handled analogously to std::string.
² Decoded using c++filt from the binutils package.
³ The return value type doesn't need to be part of the symbol name because it doesn't get used for overload resolution.

New C++ documentation section in the sidebar

2016-06-26T00:00:00+02:00

I started a new section on this website that attempts to make C++ language constructs more accessible, it's called C++ for Humans. For now, there are two articles: one about the details of variable initialization and one about the types of storage duration C++ supports.

Building External Libraries with Boost Build

2016-06-18T00:00:00+02:00

I recently ran into the problem of having to use an external library not available for our distribution. Of course, it would be possible to build Debian or RPM packages, deploy them, and then link against those. However, building sanely-behaving library packages is tricky.

The easier way is to include the source distribution in a sub-folder of the project and integrate it into our build system, Boost Build. It seemed prohibitive to create Boost Build rules to build the library from scratch (from *.cpp to *.a/*.so) because the library had fairly complex build requirements. The way to go was to leverage the library's own build system.

Easier said than done. I only arrived at a fully integrated solution after seeking help on the Boost Build Mailing List (thanks Steven, Vladimir). This is the core of it:

path-constant LIBSQUARE_DIR : . ;

make libsquare.a
    : [ glob-tree *.c *.cpp *.h : generated-file.c gen-*.cpp ]
    : @build-libsquare
    ;
actions build-libsquare
{
    ( cd $(LIBSQUARE_DIR) && make )
    cp $(LIBSQUARE_DIR)/libsquare.a $(<)
}

alias libsquare
    : libsquare.a                                        # sources
    :                                                    # requirements
    :                                                    # default-build
    : <include>$(LIBSQUARE_DIR) <dependency>libsquare.a  # usage-requirements
    ;

The first line defines a filesystem constant for the library directory, so the build process becomes independent of the working directory.

The next few lines tell Boost Build about a library called libsquare.a, which is built by calling make. The cp command copies the library to the target location expected by Boost Build ($(<)). glob-tree defines source files that should trigger library rebuilds when changed, taking care to exclude files generated by the build process itself.

The alias command defines a Boost Build target to link against the library. The real magic is the <dependency>libsquare.a directive: it is required because the library build may produce header files used by client code. Thus, build-libsquare must run before compiling any dependent C++ files. Adding libsquare.a to an executable's dependency list won't quite enforce this: it only builds libsquare.a in time for the linking step (*.o → executable), but not for the compilation (*.cpp → *.o). In contrast, the <dependency>libsquare.a directive propagates to all dependent build steps, including the compilation, and induces the required dependencies.

I created an example project on GitHub that demonstrates this. It builds a simple executable relying on a library built via make. The process is fully integrated in the Boost Build framework and triggered by a single call to bjam (or bb2).

If you need something along these lines give this solution a try!

Working with angles (is surprisingly hard)

2016-06-14T00:00:00+02:00

Due to the periodic nature of angles, especially the discontinuity at 2π / 360°, working with them presents some subtle issues. For example, the angles 5° and 355° are "close" to each other, but are numerically quite different. In a geographic (GIS) context, this often surfaces when working with geometries spanning Earth's date-line at -180°/+180° longitude, which often requires multiple code paths to obtain the desired result.

I've recently run into the problem of computing averages and differences of angles. The difference should increase linearly, it should be 0 if they are equal, negative if the second angle lies to one side of the first, positive if it's on the other side (whether that's left or right depends on whether the angles increase in the clockwise direction or in counter-clockwise direction). Getting that right and coming up with test cases to prove that it's correct was quite interesting. As Bishop notes in Pattern Recognition and Machine Learning, it's often simpler to perform operations on angles in a 2D (x, y) space and then back-transform to angles using the atan2() function. I've used that for the averaging function; the difference is calculated using modulo arithmetic.

Here's the Python version of the two functions:

def average_angles(angles):
    """Average (mean) of angles

    Return the average of an input sequence of angles. The result is between
    ``0`` and ``2 * math.pi``.
    If the average is not defined (e.g. ``average_angles([0, math.pi]))``,
    a ``ValueError`` is raised.
    """

    x = sum(math.cos(a) for a in angles)
    y = sum(math.sin(a) for a in angles)

    if x == 0 and y == 0:
        raise ValueError(
            "The angle average of the inputs is undefined: %r" % angles)

    # To get outputs from -pi to +pi, delete everything but math.atan2() here.
    return math.fmod(math.atan2(y, x) + 2 * math.pi, 2 * math.pi)


def subtract_angles(lhs, rhs):
    """Return the signed difference between angles lhs and rhs

    Return ``(lhs - rhs)``, the value will be within ``[-math.pi, math.pi)``.
    Both ``lhs`` and ``rhs`` may either be zero-based (within
    ``[0, 2*math.pi]``), or ``-pi``-based (within ``[-math.pi, math.pi]``).
    """

    return math.fmod((lhs - rhs) + math.pi * 3, 2 * math.pi) - math.pi

The code, along with test cases can also be found in this GitHub Gist. Translation of these functions to other languages should be straight-forward, sin()/cos()/fmod()/atan2() are pretty ubiquitous.

Resource management with Python

2016-05-28T00:00:00+02:00

There should be one – and preferably only one – obvious way to do it.

Tim Peters in The Zen of Python

There are multiple ways to manage resources with Python, but only one of them is save, reliable and Pythonic.

Before we dive in, let's examine what resources can mean in this context. The most obvious examples are open files, but the concept is broader: it includes locked mutexes, started client processes, or a temporary directory change using os.chdir(). The common theme is that all of these require some sort of cleanup that must reliably be executed in the future. The file must be closed, the mutex unlocked, the process terminated, and the current directory must be changed back.

So the core question is: how to ensure that this cleanup really happens?

Failed solutions

Manually calling the cleanup function at the end of a code block is the most obvious solution:

f = open('file.txt', 'w')
do_something(f)
f.close()

The problem with this is that f.close() will never be executed if do_something(f) throws an exception. So we'll need a better solution.

C++ programmers see this and try to apply the C++ solution: RAII, where resources are acquired in an object's constructor and released in the destructor:

class MyFile(object):
    def __init__(self, fname):
        self.f = open(fname, 'w')

    def __del__(self):
        self.f.close()

my_f = MyFile('file.txt')
do_something(my_f.f)
# my_f.__del__() automatically called once my_f goes out of scope

Apart from being verbose and a bit un-Pythonic, it's also not necessarily correct. __del__() is only called once the object's refcount reaches zero, which can be prevented by reference cycles or leaked references. Additionally, until Python 3.4 some __del__() methods were not called during interpreter shutdown.

A workable solution

The way to ensure that cleanup code is called in the face of exceptions is the try ... finally construct:

f = open('file.txt', 'w')
try:
    do_something(f)
finally:
    f.close()

In contrast to the previous two solutions, this ensures that the file is closed no matter what (short of an interpreter crash). It's a bit unwieldy, especially when you think about try ... finally statements sprinkled all over a large code base. Fortunately, Python provides a better way.

The correct solution™

The Pythonic solution is to use the with statement:

with open('file.txt', 'w') as f:
    do_something(f)

It is concise and correct even if do_something(f) raises an exception. Nearly all built-in classes that manage resources can be used in this way.

Under the covers, this functionality is implemented using objects known as context managers, which provide __enter__() and __exit__() methods that are called at the beginning and end of the with block. While it's possible to write such classes manually, an easier way is to use the contextlib.contextmanager decorator.

from contextlib import contextmanager

@contextmanager
def managed_resource(name):
    r = acquire_resource(name)
    try:
        yield r
    finally:
        release_resource(r)

with managed_resource('file.txt') as r:
    do_something(r)

The contextmanager decorator turns a generator function (a function with a yield statement) into a context manager. This way it is possible to make arbitrary code compatible with the with statement in just a few lines of Python.

Note that try ... finally is used as a building block here. In contrast to the previous solution, it is hidden away in a utility resource manager function, and doesn't clutter the main program flow, which is nice.

If the client code doesn't need to obtain an explicit reference to the resource, things are even simpler:

@contextmanager
def managed_resource(name):
    r = acquire_resource(name)
    try:
        yield
    finally:
        release_resource(r)

with managed_resource('file.txt'):
    do_something()

Sometimes the argument comes up that this makes it harder to use those resources in interactive Python sessions – you can't wrap your whole session in a gigantic with block, after all. The solution is simple: just call __enter__() on the context manager manually to obtain the resource:

cm_r = managed_resource('file.txt')
r = cm_r.__enter__()
# Work with r...
cm_r.__exit__(None, None, None)

The __exit__() method takes three arguments, passing None here is fine (these are used to pass exception information, where applicable). Another option in interactive sessions is to not call __exit__() at all, if you can live with the consequences.

Wrap Up

Concise, correct, Pythonic. There is no reason to ever manage resources in any other way in Python. If you aren't using it yet - start now!

Remarks on enable_shared_from_this

2016-05-21T00:00:00+02:00

std::enable_shared_from_this is a template base class that allows derived classes to get a std::shared_ptr to themselves. This can be handy, and it's not something that C++ classes can normally do. Calling std::shared_ptr<T>(this) is not an option as it creates a new shared pointer independent of the existing one, which leads to double destruction.

The caveat is that before calling the shared_from_this() member function, a shared_ptr to the object must already exist, otherwise undefined behavior results. In other words, the object must already be managed by a shared pointer.

This presents an interesting issue. When using this technique, there are member functions (those that rely on shared_from_this()) that can only be called if the object is managed via a shared_ptr. This is a rather subtle requirement: the compiler won't enforce it. If violated, the object may even work at runtime until a problematic code path is executed, which may happen rarely – a nice little trap. At the very least, this should be prominently mentioned in the class documentation. But frankly, relying on the documentation to communicate such a subtle issue sounds wrong.

The correct solution is to let the compiler enforce it. Make the constructors private and provide a static factory method that returns a shared_ptr to a new instance. Take care to delete the copy constructor and the assignment operator to prevent anyone from obtaining non-shared-pointer-managed instances this way.

Another point worth mentioning about enable_shared_from_this is that the member functions it provides, shared_from_this() and weak_from_this(), are public. Not only the object itself can retrieve it's owning shared_ptr, everyone else can too. Whether this is desirable is essentially an API design question and depends on the context. To restrict access to these functions, use private inheritance.

Overall, enable_shared_from_this is an interesting tool, if a bit situational. However, it requires care to use safely, in a way that prevents derived classes from being used incorrectly.

Building a computer from logic gates

2016-05-14T00:00:00+02:00

Ever wondered how computers actually work on a low level?

After Jeff Atwood's recent post about Robot Odyssey, I did.

The following is a sketch of what could work, not necessarily what modern hardware actually does. The aim is to explore how a Turing-complete, multi-purpose computation engine could in principle be built from simple logic elements.

From silicon to computation

Nearly all chips are manufactured on silicon plates called wafers. These plates are modified in a complex process to create semiconductor-based diodes and transistors. Most general-purpose processors use a technology called CMOS that arranges the transistors into logic gates – devices that carry out operations on zeros and ones. The most common gate to implement is the NAND. All other common logic gates (AND, OR, NOT, XOR, ...) can be constructed from NAND building blocks.

Memory cells can be constructed by composing multiple logic gates. Each cell stores a single bit of information. Conceptually, it has one output (VALUE) where the current value can be read. Additionally, there are two input pins: SET and SET_VALUE. For reading, SET is zero. For writing, SET is one and the SET_VALUE becomes the new value stored in the cell. It's not hard to imagine how to build a memory controller on top of an array of memory cells that allows addressing of individual cells for getting and setting their value.

How can memory be modified in practice? For example, how is it possible to invert (change 0 to 1 and vice versa) the value of a memory cell? Reading the memory, inverting it and writing it back into the memory cell leads to oscillation: when the cell value is changed it is immediately read back, and inverted and written again. This cycle repeats as quickly as the electronics can handle.

Inverting a bit this way doesn't work - uncontrolled oscillation is observed.

The solutions to this conundrum are clocks and edge-triggered flip-flops. Clocks are signals switching between 0 and 1 at a defined frequency. Edge-triggered flip-flops read their input at the rising edge of the clock (when it switches from 0 to 1) and output that value until the next rising edge. In other words, they sample their input once per clock cycle and hold that value until the next cycle. When such an element is inserted into the inversion loop, the memory value is inverted exactly once per clock cycle.

With additional edge-triggered flip-flop, the bit is inverted exactly once per clock tick.

Based on this technique other operations can be implemented as well, such as adding or multiplying memory cells, copying memory contents to other locations, performing bitwise operations, and so on.

General-purpose processors

For each of those operations the logic gates would have to be arranged differently, though. In contrast, real general-purpose CPU's have fixed logic circuits, their gate configuration doesn't change during runtime. Instead, the operations to execute are read from memory and interpreted according to the chip's instruction set.

For our analysis, let's assume the command is read from separate input lines instead. We'll return to reading commands from memory later on.

How could one design and implement an instruction set? Let's say we have a machine with 8 lines (bits) of input and four 8-bit registers A, B, C, D. External memory is addressed in chunks of 8 bits and is attached via 8 address lines that select the location, 8 lines for reading/writing the 8-bit value, and one line to switch between reading and writing. What operations could we have?

Opcode	Mnemonic	Description
`00RRVVVV`	`SetHi VVVV, RR`	Set the 4 highest bits of register RR to VVVV.
`01RRVVVV`	`SetLo VVVV, RR`	Set the 4 lowest bits of register RR to VVVV.
`1000RRSS`	`Mov RR, SS`	Copy the value of register RR into register SS.
`100100RR`	`Read [RR]`	Read from memory address stored in RR, store the result in register RR.
`100110RR`	`Not RR`	Logically invert the value of register RR.
`100111RR`	`Inv RR`	Negate (one's complement) the value of register RR.
`1010RRSS`	`Add RR, SS`	Add registers RR and SS, store the result in SS.
`1011RRSS`	`Mul RR, SS`	Multiply registers RR and SS, store the result in SS.
`1100RRSS`	`And RR, SS`	Logical AND of registers RR and SS, store the result in SS.
`1101RRSS`	`Or RR, SS`	Logical OR of registers RR and SS, store the result in SS.
`1111RRSS`	`Write RR, [SS]`	Write the value of register RR to the memory address stored in register SS.

It's not very efficient, but it enables a good amount of computation. How could it be implemented? All the separate opcodes could be realized as separate logic blocks on a chip. Each of them individually should be relatively easy to implement. Selecting which block to run (depending on the opcode) is a bit tricky. The easiest way to handle this is to run them all, but only enable output to the registers and memory for the single command that is desired by the input. On every cycle, all possible commands would be computed simultaneously, but only the desired one would be allowed to write to registers and memory. Is it efficient? No. Would it work? Yes.

Finally, we can address the problem of reading instructions from memory. Given the system described in the previous paragraphs, it shouldn't be too hard to add a separate component that reads instructions from memory and feeds it to this computation engine. The two components would communicate via an instruction-pointer register. The instruction set could be expanded to include (conditional) jumps, making the overall system Turing complete.

Conclusion

There are several small problems with what I've described, e.g. how to deal with instructions that consume multiple clock cycles, but all of them are solvable without too much trouble.

Thinking this topic is an interesting exercise. On the transistor level, it's hard to see how a real processor could ever be constructed from these primitives. Possible in principle – but hard to see how to do in practice. Three levels of abstractions above, after gates and memory cells there are suddenly memory blocks that are addressable via a parallel protocol. Every abstraction step is comprehensible, yet complexity is built up quickly. Two levels of abstraction further we suddenly have an 8-bit microprocessor.

It must have been an exciting opportunity to figure all of this out in the middle of the last century.

Boost Range Highlights

2016-05-07T00:00:00+02:00

Last week, I presented Boost Range for Humans: documentation for the Boost Range library with concrete code examples. This week we'll talk about some of the cool features in Boost Range.

irange

boost::irange() is the C++ equivalent to Python's range() function. It returns a range object containing an arithmetic series of numbers:

boost::irange(4, 10)    -> {4, 5, 6, 7, 8, 9}
boost::irange(4, 10, 2) -> {4, 6, 8}

Together with indexed() (see below), it serves as a range-based alternative to the classic C for loop.

combine

boost::combine() takes two or more input ranges and creates a zipped range – a range of tuples where each tuple contains corresponding elements from each input range.

The input ranges must have equal size.

std::string str = "abcde";
std::vector<int> vec = {1, 2, 3, 4, 5};
for (const auto & zipped : boost::combine(str, vec)) {
    char c; int i;
    boost::tie(c, i) = zipped;

    // Iterates over the pairs ('a', 1), ('b', 2), ...
}

Algorithms

Most if not all algorithms from the C++ standard library that apply to containers (via begin/end iterator pairs) have been wrapped in Boost Range. Examples include copy(), remove(), sort(), count(), find_if().

Adaptors

Adaptors are among the most interesting concepts Boost Range has to offer.

There are generally two ways to use adaptors, either via function syntax or via a pipe syntax. While the former is handy for simple cases, while the latter allows chaining data transformations into an easy-to-read pipeline.

bool is_even(int n) { return n % 2 == 0; }
const std::vector<int> vec = { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 };

// Function-style call:
for (int i : boost::adaptors::filter(vec, is_even)) { ... }

// Pipe-style call:
for (int i : vec | boost::adaptors::filtered(is_even)) { ... }

To see the power of the latter syntax, consider a transformation pipeline:

int square(int n) { return n*n; }
std::map<int, int> input_map = { ... };

using boost::adaptors;
auto result = input_map | map_values
                        | filtered(is_even)
                        | transformed(square);

indexed

The boost::adaptors::indexed() adapter warrants special mention: it is analogous to Python's enumerate(). Given a Range, it gives access to the elements as well as their index within the range. Boost 1.56 or higher is required for this to work properly.

accumulate

boost::accumulate() by default sums all the items in an input range, but other reduction functions can be supplied as well. Together with range adapters, this makes map-reduce pipelines easy to write.

std::vector<int> vec = {1, 2, 3, 4, 5};
int product = boost::accumulate(vec, 1, std::multiplies<int>());

as_literal

boost::as_literal() may be less a highlight and more of a crutch, but it bears mentioning still. Boost Range functions accept a wide variety of types, among them strings. C++ style strings (std::string) always work as expected, but with character arrays (char[]), there is an ambiguity as to whether the argument should be interpreted as array (including the terminal '\0' character) or as string (excluding the terminator).

By default, Boost Range treats them as arrays, which they are, after all. In practice, this is often a pitfall for newcomers. If any string-related range operations don't work as expected, this is a common reason.

To force the library to treat character arrays as strings, they can be wrapped in an as_literal() call. Alternatively, the C strings can be cast to std::string as well.

Summary

There are several interesting aspects about Boost Range. It plays very well with C++11's range-based for loops and makes code operating on containers much easier to write and (most importantly) read. In addition, it makes it possible to lay out data processing pipelines a lot more clearly.

Container iteration and modification becomes as easy as it is in modern scripting languages, which is a huge, huge step for the C++ language.

Let's hope that C++17 brings similar capabilities in the standard library. Until then, Boost Range is the way to go, so check out the docs and try it yourself.

Boost Range for Humans

2016-04-30T00:00:00+02:00

Boost Range encapsulates the common C++ pattern of passing begin/end iterators around by combining them into into a single range object. It makes code that operates on containers much more readable. One wonders why such functionality was not included in the C++ standard library in the first place, and indeed, similar ideas could be added to C++17, see N4128 and Ivan Cukic's Meeting C++ presentation. In my opinion, Boost Range is something that every C++ programmer should know about.

The library is reasonably well documented, but I was often missing concrete code examples and an explicit mention what headers are required for which function. Since this presumably happens to other people as well, I invested the time to change that situation.

Thus, I present Boost Range for Humans. It contains working example code for every function in Boost Range, along with required headers and links to the official documentation and the latest source code. I hope it will make Boost Range more accessible and furthers its adoption.

Next week, we'll look into some of the highlights of what Boost Range can offer.

Debugging riddle of the day

2016-04-23T00:00:00+02:00

One of our services failed to start on a test system (Ubuntu 12.04 on amd64). The stdout/stderr log streams contained only the string “Permission denied” – less than helpful. strace showed that the service tried to create a file under /run, which it doesn't have write permissions to. This caused the it to bail out:

open("/run/some_service", O_RDWR|O_CREAT|O_NOFOLLOW|O_CLOEXEC, 0644) = -1
    EACCES (Permission denied)

Grepping the source code and configuration files for /run didn't turn up anything that could explain this open() call. Debugging with gdb gave further hints:

Breakpoint 2, 0x00007ffff73e3ea0 in open64 () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) bt
#0  0x00007ffff73e3ea0 in open64 () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007ffff7bd69bf in shm_open () from /lib/x86_64-linux-gnu/librt.so.1
#2  0x0000000000400948 in daemonize () at service.cpp:93
#3  0x00000000004009ac in main () at main.cpp:24
(gdb) p (char*)$rdi
$1 = 0x7fffffffe550 "/run/some_service"
(gdb) frame 2
#2  0x0000000000400948 in daemonize () at service.cpp:93
9           int fd = shm_open(fname.c_str(), O_RDWR | O_CREAT, 0644);
(gdb) p fname
$2 = {...., _M_p = 0x602028 "/some_service"}}

The open("/run/some_service", ...) was caused by an shm_open("/some_service", ...).

This code is working on other machines, why does it fail on this particular one? Can you figure it out? Bonus points if you can explain why it is trying to access /run and not some other directory. You might find the shm_open() man page and source code helpful.

I'll be waiting for you.

Photo by Stefan Tell, CC-BY-2.0

The solution is pretty evident after examining the Linux version of shm_open(). By default, it tries to create shared memory files under /dev/shm. If that doesn't exist, it will pick the first tmpfs mount point from /proc/mounts.

In Ubuntu 12.04, /dev/shm is a symlink to /run/shm. On this machine the symlink was missing, which caused shm_open() to go hunting for a tmpfs filesystem, and /run happened to be the first one in /proc/mounts.

Re-creating the symlink solved the problem. Why it was missing in the first place is still unclear. In the aftermath, we're also improving the error messages in this part of the code to make such issues easier to diagnose.

libconf - a Python reader for libconfig files

2016-04-16T00:00:00+02:00

This weekend, I uploaded my first package to PyPI: libconf, a pure-Python reader for files in libconfig format. This configuration file format is reminiscent of JSON and is mostly used in C/C++ projects through the libconfig library. It looks like this:

version = 7;
window: {
   title: "libconfig example"
   position: { x: 375; y: 210; w: 800; h: 600; }
};
capabilities: {
   can-do-lists: (true, 0x3A20, ("sublist"), {subgroup: "ok"})
   can-do-arrays: [3, "yes", True]
};

There are already two Python implementations: pylibconfig2 is a pure-Python reader licensed under GPLv3 and python-libconfig provides bindings for the libconfig C++ library. The first one I didn't like because of it's licensing, the second one I didn't like because of the more involved installation procedure. Also, I kind of enjoy writing parsers.

So, I set down during the easter weekend and wrote libconf. It's a pure-Python reader for libconfig files with an interface similar to the Python json module. There are two main methods: load(f) and loads(string). Both return a dict-like data-structure that can be indexed (config['version']), but supports attribute access as well (config.version):

import libconf
>>> with open('example.cfg') as f:
...     config = libconf.load(f)
>>> config['window']['title']
'libconfig example'
>>> config.window.title
'libconfig example'

It was a fun little project. Creating a recursive descent parser is pretty straightforward, especially for such a simple file format. Writing documentation, packaging and uploading to GitHub and PyPI took longer than coding up the implementation itself.

Data access times

2016-04-06T00:00:00+02:00

Recently, I looked up how fast data can be accessed on modern computers. To remember this a bit better, I made a graphic that gives a nicer (to me) overview than the more common block diagrams:

Approximate data access times on 2010-2020 class hardware

I hope other people find it useful as well.

Requesting certificates the whirlwind way

2016-03-30T00:00:00+02:00

As a follow-up to last post, let's take a closer look at certificates. Specifically, when you'd want one and how you'd go about obtaining it.

When would you need a certificate?

The most common reason is to provide access to a service over a secure connection, e.g. HTTPS for web traffic or IMAPS for emails. This requires a certificate signed by a CA that's recognized by browsers and operating systems. Signatures by these CAs indicate control over the internet domain for which the certificate was issued (e.g. caichinger.com). They are used to prove to users that they are talking to the real thing, not to some scammer who hijacked their internet connection. This kind of certificate is often called "server certificate".

Instead of proving the identity of a server, certificates can also be used to authenticate a client. Unsurprisingly, this is called "client certificate" and can serve as a more secure alternative to password-based authentication. For example, Apple Push Notifications (for sending notifications to apps on an iPhone) require a certificate signed by Apple to prove that someone is allowed to send push notifications. Several VPN technologies also rely on client certificates for authentication.

How do you go about obtaining a certificate?

You need to create a certificate signing request (CSR) and submit it to the CA. Last time we saw that a certificate consists of a keypair identifier (hash), metadata, and signatures. A CSR contains only the first two of these: a hash that uniquely identifies a keypair and metadata relating to the intended use of the certificate.

Obviously, this means there can be no CSR without a keypair!¹

For most purposes, it's best to start with a new keypair that hasn't been used anywhere else. So we'll first generate that and then create a CSR for it.

Those are two pretty simple operations, they just get complicated by the horrific user interface of OpenSSL. For most people, these commands equate to mystical incantations that must be recited to work their magic. That's unfortunate: it just deepens the confusion about how this certificate shebang is supposed to work. OpenSSL is the most widely available and popular tool for the job, though, so for better or worse we'll have to put up with it.

On Linux, OpenSSL should be available from the package manager, if it isn't installed already. On MacOS, OpenSSL is probably pre-installed, if not, it can be obtained from Homebrew. Windows binaries can be found here.

Creating a CSR

First, generate a keypair:

openssl genrsa -out certtest.key 2048

This will write a new keypair to the file "certtest.key". The number 2048 is the desired key size, with larger keys being more secure but also slower. As of 2016, 2048 bits is a reasonable default.

Next create a CSR for that keypair:

openssl req -new -sha256 -key certtest.key -out certtest.csr

OpenSSL will ask some questions to set the CSR metadata. For server certificates, the crucial field is "Common Name", which has to be the internet address of the server the certificate is intended for. The other fields are for informational purposes only. You may set them, or you may write a single dot (".") to leave the field blank. Not typing anything at all at the question prompts will give you OpenSSL's default values, don't do that.

Here is a screen capture of the process:

Screen capture of creating a CSR - it's super simple!

Before submitting the CSR for signing, it's a good practice to re-check the metadata fields:

openssl req -noout -text -in certtest.csr

This outputs several lines of data, the important part is the "Subject" line at the very top. Check that all the fields are what you expect them to be.

Submitting the CSR for signing

The details of submitting a CSR for signing vary from CA to CA, typically it involves an upload on a web page. Depending on the intended use of the certificate, additional steps may be required, such as proving your real-life identity and/or proving that you have control over some domain (for server certificates). Fortunately, these steps are typically quite thoroughly described in the CA's documentation or on the internet.

After the submission, it may take some time for the certificate to be signed, then you can download it from the CA's web page.

Pretty simple, right?

¹ For users of the Apple Keychain application this can be a bit confusing, because there is an assistant to generate CSR's, but it doesn't mention keypairs anywhere. Under the hood, it does create a new keypair and add it to the keychain, it just doesn't tell you.

A whirlwind introduction to the secure web

2016-03-23T00:00:00+01:00

How secure internet connections work is often a mystery, even for fairly technical people – let's rectify this!

Although some fairly complicated mathematics lays the foundation, it's not necessary to grasp all of it to get a good high-level understanding of the secure web. In this post, I'll give an overview of the major components of the secure web and of how they interact. I want to shed some light on the wizardry that both your browser and webservers do to make secure communication over the internet possible. Specifically, the focus is on authentication: how the public key infrastructure (PKI) guarantees you that you are really talking to your bank not to some scammer.

This is going to be a high-level overview with a lot of handwaving. We'll gloss over some of the nitty gritty details, and focus on the big picture.

Keypairs

Let's start off with the basic building block of public-key cryptography, the public/private keypair. It consists of two halves: the public part can be shared with the world, the private half must stay hidden, for your eyes only. With such a keypair, you can do three pretty nifty things:

People can encrypt data with your public key, and only you can decrypt it.
You can prove to other people (who have your public key) that you know the private key. In some sense, they can verify your identity.
You can sign data with your private key, and everyone with the public key can verify that this data came from you and was not tempered with.

Let's look at an example to see how cool this is: let's say your bank has a keypair and you know their public key (it could be printed in large letters on the walls of their building, it's public after all). You can then securely communicate with your bank, without anyone being able to listen in, even if they can intercept or modify the messages between you and the bank (encryption). You can be sure you really are talking to the bank, and not to a fraudster (verification). And the bank can make statements that anyone who has their public key can ascertain is genuine, i.e. it really came from the bank (signing).

The last part is nice because the bank can sign a statement about your balance, give it to you, and you can forward it to your sleazy landlord who wants proof of your financial situation. The bank and the landlord never directly talk with each other, nevertheless the latter has full certainty that the statement was made by the bank, and that you didn't tamper with it.

So it's cool that we can securely communicate with our banks. We can do the same with websites: once we have the public key of e.g. Google, it's easy to setup an encrypted communication channel. Via the verification function of keypairs, it's also easy to prove we really are talking to that Google, and not to some kid who's trying to steal our password to post dog pictures in our cat groups.

How do we get Google's public key? — This is where things start going downhill.

In the bank example, we'd gotten the public key personally from the bank (written on its front wall). With Google, it'd be kind of difficult to travel to Mountain View just to get get their public key. And we can't just go and download the key from google.com, the whole point is that we're not sure that the google.com we're talking to is the real Google.

Are we completely out of luck? Can we communicate securely over the internet only if we manually exchange keys before, which we usually can't? It turns out we are only sort-of out of luck: we are stuck with certificates and the halfway-broken system of certificate authorities.

Certificates

A certificate contains several parts:

an identifier (a hash) that uniquely identifies a keypair
metadata that says who this keypair belongs to
signatures: statements signed by other keys that vouch that the keypair referenced here really belongs to the entity described in the metadata section

It's important to note that certificates don't need to be kept secret. The keypair identifier doesn't reveal the private key, so certificates can be shared freely. The corollary of this is that a certificate alone can't be used to verify you're talking to anyone in particular. To be used for authentification, it needs to be paired with the associated private key.

With that out of the way, let's look at why certificates are useful. Say someone gives you a certificate and proves they have the associated private key. You've never met this person. However, the certificate carries signatures from several keys that you know belong to close friends of yours. All of those signatures attest that this person is called "Hari Seldon". If you trust your friends, you can be pretty certain that the person is really called that way.

When you think about this, it's kind of neat. A stranger can authenticate to you (prove that they say who they are) just because someone you trust made a statement confirming the stranger's identity. That this statement is really coming from your trusted friend is ensured, because it's signed with their private key.

The same concept can be applied websites. As long as there's someone you trust and you have their public key, they can sign other people's certificates to affirm that identity to you. For example, they can sign a certificate for Google that says "This really is the real google.com". When you see that certificate and verify that the other party has the associated private key, you'll have good reason to believe that you really are talking to Google's google.com server, not some scam version by a North Korean hacker.

Certificate Authorities

So how do you find someone you can trust? And how does that person make sure that the certificate they are signing really belongs to Google? They face the same problems confirming that fact as you did! Does this even improve the situation in any way?

It does – let's take the questions in order. The reality on the internet is: it's not you trusting someone, it's your browser that does the trusting. Your browser includes public keys from so-called "certificate authorities" (CA's). You can find the list of CA's trusted by your own browser in its options, under Advanced / Security / Certificates / Authorities. If the browser sees certificates signed by any one of these keys, it believes them to be true. It trusts CA's not to sign any bogus certificates.

Why are these keys trustworthy? Because CA's are mostly operated by large companies that have strict policies in place to make sure they only sign stuff that's legit. How do they do that? After all, as an individual you'd have a pretty tough time verifying that the public key offered by google.com really belongs to Google. Don't CA's face the same problem?

Not really. There are billions of people accessing google.com. There are only about 200 CA's that are trusted by the common browsers. And Google needs a signed certificate by only one of them (one signature is enough to earn the browser's trust). So Google can afford to prove it's identity to a CA: by sending written letters, a team of lawyers, or whatever. Once Google gets a certificate for google.com signed by any reputable CA, it is recognized by pretty much every device in the world.

Similarly, I, as a private person, can get a certificate for caichinger.com by proving my identity and my ownership of this domain to the CA. The identity part is usually done by submitting a scan of a driver's license or passport. Ownership of the domain can be shown by uploading a file supplied by the CA to the webserver. Once the CA confirms that the file is there, it knows I have control of that domain.

So instead of me having to prove my identity to every single user visiting this website, I can prove it once to a CA, and all browsers coming here will recognize this as good enough. This way, CA's make the problem of authentication of servers ("I'm the real google.com, not a cheap fake") tractable. It's a system that has made the large-scale deployment of secure internet traffic via HTTPS possible.

The half-broken part

Let's get back to the analogy of a stranger authenticating to you via a certificate signed by someone you know. What if the signature wasn't from a close friend of yours, but from a seedy guy you meet occasionally when going out? Would you still have full confidence in the certificate? Hopefully not.

What does this mean for the web?

Not all of the CA's included in the common web browsers are the equivalent of a trusted friend:

They may be in control of some government who wants a certificate for gmail.com, so it can read dissident's emails
An employee with access to the certificate authority key may create certificates for bank websites and sell them on the black market
The computer network where the CA keys are stored could have been hacked

I'm pretty sure all three of those have actually happened in the past. Given that a single forged certificate can be used to attack millions of users, CA's are juicy targets. As soon as forged certificates are detected in the wild, they tend to be blacklisted (blocked by browsers) very quickly, but there is still a window of vulnerability.

For this reason, the whole CA system has been questioned over the last few years, but replacing it does not seem feasible at the moment. There are techniques (such as public key pinning) to augment the CA-based authentication, but it takes time for them to be picked up by website owners.

While this is a problem, it mostly affects the largest websites (obtaining a forged certificate is difficult and costly). Together with browser vendors, they are developing new mitigation techniques against forged certificates. In the meantime, the rest of us is still pretty well served by the current CA system, even though it is not perfect.

Wrapup

So, this is it for an overview of the public key infrastructure that enables secure connections to internet sites, from the basics of public key cryptography to certificate authorities. If you want to dig deeper, I recommend starting with the Wikipedia articles I linked throughout the article. If you are interested in cryptography in general, I highly recommend Bruce Schneier's book Applied Cryptography. It's 20 years old now, and still enormously relevant today.

I hope this text helps a bit to clear up the confusion associated with public key cryptography and the secure web. If you liked it, or if you have any suggestions for improvement, please let me know in the comments!

dh_virtualenv and long package names (FileNotFound error)

2016-03-01T00:00:00+01:00

Interesting tidbit about Linux:

A maximum line length of 127 characters is allowed for the first line in a #! executable shell script.

from man execve

Should be enough, right? Wrong.

Well, not if you are using dh_virtualenv with long package names, anyway:

Installing pip...
  Error [Errno 2] No such file or directory while executing command /tmp/lo...me/bin/easy_install /usr/share/python-virtualenv/pip-1.1.tar.gz
...Installing pip...done.
Traceback (most recent call last):
  File "/usr/bin/virtualenv", line 3, in <module>
virtualenv.main()
  File "/usr/lib/python2.7/dist-packages/virtualenv.py", line 938, in main
never_download=options.never_download)
  File "/usr/lib/python2.7/dist-packages/virtualenv.py", line 1054, in create_environment
install_pip(py_executable, search_dirs=search_dirs, never_download=never_download)
  File "/usr/lib/python2.7/dist-packages/virtualenv.py", line 643, in install_pip
filter_stdout=_filter_setup)
  File "/usr/lib/python2.7/dist-packages/virtualenv.py", line 976, in call_subprocess
cwd=cwd, env=env)
  File "/usr/lib/python2.7/subprocess.py", line 679, in __init__
errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1249, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
Traceback (most recent call last):
  File "/usr/bin/dh_virtualenv", line 106, in <module>
sys.exit(main() or 0)
  File "/usr/bin/dh_virtualenv", line 83, in main
deploy.create_virtualenv()
  File "/usr/lib/python2.7/dist-packages/dh_virtualenv/deployment.py", line 112, in create_virtualenv
subprocess.check_call(virtualenv)
  File "/usr/lib/python2.7/subprocess.py", line 511, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['virtualenv', '--system-site-packages', '--setuptools', 'debian/long-package-name/usr/share/python/long-package-name']' returned non-zero exit status 1
make: *** [binary-arch] Error 1
dpkg-buildpackage: error: fakeroot debian/rules binary gave error exit status 2

dh_virtualenv is used to create Debian packages that include Python virtualenvs. It is one of the better ways of packaging Python software, especially if there are Python dependencies that are not available in Debian or Ubuntu. When building a .deb package, it creates a virtualenv in a location such as:

/<build-directory>/debian/<packagename>/usr/share/python/<packagename>

This virtualenv has several tools under its bin/ directory, and they all have the absolute path of the virtualenv's Python interpreter hard-coded in their #! shebang line:

#!/<build-directory>/debian/<packagename>/usr/share/python/<packagename>/bin/python

Given that <build-directory> often contains the package name as well, it's easy to overflow the 128 byte limit of the #! shebang line. In my case, with a ~30 character package name, the path length grew to 160 characters!

Consequently, the kernel couldn't find the Python executable anymore, and running any of the tools from the bin/ directory gave an ENOENT (file not found) error. This is what happened when virtualenv tried to install pip during the initial setup. The root cause of this error is not immediately obvious, to say the least.

To check whether this affects you, check the line length of any script with wc:

head -n 1 /path/to/virtualenv/bin/easy_install | wc -c

If that's larger than 128, it's probably the cause of the problem.

The fix is to change the package name and/or the build location to something shorter. The alternative would be to patch the Linux kernel, which – depending on your preferences – sounds either fun or really unpleasant. Suit yourself!

Behold the power of perf-tools

2016-02-10T00:00:00+01:00

perf-tools is a collection of scripts for system-wide tracing on Linux. It's really, really cool. It's what the perf command should have included from day one, but didn't.

It is packaged in Debian and Ubuntu, but those versions miss some key features. As perf-tools consists of shell scripts (no compilation necessary), I recommend using the GitHub version directly:

git clone https://github.com/brendangregg/perf-tools.git

Two tools that are included are execsnoop and opensnoop, which trace new program executions and open() calls across the whole system.

$ sudo ./execsnoop
TIME        PID   PPID ARGS
21:12:56  22898  15674 ls --color=auto -la
21:12:56  22899  15674 git rev-parse --is-inside-work-tree
21:12:56  22900  15674 git rev-parse --git-dir
...

$ sudo ./opensnoop
Tracing open()s. Ctrl-C to end.
COMM             PID      FD FILE
opensnoop        22924   0x3 /etc/ld.so.cache
gawk             22924   0x3 /usr/lib/locale/locale-archive
top              15555   0x8 /proc/1/stat
...

Maybe the most interesting tool is uprobe. It's magic: it traces function calls in arbitrary user-space programs. With debugging symbols available, it can trace practically every function in a program. Without them, it can trace exported functions or arbitrary code locations (specified by raw address). It can also trace library code, e.g. libc). Having these possibilities on a production system without any prior setup is staggering.

$ sudo user/uprobe -F -l /tmp/a.out | grep quicksort
_Z9quicksortN9__gnu_cxx17__normal_iteratorIPiSt6vectorIiSaIiEEEES5_
$ sudo user/uprobe -F p:/tmp/a.out:_Z9quicksortN9__gnu_cxx17__normal_iteratorIPiSt6vectorIiSaIiEEEES5_
Tracing uprobe _Z9quicksort[snip] (p:_Z9quicksort[snip] /tmp/a.out:0x8ba). Ctrl-C to end.
   a.out-23171 [000] d... 1860355.891238: _Z9quicksort[snip]: (0x80488ba)
   a.out-23171 [000] d... 1860355.891353: _Z9quicksort[snip]: (0x80488ba)
   ...

(To demangle the C++ function names, use the c++filt tool.)

perf-tools really shows the power of the Linux perf/ftrace infrastructure, and make it usable for the broad masses. There are several other tools that analyze latency and cache hit rates, trace kernel functions, and much more. To finally have such functionality in Linux is fabulous!

Running strace for multiple processes

2016-02-09T00:00:00+01:00

Just a quick note about strace, the ancient Linux system-call tracing tool.

It can trace multiple processes at once: simply start it with multiple -p arguments (the numbers give the processes' PIDs):

sudo strace -p 2916 -p 2929 -p 2930 -p 2931 -p 2932 -o /tmp/strace.log

This is great for tracing daemons which use separate worker processes, for example Apache with mpm-prefork.

Plotting maps with Folium

2016-01-27T00:00:00+01:00

Data visualization in Python is a well solved problem by now. Matplotlib and it's prettier cousin Seaborn are widely used to generate static graphs. Bokeh generates HTML files with interactive, JavaScript-based graphs. It's a great way of sharing data with other people who don't have a Python development environment ready. Several other libraries exist for more specialized purposes.

What has been missing for a long time was good map libraries. Plotting capabilities were fine, but basemap support of the existing libraries was very limited. For example, the popular Matplotlib-basemap has great plot types (contour maps, heatmaps, ...) but can't show any high-resolution maps: it only has country/state shapes or whole-world images. Consequently, it's useless for drawing city or street level maps, unless you want to set up your own tile server (you don't).

Along comes Folium, a library that generates interactive maps in HTML format based on Leaflet.js. It supports, among others, OpenStreetMap and MapBox base layers which look great and provide enough details for large-scale maps.

Here is an example that shows some GPS data I cleaned up with a Kalman filter:

def plot(points, center):
    map_osm = folium.Map(location=center, zoom_start=16, max_zoom=23)
    map_osm.line(locations=points)
    map_osm.create_map(path='folium-example.html')

Here's what it looks like. I find it pretty neat, especially given that it took only 3 lines of code to create:

Mixing C runtime library versions

2016-01-01T00:00:00+01:00

When compiling code with Microsoft's Visual C/C++ compiler, a dependency on Microsoft's C runtime library (CRT) is introduced. The CRT can be linked either statically or dynamically, and comes in several versions (different version numbers as well as in debug and release variants).

Complications arise when libraries linked into the same program use different CRTs. This happens if they were compiled with different compiler versions, or with different compiler flags (static/dynamic CRT linkage or release/debug switches). In theory, this could be made to work, but in practice it is asking for trouble:

If the CRT versions differ (either version number or debug/release flag), you can't reliably share objects generated by CRT A with any code that uses CRT B. The reason is that the two CRTs may have a different memory layout (structure layout) for the that object. The memory location that CRT A wrote the object size to might be interpreted by CRT B as a pointer, leading to a crash when CRT B tries to access that memory.
Even if the same CRT version is included twice (once statically, once dynamically linked), they won't share a heap. Both CRTs track the memory they allocate individually, but they don't know anything about objects allocated by the other CRT. If CRT A tries to free memory allocated by CRT B, that causes heap corruption as the two CRTs trample on each others feet. While you can freely share objects between CRTs, you have to be careful whenever memory is allocated or freed. This can sometimes be managed when writing C code, but is very hard to do correctly in C++ (where e.g. pushing to a vector can cause a reallocation of its internal buffer).

Accordingly, having multiple CRTs in the same process is fragile at best. When mixing CRTs, there are no tools to check whether objects are shared in a way that's problematic, and manual tracking is subtle, easy to get wrong, and unreliable. Mistakes will lead to difficult-to-diagnose bugs and intermittent crashes, generally at the most inconvenient times.

To keep your sanity, ensure that all code going into your program uses the same CRT.¹ Consequently, all program code, as well as all libraries, need to be compiled from scratch using the same runtime library options (/MD or /MT). Pre-compiled libraries are a major headache, because they force the use of a specific compiler version to get matching CRT version requirements. If multiple pre-compiled libraries use different CRT versions, there may not be any viable solution at all.

This situation will improve with the runtime library refactoring in VS2015, which promises CRT compatibility to subsequent compiler versions. Thus, this inconvenience should mostly be solved in the future.

¹ Dependency Walker can be used to list all dynamically linked CRT versions. I'm not sure whether that can be done for statically linked CRTs, I generally avoid those.

Debian's most annoying warning message: "Setting locale failed"

2015-12-03T00:00:00+01:00

You ssh into a server or you enter a chroot, and the console overflows with these messages:

perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
        LANGUAGE = "en_US:en",
        LC_ALL = (unset),
        LC_TIME = "en_US.utf8",
        LC_CTYPE = "de_AT.UTF-8",
        LC_COLLATE = "C",
        LC_MESSAGES = "en_US.utf8",
        LANG = "de_AT.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory

Gaaaaarrrrh!

Fortunately the fix is easy (adjust the locale names for your situation):

locale-gen en_US.UTF-8 de_AT.UTF-8

Finally, peace on the console.

Profiling

2015-11-28T00:00:00+01:00

Profiling is hard. Measuring the right metric and correctly interpreting the obtained data can be difficult even for relatively simple programs.

For performance optimization, I'm a big fan of the poor man's profiler: run the binary to analyze under a debugger, periodically stop the execution, get a backtrace and continue. After doing this a few times, the hotspots will become apparent. This works amazingly well in practice and gives a reliable picture of where time is spent, without the danger of skewed results from instrumentation overhead.

Sometimes it's nice to get a more fine-grained view. That is, not only find the hotspot, but get an overview how much time is spent where. That's where 'real' profilers come in handy.

Under Windows, I like the built-in "Event Tracing for Windows" (ETW), which produces files that can be analyzed with Xperf/Windows Performance Analyzer. It is a really well thought out system, and the Xperf UI is amazing in the analyzing abilities that it offers. Probably the best place to start reading up on this is ETW Central.

Under Linux, I haven't found a profiler I can really recommend, yet. gprof and sprof are both ancient and have severe limitations. OProfile may be nice, but I haven't had a chance to use it yet, as it wasn't available for my Ubuntu LTS release.

I have used Callgrind from the Valgrind toolkit in combination with the KCachegrind GUI analyzer. I typically invoke it like this:

valgrind --tool=callgrind --callgrind-out-file=callgrind-cpu.out ./program-to-profile
kcachegrind callgrind-cpu.out

Callgrind works by instrumenting the binary under test. It slows down program execution, often by a factor of 10. Further, it only measures CPU time, so sleeping times are not included. This makes it unsuitable for programs that wait a significant amount of time for network or disk operations to complete. Despite these drawbacks, it's pretty handy if CPU time is all that you're interested in.

If blocking times are important (as they are for so many modern applications - we generally spend less time computing and more time communicating), gperftools is a decent choice. It includes a CPU profiler that can be run in real-time sampling mode, and the results can viewed in KCachegrind. It is recommended to compile libprofiler.so into the binary to analyze, but using LD_PRELOAD works decently well:

CPUPROFILE_REALTIME=1 CPUPROFILE=prof.out LD_PRELOAD=/usr/lib/libprofiler.so ./program-to-profile
google-pprof --callgrind ./program_to_profile prof.out > callgrind-wallclock.out
kcachegrind callgrind-wallclock.out

If it works, this gives a good overall profile of the application. Unfortunately, it sometimes fails: on amd64, there are sporadic crashes from within libunwind. It's possible to just ignore those and rerun the profile, at least interesting data is obtained 50% of the time.

The more serious problem is that CPUPROFILE_REALTIME=1 causes gperftools to use SIGALARM internally, conflicting with any applications that want to use that signal for themselves. Looking at the profiler source code, it should be possible to work around this limitation with the undocumented CPUPROFILE_PER_THREAD_TIMERS and CPUPROFILE_TIMER_SIGNAL environment variables, but I couldn't get that to work yet.

You'd think that perf has something to offer in this area as well. Indeed, it has a CPU profiling mode (with nice flamegraph visualizations) and a sleeping time profiling mode, but I couldn't find a way to combine the two to get a real-time profile.

Overall, there still seems to be room for a good, reliable real-time sampling profiler under Linux. If I'm missing something, please let me know!

Returning generators from with statements

2015-11-12T00:00:00+01:00

Recently, an interesting issue came up at work that involved a subtle interaction between context managers and generator functions. Here is some example code demonstrating the problem:

@contextlib.contextmanager
def resource():
    """Context manager for some resource"""

    print("Resource setup")
    yield
    print("Resource teardown")


def _load_values():
    """Load a list of values (requires resource to be held)"""

    for i in range(3):
        print("Generating value %d" % i)
        yield i


def load_values():
    """Load values while holding the required resource"""

    with resource():
        return _load_values()

This is the output when run:

>>> for val in load_values(): pass
Resource setup
Resource teardown
Generating value 0
Generating value 1
Generating value 2

Whoops. The resource is destroyed before the values are actually generated. This is obviously a problem if the generator depends on the existence of the resource.

When you think about it, it's pretty clear what's going on. Calling _load_values() produces a generator object, whose code is only executed when values are requested. load_values() returns that generator, exiting the with statement and leading to the destruction of the resource. When the outer for loop (for val) comes around to iterating over the generator, the resource is long gone.

How do you solve this problem? In Python 3.3 and newer, you can use the yield from syntax to turn load_values() into a generator as well. The execution of load_values() is halted at the yield from point until the child generator is exhausted, at which point it is safe to dispose of the resource:

def load_values():
    """Load values while holding the required resource"""

    with resource():
        yield from _load_values()

In older Python versions, an explicit for loop over the child generator is required:

def load_values():
    """Load values while holding the required resource"""

    with resource():
        for val in _load_values():
            yield val

Still another method would be to turn the result of _load_values() into a list and returning that instead. This incurs higher memory overhead since all values have to be held in memory at the same time, so it's only appropriate for relatively short lists.

To sum up, it's a bad idea to return generators from under with statements. While it's not terribly confusing what's going on, it's a whee bit subtle and not many people think about this until they ran into the issue. Hope this heads-up helps.

A better way for deleting Docker images and containers

2015-06-11T00:00:00+02:00

In one of my last posts, I described the current (sad) state of managing Docker container and image expiration. Briefly, Docker creates new containers and images for many tasks, but there is no good way to automatically remove them. The best practice seems to be a rather hack-ish bash one-liner.

Since this wasn't particularly satisfying, I decided to do something about it. Here, I present docker-cleanup, a Python application for removing containers and images based on a configurable set of rules.

This is a rules file example:

# Keep currently running containers, delete others if they last finished
# more than a week ago.
KEEP CONTAINER IF Container.State.Running;
DELETE CONTAINER IF Container.State.FinishedAt.before('1 week ago');

# Delete dangling (unnamed and not used by containers) images.
DELETE IMAGE IF Image.Dangling;

Clear, expressive, straight-forward. The rule language can do a whole lot more and provides a readable and intuitive way to define removal policies for images and containers.

Head over to GitHub, give it a try, and let me know what you think!

Using Python slice objects for fun and profit

2015-06-04T00:00:00+02:00

Just a quick tip about the hardly known slice objects in Python. They are used to implement the slicing syntax for sequence types (lists, strings):

s = "The quick brown fox jumps over the lazy dog"

# s[4:9] is internally converted (and equivalent) to s[slice(4, 9)].
assert s[4:9] == s[slice(4, 9)]

# 'Not present' is encoded as 'None'
assert s[20:] == s[slice(20, None)]

slice object can be used in normal code too, for example for tracking regions in strings: instead of having separate start_idx and end_idx variables (or writing a custom class/namedtuple) simply roll the indices into a slice.

# A column-aligned table:
table = ('REPOSITORY   TAG      IMAGE ID       CREATED       VIRTUAL SIZE',
         '<none>       <none>   0987654321AB   2 hours ago   385.8 MB',
         'chris/web    latest   0123456789AB   2 hours ago   385.8 MB',
        )
header, *entries = table

# Compute the column slices by parsing the header. Gives a list of slices.
slices = find_column_slices(header)

for entry in entries:
    repo, tag, id, created, size = [entry[sl].strip() for sl in slices]
    ...

This is mostly useful when the indices are computed at runtime and applied to more than one string.

More generally, slice objects encapsulate regions of strings/lists/tuples, and are an appropriate tool for simplifying code that operates on start/end indices. They provide a clean abstraction, make the code more straight-forward and save a bit of typing.

A neat Python debugger command

2015-05-28T00:00:00+02:00

pdb is a console-mode debugger built into Python. Out of the box, it has basic features like variable inspection, breakpoints, and stack frame walking, but it lacks more advanced capabilities.

Fortunately, it can be customized with a .pdbrc file in the user's home directory. Ned Batchelder has several helpful commands in his .pdbrc file:

pl: print local variables
pi obj: print the instance variables of obj
ps: print the instance variables of self

Printing instance variables is great for quickly inspecting objects, but it shows only one half of the picture. What about the class-side of objects? Properties and methods are crucial for understanding what can actually be done with an object, in contrast to what data it encapsulates.

Since I couldn't find a readily available pdb command for listing class contents, I wrote my own:

# Print contents of an object's class (including bases).
alias pc for k,v in sorted({k:v for cls in reversed(%1.__class__.__mro__) for k,v in cls.__dict__.items() if cls is not object}.items()): print("%s%-20s= %-80.80s" % ("%1.",k,repr(v)))

pc lists the contents of an object's class and its base classes. Typically, these are the properties and methods supported by the object. It is used like this:

# 'proc' is a multiprocessing.Process() instance.
(Pdb) pc proc
...
proc.daemon              = <property object at 0x036B9A20>
proc.exitcode            = <property object at 0x036B99C0>
proc.ident               = <property object at 0x036B9A50>
proc.is_alive            = <function BaseProcess.is_alive at 0x033E4618>
proc.join                = <function BaseProcess.join at 0x033E45D0>
proc.name                = <property object at 0x036B99F0>
proc.pid                 = <property object at 0x036B9A50>
proc.run                 = <function BaseProcess.run at 0x033E4A98>
proc.start               = <function BaseProcess.start at 0x033E4DB0>
proc.terminate           = <function BaseProcess.terminate at 0x033E4DF8>

Note the difference to pi, which lists the contents of the proc instance:

(Pdb) pi proc       # In contrast, here is the image dictionary.
proc._args          = ()
proc._config        = {'authkey': b'\xd0\xc8\xbd\xd6\xcf\x7fo\xab\x19_A6\xf8M\xd4\xef\x88\xa9;\x99c\x9
proc._identity      = (2,)
proc._kwargs        = {}
proc._name          = 'Process-2'
proc._parent_pid    = 1308
proc._popen         = None
proc._target        = None

In general, pc focuses on the interface while pi examines the state of the object. The two complement each other nicely. Especially when working with an unfamiliar codebase, pc is helpful for quickly figuring out how to use a specific class.

pc works with both Python 2 and Python 3 (on Python 2 it only shows new-style classes). Add it to your .pdbrc and give it a try. Let me know what you think!

Python's GIL and atomic reference counting

2015-05-23T00:00:00+02:00

I love Python because it's an incredibly fun, expressive and productive language. However, it's often criticised for being slow. I think the correct answer to that is two-fold:

Use an alternative Python implementation for impressive speed-ups. For example, PyPy is on average 7 times faster than the standard CPython.
Not all parts of a program have to be blazingly fast. Use Python for all non-performance-critical areas, such as the UI and database access, and drop to C or C++ only when it's required for for CPU intensive tasks. This is easy to achieve with language binding generators such as CFFI, SWIG or SIP.

A further performance-related issue is Python's Global Interpreter Lock (GIL), which ensures that only one Python thread can run at a single time. This is a bit problematic, because it affects PyPy as well, unless you want to use its experimental software transactional memory support.

Why is this such a big deal? With the rise of multi-core processors, multithreading is becoming more important as well. This not only affects performance on large servers, it impacts desktop programs and is crucial for battery life on mobile phones (race to idle). Further, other programming languages make multi-threaded programming easier and easier. C, C++, and Java have all moved to a common memory model for multithreading. C++ has gained std::atomic, futures, and first-class thread support. C# has async and await, which is proposed for inclusion in C++ as well. This trend will only accelerate in the future.

With this in mind, I decided to investigate the CPython GIL. Previous proposals for its removal have failed, but I thought it's worth a look — especially since I couldn't find any recent attempts.

The results were not encouraging. Changing the reference count used by Python objects from a normal int to an atomic type resulted in a ~23% slowdown on my machine. This is without actually changing any of the locking. This penalty could be moderated for single-threaded programs by only using atomic instructions once a second thread is started. This requires a function call and an if statement to check whether to use atomics or not in the refcount hot path. Doing this still results in an 11% slowdown in the single-threaded case. If hot-patching was used instead of the if, a 6% slowdown remained.

Atomic refcounts slow down the overall Python execution speed by >20%.

The last result looks promising, but is deceiving. Hot-patching would rely on having a single function to patch. Alas, the compiler decided to mostly inline the Py_INCREF/Py_DECREF function calls. Disabling inlining of these functions gives a 16% slowdown, which is worse than the "call + if" method. Furthermore, hot-patching is probably not something that could be merged in CPython anyway.

So what's the conclusion? Maybe turning Py_INCREF and Py_DECREF into functions and living with the 11% slowdown of the single-threaded case would be sell-able, if compelling speed-ups of multithreaded workloads could be shown. It should be possible to convert one module at a time from the GIL to fine-grained locking, but performance increases would only be expected once at least a couple of core modules are converted. That would take a substantial amount of work, especially given the high risk that the resulting patches wouldn't be accepted upstream.

Where does this leave Python as a language in the multi-threaded world? I'm not sure. Since PyPy is already the solution to Python's performance issue, perhaps it can solve the concurrency problem as well with its software transactional memory mode.

PS: My profiling showed that the reference counting in Python (Py_INCREF() and Py_DECREF()) takes up to about 5–10% of the execution time of benchmarks (not including actual object destruction), crazy!

Subclassing C++ in Python with SIP

2015-05-17T00:00:00+02:00

I use SIP in MapsEvolved to generate bindings for interfacing Python with C++. I really like SIP due to its straight-forward syntax that mostly allows just copying class definitions over from C++. Further, it's really well thought out and contains support for a number of advanced use cases.

One such feature is implementing a C++ interface in Python. The resulting class can then even be passed back to C++, and any methods called on it will be forwarded to the Python implementation. Sweet!

Here is an example I originally wrote for this Stack Overflow question. It illustrates how ridiculously easy it is to get this working:

visitor.h:

class EXPORT Node {
public:
    int getN() const;
    ...
};
struct EXPORT NodeVisitor {
    virtual void OnNode(Node *n) = 0;
};
void visit_graph_nodes(NodeVisitor *nv);

visitor.sip:

%Module pyvisit

%ModuleHeaderCode
#include "visitor.h"
%End

class Node {
public:
    int getN() const;
    ...
};

struct NodeVisitor {
    virtual void OnNode(Node* n) = 0;
};

void visit_graph_nodes(NodeVisitor *nv);

Using it from Python:

>>> import pyvisit
>>> class PyNodeVisitor(pyvisit.NodeVisitor):
>>>     def OnNode(self, node):
>>>         print(node.getN())
>>> pnv = PyNodeVisitor()
>>> visit_graph_nodes(pnv)
1
2
3
...

Here, the C++ function visit_graph_nodes() calls the Python method pnv.OnNode() for every node in its (internal) graph. A zip file with the full working source code of this example can be downloaded here.

The subclassing capabilities of SIP don't stop at interfaces, either. It's possible to derive from any C++ class, abstract or not, inheriting (or overriding) existing method implementations as needed. This gives a lot of flexibility and makes it easy to have classes with some parts implemented in C++, and others being in Python.

Configuring SSL on Apache 2.4

2015-05-11T00:00:00+02:00

Configuring a modern web server to employ strong encryption and forward secrecy doesn't have to be hard. There is excellent documentation from Mozilla and from the OWASP.

Obtaining an SSL certificate

One major stumbling block is where to obtain an SSL certificate. In the future, this should hopefully be easy with Let's Encrypt. Until that is actually functional, StartSSL offers free SSL certificates. The process takes a bit of patience, but it's not difficult. There's also a StartSSL HOWTO from h-online.com.

While I've used StartSSL in the past, I had some trouble with them because 10 years after I registered greek0.net, someone grabbed greekO.net and StartSSL was alleging I was trying to mislead users?! So that was the end of my business with them...

I've now switched to Comodo's Positive SSL Certificate, which I like for a couple of reasons:

it lasts for 3 years,
it's really uncomplicated, and
it's crazy cheap: 7.45$ per year.

The process of getting the cert from them was super easy, simpler than StartSSL. About 3 hours from going to their website to having the certificate installed on my server, with most of it waiting email verifications. Credit card payment was quick and easy. 10/10, would buy again :-)

Apache configuration

With the certificate acquisition out of the way, here are the juicy bits from my Apache config.

mod_ssl config:

# Enable only cyphers that support forward secrecy.
# See these two links for reference:
# https://stackoverflow.com/questions/17308690
# https://wiki.mozilla.org/Security/Server_Side_TLS#Non-Backward_Compatible_Ciphersuite
SSLCipherSuite ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!3DES:!MD5:!PSK

# Use server priorities for cipher algorithm choice.
SSLHonorCipherOrder on

# With Apache 2.4, SSLv2 is gone and only SSLv3 and TLSv* are supported.
# Disable SSLv3, all TLS protocols are OK.
SSLProtocol all -SSLv3

# Enable OCSP stapling
# With this, the client can verify that our certificate isn't revoked
# without having to query an external OCSP service.
SSLUseStapling On
SSLStaplingCache shmcb:${APACHE_RUN_DIR}/ssl_stapling(32768)

The per-site configuration:

SSLEngine On
SSLCertificateKeyFile   /path/to/serverkey.key    # The private server key.
SSLCertificateFile      /path/to/certificate.crt  # The certificate provided by CA.
SSLCertificateChainFile /path/to/cert-bundle      # A separate download from your CA.


# Use a customized prime group for DH key exchange (vs Logjam attack).
# Generate a DH group file with:
#    openssl dhparam -out dhparams.pem 2048
#
# Newer Apache versions support the following command to set the dhparams:
SSLOpenSSLConfCmd DHParameters "/path/to/dhparams.pem"

# If Apache reports an error for the above line, remove it and include
# the dhparams in the certificate:
#   cat <CERT>.crt dhparams.pem > cert-with-dhparams.crt
#   SSLCertificateFile cert-with-dhparams.crt


# HSTS: Force browsers to require SSL for this domain for the next year.
# Down-grade to HTTP will cause browsers to abort with a security error.
Header always set Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"


# HPKP: Pin the current key for the next two months.
# Generate hash using:
#   openssl rsa -in <serverkey>.key -outform der -pubout | \
#   openssl dgst -sha256 -binary | openssl enc -base64
#
# You ideally want to generate a backup key and include that here as well,
# in case the primary key is lost or compromised.
# Also note the implications for key rollover.
# See: https://developer.mozilla.org/en-US/docs/Web/Security/Public_Key_Pinning
Header always set Public-Key-Pins "pin-sha256=\"<HASH>\"; max-age=5184000; includeSubDomains"


# Disable compression to avoid BREACH HTTPS/SSL attack.
<Location />
    SetEnv no-gzip
</Location>

That should cover the basics.

Testing

As for SSL connection testing, I found the Qualys SSL Labs Test helpful. It shows what browsers (browser versions) will get which encryption quality (forward secrecy or not) and highlights common problems such as certificate chain issues.

Hope this helps someone out there!

Cleaning up unused Docker images and containers

2015-05-06T00:00:00+02:00

Docker doesn't delete old/unused images or containers by itself, even if they weren't used for a long time or were only intermediary steps on the way to another image. This leads to an image sprawl that eats up a lot of disk space if not kept in check.

The right way to solve this would be to parse the output of docker inspect and remove containers and images based on certain policies. Unfortunately, a quick internet search did not turn up a script that does this.

Since I didn't want to spend the time to write such a thing myself, I resorted to what – sadly – seems to be state-of-the-art docker image management: a cronjob running those two lines:

docker ps -a | grep 'weeks ago' | awk '{print $1}' | xargs --no-run-if-empty docker rm
docker images -f "dangling=true" -q | xargs --no-run-if-empty docker rmi >/dev/null 2>&1

The first line removes containers that are older than two weeks and are not currently running (docker rm simply will not remove running containers). The second line removes images that are not used by any container and are not tagged (i.e. don't have proper repository name).

These two invocations are based on this Stack Overflow question and on this blog post by Jim Hoskins.

This solution works well enough, you probably shouldn't use it on production servers, though. :-)

Manually creating Docker images

2015-04-30T00:00:00+02:00

Docker is a virtualization solution that's been gaining a lot of momentum over the last few years. It focuses on light-weight, ephemeral containers that can be created based on simple config files.

Docker's main target platform is amd64, but it also works on x86. However, practically all official container images in the Docker registry are amd64 based, which means they can't be used on an x86 machine. So, it's necessary to manually create the required base images. As you might have guessed, my server runs Docker on x86, so I've had to find a solution for that problem.

Fortunately, creating images from scratch is really easy with the mkimage.sh script that comes bundled with Docker. On Debian systems, its installed in /usr/share/docker.io/contrib/mkimage.sh, on Fedora it has to be obtained from the Docker git repository:

$ git clone https://github.com/docker/docker.git

The script can then be found under docker/contrib/mkimage.sh.

Creating a Debian Jessie image is straight-forward:

# mkimage.sh -t debootstrap/minbase debootstrap --variant=minbase jessie

This command will create a minimal Debian Jessie image using Debootstrap, and import it into Docker with the name debootstrap/minbase. Further options can set a specific Debian mirror server and a list of additional packages to install:

# mkimage.sh -t debootstrap/minbase debootstrap \
             --include=locales --variant=minbase \
             jessie http://httpredir.debian.org/debian

This will use httpredir.debian.org as mirror and install the locales package in the image.

mkimage.sh has backends to bootstrap Arch Linux, Busybox, Centos, Mageia, and Ubuntu. Fedora images doesn't seem to be supported directly, but they can be generated by following instructions compiled by James Labocki.

Finally, it's worth mentioning that this should only be used to generate base images. You'd then use Docker itself (cf. Dockerfile) to create images that actually do something interesting, based on these base images. This will save both time and memory, due to Docker's caching and copy-on-write mechanisms.

Installing CyanogenMod 11 on a Samsung Galaxy S2

2015-04-15T00:00:00+02:00

I bought my Samsung Galaxy S2 in 2011, and it's still going strong. It really was a great phone for the time and held up incredibly well. Unfortunately, Samsung's support has ended long ago, and users are stranded with an obsolete (and insecure) firmware.

Fortunately, CyanogenMod still provides relatively recent images for the device. As of this writing, snapshots of CM11 (based on Android 4.4) are available, but there are no images of CM12.

Here is how I flashed CM11 to my phone. This is based on the official CyanogenMod wiki page for the SGS2 and on this xda-developers post. Since you can brick your phone if you don't know what you are doing, I suggest reading both of these pages. Note that you will need to factory-reset your phone, so backup all your data (files, apps, SMS, contacts, ...).

All the following steps have to be performed on a root shell on Linux.

To start from a clean slate, create a new Debian Jessie chroot (you may need to install debootstrap first). Don't use LXC/Docker/VMWare here, you need raw hardware access:

host#  mkdir sgs2-flash-chroot
host#  cd sgs2-flash-chroot
host#  debootstrap jessie .
host#  mount --bind /dev/ dev
host#  mount --bind /sys sys
host#  mount --bind /proc proc

Copy the following files to sgs2-flash-chroot/tmp:

cm-11-20141115-SNAPSHOT-M12-i9100.zip: CyanogenMod image
gapps-kk-20140105-signed.zip: Google Apps package; note that version >= 20140606 may not work: check the WARNING section on the CM wiki.
zImage (from GT-I9100_JB_ClockworkMod-Recovery_6.0.2.9.tar): Recovery kernel
Recovery_CWM_6.0.4.7_I9100.zip: Newer recovery image required to install CM >= 11.

Boot the phone into download-mode (shutdown, then VOLDOWN + HOME + POWER) and connect to the Linux computer.

host#  chroot .
chroot#  apt-get install heimdall-flash android-tools-adb
chroot#  heimdall print-pit
chroot#  cd /tmp
chroot#  heimdall flash --KERNEL zImage --no-reboot

Disconnect the USB cable and hold POWER until the phone shuts down. Reboot into recovery (VOLUP + HOME + POWER, let go of POWER after 5 seconds or you'll trigger a reboot). Then reconnect the USB cable.

chroot#  adb devices    # Check if device recognized.
chroot#  adb push Recovery_CWM_6.0.4.7_I9100.zip /emmc

In recovery, select "install from zip file" to flash the new recovery image. Then go into advanced -> "reboot recovery". Mount /storage/sdcard0 in the recovery menu, then reconnect the USB cable.

chroot#  adb devices    # Check if device recognized.
chroot#  adb push cm-11-20141115-SNAPSHOT-M12-i9100.zip /storage/sdcard0
chroot#  adb push gapps-kk-20140105-signed.zip /storage/sdcard0

Again, in recovery, select "install from zip files", first install the CM image, then the GApps package. Select "reboot" to boot into CyanogenMod. Shut down again, reboot into recovery, wipe cache and perform factory reset, reboot into CM (avoid factory reset with stock kernel due to the "super brick" problem).

Done. You should now have a not-so-shiny-anymore Galaxy S2 running a new-and-shiny CyanogenMod 11. Enjoy :-)

Checklib finally announced

2006-09-27T00:00:00+02:00

On Monday, checklib was finally announced on debian-devel-announce, thanks to Andreas Barth for sponsoring the mail.

I got very positive reactions from a number of people, which is great. I got less friendly comments prior to the announcement (by one person) , and I'm happy this reaction wasn't representative for the rest of Debian.

It's nice that people show interest in the problem, there's currently a discussion on debian-devel if and how automatic checking (and fixing) could be added to debhelper. That would seriously rock, as it would be one of the faster ways to get the number of affected packages down.

It's also cool to hear that the GNOME people are fixing their .la files with 2.16 in order to cut down dependencies introduced by broken libtool files.

There are some other interesting things on the horizon on the technical side of the project, as automatically built dbgsym packages (containing debug symbols, Ubuntu does that already), and the idea Simon Richter already talked about, which could really cut down the work the release team has with library transitions.

Installing Debian on an oldworld PPC

2006-09-20T00:00:00+02:00

The victim was a PowerMac 9500 with a 300MHz G3 CPU, 200Mb RAM, a 9G HDD (with OS9 installed), and a 2G HDD (blank).

I hooked up a PowerBook to see the serial console output, since OpenFirmware only talks over the serial line. Then I finally found floppy images that would boot actually boot. They were from Woody, so I did a netinst using boot-floppies.

Unfortunately the Linux didn't come up after reboot. After resetting the nvram I could at least boot MacOS again and then start Linux via BootX. It took quite some fiddling with quik and the nvram till I had the direct boot working.

Subsequently I upgraded to etch, but I couldn't get a 2.6 kernel to boot. First I got a bus error from the IMS TwinTurbo graphics card driver, then the kernel "forgot" where the initramfs was loaded, which turned out to be a grave bug in the Debian kernel images (#366620).

I wrote patches for both bugs and now I finally have Etch with Linux 2.6 working. Whee!

The only sad thing is that all this took the better part of last week :-/

Correct use of hyphens in man-pages

2006-09-06T00:00:00+02:00

When writing manual pages the question comes up when to use "-" and when to use "\-". The answer is actually quite simple. Use "-" whenever you want a hyphen and "\-" when you want a minus sign.

There are two exceptions though: In the name section, "\-" is used to separate program name from short description, as in "man \- an interface to on-line manuals".

The other exception is that you have to use "\-" for options/switches (-h, --foo, etc.). "\-" causes man to emit an U+002d Hyphen-Minus character, whereas "-" results in U+2010 Hyphen (in a unicode locale).

U+2d is the normal ASCII hyphen char, the one programs use to test for switches. So "\-" allows copy&paste from the manpage, while "-" doesn't.

ELF talk

2006-08-19T00:00:00+02:00

Last monday I held a short talk about ELF objects and dynamic linking for the Debienna crowd. It went semi-well; people were quite interested but somtimes didn't seem to grasp what I was talking about. Which was probably my fault because I didn't spend enough time preparing the talk, being on a difficult subject to begin with.

Perhaps I'll talk about the subject again for maks, Rhonda and baumgartner (if they are still interested), since they weren't able to attend.

In case anyone cares, I've written up some notes about ELF, dynamic linking, symbol lookup and related stuff, covering most the thinks I talked about.

Cross-compiler fun

2006-08-12T00:00:00+02:00

I needed to fix the elfutils build failure on ia64, but I didn't have access to such a machine. Fortunately Herbert Pötzl pointed out ski, an ia64 emulator for Linux.

Ski needs a custom guest kernel however, so I had to cross-compile that for ia64.

Setting up a cross-compiling toolchain on Debian is really easy nowadays; there's even a nice HOWTO describing the needed steps. For lazy people pre-built packages are available.

When compiling the toolchain yourself, note that you may need more/other library packages then listed in the HOWTO. This depends on the target architecture, e.g. for ia64 you will need libunwind7-dev, libatomic-ops-dev, and further libc6.1 instead of libc6. Otherwise gcc will complain about missing build-dependencies.

For ia64 I ran into a linker error when building gcc, however a patch from Bertl's cross-compiling corner solved that.

While doing all this I wrote some scripts to automate the process, so compiling a cross-toolchain (for any architecture) is now a matter of 5 minutes configuration and one ./driver run. Whee!

Using mutt's header_cache feature

2006-08-08T00:00:00+02:00

In the past I needed to move old mails out to a backup folder occasionally, since it took too long for mutt to load all the headers when opening the maildir (often some 10 seconds for about 6000 mails).

Today I remembered that mutt should actually do header caching, so I looked it up in the docs, and saw that the config option was missing in my .muttrc. I put it in, restarted mutt, and after the cache was initialized the performance was noticeably better:

set header_cache="~/.mutt/headercache"

Mutt can actually use 3 kinds of database backends for the cache, bdb, gdbm and qdbm. The default in Debian is bdb, to use one of the others you have to flip a switch in debian/rules and recompile the package.

On my machine qdbm was fastest (gdbm slightly slower, dbm far behind), so I stuck with that. Since I use a self-compiled mutt package anyway (because I want my index_color patch included), it's not much of a problem.

Automatically syncing files between hosts without compromising security

2006-08-05T00:00:00+02:00

The problem

The goal is to automatically synchronize files between several hosts without compromising the integrity of the separate machines. A nice tool for 2-way sync is unison. To sync files between different machines The Right Way (TM) is to tunnel the unison protocol over ssh. This is well supported by unison.

To run sync automatically (e.g. via cron), you need to create an SSH keypair without passphrase, so unison can log into the other machine without human interaction. This is where the problems start, since anyone who got access to the private key (e.g. by compromising or stealing the machine the private key was on) can log into the other host.

Now ssh has a nice way to restrict what you can do with a specific key, so you can e.g. use the following in the remote hosts ~/.ssh/authorized_keys:

command="/usr/bin/unison -server" ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA8K2cd0yemw...

That way someone who has the private key can't execute arbitrary commands, but just unison in server mode. However it's still possible to tell the unison server to overwrite arbitrary files (which the user has write access to). This is a major problem, since also files like ~/.bashrc can be overwritten, so the next time the user logs in, arbitrary commands will be executed.

A possible solution

One solution is to simply create a new user on the remote host with a disabled password, and let unison run as that user (via adding the appropriate line to $HOME/.ssh/authorized_keys, and telling the local unison to use that username).

That's possible, but the .bashrc trick still works, it's just less likely that the code there is ever executed (root would have to use su to become that user).

For me this solution didn't work out since I wanted to sync my maildir, and it was hard to ensure that file permissions were set in a way that both allowed me to read my mail and allowed unison (running under user unison-sync) to sync the files.

The Right Solution (TM)

All the problems vanish as soon as you run unison under the user you'd normally use, but in a chroot. Now a full-blown chroot takes up a lot of space, and there's once again the danger that someone might enter the chroot and run some kind of shell (though the risk is even lower).

It's best to use a chroot which only contains the bare minimum of files necessary to run unison -server.

You get numerous advantages:

No problems with file permissions
No shell inside the chroot that would read startup files from $HOME.
Hardly any space wasted. The whole chroot is about 4Mb in size
Since the chroot is pretty much empty, many common exploits (well, shell codes) won't work

How to do it

greek0@orest:/home/chroots/unichroot$ cat ~/.ssh/authorized_keys
command="/usr/bin/dchroot -q -c unison -- /usr/bin/unison -server" ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA8K2cd0.....

greek0@orest:/home/chroots/unichroot$ grep unison /etc/dchroot.conf
unison /home/chroots/unichroot

greek0@orest:/home/chroots/unichroot$ find . -maxdepth 3 | xargs ls -ld
drwxr-xr-x   2 root   root      4096 2006-08-04 16:39 ./bin
-rwxr-xr-x   1 root   root    576100 2006-08-04 15:06 ./bin/sash
lrwxrwxrwx   1 root   root         4 2006-08-04 16:39 ./bin/zsh -&gt; sash
drwxr-xr-x   3 root   root      4096 2006-08-04 15:10 ./home
drwx------   4 greek0 greek0    4096 2006-08-04 15:18 ./home/greek0
drwx------  31 greek0 greek0    4096 2006-08-04 13:47 ./home/greek0/Maildir
drwx------   2 greek0 greek0    4096 2006-08-04 15:47 ./home/greek0/.unison
drwxr-xr-x   2 root   root      4096 2006-08-04 15:07 ./lib
-rwxr-xr-x   1 root   root     88164 2006-08-04 14:58 ./lib/ld-linux.so.2
-rwxr-xr-x   1 root   root   1151644 2006-08-04 14:56 ./lib/libc.so.6
-rw-r--r--   1 root   root      9592 2006-08-04 14:56 ./lib/libdl.so.2
-rw-r--r--   1 root   root    141040 2006-08-04 14:55 ./lib/libm.so.6
-rw-r--r--   1 root   root      9656 2006-08-04 14:55 ./lib/libutil.so.1
drwxr-xr-x   3 root   root      4096 2006-08-04 14:53 ./usr
drwxr-xr-x   2 root   root      4096 2006-08-04 14:55 ./usr/bin
lrwxrwxrwx   1 root   root        14 2006-08-04 15:12 ./usr/bin/unison -&gt; unison-2.13.16
-rwxr-xr-x   1 root   root    955784 2006-08-04 14:54 ./usr/bin/unison-2.13.16

The zsh symlink is there because I have /bin/zsh as my shell in /etc/passwd, and dchroot also wants to use it in the chroot (for launching unison).

/home/greek0/Maildir is bind-mounted from outside the chroot, bind-mounting is done at boot-time via /etc/fstab.

The chroot was created manually, simply by copying the files from the host. You obviously need /usr/bin/unison plus all the libraries it depends on. You can find those via readelf -d /usr/bin/unison | grep NEEDED. Additionally you need the dynamic linker /lib/ld-linux.so.2 (seen from readelf -l /usr/bin/unison | grep INTERP -A 1).

One thing to pay attention to is that most of the files copied from /lib are symlinks. Be sure to either use cp without arguments, or use cp -a and copy the link targets too.

Installing Windows 95 inside QEMU on Linux

2006-08-05T00:00:00+02:00

If you can just install Windows 95 inside QEMU and it magically works consider yourself lucky. I tried this and got an error message that I need to create a FAT partition first where the installer can place some files. That's where it all began.

Installing

Create the disk image. dd if=/dev/zero of=hda bs=$(( 1024*1024 )) seek=1000 count=0. That way you create a sparse 1GB file.
Get a FreeDOS Floppy and CD image and install FreeDOS inside QEMU, there you can partition your disk. qemu -hda hda -fda fd0 -cdrom cdrom-img -boot a.
Insert the windows CD, run qemu with -cdrom /dev/cdrom -boot c so you get to the FreeDOS prompt, then go to the cdrom drive and run setup. The setup should work then

Getting networking up inside Windows

You need the Realtek rtl8139 driver, other network cards won't work (ne2000 at least doesn't). So run qemu with -net user -net nic,model=rtl8139.
Windows 95 unfortunately doesn't have the driver for the rtl8139, so you need to get it onto the system. Download it from well, the Realtek driver download page. Unzip it, put it onto a floppy, create an image of that floppy and run qemu with -fda image.
Then setup the networking under windows, reboot and you should be able to access the internet.

Tools for mutt

2006-08-02T00:00:00+02:00

Tools

mutt-bug

This is a tool that displays Debian bug reports in mutt. You can then directly read all messages sent to the bug and reply. The messages are fetched directly from the web interface, so there is no delay between requesting bug and getting it per email.

This tool was originally written by Christoph Berg, I've made some modifications to make it work in arbitrary directories.

Useage:

$ mutt-bug bugnumber

Download: mutt-bug

gpgverify

gpg --verify is quite slow when you have large keyrings included (like the debian keyring). This is nasty, since mutt has to wait until gpg is finished when displaying a gpg signed message (with signature verification on). So I've written a tool that splits a huge keyring into a lot of smaller keyrings (one key per keyring) and a shell script to verify signatures, to be used from within mutt. The former tool is called splitkeyring.sh. The latter one is gpgverify.sh.

gpgverify.sh first invokes gpg --verify as normal and captures its output. If gpg failed because the key was not found in any keyring, the script looks if the key is in one of the splitted keyrings, and if so, reruns gpg with that keyring included. Otherwise the gpg error is returned.

These scripts are still hacky, if you want to use them you'll probably have to modify them a bit. They aren't too big, so this shouldn't be too much of a problem.

Download: gpgverify

A mutt patch for a more colorful index

2006-08-01T00:00:00+02:00

This patch enables different colorings for different parts of the index display. For example you can choose one color for the subject, another one for the author, and a third one for the flags.

Screenshots:

Screenshot #1: Just a random screenshot...
Screenshot #2: Notice how messages written by me are differently colored then those from other authors. This is done using indexcolor's author pattern matching.

Downloads:

mutt-1.5.12-indexcolor-3+cb.diff: indexcolor-3 forward-ported to mutt 1.5.12 by Christoph Berg. Thanks a lot!
mutt-1.5.8-indexcolor-3.diff: Added pattern to index_author and index_color command. Documentation updates and some other fixes. This one is stable and has the biggest feature-set.
mutt-1.5.8-indexcolor-2+cb.diff: -2 with additional fixes by Christoph Berg (thanks!). This one works really well.
mutt-1.5.8-indexcolor-2.diff: Bugfix release
mutt-1.5.8-indexcolor-1.diff: First release of the indexcolor patch. It's undocumented and buggy.

Vim notes

2006-07-13T00:00:00+02:00

Scripts

UpdateModDate.vim is a vim script I have written for updating the date-stamp of these pages automatically every time I save the files from within vim.

It searches for lines marked with %DATE_TAG% and updates the date on these lines every time the file is saved. Example:

Last Modified: Tue Jun  2 04:44:58 UTC 2004     // %DATE_TAG%

Bugs

I've hit a bug in vim related to html highlighting that would cause get really really slow. This has already been fixed though. The interested might look at the detailed description.

Debian on Linksys WRT54GS (German)

2005-08-28T00:00:00+02:00

Notizen zu einem kurzen Vortrag, den ich am 28.8.2005 beim Debienna Treffen gehalten habe.

Vortrag

Was ist der WRT54GS?
Was sind die Vorteile einer eigenen Firmware?
Spielerei
Shell auf dem WRT
Mehr Möglichkeiten als die offizielle Linksys Firmware z.B: OpenVPN
Meine Wahl: OpenWRT
Paketsystem, ähnlich Debian
Wieso zusätzlich noch Debian installieren?
Spielerei
Mipsel Architektur
Gcc/Binutils -> Code schreiben
Debugging
Wie macht man's?
Debootstrap will nicht (Bashisms)
CDebootstrap muss her
Debian Binary funktioniert auf OpenWRT nicht (glibc <-> uclibc)
Man muss cdebootstrap (und libdebian-installer) .ipkg Pakete mit der OpenWRT Toolchain bauen
Nicht genug Platz am WRT (nur 8Mb Flash) -> NFS share
Bootstrap am WRT läuft nicht glatt durch weil dpkg fehlt: Das lässt sich zwar in Busybox einkompilieren, das erfordert aber etwas mehr Änderungen, und man muss den ganzen OpenWRT Tree bauen.
Einfacherer Weg (etwas hacky): cdebootstrap --arch am NFS server im Share, das scheitert irgendwann, weil Arch-spezifische Sachen fehlschlagen. Dpkg ist dann aber schon entpackt. Danach lässt man cdebootstrap am WRT laufen (selbe Optionen), das merkt, dass schon einiges da ist, und macht dort weiter wo cdebootstrap am Host aufgehört hat
Danach: Chroot & Freude an Debian haben!
Fragen?
Danke für's Zuhören

Source & Binaries

Liegen in /div/wrt/ herum.

wmbutton: dockapp displaying configureable buttons

2005-02-20T00:00:00+01:00

Introduction

I have been using wmbutton for quite some time, and I think it's really nice. One thing that has always sucked was the need to recompile just to change the images. So I got the source code of the wmbutton Debian package and added the capability to change the images without a recompile. One day later I discovered that someone (ehflora) had already done this. I got his version and it looked a bit better then mine. While playing around I noticed that wmbutton was leaking memory, so I ran valgrind on it, which confirmed my suspicions. I fixed the Leaks and did some more code cleanup.

I've increased the version number to 0.6. The code is still quite a mess, but at least it's better then before. Feel free to further improve it if you like, there's plenty room for it ;-).

Details

This program is released under the GPL version 2. A copy of the license may be obtained at http://www.gnu.org/licenses/gpl.txt.

Screenshots and more can be found at: http://www.dockapps.org/file.php/id/241

Sourcecode Download

Debian packages

wmbutton is available directly from Debian. Check out the package page.

Christian Aichinger's posts

Separate development feedback and performance evaluation

Git Best Practices

Staging changes: git add -p and git diff --cached

Rewriting history: git commit --amend

Rewriting history: git rebase -i HEAD~10

Atomic commits

Commit messages

Git branching patterns

Covid variant competition and a longer-term strategy

How to create pixel-perfect PDF reports in 2021

Capture Bluetooth Traffic from Android

Force enable dark mode in Chrome, Safari, Firefox

Chrome

On Android

On iOS

On Windows

Safari

Changing the system setting on iOS

Changing the system setting on MacOS

Firefox

LDAP basics for users

Concepts

Querying

Converting Unix timestamps

The Kelly Criterion: Comparison with Expected Values

The big reveal

Different kinds of playing

A dubious game

Conclusion

The Kelly criterion

The Kelly Criterion: Where does the logarithm come from?

The Kelly Criterion: Multiple Investment Opportunities

Conclusion

The Kelly Criterion: Introduction

Derivation of the Kelly criterion

Extension to games with more than two possible outcomes

Exploring GDB's Python API with Jupyter

PyDays 2017

GCC compatibility: inline namespaces and ABI tags

New C++ documentation section in the sidebar

Building External Libraries with Boost Build

Working with angles (is surprisingly hard)

Resource management with Python

Failed solutions

A workable solution

The correct solution™

Wrap Up

Remarks on enable_shared_from_this

Building a computer from logic gates

From silicon to computation

General-purpose processors

Conclusion

Boost Range Highlights

irange

combine

Algorithms

Adaptors

indexed

accumulate

as_literal

Summary

Boost Range for Humans

Debugging riddle of the day

libconf - a Python reader for libconfig files

Data access times

Requesting certificates the whirlwind way

When would you need a certificate?

How do you go about obtaining a certificate?

Creating a CSR

Submitting the CSR for signing

A whirlwind introduction to the secure web

Keypairs

Certificates

Certificate Authorities

The half-broken part

Wrapup

dh_virtualenv and long package names (FileNotFound error)

Behold the power of perf-tools

Running strace for multiple processes

Staging changes: `git add -p` and `git diff --cached`

Rewriting history: `git commit --amend`

Rewriting history: `git rebase -i HEAD~10`