dionhaefner.github.io - blog

Where’d all the time go

2026-03-10T00:00:00+01:00

Some people talk about how life has seasons. I prefer the term phase transitions. Ice melts and turns into liquid – boom! The same substance suddenly feels completely different.

That choice of words is no accident. To me, “it feels different” is the most poignant, perhaps only truly correct, description of what makes a phase transition. You can’t even verbalize what it is without sounding preachy, or full of it, or eye-roll-inducing cheesy. You just have to live it, and then it makes sense and all the good advice comes too late, too bad. I’ll try anyway.

There’s another reason why I like “phase transitions” more than “seasons”. Unlike seasons, which come and go in a fixed sequence, phase transitions can come in any order, at any time, which also means we have control over them, at least sometimes. I suppose one reason why I’m writing this article is to remind me and everyone else that phase transitions are a thing and life can feel completely different one moment to the next, for better and worse (there’s the cheese, don’t say I didn’t warn you). It’s also a reminder that it’s up to us to set ourselves up for the phase transitions we want, and to prepare for (or counteract) the ones we don’t.

The most impactful phase transitions of my adult life have been becoming a parent and transitioning together with an early startup into a not quite so early startup. I’m mentioning those in the same breath not because they matter equally much, but because they happened roughly around the same time, and because the quality of the phase transitions is so similar it’s impossible for me to separate them.

It’s no secret that children, like companies, undergo phase transitions left and right. That’s not what I’m talking about. To me, the less expected part is the phase transition that is forced upon the people going along for the ride, that is, the freshly baked parent or engineer who suddenly has names below them in the org chart and has something to say about everything because they helped build everything.

Did you guess yet what both phase transitions have in common? It’s going from having nothing but time to having no time at all, personally and professionally. Since this is a (mostly) professional blog I’ll try and stick to the professional side of things.

The curse: There’s disappointment everywhere¶

There’s simply more to do than hours in the day. More people who need your attention than attention you can give. More problems worth solving than you can possibly solve. Every “yes” to one thing is an implicit “no” to something else, and often that something else is a person who deserved better. Ring ring, reality called: The rules of the game changed, and it’s now zero-sum.

Before the phase transition, in the age of abundance, I could be everything to everyone in my small circle. I could respond to every message promptly, attend every meeting fully present, and still have energy left over to do the actual work and overdeliver. Now, I’m constantly triaging. The colleague who needed a thorough code review gets a quick skim. The coworker who wanted to float some ideas by you gets rescheduled. The kid who wants to play gets “just five more minutes” while I finish one last email.

You learn to live with the low-grade guilt. You learn to apologize gracefully, and mean it. You learn to forgive yourself for being finite. But it also means that every day is a little bit uncomfortable, and you need to get used to that. But where there’s discomfort, there’s also growth.

The blessing: Everything you do matters¶

When time becomes scarce, what you choose to do with it becomes meaningful by definition. You can’t waste time on things that don’t matter because there’s no time to waste in the first place.

The result is that you end up doing more good, meaningful stuff than you ever did when time was abundant, precisely because of the constraints.

When you have unlimited time, every problem looks worth solving. Every rabbit hole looks worth exploring. Every refactor looks worth doing. When you have no time, you develop a ruthless filter. Some problems solve themselves if you ignore them long enough. Some “critical” tasks turn out to be entirely optional. Some meetings could have been emails, and some emails could have been nothing at all.

Here’s the funny thing: I used to be a major procrastinator. Early on I learned to combat this with concurrency and task switching. When I got bored with task A, I’d go to B, then C, then D, then back to A. The phase transition turned this coping mechanism into a superpower. Now B, C, and D are likely just as important as A. The backlog is so deep that even procrastination is productive. In many ways I feel much more content with my output now than I ever did before.

The shape of time changes¶

Before the phase transition I perceived time as hairless and undifferentiated. I could think about a problem whenever I wanted. My brain was always on, processing in the background, making connections.

After the phase transition, time develops texture. There’s work time, family time, chore time. Each has its own character and constraints, and some of these boundaries are sharper than others. I can’t process a technical problem while reading a bedtime story, and I owe others being present during quality time.

This is not to say I don’t think through a design decision while going on a walk or folding laundry or, to hell with it, kicking a ball in the park. The daily chores (and, let’s be honest, frequent boredom) of parenting make for decent background processing time. What’s changed is that the transitions between modes are more abrupt, the context switches are harder, and their bite stings a little more.

Leverage becomes everything¶

When you can’t do all the work yourself, you become obsessed with leverage. To me this means constantly zooming out when others are zooming in. The team is losing too much time manually bumping dependencies? Let me quietly write a bot on my own time that cuts that friction in half, then roll it out once it’s ready. Just explained a difficult concept to a colleague and I like the ring of my own words? Put it into a blog post and share it with the team.

This is where the transition from individual contributor to leader becomes most palpable. You stop engaging with every problem and start creating environments where problems get solved without you. Your individual contributions matter less than your ability to multiply the contributions of others. The technical skills that got you here are still there when you need them, but you catch yourself reaching for a different skillset more often: communication, prioritization, creating clarity.

Finding equilibrium¶

Phase transitions eventually reach a new equilibrium. Water doesn’t keep boiling forever. Ice doesn’t keep melting. You find a new normal.

For me, the new equilibrium looks like this: I’ve become comfortable with being uncomfortable. I’ve developed new skills to survive and thrive in a resource-constrained habitat. I’ve learned to delegate not just tasks but also trust. I’ve accepted that some things I used to do well, I now do adequately, and that’s okay.

The new equilibrium also means recognizing that both phases have value. The early days of having unlimited time, of being able to go deep on problems, were necessary to build the foundation of who I am and what I do best. The current phase of having no time is necessary to scale that foundation into something bigger.

And here’s the thing about phase transitions: they go both ways. Ice can melt, and water can freeze. The busy season will eventually yield. The startup will mature, the kids will grow older, the calendar will loosen. Until then, I’m here for the craziness one day at a time.

Different properties, same substance.

My personal philosophy

2026-01-19T00:00:00+01:00

I’ve seen other people write about their personal philosophies, and I thought it would be fun to write mine down too. I don’t have a grand unifying theory of life, and I don’t really think it’s possible to have one (which I guess is a philosophical statement on its own). Anyhow, here’s a collection of loosely held beliefs and values that guide me.

No single philosophy works when taken whole. Every sufficiently simple system contains insight and overreach in equal measure. What follows isn’t a doctrine or an optimization strategy; it’s a working description of how I move through the world, shaped by uncertainty, beauty, and the fact that this all ends.

Beauty matters. Not in the form of ornament, luxury, or art objects curated for consumption, but as something emergent and real: awe in nature, intimacy with another person, clarity after confusion, looking back on the great things you’ve built and gleaming with pride, presence in a fleeting moment. Real beauty is nourishment, but if it’s hollow, performative, or abstracted it doesn’t feed you.

Subjectivity is unavoidable. It’s impossible to escape your own perspective. Whether there is an objective reality underneath it all is unknowable. Act accordingly. Treat subjectivity not as a flaw to be corrected, but as the medium through which everything meaningful arrives. Your inner life is yours to shape and account for, and its effects ripple outward.

The moment is all you ever touch. Don’t waste it pretending you’ll live later.

Happiness = joy + satisfaction. Satisfaction = outcomes / expectations. Inflate expectations and even good outcomes feel like failure; adjust them thoughtfully and ordinary moments can become miraculous. And don’t forget to optimize for joy! The best things in life are free.

The universe does not care about morality, yet it’s what makes us human. Ethics are chosen, enacted, and sustained locally, between people who can feel harm and care. Any moral system that forgets the individual in favor of abstraction has already failed: Metrics that become targets rot and rules that can’t bend break people.

Think globally, act locally. The world is a big place, and it’s easy to feel small and powerless. Focus on the people around you, the community you’re part of, and the actions you can take that align with your values.

Free will is unresolved. But you still experience choice, intention, regret, and responsibility. There’s no alternative to living as though your actions matter.

The stars are indifferent to astronomy. Science is about identifying models that work: patterns, compressed experience, and predicted behavior within the limits of human cognition. Mathematics discovers relationships inside invented systems. But conflating usefulness with truth is a category error. Certainty is seductive and usually false, intellectual humility is harder and more honest.

Consciousness is the mystery. How strange and wonderful is the fact that anything feels like anything at all. Sensations arise, beauty appears, pain intrudes; becoming numb to that is the real tragedy. A worldview that treats experience as incidental has missed the point.

Existence is the miracle. The prospect of non-existence cuts deeper than any abstract argument can resolve. Wanting an afterlife is a refusal to casually discard the value of experience. The real task is to live in such a way that, when the end approaches, fear loosens its grip.

Life is music. There is no guaranteed meaning, no built-in purpose, and no final explanation that makes everything click. The way to face this strange trip is with laughter in our hearts, and to listen to the music that is life.

Each other is all we’ve got.

The totally reasonable effectiveness of execution-driven science

2025-04-02T00:00:00+02:00

This post has originally been published on Pasteur Labs Insights.

At Pasteur Labs, we work every day to uplift simulation workflows to the machine age. Concretely, this means we turn bleeding-edge research and technology (Simulation Intelligence) into real-world capabilities:

We implement and modify recently published AI methods to work at industrial scales with real & simulated data.
We measure how experimental differentiable physics routines hold up in end-to-end applications (in computational engineering, for example).
We build simulators that are data-driven, GPU-native, and automatically differentiable.
We automate the definition, training, evaluation, and deployment of hybrid AI simulators.
All the while, we favor robust, general workflows over one-off solutions.

This particular mix — where science meets engineering, technology enthusiasm meets sobering reality, pie-in-the-sky meets down-to-earth thinking — provides many opportunities but also unique challenges. Scientists and engineers at Pasteur Labs are constantly faced with compromise: How do we answer fundamental questions in a way that leads to measurable impact without getting lost in details that don’t matter, nor providing superficial solutions that don’t hold up in the real world? We believe part of the answer lies in two paradigms that are pervasive in everything we do, namely, use-inspired science and execution-driven science.

Use-inspired science — the what. (covered elsewhere)¶

Use-inspired science encodes the fact that neither application-ignorant basic science nor singular-purpose applied science is sufficient to solve classes of fundamental, real-world problems. This is ingrained in Pasteur Labs at the deepest level (including the name of our company).

Execution-driven science — the how. (covered here!)¶

Execution-driven science is the art of navigating complexity by scoping, iterating on, surgically modifying, and stacking existing solutions, and failing fast when encountering dead ends. In particular, execution-driven science allows us to build end-to-end systems of non-trivial complexity — emphasizing execution makes the immense challenges and combinatorial search spaces we are faced with tractable. It helps us guide our day-to-day work towards what is possible, feasible, and worthwhile.

The how of research is typically much less clearly defined than the what, although it is at least as important. Execution-driven science is a particularly powerful way to define this process when faced with the need to build systems that withstand the complexities of the real world. And in the moment, it’s a valuable reminder to the individual researcher facing an overwhelming or constantly-moving research problem.

Simulating the world¶

I clearly remember the moment I fell in love with physics. It was the moment I realized it made reality computable.

Suddenly, I looked at the world with fresh eyes, computing the time it would take for objects to fall or trains to stop or elevators to arrive. How could a simple mathematical equation like $t=\sqrt{2 s / a}$ generate predictions that we can observe in the real world?

I later realized that it wasn’t so easy, and that the simplified physical laws we learn in school don’t always hold up to the complexity of the real world. A second revelation came when I saw we could model complex physical systems by creating approximate solutions, solved by computers. Later still, I decided to write a simulator to forecast ocean dynamics (see Fig. 1).

Fig. 1: A high-resolution ocean simulation, executed on a single high-end workstation about the size of an AC unit. From https://agupubs.onlinelibrary.wiley.com/doi/10.1029/2021MS002717.

The ocean is home to a range of complex physical phenomena, like mesoscale turbulence (the vortices or “eddies” most visible around the equator), boundary currents (like the Gulf Stream), an entire zoo of internal and surface waves, and an all-encompassing overturning circulation. Yet it is governed by only 8 so-called primitive equations.

After deriving them from first principles, we can easily write down the mathematical rules that generate the entire wealth of observed ocean dynamics. To actually simulate them, we need to conjure an unholy stack of tricks, approximations, numerical hacks, heuristics, and empirical laws, adding up to well above ten thousand lines of code.

As a grad student, seeing an ocean simulation unfold on a supercomputer in real time was sublime. For the longest time, I believed that science was inherently about simplicity: finding the most concise explanation for a phenomenon, or an elegant theory that predicts observations. Today I know there are many ways to do science. One of them is building systems — like climate models, fusion reactors, and space telescopes — via execution-driven science, bridging the gap between research lab and real world. Execution-driven science still emphasizes simplicity (or rather, parsimony), but introduces compounding as an additional guiding principle.

Science that’s more than the sum of its parts¶

Execution-driven science is about standing on the shoulders of giants and making sure they’re all standing up straight. It’s about taking the pieces others have built, polishing them, and stacking them in a way that pushes the boundaries of what’s possible. And then adding some new pieces where they matter the most, and shatter records.

Fig. 2: A prime example of execution-driven science: NeuralGCM, an AI-enabled system for climate modelling that is on par with some of the best traditional climate models and orders of magnitude faster. Each of the depicted boxes is unremarkable in isolation, but in combination they create something revolutionary. From https://arxiv.org/pdf/2311.07222.

NeuralGCM is an AI-driven system for atmospheric simulations (the atmosphere is typically the most computationally expensive component of a climate model). Taken at face value, each of its parts seems unremarkable (Fig. 2): a simple encoder-decoder architecture; a differentiable dynamical core (that is, hand-written traditional simulator) in Python; a feed-forward neural network to machine-learn a data-driven correction; and an off-the-rack solver for ordinary differential equations (ODEs). But acting together, this is the first system that demonstrates that AI can aid with simulations on climatic time scales, consisting of tens of thousands to millions of iterations, at an efficiency that has never been seen before:

For both weather and climate, our approach offers orders of magnitude computational savings over conventional GCMs (Kochkov et al. 2023).

The point here is not that all it took was to combine some existing methods and reap the benefits. What actually happened is much more subtle, where success came from a combination of different factors:

A rock solid understanding of the problem, to focus on what matters. That is, using AI to speed up computations, while keeping existing knowledge around in the form of a differentiable dynamical core, iterative ODE solver, and structure of the problem. This particular combination of methods fixes many existing shortcomings of pure-AI and pure-simulator systems.
Pragmatic, high-quality execution. All code is implemented in the high-performance Python framework JAX and executed on accelerators (Tensor Processing Units, TPU). This is not just computationally efficient. — keeping things entirely in Python also allows for rapid experimentation. This combination is likely close to Pareto-optimal: Yes, the system could potentially run faster if painstakingly optimized, at a huge expense in terms of human time. Or it could be much easier to develop, but not be nearly as powerful. Finding the right trade-off is precisely what it takes to iterate fast and effectively.
Unprecedented scale. A resource-efficient implementation allows researchers to scale up aggressively and reach real-world scales. At its finest resolution, NeuralGCM is trained for 3 weeks on 256 TPU devices, which is way beyond the norm in AI for climate, and AI in general.

This is an example of execution-driven science at its best, and demonstrates several techniques on how to wield it to push the envelope. There are many more examples like it throughout science and industry, and they follow similar patterns; for example:

PKDGRAV3, a massive-scale simulator for astrophysics that leverages GPUs and advances in numerical methods to achieve unseen efficiency.
PySR, which made symbolic regression widely accessible through a high-performance Julia backend and a Python frontend.
AlphaGo, a breakthrough in reinforcement learning that pushed the boundaries of what’s possible with AI, based around a relatively simple but efficient tree search algorithm.
(Everything we build at Pasteur Labs to realize the platform for Simulation Intelligence.)

💡 Notice how some of those examples are clearly use-inspired, while others are not? This illustrates how use-inspired science and execution-driven science act along two different dimensions — you can have one without the other, or be neither use-inspired nor execution-driven.

Explore violently, fail fast, be pragmatic¶

Let’s take a closer look at what it takes to apply execution-driven science in practice to solve hard, real-world problems.

Get really good at executing stuff. Good execution relies on top-notch technical skills, and a deep understanding of the fundamentals. (No one said this was easy, sorry!)
Build end-to-end as early as possible. Prefer to build the first prototype that has all major parts of the final system as soon as you possibly can. Ideally, this computes data that looks like the final outputs all the way from the initial inputs. That prototype will probably perform horribly! But it allows you to see the system in its entirety, and makes it crystal clear whether you missed something important. It will also provide you a performance baseline, so you can start monitoring whether your changes make things better straight away.
Fail fast when things don’t work out. This is a tricky one. How do we decide that things won’t work out, considering all the situations when we just need to keep at it a little longer? For me the best visualization of a good heuristic comes from the A* algorithm for pathfinding (see Fig. 3). Rather than going down rabbit holes indefinitely, we alternate between explore and exploit, pursue strategies that work, but stay flexible and switch things up once hitting a wall. This is true to the spirit of execution-driven science: the value is in the system we build to solve problems, not sticking with any particular component, and always keeping sight on the true objective (like the A* heuristic).

Fig. 3: Illustration of the A* algorithm navigating major cities (Chicago and Rome). A healthy mix of doubling down on promising routes and switching lanes when getting stuck allows the algorithm to navigate a massive search space without falling back to random exploration nor mindless exploitation.

Refine the things that matter, always. To navigate complexity with combinatorial search spaces (that arise from stacking many different components) we must be selective about the battles we pick. Don’t sink time into devising a clever solution to a problem that doesn’t manifest. And remember, a major benefit to building end-to-end prototypes early in the process is the ability to measure actual bottlenecks right away. Make use of it!
Consider scaling up. Good execution allows you to do more within the same budget (of people, compute, time, …). Spend some of your efficiency gains on more processed data, bigger models, and more powerful pipelines where it provides the largest benefit.
Demonstrate it works. It’s easy to get confused in complex systems. Ensure you have proper end-to-end testing in place, so that there can be no ambiguity whether problems are actually solved (and remember that it’s easiest to fool oneself).

And in the end, no amount of good advice will replace your best judgment, so don’t be afraid to use it.

Science or engineering?¶

People ask us sometimes: is this really science, or is it engineering? We don’t believe the distinction is a useful one — there’s certainly a lot of both, often in synergistic ways. Regardless, well-conducted execution-driven and use-inspired science exhibits many of the core values of the scientific method:

It aims to provide fundamental solutions to a class of (real-world) problems, rather than specific instances of them.
It depends on a thorough understanding of the problem at hand, involved components, and resulting system, to navigate complexity and identify minimally invasive changes with maximum impact.
It is reproducible, transparent, and extensible, allowing other researchers to build on top of results.

Execution-driven science in the wild¶

At Pasteur Labs, the journey towards next-gen simulation continues — we’re still far away from humanity’s dream to simulate everything. Models need to become orders of magnitude faster, cheaper, more accurate, grounded in reality and causality. Today, we believe that data-driven, AI-infused methods are a central piece to compute the parts of reality that elude classical approaches, and to tackle problems that do not have a concise mathematical representation. Execution-driven science has proven to be a powerful tool in this quest, and we’re excited to see where it will take us next.

And while the musings above are my earned insights, they are in great part realized because of the shared pursuits, rigor, continous support, and lived examples of top-notch use-inspired and execution-driven research from my teammates at Pasteur Labs.

Eager to help? 👉 pasteurlabs.ai/careers

Supercharged high-resolution ocean simulation with JAX

2021-12-03T00:00:00+01:00

Our Python ocean model Veros (which I maintain) now fully supports JAX as its computational backend. As a result, Veros has much better performance than before on both CPU and GPU, while all model code is still written in Python. In fact, we can now do high-resolution ocean simulations on a handful of GPUs, with the performance of entire CPU clusters!

The turbulent ocean. This high-resolution (0.1°) snapshot of the ocean was simulated with Veros on 16 A100 GPUs on a single Google Cloud VM, faster than 2000 CPUs running a Fortran model.

So, what does this mean, and how did we pull this off? In this blog post I will give you an introduction to high-performance ocean modelling, show you how JAX fits into the picture, and show some benchmarks to prove to you that Python code can be competitive with hand-written Fortran (while also having great GPU performance).

If you want to know all the details, you should make sure to also check out our article “Fast, cheap, & turbulent — Global ocean modelling with GPU acceleration in Python” that was published in the Journal of Advances in Earth System Modelling (JAMES) today.

Ocean modelling in a nutshell¶

Ocean models simulate how the oceans react to external forcings, like irradiation from the sun, wind patterns, or freshwater influx from rivers and glaciers. As such, they are a major component of every climate model (other parts being for example atmosphere, ice, and land models), and help us understand the complex processes taking place in the real oceans.

In the following sections I will show you how oceans can be modelled mathematically, and how we can solve these equations with computers, before we dive deeper into using JAX for high-performance computing.

The primitive equations¶

The starting point for almost all fluid dynamics are the Navier-Stokes and continuity equations, which are derived from momentum and mass conservation within the fluid. In their most general form these equations are too unwieldy for ocean modelling, but after a few reasonable approximations (like assuming a constant background density and small vertical velocities), we arrive at the so-called primitive equations.

I won’t go into too much detail here, but I think it is still nice to show the equations in full so you can get an idea of the complexity of the problem we are trying to solve. They can be written like this:

$$ \frac{\partial \vec{v}_h}{\partial t} + (\vec{v} \cdot \nabla) \vec{v}_h + f \hat{k} \times \vec{v}_h + \frac{1}{\rho_0} \nabla_h p' = \vec{\mathcal{F}} \\ \nabla_h \cdot \vec{v}_h + \frac{\partial w}{\partial z} = 0 $$

$$ \frac{\partial p'}{\partial z} = -g \rho' $$

$$ \rho' = \rho(\theta, S, p_0(z)) - \rho_0 $$

$$ \frac{\partial \theta}{\partial t} + (\vec{v} \cdot \nabla) \theta = \mathcal{Q}_\theta $$

$$ \frac{\partial S}{\partial t} + (\vec{v} \cdot \nabla) S = \mathcal{Q}_S $$

This is a set of 7 coupled, nonlinear partial differential equations. The primitive equations describe the evolution of velocity $\vec{v} = (u, v, w)$ ($\vec{v_h} = (u, v)$), pressure $p$, density $\rho$, temperature $\theta$, and salinity $S$ in time $t$ and space $(x, y, z)$. $f, \rho_0$, and $g$ are constants; and $\vec{\mathcal{F}}$, $\mathcal{Q}_\theta$, and $\mathcal{Q}_S$ represent dissipation and forcings (which are usually quite complex terms, too).

So how can we even solve complex equations like these on a computer? One of the simplest ways is to discretize them using a finite difference method.

Discretization¶

The basic idea is to define all quantities (like pressure and velocity) at fixed locations on a computational grid. This implies that we are now dealing with discrete quantities $(y_0, y_1, \ldots, y_N)$ instead of their true continuous versions $y(x)$. If you are familiar with calculus, you probably know that a derivative can be written as a difference between neighboring grid cells, divided by a small step size:

$$ \left. \frac{\partial y}{\partial x} \right|_{x_i} \quad \sim \quad \frac{y_{i+1} - y_i}{\Delta x} $$

This converges to the “true” value in the limit of smaller and smaller $\Delta x$. Writing gradients like this is the central idea of finite difference discretizations, and although there are many different ways to write these discrete gradients — with different numerical accuracies and stability properties — the principle is always the same.

For time derivatives, we typically use a multi-step method:

$$ \frac{\partial y}{\partial t} = f(t, y) \quad \sim \quad \frac{y^{n+1}-y^n}{\Delta t} = \frac{3}{2} f(t^{n}, y^{n}) - \frac{1}{2} f(t^{n-1}, y^{n-1}) $$

This requires us to store the solution at different time steps, but is otherwise not more difficult than a simple forward difference.

By replacing all gradients in the primitive equations through their discrete counterparts, we arrive at a set of discrete equations that can be stepped forward in time. To solve them on a computer, all we need to do is to define our physical variables as arrays (where array indices correspond to grid locations), and perform the right finite difference operations. For example, in vectorized Python code, we can write a gradient operation like the one above through index shifts:

# discrete version of ∂y/∂x
#
#             y_{i+1}   y_{i}
#                |        |
dy_dx[1:-1] = (y[2:] - y[1:-1]) / dx

For this to work, we need to pad all arrays by 1 extra element along each dimension (so the original data is y[1:-1]). Then, we can compute finite differences by shifting slices. The extra elements y[0] and y[-1] are called “ghost cells” or “overlap”.

Parallelization¶

Solving the primitive equations is very computationally expensive. Our model domain is typically the whole globe, and the non-linear nature of the primitive equations impose tight constraints on the largest time steps we can take. To make things worse, the ocean has a long memory of hundreds of years, so we need to run very long simulations until they reach a steady state — it’s not uncommon that a setup runs for more than 1 million time steps.

Because of this, even relatively low-resolution models like 3×3° (with about 600,000 grid elements) should be run on more than one process, unless you are willing to wait on results for months. High-resolution setups typically run on thousands of processes (CPU cores) across dozens of computational nodes.

The basic idea to execute the model in parallel is a simple domain decomposition. Every process takes ownership of a chunk of the total domain, and iterates it forward in time. Of course, neighboring processes need to exchange information from time to time, e.g. by sending messages through MPI (Message Passing Interface). Luckily, we can re-use our overlap cells for this! So by filling the ghost cells of each chunk with the current solution from the process neighbor we can ensure that the final solution is identical to that of the sequential model. This process is also called halo exchange.

Distributed modelling via halo exchange. Each process (P1-P9) owns a chunk of the total domain and exchanges information with its neighbors. Received information is written into the chunk’s overlap cells.

How often communications need to happen depends on how far each index is shifted, and the size of the overlap region. An overlap of 2 on each edge gives us enough leeway to execute 2 forward operations on the same array before having to communicate.

Now we are ready solve the primitive equations on an arbitrary number of processes with vectorized array operations. JAX is an excellent fit for this task, as we will see in the next section.

JAX for high-performance computing¶

Even though most people use JAX for machine learning, it is actually a great choice for high-performance computing. Through its just-in-time (JIT) compiler, JAX has good all-round performance on CPU and GPU, and the API is close enough to NumPy to make it easy to port existing code.

To demonstrate how this works in practice, I will show you how to implement the time stepping for a simple partial differential equation in JAX. Here is the equation, and the resulting JAX code:

$$ \frac{\partial h}{\partial t} = - \frac{\partial f_e}{\partial x} - \frac{\partial f_n}{\partial y} $$

# compile function with jit for speed
@jax.jit
def update_h(h, dh, fe, fn):
    """Step h forward in time."""

    # compute right hand side
    dh_new = dh.at[1:-1, 1:-1].set(
        -(fe[1:-1, 1:-1] - fe[1:-1, :-2]) / dx
        - (fn[1:-1, 1:-1] - fn[:-2, 1:-1]) / dy
    )

    # step in time via multistep integration
    h = h.at[1:-1, 1:-1].add(
        dt * (1.5 * dh_new[1:-1, 1:-1] - 0.5 * dh[1:-1, 1:-1])
    )

    # enforce cyclic boundaries and handle inter-process communication
    h = enforce_boundaries(h)
    return h

Applying this function to a model state with variables {h, dh, fe, fn} yields a new state, with h stepped forward in time by dt. And because we can write finite difference operations in a fully vectorized way with JAX (via index shifts), the implementation ends up clean and fast.

The only big obstacle left to figure out is communication between processes — i.e., what happens inside enforce_boundaries. In functions decorated with @jax.jit, arrays can only be manipulated through transformations that are known to the underlying compiler, XLA. This means that we would have to break control flow at every communication operation to leave JIT, perform the operation, and then re-enter a JIT block. This is ugly: it complicates the code structure, and we are leaving performance on the table by applying JIT to smaller blocks at a time.

To solve this problem I co-developed mpi4jax, which registers MPI operations with XLA, so we can use them within JIT blocks. Here is a simplified implementation of enforce_boundaries, where we use mpi4jax.sendrecv to exchange information:

@jax.jit
def enforce_boundaries(arr, grid):
    """Exchange overlap between processes."""
    token = None
    send_order = ("west", "east")
    recv_order = ("east", "west")

    # loop over neighbors
    for send_dir, recv_dir in zip(send_order, recv_order):
        # determine neighboring processes
        send_proc = proc_neighbors[send_dir]
        recv_proc = proc_neighbors[recv_dir]

        # determine data to send
        send_idx = overlap_slices_send[send_dir]
        send_arr = arr[send_idx]

        # determine where to place received data
        recv_idx = overlap_slices_recv[recv_dir]
        recv_arr = jnp.empty_like(arr[recv_idx])

        # execute send-receive operation through mpi4jax
        recv_arr, token = mpi4jax.sendrecv(
            send_arr,
            recv_arr,
            source=recv_proc,
            dest=send_proc,
            comm=mpi_comm,
            token=token,
      )

        # update array with received data
        arr = arr.at[recv_idx].set(recv_arr)

      return arr

(for a full, working example see the mpi4jax repository)

With this in place, we can do fully distributed simulations on CPU and GPU, with just a few lines of Python code.

Veros + JAX in action¶

Because the whole model is written in Python, getting started with Veros is pretty easy. For example, if you already have a working Python installation and current CUDA drivers, the following screencast shows you all you need to do to:

Install Veros and all dependencies
Run a global 1x1° setup on GPU

From an empty environment to running Veros on GPU in a handful of commands. Screencast in 2x speed.

Leaving this setup running on a high-end GPU for about 24 hours finally leads to output like this, which shows us all the major ocean circulations:

Output of the global 1x1° setup. Barotropic streamfunction after 10 model years. The ocean circulation runs along the plotted streamlines.

Similarly, we can run Veros on multiple CPU cores (in this case 4) like this:

$ mpirun -n 4 veros run global_1deg -b jax -n 2 2

(after installing MPI, mpi4py, and mpi4jax)

Turns out it’s pretty fast¶

Now we can finally run some benchmarks of the new JAX backend. Because the dynamical core of Veros is a one-to-one translation of a Fortran model to Python, we can also do a direct comparison between JAX and the original Fortran code.

First up, we compare the performance on a single computer with 24 CPU cores and a Tesla P100 GPU. This benchmark shows you how the computational efficiency depends on the number of grid elements:

JAX performance is very close to Fortran, both with and without multiprocessing (via MPI), while a single GPU easily outperforms 24 CPUs. Shown are full model benchmarks on a single machine with a varying number of grid elements.

We can see that single-process NumPy is 3-4x slower than single-process Fortran, while JAX is a bit faster (mostly because JAX has some thread parallelism under the hood). On all 24 CPU cores, JAX is marginally slower than Fortran, while JAX on GPU outperforms everything.

But this is only what we get on a single computational node. Realistic ocean models need to run much faster than that, so we have to study how Veros scales to multiple nodes in a compute cluster. For CPU, we measured this:

JAX performance is very close to Fortran, even when using hundreds of CPU cores. Shown are full model benchmarks on a CPU cluster with fixed number of grid elements (6M) and varying number of processes.

Again, JAX is only slightly slower than Fortran or breaks even, with NumPy far behind. This means that we are able to match the performance of Fortran, a highly optimized language made for high-performance computing, with our pure Python model + the JAX compiler, without any of the baggage that comes with Fortran models.

But the real star is this benchmark, where we see how Veros / JAX scales to multiple GPUs:

For big problems that completely fill each GPU, scaling to multiple GPUs is almost perfect. Shown are full model benchmarks on a a2-megagpu-16g Google Cloud instance with 16 NVIDIA A100 GPUs. Fixed number of grid elements (weak scaling; left) and fixed number of grid elements per GPU (strong scaling; right). x-axis shows number of GPUs.

These results are a bit difficult to unpack, but the gist is this: If we can decompose the computational domain in such a way that every GPU is fully utilized, scaling to more devices is almost perfect. This is what allowed us to run a very high resolution simulation (global 0.1°) on a single Google Cloud instance with 16 GPUs — which people typically run on at least 2000 Fortran processes. You could already see the result at the start of this article, but here it is again:

The turbulent ocean. This high-resolution (0.1°) snapshot of the ocean was simulated with Veros on 16 A100 GPUs on a single Google Cloud VM, faster than 2000 CPUs running a Fortran model.

Differentiable physics¶

Now that we have a fast ocean model in Python, what’s next? Most of all, I hope that people will simply find Veros enjoyable to work with, and use it to understand our earth and climate.

But there is one more new, interesting direction, namely the integration of machine learning models into the physical simulation. This is possible because JAX offers more than just a JIT compiler: JAX functions are also differentiable, which opens up a whole new of possibilities (Veros is not fully differentiable yet, but could be with some more effort).

In particular, there is an emerging field of physics-based deep learning that integrates machine learning with physical modelling. In case of a differentiable physical model, these “hybrid” systems can be trained end-to-end, which tends to make training much more efficient. There are already differentiable models for fluid dynamics in JAX (namely PhiFlow and jax-cfd), but — as far as I know — Veros is the first that supports realistic ocean setups.

Finally, I hope that I managed to share some of my excitement about working on modern physical models! If so, you are welcome to contribute.

Bayesian histograms for rare event classification

2021-09-23T00:00:00+02:00

Contents

Extreme events call for extreme measures
The problem with histograms for rare events
Bayes to the rescue
Significant bins only!
It works™

Bayesian histograms are a stupidly fast, simple, and nonparametric way to find how rare event probabilities depend on a variable (with uncertainties!).

My implementation of Bayesian histograms is available as the Python package bayeshist. So if you think this could be useful, just install the package and try it out:

$ pip install bayeshist

Extreme events call for extreme measures¶

→

(1) 1 million samples containing a binary rare event $y$, depending on a parameter $x$. This is what the model sees. (2) Bayesian histogram estimate of event rate $p(y=1 \mid x)$

Suppose you want to estimate how the risk of some rare event depends on certain factors. For example, given some variables about a person, you want to know how likely it is they will develop a rare disease, or fail to pay back their mortgage.

In both cases, the probability that this will happen is probably very small, no matter how predisposed someone is, which makes it a rare event. Additionally, the probability of such an event to happen is always larger than zero. In machine learning, we say that the data is not separable: The distributions of positive and negative labels overlap, so there will be no decision boundary — no matter how complicated — that will separate them perfectly.

This is a hard problem that most machine learning algorithms are not equipped to solve. Most classification models assume that the data is separable, or output badly calibrated probabilities, which is a no-go for rare events (because probabilities are what we are interested in). On top of this, we often have a massive amount of data points, but only few interesting ones.

I came across this problem during my own research on extreme ocean waves (rogue waves). There is always a small probability to encounter a rogue wave, but how does this probability vary in different conditions? And do we even have enough data to tell anything?

To answer these questions, I use something that I call Bayesian histograms, which tell us how the probability of a rare event changes, and how certain we are in this estimate. In short, they help us to estimate event rates from samples (as in the figure above).

In this post I will introduce the idea behind the method and explain how it works. If you’re just interested in the results, feel free to skip ahead.

The problem with histograms for rare events¶

We are given a dataset of binary samples $y$ and a parameter $x$, and our task is to find out how the probability of $y=1$ changes with $x$, i.e., $p(y=1 \mid x)$ (we call this the event rate). We decide to plot your samples up in a scatter plot, and what we get is this:

1 million binary samples. With only 1000 positive samples, $y=1$ is a rare event.

Unfortunately, this tells us nothing about $p(y=1 \mid x)$. We can see that there are less positive samples for more extreme values of x, but there are also less negative samples. To see what’s going on, we make a histogram next:

Histograms of positive and negative samples, raw counts (left) and normalized (right).

Now we see that there are always a lot more negative than positive samples, and we also see that there seem to be some local peaks in the distribution of $y=1$. But what we actually want is the frequency of $y=1$ relative to $y=0$. We can get a first estimate of $p(y=1 \mid x)$ by computing the ratio of the 2 histograms:

A first estimate of the event rate $p(y=1 \mid x)$.

This is pretty useful already: It looks like the event rate is higher towards negative values of x, and there seem to be 2 local peaks. But there are still some problems with this approach. For example, do we even have enough data to determine the event rate for every bin? What about bins with no positive samples? And how many bins should we choose?

Bayes to the rescue¶

Bayesian histograms address these issues by adding uncertainties to $p(y=1 \mid x)$. For this, we need to make some assumptions about the data generating processes.

We assume that, within each bin $i$, the number of positive samples $n^+_i$ is drawn independently with fixed probability $p_i$ (this is what we want to estimate) and number of negative samples $n^-_i$. Then, $n^+_i$ follows a binomial distribution:

$$ n^+_i \sim \operatorname{Binom}(n^-_i, p_i) $$

Our goal is to estimate $p_i$ via Bayesian inference. For this, we still need a prior for the variable $p$, which encodes our belief of what $p$ is before measuring any data (this also takes care of empty bins). A convenient choice is a beta distributed prior with 2 parameters $\alpha_0, \beta_0$:

$$ p(y=1) \sim \operatorname{Beta}(\alpha_0, \beta_0) $$

There are many ways to choose the prior parameters. Popular choices are complete ignorance ($\alpha_0 = \beta_0 = 0$; all values of $p$ are equally likely) and Jeffrey’s prior ($\alpha_0 = \beta_0 = 1/2$). In bayeshist, we use a weakly informative prior by default that ensures that the prediction of an empty bin is the global mean event rate with a big uncertainty.

Now we are ready to compute the posterior distribution of $p_i$. According to Bayes’ theorem, it is given by the normalized product of (beta) prior and (binomial) likelihood. A neat property of the beta prior is that it is conjugate to the binomial likelihood, so the posterior is a beta distribution, too:

$$ p(y=1 \mid n^+_i, n^-_i) \sim \operatorname{Beta}(n^+_i + \alpha_0, n^-_i + \beta_0) \tag{1} $$

To get uncertainties on our event rate estimates, all we need to do is to compute a credible interval of the posterior (1) via its quantiles. In Python this looks like this:

# compute number of positive and negative samples per bin
>>> n_plus = 10
>>> n_minus = 10_000_000

# Jeffrey's prior
>>> alpha_0 = beta_0 = 0.5

# define posterior
>>> p_posterior = scipy.stats.beta(n_plus + alpha_0, n_minus + beta_0)

# evaluate posterior mean
>>> p_posterior.mean()
1.0499988450012706e-06

# 98% credible interval
>>> p_posterior.ppf([1e-2, 1 - 1e-2])
array([4.44859565e-07, 1.94660572e-06])

This tells us that, with 98% certainty, the event rate for this sample lies between $4.5 \cdot 10^{-7}$ and $1.9 \cdot 10^{-6}$, with a mean value of $1.0 \cdot 10^{-6}$.

Computing quantiles of a beta distribution is very fast, so we can perform this calculation for every histogram bin with almost no additional compute cost. This is what bayeshist does to compute a Bayesian histogram.

from bayeshist import bayesian_histogram, plot_bayesian_histogram

bins, bin_posterior = bayesian_histogram(X, y, bins=20, pruning_method=None)
plot_bayesian_histogram(bins, bin_posterior)

Bayesian histogram estimate of event rate $p(y=1 \mid x)$.

Now we have an estimate for all bins (even empty ones), and we can see that we have enough data to say that the variation we see for $|x| \leq 2$ is statistically significant.

But there is still one major open question: How many bins should we use?

If we use too many bins, the sample size per bin will be tiny, so our uncertainties will be huge.
If we use too few bins, there will be a considerable variation of $p(y=1 \mid x)$ within each bin, which means that one of our main assumptions (that the event rate is constant) is violated.

We address this in the next section.

Significant bins only!¶

If we can come up with a way to determine whether 2 bins are significantly different, we could start with a high number of bins and merge non-significant bins until we are left with an optimal binning. This is the main idea behind histogram pruning.

Imagine that we want to compare 2 bins with samples $(n_1^+, n_1^-)$ and $(n_2^+, n_2^-)$. We formulate the following hypotheses:

H1: Sample 1 is drawn from bin 1 with
$$ p_1 \sim \operatorname{Beta}(\alpha_1 = n_1^+ + \alpha_0, \beta_1 = n_1^- + \beta_0) $$
and sample 2 is drawn from bin 2 with
$$ p_2 \sim \operatorname{Beta}(\alpha_2 = n_2^+ + \alpha_0, \beta_2 = n_2^- + \beta_0) $$
H0: Both samples are drawn from the merged version of both bins with
$$ p_{tot} \sim \operatorname{Beta}(\alpha_{tot} = n_1^+ + n_2^+ + \alpha_0, \beta_{tot} = n_1^- + n_2^- + \beta_0) $$

Now we want to know: how much more likely is it to measure the samples under H1 compared to H0? To answer this, we have to compute the likelihood of each sample, while accounting for every possible value of the unknown event rate $p$ (the result is also called a Bayes factor). This is typically done by integrating out unknown parameters from the posterior (marginalization). In our case of a binomial likelihood with beta-distributed event rate, the result is a Beta-binomial distribution:

$$ n^+_i \sim \operatorname{Betabinom}(n^-_i, \alpha_i, \beta_i) $$

Now we can evaluate the likelihood of each sample under H1 and H0, and compute the ratio (Bayes factor). In Python code, this looks like this:

def sample_log_likelihood(n_plus, n_minus, alpha, beta):
    """Probability to measure `n_plus` positive and `n_minus` negative events"""
    sample_posterior = scipy.stats.betabinom(
        n_plus + n_minus, alpha, beta
    )
    return sample_posterior.logpmf(n_plus)

def sample_bayes_factor(n_plus_1, n_minus_1, n_plus_2, n_minus_2, alpha_0, beta_0):
    """Bayes factor to decide between separate and merged bins

    (higher values -> splitting more favorable)
    """
    alpha_1 = n_plus_1 + alpha_0
    beta_1 = n_minus_1 + beta_0

    alpha_2 = n_plus_2 + alpha_0
    beta_2 = n_minus_2 + beta_0

    alpha_tot = n_plus_1 + n_plus_2 + alpha_0
    beta_tot = n_minus_1 + n_minus_2 + beta_0

    return np.exp(
        sample_log_likelihood(n_plus_1, n_minus_1, alpha_1, beta_1)
        + sample_log_likelihood(n_plus_2, n_minus_2, alpha_2, beta_2)
        - sample_log_likelihood(n_plus_1, n_minus_1, alpha_tot, beta_tot)
        - sample_log_likelihood(n_plus_2, n_minus_2, alpha_tot, beta_tot)
    )

With this we can finally find the optimal number of bins! The algorithm works like this:

Start with a relatively high number of bins (for example 100).
Compare a neighboring pair of bins with samples $(n_1^+, n_1^-)$ and $(n_2^+, n_2^-)$. If the data is at least $\epsilon$ times more likely under H1, we do nothing. Otherwise, we revert the split by merging the 2 bins, and replace them by a single bin with $(n_1^+ + n_2^+, n_1^- + n_2^-)$.
Proceed with the next pair, or start over with the first pair when reaching the end of the domain.
Stop when no more neighbors can be merged.

This is how this looks in action:

Your browser does not support the video tag.

Bayesian histogram pruning.

(If you prefer a Frequentist method, bayeshist also supports Fisher’s exact test to test whether the Beta distributions of neighboring bins differ significantly. The results are very similar.)

It works™¶

Before I show you an example of a case where Bayesian histograms work well, let me warn you loud and clear: You can only trust Bayesian histograms if their underlying assumptions are fulfilled. That is, events must occur independently, and $p(y=1)$ must be approximately constant within every bin. Histogram pruning can help with finding a partition that satisfies the latter condition, but comes with another caveat: the resulting bins are only reasonable if the parameter space is well resolved. If you have big gaps in your data coverage that miss a lot of variability of $p(y=1 \mid x)$, bins will be merged too aggressively. So in practice, it can be a good idea to use both pruned and unpruned Bayesian histograms.

With that out of the way, here is the result on the example above, and the true event rate that the samples are generated from:

In this case, a pruned Bayesian histogram is a good representation of the true event rate.

See how bins get smaller the more variability there is in the true event rate? This is the effect of histogram pruning. If we had more samples, we could also resolve the oscillations towards the edges of the figure, but at this level of significance they are grouped into a single bin. Pruned Bayesian histograms are especially good at resolving local maxima in the event rate, which are often the most interesting regions — they even manage to get the full peak height right (within the uncertainty).

We can also compare pruned and unpruned histograms on this task:

Output of pruned and unpruned Bayesian histograms on the same task.

Unpruned histograms are a bit more faithful when it comes to regions with very little data close to the edges of the figure (they show accurately where there are gaps in the data coverage with huge uncertainty). But pruned histograms are much better at resolving small-scale features with reasonable confidence.

Finally, we can compare the performance of bayeshist to the Python package optbinning. optbinning is much more powerful and does a lot more than estimating binary event rates. But on this particular task, it looks like Bayesian histograms work better:

Compared to optbinning, Bayesian histograms are able to resolve both local peaks at full height.

So why don’t you give bayeshist a try, and let me know what you think!

Creating a better science conference poster

2021-09-10T00:00:00+02:00

My recent posters that I describe in this blog post. (click for higher resolution)

Good and bad posters¶

By far the most popular post on this blog is this one: Creating a Science Conference Poster with Inkscape. I have had mixed feelings about this for a while, because by now I think that the poster is not very good:

Yikes. So much text!

Don’t get me wrong, I still think it looks decent. But I made this as a part of a class project during my Master’s, before I had ever been to a conference (yet alone presented a poster). This has the effect that it just doesn’t work.

But let’s take a step back. What are science posters supposed to do? Here’s what I came up with:

Attract initial attention.
Ensure that people register your main finding, even if they just walk by.
Tell a clear story of your research.
Contain enough detail so you can use it to have a nuanced discussion during the poster session.
Promote your publications.
After the conference is over, act as eyecandy in the hallways of your institute.

I think the poster above does well in some of these points — it certainly catches attention, and it contains a lot of (too much) detail — but it does a terrible job at promoting the main results. Let’s fix that.

#betterposter — Less is more¶

#betterposter is a movement advocating for a new style of posters, sparked by this tweet:

Let's fix academic posters! Prepping a poster for #SIOP19 and sick of the old “wall-of-text” poster design? Watch this cartoon to see a new, faster approach to designing research posters. Includes templates. #betterposter https://t.co/wyXhiP5CKl pic.twitter.com/UBYB2GzIv9
— Mike Morrison (@mikemorrison) March 25, 2019

It also comes with this template:

A prototypical #betterposter

Now while I don’t think this is perfect, it does get so much right. Putting your main finding at the most prominent part of the poster is immensely helpful to promote your research to others, even if they just stroll by, and it helps you to tell a clear story in your poster.

Inspired by this, I designed these 2 posters (presented at the virtual EGU general assembly 2021):

#betterposter 1: Real-world rogue wave probabilities. (click for higher resolution)

#betterposter 2: Higher-level geophysical modelling. (click for higher resolution)

Keep in mind that this was a virtual conference, so these posters are optimized for screen reading rather than printing. For example, this means that they are in 16:9 aspect ratio (1.77 rather than 1.42 like A0 paper), and they contain less detail than I would put on a physical poster (because I only had a couple of minutes to show the poster).

Both posters worked great, and I had the feeling that people were generally engaged in the discussions. For the first poster I actually won an “Outstanding Student and PhD candidate Presentation” award at EGU 2021, so it seems like it served its purpose well.

So you like fancier posters?¶

Alright, I get it. Sometimes you need to feast your eye on something pretty.

For a recent physical conference I wanted to see whether I could salvage the poster from my previous blog post. This is what I came up with:

Visual poster: Inferring the causes of real-world rogue waves. (click for higher resolution, or download PDF version)

This is closer to what people typically think of when they hear the term “poster”: Something that is nice to look at, and where the visual impact helps to drive the message home.

Compared to its previous iteration, this design has undergone significant changes. I removed all text boxes across the borders to let the design breathe and put the focus on the important parts. I personally think that the figures should do most of the talking, so all text on the poster just serves to explain what’s shown in the figures.

This time, I decided not to use a main finding as the title, simply because this is research in progress, so I don’t have a main finding yet. This was for a smaller conference with a higher ratio of experts among the attendees, so I think a bigger focus on details is fine in this case.

Unfortunately I couldn’t attend the conference, so I don’t know how it was received, but I am really satisfied with this design.

Which tool should I use?¶

My original post was about making posters in Inkscape, but I think that you can make great posters with any vector tool.

You can use Inkscape, the Adobe suite (Illustrator / InDesign), or a presentation tool like PowerPoint or Keynote. Actually, I made all posters above in Keynote, and it worked great.

My only piece of advice is: for the love of God, do not use LaTeX. I know there are some people who hate the thought of having to use PowerPoint for science. But LaTeX is just not a very good tool to make great posters. Posters have to work visually, which is super hard to get right with the edit-compile-review workflow. And things that LaTeX is great at (like page layouts) just don’t matter for posters.

Do yourself a favor and use a tool where you can drag a box around until it just looks right.

Now go and make your own!¶

Feel free to use my designs to make your own poster, and let me know if you want to show off your work or have anything else to add (e.g. in the comments or on Twitter).

Higher-level geophysical modelling

2021-04-20T00:00:00+02:00

By Roman Nuterman, Dion Häfner, and Markus Jochum.
(Niels Bohr Institute, Copenhagen, Denmark)

This post is the display material for the vEGU 2021 abstract “Higher-level geophysical modelling” in session AS1.1 (Recent Developments in Numerical Earth System Modelling).

Therefore, it is a bit more technical than my usual blog posts, and assumes that you are somewhat familiar within the field of numerical modelling.

Contents

A paradigm shift
The taxonomy of high-level modelling
Let’s talk about performance
Abstraction is key
Quo vadis?

A paradigm shift¶

The year is 2021, but numerical modelling and high-performance computing are still bastions of low-level programming languages. Most (finite difference) models are written in Fortran or C, which have been around since the early days of computing.

This is not surprising on the very largest compute scales like the CMIP climate model ensembles, which run on the world’s largest supercomputers. In this case, even small performance drops could end up consuming funding for several human positions. But this is an extreme example, and typically, human time tends to be more valuable than computer time. (Just think of your poor PhD students trying to compile the model code or get their setup to work.)

On top of this, the efficiency of GPUs has increased dramatically in tandem with the recent machine learning boom. This has also lead to more heterogeneous compute architectures than ever. For example, Finland’s LUMI supercomputer will consist of 550 PFLOP/s worth of GPUs (on top of about 200,000 CPU cores).

We think that these two developments call for a new, flexible generation of geophysical models:

Flexible to run. The same code needs to be able to run on CPU and GPU hardware stacks.
Flexible to use. Simple to install and get started.
Flexible to modify. Readable code with helpful abstractions. Easy to re-build.
Flexible to integrate. Simple to interface with external libraries for plotting, post-processing, machine learning, other models, …

We call these flexible models “high-level”, and to meet these design goals, at least a part of them needs to be implemented in a high-level programming language.

The taxonomy of high-level modelling¶

Essentially, high-level modelling comes in 3 different flavors. Each of these is a good way forward, and definitely a step up from the status quo.

Type I: High-level frontend, low-level backend

A typical example is climt, an atmospheric model that wraps modular computational kernels written in Fortran with a Python user interface, or PyOM2, an ocean model with a similar structure.

Those models have great CPU performance out of the box, and are straightforward to implement.

Our biggest concern with models of this type is the lack of GPU support. Developers need to maintain a seperate backend implementation, for example in CUDA, to run on GPU. Developing and maintaining 2 implementations of the same code is more than many academic projects can handle.
Type II: High-level model in a niche programming language

This is the approach pursued by the Climate Modelling Alliance‘s Oceananigans.jl model, implemented in Julia.

Julia in particular has excellent performance, first-class GPU support, and a growing scientific library ecosystem. Therefore, it has tremendous potential to become the dominant language for scientific computing.

On the other hand, Julia’s focus on scientific applications is both blessing and curse. In this day and age, a lot of the progress in computing is driven by applications outside academia (mostly through machine learning). Sticking to a programming language that is not (yet) widely established means that such synergies can’t be exploited.
Type III: High-level model in a widely used programming language

Currently, Python is the only programming language that fits this category. Python is being used extensively both inside and outside academia, and probably has the largest scientific library ecosystem of any programming language.

The only example we know for this type is our Python ocean model Veros. (We’re sure there’s more - if you know or maintain a high-performance model in Python, please reach out.)

Although we have the highest respect for Type I and II projects, we argue that Type III is the most valuable type of model, because it is easier to use / modify (more people are already familiar with the language and ecosystem) and integrate (larger library support) than the other types.

If you need some evidence for that last statement, just look how easy it is to install and use Veros from a clean Linux environment:

Installing and running Veros, starting from a fresh environment. Screencast in real time.

Unfortunately, this type is also the hardest to get right. The main problem is finding the right trade-off between readability, performance, and abstraction.

For the remainder of this blog post, we will discuss what it takes to build high-performance models in Python, and where we should go as a community to make this as painless as possible.

Let’s talk about performance¶

Model performance is the elephant in the room whenever we discuss high-performance computing in Python.

As we argue in the introduction, human time is often more valuable than computer time, so we don’t think that it should be prioritized at all cost. However, model performance is a part of the user experience. A long feedback loop between designing an experiment and examining its results is catastrophic to overall productivity.

So, we have to care about performance to some degree. Luckily, this is largely a solved problem. It is already possible to match native Fortran performance in Python (within ±10%).

Pure Python / NumPy is of course nowhere close to Fortran — in our experience, a model written in NumPy is about 5x slower than its Fortran equivalent. But fortunately, there is a rich library ecosystem to accelerate Python code. Most of these libraries are geared towards machine learning, but nothing prevents us from using them for scientific computing instead. (Remember when we mentioned synergy as a major asset of Type III models?)

The following plots are from pyhpc-benchmarks, a repository we created to compare the performance of various Python frameworks on subroutines of our ocean model Veros:

Performance of various Python frameworks on TEOS-10 equation of state. Numba performance is similar to that of the underlying Fortran code.

Performance of various Python frameworks on Veros isoneutral mixing subroutine. Numba performance is similar to that of the underlying Fortran code.

There are two Python frameworks that show particularly strong performance, Numba and JAX. To give you an idea how this works, here is the same code snippet in Fortran, NumPy, Numba, and JAX:

Fortran

tke_surf_corr = 0.0
do j=js_pe,je_pe
  do i=is_pe,ie_pe
    if (tke(i,j,nz,taup1) < 0.0 ) then
      tke_surf_corr(i,j) = -tke(i,j,nz,taup1)*(0.5*dzw(ke)) /dt_tke
      tke(i,j,nz,taup1) = 0.0
    endif
  enddo
enddo

NumPy

mask = tke[2:-2, 2:-2, -1, taup1] < 0.0
tke_surf_corr = np.zeros_like(maskU[..., -1])
tke_surf_corr[2:-2, 2:-2] = np.where(
    mask,
    -tke[2:-2, 2:-2, -1, taup1] * 0.5 * dzw[-1] / dt_tke,
    0.
)

Numba

tke_surf_corr = np.zeros((nx, ny))
for i in range(2, nx-2):
    for j in range(2, ny-2):
        if tke[i, j, -1, taup1] >= 0.:
            continue
        tke_surf_corr[i, j] = -tke[i, j, -1, taup1] * (0.5 * dzw[-1]) / dt_tke
        tke[i, j, -1, taup1] = 0.

JAX

mask = tke[2:-2, 2:-2, -1, taup1] < 0.0
tke_surf_corr = jnp.zeros_like(maskU[..., -1])
tke_surf_corr = tke_surf_corr.at[2:-2, 2:-2].set(
    jnp.where(
        mask,
        -tke[2:-2, 2:-2, -1, taup1] * 0.5 * dzw[-1] / dt_tke,
        0.
    )
)

See how the Numba implementation is basically translated Fortran, including the explicit loops? Numba’s JIT compiler is very efficient at transforming explicit loops like this one, essentially giving us the same performance as the Fortran code after compilation.

Unfortunately, there is no way to use the same Numba implementation for both CPU and GPU, as efficient GPU code generation typically requires a vectorized approach instead of explicit loops.

JAX on the other hand reads very similarly to NumPy, but its JIT compiler generates code that is competitive with Numba / Fortran on CPU and has great performance on GPU (with speedups of 40x — 3000x over NumPy). The only major restrictions are that JAX arrays are immutable and all JAX functions have to be pure (i.e., have no side effects).

We have therefore decided on JAX as the new computational backend for Veros. First benchmarks of the full model show that, with JAX, high-resolution setups are ~10% slower than Fortran on CPU, and about as fast as 50 Fortran CPUs on a single high-end GPU.

We have not measured power consumption yet, but with some back-of-the-envelope math, this should yield a GPU model that is at least 2x more energy efficient than its Fortran equivalent.

We think that JAX shows exceptional promise to become the de-facto computational high-performance backend for Python, because of its user friendliness and consistently high performance on both CPU and GPU.

Abstraction is key¶

Surprisingly, the main problem with high-level models is not performance, but to find the right level of abstraction.

Just translating model code from Fortran to Python does not make it more readable.

In fact, it can be significantly less readable, because in Python, there is a delicate balance between performance and readability. The most readable way to write the code will not be performant, and the most performant way to write the code will not be readable. In our experience, finding the perfect middle ground is extremely hard, and the different performance characteristics of each computational framework make it so there is no universal answer to this.

Additionally, people have been writing model code in Fortran since the 1970s, but are only starting to do so in Python / NumPy / JAX. This means that there are no established community standards on how to write models.

In our experience, more abstraction is needed at all levels of a Python model:

At the lowest level, to separate numerics from physics.

dydx(salt, order=1) represents the intent of the code much better than (salt[1:] - salt[:-1]) / dx. Numerical computations should also be aware of physical units and be able to perform conversions between them (e.g. via pint). Additionally, the code representing the physics should looks the same regardless of the computational backend used (NumPy or JAX or Fortran or something else).
At the intermediate level, to encapsulate model state and define the data flow between model routines.

There are several projects that address this, including sympl, xarray-simlab, and climlab. But neither has been adopted by more than a handful of projects yet, which we interpret as evidence that they are not flexible or approachable enough just yet.
At the highest level, to provide a common interface for setup specification, introspection, coupling, and interactive data analysis.

Ideally, running the model should happen in the same environment and use the same tools as post-processing of its output. For example, every physical model could expose its state as a self-describing xarray dataset.

We can address these points only through dialogue and collaboration. So if you have an idea or a project that can scratch one of these itches, please share.

Quo vadis?¶

While we think that high-performance modelling in Python has several decisive advantages, the future is still unclear.

To be fair, there is a movement to revive Fortran called “modern Fortran”, and a modern LLVM-based Fortran compiler LFortran, which are efforts we applaud. Fortran is still immensely powerful and a good language to write performant CPU models in.

But if the current trend continues, first-class GPU support will become more and more important. This alone means that there is no turning back (usability issues aside).

We are therefore convinced that Type II and Type III models will eventually take over, but there is still a long way to go. We as a community need to find a way to handle the increased complexity of more dynamical languages, but it can be done.

The future of high-level modelling is bright.

If you enjoyed this post, make sure to visit my vPICO presentation during vEGU 2021 (Tue, 27 Apr), and join the discussion afterwards.

I’m also happy to respond to your comments below.

Learning to play Yahtzee with Advantage Actor-Critic (A2C)

2021-04-01T00:00:00+02:00

My in-laws are really into the dice game Yatzy (the Scandinavian version of Yahtzee).

If you’re unfamiliar with the game, here’s a brief summary of the rules from Wikipedia:

Players take turns rolling five dice. After each roll, the player chooses which dice to keep, and which to reroll. A player may reroll some or all of the dice up to two times on a turn. The player must put a score or zero into a score box each turn. The game ends when all score boxes are used. The player with the highest total score wins the game.

Sounds easy enough, right?

My in-laws, who have much more experience than me, made very quick decisions, but I couldn’t see if they were really better than mine. Is it better to give up on getting a Yahtzee early on, or should you delay that for as long as possible? Is going for straights even worth it? How important is the bonus, really?

What to go for? A straight? Or keep the sixes? The right answer isn’t obvious, and depends on both your and your opponent’s scorecard.

While playing (and losing), I could never shake the feeling that I had no idea whether my strategy was good or not. Yahtzee is luck-based to a large degree, so it’s hard to judge whether you suck at the game or whether you’re just unlucky.

So, I finally thought to myself:

This sounds like a game that should be easy to learn for a bot. Maybe it can teach me how to play!

Specifically, I wanted to build a bot that could learn to play Yahtzee close to perfection through self-play (via reinforcement learning, RL). Turns out, I was wrong about the easy part, but a few weeks of intensive labor later I was done with my creation.

Lessons

Throughout this article, you will find some of the more salient lessons I learned in boxes like this one.

Contents

The Making of Yahtzotron
The Results Are In
- Showmatch Time!
- The Numbers
Final Thoughts
- Why Was This So Hard?
- Human learning — Machine Teaching

The Making of Yahtzotron¶

The nerve on this guy.

Some spoilers first:

Yahtzotron’s average final score is about 5% below perfect play, which is definitely competitive with experienced human players. Training time of the final agent is about 2 hours on a single CPU. And it’s available on GitHub, for you to try!

However, for a long time, it seemed like I bit off more than I could chew. As a first reinforcement learning project this was definitely a challenge. So, let me present to you the long and winding path towards a strong reinforcement learning agent, so that others may learn from my hubris.

If you want to skip ahead to the fun part, you can watch me play a full game at the end of this article.

Why Reinforcement Learning?¶

Yahtzee has a known solution for perfect play (and so does Yatzy). So, why go through all of this to learn a “solved” game?

I believe that a strong self-taught agent is still valuable, even if there is a known solution to the game.

Perhaps the biggest factor is efficiency. Exact solutions to probabilistic games have to search through all possible outcomes to identify the best action. This is computationally costly (without optimization, this scales exponentially with the number of game steps). Common optimization strategies such as dynamic programming are difficult to implement correctly, and efficient implementations need to be tailored to the problem at hand.

A second factor is flexibility. An agent that learns through self-play is robust to minor rule changes (this is why we can learn Yahtzee and Yatzy with the same agent). We can also change the objective. Most exact solutions to Yahtzee optimize the average game score, but I personally tend to play to win (a good winning agent might have to take more risks when it is behind, and play it safe when it’s ahead).

And finally, it can help us understand how to build efficient RL agents, so we can eventually tackle problems that do not have a known solution.

RL Frameworks — The JAX + DeepMind Stack¶

I wanted to use this opportunity to learn a new framework, so I decided to implement the training loop in JAX.

Unlike Pytorch and Tensorflow, JAX doesn’t supply a high-level interface for machine learning (instead, it relies on third-party libraries for that). I settled on the DeepMind stack: Haiku for neural networks, optax for optimization, rlax for reinforcement learning components.

Overall, the experience was pleasant, but not without obstacles. I had to file several bug reports while working on Yahtzotron. For more serious projects I would probably just go with Pytorch right now.

Lesson 1

Only go with the JAX ecosystem if you are prepared to implement most of the logic yourself. Also, be ready to work around bugs or performance issues.

(JAX is evolving fast. This advice is probably outdated soon, so make sure to give JAX a chance. Most of it is awesome already.)

Implementing Yahtzee / Yatzy¶

I started with coding up the rules for Yahtzee and Yatzy in pure Python.

For this, I introduced 2 classes: Ruleset and Scorecard. Ruleset encapsulates the core rules of the game: Which categories they are, how they are scored, and what bonuses there are (see e.g. the ruleset for Yatzy). Scorecard uses a Ruleset internally to keep track of filled categories and scores for each player.

In hindsight, I do like this extensible approach. However, I made one crucial mistake, which was not to wrap the game in OpenAI’s gym. I didn’t see how I could make it work with the changing action space, so I didn’t bother because I thought I didn’t need it.

I later realized that not having the game implemented in gym was a huge disadvantage. It made it so I couldn’t test my agent on other, simpler problems while debugging. And it also made it that I couldn’t switch out the A2C agent for a different one (such as PPO) when I would have liked to.

Lesson 2

Implement your problem in gym. You might think you don’t need to, but you probably will.

Detour: Genetic Optimization¶

I was intrigued to try genetic optimization as a baseline. This article gives an overview how genetic optimization can work in RL contexts. The idea is deceptively simple:

Initialize a league of agents with random weights.
Let them play each other repeatedly.
Compute a fitness based on how well each agent did.
Populate a new league by drawing agents at random, weighted with their respective fitness. (You could also do sexual procreation by drawing 2 agents and combining their weights every time, but it’s not really necessary.)
Mutate the weights of each agent by a small random number (e.g. drawn from a Gaussian with zero mean and small variance).
Repeat for as long as necessary.

Genetic optimization in a nutshell.

As expected, genetic optimization was easy to implement. All we need is an agent that can play games - no loss functions, no optimizers. You can treat your agent as a black box that you just evaluate based on its fitness.

But this simplicity comes as a cost. In my tests, the league advanced quickly at first, up to a mean score of about 130. But by then, progress had slowed down so much that it seemed stuck. (A decent game score is 200+.)

It is important to remember that real-life evolution needs one key ingredient to work: time. Unfortunately, we don’t have millions of years on our hands to wait for the perfect Yahtzee machine to evolve. So let’s get back to something more intelligent.

Lesson 3

Genetic optimization is simple to implement and can give you a decent baseline performance with little effort, but don’t expect super-human agents to come out of this.

Advantage Actor-Critic¶

After the detour to genetic optimization, it is time to return to reinforcement learning. I decided to try the A2C (advantage actor-critic) algorithm next.

If you are not familiar with A2C, here is an amazing introduction in the form of a cartoon.

The basic idea is pretty straightforward. The agent consists of 2 parts, the actor and the critic. Both receive the current state of the game as input.

The critic predicts the value of the current state in the form of what it thinks the total reward will be at the end of the game (in our case, the final score of the agent + an optional winning bonus).

The critic’s loss is essentially the accuracy of those predictions, evaluated through a mechanism called temporal difference learning or TD-λ. Temporal differencing accounts for the fact that the near future is safer to predict than the far future, and that rewards now are better than equal rewards later.
The actor predicts a probability with which each action should be taken, according to the current policy.

Its loss is based on something called the advantage: If the picked action was better than expected (based on what the critic estimated for it), the actor should take it more often, and its probability increases (and vice-versa for an action that was worse than predicted).

The total loss of the agent is then the summed loss of both actor and critic, plus an additional entropy loss that makes sure that the model keeps exploring different options.

For the implementation in JAX / Haiku, I used bsuite’s A2C agent as a template. One nice thing about JAX is how readable and “non-magical” the loss function looks:

def loss(
        weights,
        observations,
        actions,
        rewards,
        td_lambda=0.2,
        discount=0.99,
        policy_cost=0.25,
        entropy_cost=1e-3,
    ):
        """Actor-critic loss."""
        logits, values = network(weights, observations)
        values = jnp.append(values, jnp.sum(rewards))

        # replace -inf values by tiny finite value
        logits = jnp.maximum(logits, MINIMUM_LOGIT)

        td_errors = rlax.td_lambda(
            v_tm1=values[:-1],
            r_t=rewards,
            discount_t=jnp.full_like(rewards, discount),
            v_t=values[1:],
            lambda_=td_lambda,
        )
        critic_loss = jnp.mean(td_errors ** 2)

        if type_ == "a2c":
            actor_loss = rlax.policy_gradient_loss(
                logits_t=logits,
                a_t=actions,
                adv_t=td_errors,
                w_t=jnp.ones(td_errors.shape[0]),
            )
        elif type_ == "supervised":
            actor_loss = jnp.mean(cross_entropy(logits, actions))

        entropy_loss = -jnp.mean(entropy(logits))

        return policy_cost * actor_loss, critic_loss, entropy_cost * entropy_loss

(These are actually 2 losses in 1, either A2C or a supervised loss for the actor. This becomes relevant during pre-training, see next section.)

The only real struggle was to figure out how to handle action space constraints. In Yahtzee, the first 2 actions of the turn are keep actions (which dice should be kept for the next roll). The last action is a category action (which score category we should use for the roll). Both have a different number of possible actions. How do we encode this with a single output?

I decided to bake this into the network architecture, and replace the predicted logits with -inf if the action was invalid. By doing this however I opened Pandora’s Box, because NaN values started to pop up everywhere (where JAX functions couldn’t handle infinities). To work around this, in the loss function, I replace -inf with the smallest possible float instead.

(I have since read that people usually just give a negative reward to impossible actions and let the model learn the rules by itself. Perhaps I should have done that instead.)

Finally, after implementing A2C, I had it all laid out. Here is how Yahtzotron plays a turn:

How Yahtzotron plays a turn. The agent uses its value output to determine the value of the strongest opponent, which is used as an input later on. Then, it uses its policy output to select actions, which finally lead to a turn score (reward).

If you look closely, you will find some more features that I haven’t mentioned yet: Once, at the start of the turn, Yahtzotron uses its value output (from the critic, see above) to predict the value of the currently strongest opponent. This is used as an input to all decisions if Yahtzotron is playing to win (as opposed to maximizing expected score). Another thing I haven’t mentioned yet is the strategy output, which we will return to later.

Unfortunately, there’s no way to sugarcoat it: initial performance of the agent was terrible. It seemed to be unable to learn anything more sophisticated than super greedy, semi-random play with a mean score of about 100.

In the following sections, I will describe how I managed to convince the agent to go above this local maximum.

Lesson 4

A more complicated model could mean that you need to try harder to make it work (with potentially greater reward).

Pre-training via Greedy Look-up Table¶

To help the agent learn a better strategy, I decided to pre-train it on a simple baseline policy via supervised learning.

As the baseline policy I first used a naive, greedy strategy: Pick the action that yields the maximum expected score across all categories after the next roll. This can be pre-computed in a look-up table within a few seconds (there are only 252 distinct roll combinations for a single roll).

Unfortunately, this proved to be a very weak baseline. Being this greedy is highly suboptimal in Yahtzee, because you end up filling valuable categories early on that you might need as a buffer later (like Chance). With a mean score of around 120, this was able to make the agent somewhat better, but nowhere near optimal play.

Pre-training via Advantage Look-up Table¶

Next, I thought about how I as a human approach the game. It occurred to me that human players don’t think in discrete “keep” actions. Rather, they decide for a category to go for, and then pick the keep action that they think maximizes the score for this category.

But how to pick the optimal category? As a human I’m playing mostly opportunistic. If my current roll looks like it might become a better-than-average result for a category, I go for it.

For example, rolling 1 6 6 6 6 is certainly better than average for the “sixes” category, worse than average for the “ones” category, and much better than average for the “Yahtzee” category. I would try to roll a Yahtzee here (and keep the sixes).

So, this is the quantity that I use to pick the best category for this agent: The maximum expected score of each category given the current roll, minus the average score for this category across all rolls (I call that quantity advantage, as in A2C learning). We can pre-compute this with the same look-up table as in the greedy case - we just need to also compute the expected score for each category across all rolls.

This baseline is much much stronger, with an average score of about 220 for Yahtzee und 200 for Yatzy. This is a great baseline to lift our agents to the next level.

Lesson 5

Think about how you as a human approach the game. Simple heuristics often make for a strong baseline that you can use to pre-train your model.

Parameter Tuning¶

What followed next was lots. of. parameter. tuning.

RL agents have a large number of hyperparameters:

network architecture (e.g. number of layers and neurons);
learning rate & number of epochs;
reward discount & TD-λ;
reward norm & winning reward;
relative weight of policy, value, entropy loss terms;
batch size (here: number of players per game).

(possibly repeated for multiple learning stages)

I found that parameter tuning was even more important for this RL application than what I’m used to from “regular” Deep Learning.

Especially λ (as in TD-λ) had a huge influence on performance. λ can range from 0 to 1, where 0 implies maximum greed (only care about the reward of the next action), and 1 maximum patience (rewards later are as good as rewards now). I first started with a high λ of around 0.9 because I thought that patience was the right strategy, but it also made it harder for my agent to learn the right patterns.

Ultimately, I found it best to start out with a low λ (0.2), which I gradually increase during training to 0.8. This lets the agent learn simple greedy patterns first before strategizing more about the long run.

To give you an idea of the complexity of this tuning process, here is the function responsible for varying some of the hyperparameters during training:

def get_default_schedules(pretraining=False):
    """Get schedules for learning rate, entropy, TDlambda."""
    if pretraining:
        return dict(
            learning_rate=optax.constant_schedule(5e-3),
            entropy=optax.constant_schedule(1e-3),
            td_lambda=optax.constant_schedule(0.2),
        )

    return dict(
        learning_rate=optax.exponential_decay(1e-3, 60_000, decay_rate=0.2),
        entropy=(
            lambda count: 1e-3 * 0.1 ** (count / 80_000) if count < 80_000 else -1e-2
        ),
        td_lambda=optax.polynomial_schedule(0.2, 0.8, power=1, transition_steps=60_000),
    )

Lesson 6

When doing RL, don’t write your model off before doing at least some parameter tuning. In particular, try varying your time differencing parameter early on (λ).

Other Stuff I Did That Didn’t End Up Working¶

Predict only categories: I thought it would be more human-like to predict a category to go for and just take the keep actions that maximize expected score for that category (similar to the baseline agent). This did lead to somewhat faster training and the same final performance, but ultimately I knew that perfect play would be impossible with this, so I removed it.

Don’t dumb down your environment, your agent will learn to deal with complication.

Deterministic rolls: I figured it could help to give all agents in a game the same dice rolls, so they could explore different options without luck picking the winner. Nope. Made things worse.

Don’t remove randomness if it’s an integral part of your task.

Tailored neural network architectures: Instead of a “dumb” single-head feed-forward network I tried other architectures that I thought would be more fitting for the structure of the game. For example, I put a layer with only 5 neurons (because there are 5 dice) before the keep actions output layer. There was no positive performance impact.

No need to tinker too much with the architecture, an MLP will learn just fine.

JAXify everything: I wrote (almost) the whole turn logic in JAX, only to find that it was much slower than leaving it in NumPy. This is because games are played sequentially, so all arrays just contain a few dozen elements, which is not enough to amortize the overhead of using JAX.

Stick to NumPy for small array operations.

Lesson 7

Sometimes, less is more.

The Results Are In¶

Showmatch Time!¶

Here, you can watch me play (and lose to) Yahtzotron.

Yes, it rolled a Yatzy in the first round.

If you’re interested in playing against Yahtzotron yourself, here’s the code and the instructions.

The Numbers¶

Let’s have a look at how the final agents perform.

First up, we’ll have a four-way tournament across 10 000 games. This way, we can see what the average final score is. We will also test whether an agent trained with a winning bonus is better at winning than agents without it.

$ yahtzotron evaluate pretrained/yahtzee-score.pkl pretrained/yahtzee-score.pkl pretrained/yahtzee-score.pkl pretrained/yahtzee-win.pkl -n 10000 --ruleset yahtzee

100%|███████████████████████| 10000/10000 [09:34<00:00, 17.42it/s]
Agent #1 (pretrained/yahtzee-score.pkl)
---------------------------------------
 Rank 1 | ██████ 2515
 Rank 2 | ██████ 2550
 Rank 3 | ██████ 2510
 Rank 4 | █████ 2425
 ---
 Final score: 236.2 ± 59.2

... (agent 2 and 3 similar to agent 1)

Agent #4 (pretrained/yahtzee-win.pkl)
-------------------------------------
 Rank 1 | ██████ 2581
 Rank 2 | █████ 2446
 Rank 3 | █████ 2463
 Rank 4 | ██████ 2510
 ---
 Final score: 235.8 ± 59.4

(ties are counted as the higher rank for both agents)

As it turns out, we achieve mean scores of arond 236, and the play-to-win agent is indeed winning more often despite having a slightly lower mean score! (Of course, it is also coming last more often - riskier plays don’t always pay off.)

Needless to say, the trained agent also consistently beats the greedy and random baseline agents.

Agent #1 (pretrained/yahtzee-score.pkl)
---------------------------------------
 Rank 1 | █████████████ 616
 Rank 2 | ████████ 384
 Rank 3 |  0
 ---
 Final score: 239.7 ± 62.2

Agent #2 (greedy)
-----------------
 Rank 1 | ████████ 390
 Rank 2 | █████████████ 610
 Rank 3 |  0
 ---
 Final score: 218.7 ± 52.9

Agent #3 (random)
-----------------
 Rank 1 |  0
 Rank 2 |  0
 Rank 3 | ████████████████████ 1000
 ---
 Final score: 44.2 ± 17.7

Still, 236 is not an optimal score (perfect play is around 254). The situation is better for Yatzy, where we get a mean score of 241 (perfect play is around 248).

I suspect that we see this gap because Yahtzee has some “weird” rules when rolling multiple Yahtzees in a game. More than, say, 3 Yahtzees should occur very rarely, so the agent has a hard time learning what to do. On the other hand, these unicorn games can lead to some very high final scores, thus having a (relatively) big impact on the mean score.

For the typical game, these agents shold be reasonably close to optimal.

Final Thoughts¶

Why Was This So Hard?¶

I’d like to take some time to reflect why Yahtzee is such a challenging problem for RL.

Achieving perfect play in Yahtzee is challenging (impossible?) for humans because it requires perfectly calibrated probabilities. You will need a flawless mental model to evaluate the risk that is associated with each action. This is hard because it is more quantitative than our intuition can handle.

On the other hand, machines are extremely quantitative, so learning quasi-perfectly calibrated probabilities should not be a problem. Here, the problems I’ve observed are two-fold:

Getting stuck in a local optimum. Easy to learn strategies like greedy play are too hard to beat by incremental improvements.
Missing obvious best plays — obvious to humans, that is — because the situations when they are needed are too rare.

One example are “hail mary” plays where the agent is hopelessly behind, and can only hope to win by gambling on getting a Yahtzee. Most of the time it won’t work, so the agent won’t learn that it is actually advantageous.

I’m not sure how a RL agent can solve situations like these reliably. An obvious solution are Monte-Carlo tree search methods that actually play out the consequences of each decision, but then the learning process would be dependend on the game rules again — something I wanted to avoid.

Anyhow, it took me by surprise how fast this little side project essentially turned into a research problem. Reinforcement learning is hard!

Human learning — Machine Teaching¶

Remember the introduction, when I said that I wanted Yahtzotron to teach me to get better at the game? So far, we haven’t really done anything in this direction.

Creating an agent that is good at a task is one thing. Another — and, in my opinion, much more valuable — thing is to transfer that knowledge back to us humans. This is what I call human learning.

Achieving this is incredibly hard, and I don’t think there is a universal recipe for this yet.

The way I approached this with Yahtzotron is to enable the agent to “think out loud”. For this, I trained another neural network that predicts the final (category) action taken after the first and second roll. I call this the strategy network.

The strategy network gives you its best guess what Yahtzotron might be going for when picking dice to roll. Usually, this is quite convincing:

> My turn!
> Roll #1: [3, 3, 3, 5, 6].
> I think I should go for Threes, so I’m keeping [3, 3, 3].
> Roll #2: [3, 3, 3, 3, 4].
> I think I should go for Threes or Yatzy, so I’m keeping [3, 3, 3, 3].
> Roll #3: [1, 3, 3, 3, 3].
> I’ll pick the “Threes” category for that.

The lines starting with “I think I should go for…” are based on the output of the strategy network.

(There is still a lot of luck involved, so the prediction isn’t always right.)

With this in place, it becomes somewhat more transparent how Yahtzotron is making decisions. Of course, there’s still a long way to go towards a real machine teacher.

I hope you have enjoyed this read.

If you want to give Yahtzotron a try, just visit the repository and follow the instructions.

Good luck! 🎲🎲🎲🎲🎲

Suck-less scientific Python

2016-11-30T00:00:00+01:00

In this example, we explore how evolution models can be implemented efficiently. An example of an evolution model is given in Botero et al., 2015, but many are found throughout literature. In those models, some individuals (in this example animals) form a population, and each individual randomly adjusts its genome (i.e., mutates). Depending on how well the individuals fit into their environment, they produce more or less offspring, thus altering the composition of the population (survival of the fittest).

In this simple example, the fitness of the individuals will just depend on the distance of their genome to some perfect state, and the population size is held constant by cloning / killing random animals.

In [1]:

%matplotlib inline
%load_ext cython
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
import seaborn as sns
sns.set("talk")

Naive Python implementation¶

This implementation uses two classes, one for the animals, and one for the population. During one time step, each animal mutates five times, then it reproduces. The fitness has the shape of a Gaussian around an optimal value $i=10$.

In [2]:

class PyAnimal:
    def __init__(self,initial=0):
        self.i = initial
    def mutate(self):
        self.i += np.random.randn()
    def fitness(self):
        return np.exp(-(self.i-10)**2)
    def state(self):
        return self.i
    
class PyPopulation:
    def __init__(self,animals):
        self.animals = animals
        
    def do_step(self):
        for animal in self.animals:
            for _ in range(5):
                animal.mutate()
        fitnesses = [a.fitness() for a in self.animals]
        offspring = np.random.poisson(fitnesses / np.mean(fitnesses))
        born_animals = [PyAnimal(a.state()) for a in np.repeat(self.animals, offspring)]
        if len(born_animals) > len(self.animals):
            born_animals = np.random.choice(born_animals,len(self.animals)).tolist()
        else:
            clones = np.random.choice(born_animals,len(self.animals) - len(born_animals))
            born_animals = born_animals + [PyAnimal(a.state()) for a in clones]
        self.animals = born_animals
        
    def state(self):
        return [a.state() for a in self.animals]

Let’s check the performance and the resulting distribution of genes:

In [3]:

animals = [PyAnimal() for i in range(100000)]
population = PyPopulation(animals)
%timeit population.do_step()
plt.hist(population.state());

1 loop, best of 3: 592 ms per loop

In [4]:

%prun -l 10 -q -T profile population.do_step()
with open("profile","r") as f:
    print(f.read())

 
*** Profile printout saved to text file u'profile'. 
         1400140 function calls in 1.009 seconds

   Ordered by: internal time
   List reduced from 34 to 10 due to restriction <10>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.406    0.406    1.008    1.008 <ipython-input-2-b4af97439b11>:15(do_step)
   500000    0.137    0.000    0.137    0.000 {method 'randn' of 'mtrand.RandomState' objects}
   500000    0.132    0.000    0.269    0.000 <ipython-input-2-b4af97439b11>:4(mutate)
        4    0.124    0.031    0.124    0.031 {numpy.core.multiarray.array}
   100000    0.084    0.000    0.084    0.000 <ipython-input-2-b4af97439b11>:6(fitness)
        1    0.082    0.082    0.082    0.082 {method 'choice' of 'mtrand.RandomState' objects}
   100047    0.011    0.000    0.011    0.000 <ipython-input-2-b4af97439b11>:8(state)
   100001    0.011    0.000    0.011    0.000 {range}
   100047    0.010    0.000    0.010    0.000 <ipython-input-2-b4af97439b11>:2(__init__)
        1    0.005    0.005    0.006    0.006 {method 'poisson' of 'mtrand.RandomState' objects}

The implementation seems to work (the resulting distribution is centered around 10), but it is quite slow. Profiling the code reveals that most of the time is spent directly in the do_step, mutate and fitness functions, while the NumPy functions only play a minor roll. It thus seems like the slow runtime is mainly caused by the overhead when creating and accessing the class members or calling class methods.

Cython implementation¶

Since member access is much faster in Cython, it seems obvious to try and cythonize the Python implementation. A first iteration may look like this:

In [5]:

%%cython -c=-O3
# cython: profile=True

import numpy as np
cimport numpy as np

cdef class CyAnimal:
    cdef double i
    
    def __init__(self, double initial=0):
        self.i = initial
    cpdef void mutate(self):
        self.i += np.random.randn()
    cpdef double fitness(self):
        return np.exp(-(self.i-10)**2)
    cpdef double state(self):
        return self.i

cdef class CyPopulation:
    cdef list animals
    
    def __init__(self,list animals):
        self.animals = animals
        
    cpdef void do_step(self):
        cdef list fitnesses, offspring, born_animals, clones
        
        for animal in self.animals:
            for _ in range(5):
                animal.mutate()
        fitnesses = [a.fitness() for a in self.animals]
        offspring = np.random.poisson(fitnesses / np.mean(fitnesses)).tolist()
        born_animals = [CyAnimal(a.state()) for a in np.repeat(self.animals, offspring)]
        if len(born_animals) > len(self.animals):
            born_animals = np.random.choice(born_animals,len(self.animals)).tolist()
        else:
            clones = np.random.choice(born_animals,len(self.animals) - len(born_animals)).tolist()
            born_animals = born_animals + [CyAnimal(a.state()) for a in clones]
        self.animals = born_animals
            
    cpdef list state(self):
        return [animal.state() for animal in self.animals]

In [6]:

animals = [CyAnimal() for i in range(100000)]
population = CyPopulation(animals)
%timeit population.do_step()
plt.hist(population.state())
print(len(population.state()))

1 loop, best of 3: 366 ms per loop
100000

In [7]:

%prun -l 10 -q -T profile population.do_step()
with open("profile","r") as f:
    print(f.read())

 
*** Profile printout saved to text file u'profile'. 
         1500040 function calls in 0.565 seconds

   Ordered by: internal time
   List reduced from 33 to 10 due to restriction <10>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
   500000    0.182    0.000    0.182    0.000 _cython_magic_a6767cf8e5f3cb4da40bd98caafdfcf3.pyx:11(mutate)
        1    0.169    0.169    0.565    0.565 _cython_magic_a6767cf8e5f3cb4da40bd98caafdfcf3.pyx:24(do_step)
   500000    0.062    0.000    0.244    0.000 _cython_magic_a6767cf8e5f3cb4da40bd98caafdfcf3.pyx:11(mutate (wrapper))
   100000    0.059    0.000    0.059    0.000 _cython_magic_a6767cf8e5f3cb4da40bd98caafdfcf3.pyx:13(fitness)
        4    0.056    0.014    0.056    0.014 {numpy.core.multiarray.array}
   100000    0.013    0.000    0.072    0.000 _cython_magic_a6767cf8e5f3cb4da40bd98caafdfcf3.pyx:13(fitness (wrapper))
   100000    0.012    0.000    0.015    0.000 _cython_magic_a6767cf8e5f3cb4da40bd98caafdfcf3.pyx:15(state (wrapper))
        1    0.004    0.004    0.004    0.004 {method 'repeat' of 'numpy.ndarray' objects}
   100000    0.004    0.000    0.004    0.000 _cython_magic_a6767cf8e5f3cb4da40bd98caafdfcf3.pyx:15(state)
   100000    0.004    0.000    0.004    0.000 _cython_magic_a6767cf8e5f3cb4da40bd98caafdfcf3.pyx:9(__init__)

So, just by adding a few types we gain about double the performance, which is alright, but not really mind-blowing. Also, it seems like the performance bottleneck in mutate and fitness is still present (there are still lots of Python interactions). How can we eliminate this? One solution is to implement the animal class in pure C++ and interface it through Cython.

C++ Wrapper¶

Since the animals are now implemented in C++, we cannot use the NumPy functions anymore to create random numbers, but the C++ standard library provides random number generators anyway. Just make sure to set a different random seed for every animal (otherwise your whole population will evolve into the same direction).

In [8]:

%%writefile cppanimal.h
#include <math.h>
#include <random>

class CppAnimal {
    private:
        std::default_random_engine generator;
        std::normal_distribution<double> distribution;
        double i;
    public:
        CppAnimal(double initial):i(initial), generator(std::rand()), 
                  distribution(0.0,1.0) {}
        void mutate() {i += distribution(generator);}
        double fitness() {return exp(-pow(i-10,2));}
        double state() {return i;}
};

Overwriting cppanimal.h

The class CppAnimal from the external file cppanimal.h can then be interfaced using the extern keyword from Cython. You just need to specify which functions should be interfaced to Python. Also note the different compiler flags (-+ to enable C++, and -I . to tell the compiler that it should look in the current directory for cppanimal.h).

Since CppAnimal is implemented in pure C++, we cannot put it into a Python list without wrapping it. Instead, we use a C++ vector to contain the animals, which enables us to loop over the animals in C-speed. However, this makes the animals inaccessible from Python, so we need to provide a new Cython method create_animal.

In [9]:

%%cython -+ -I . -c=-O3
# cython: profile=True

from libcpp.vector cimport vector
import numpy as np
cimport numpy as np

cdef extern from "cppanimal.h":
    cdef cppclass CppAnimal:
        CppAnimal(double initial)
        void mutate()
        double fitness()
        double state()

cdef class CppPopulation:
    cdef vector[CppAnimal] animals
    
    cpdef create_animal(self,double initial=0):
        self.animals.push_back(CppAnimal(initial))
    cpdef void do_step(self):
        cdef int i, c
        cdef np.ndarray[np.float_t,ndim=1] fitnesses = np.empty(self.animals.size(), dtype=np.float)
        cdef vector[CppAnimal] born_animals
        cdef np.ndarray[np.int_t,ndim=1] offspring, clones
        cdef double c_state
        
        for i in range(self.animals.size()):
            for _ in range(5):
                self.animals[i].mutate()
            fitnesses[i] = self.animals[i].fitness()
        offspring = np.random.poisson(fitnesses / np.mean(fitnesses))
        for i in range(self.animals.size()):
            for k in range(offspring[i]):
                born_animals.push_back(CppAnimal(self.animals[i].state()))
        if born_animals.size() > self.animals.size():
            clones = np.random.choice(np.arange(born_animals.size()), 
                                      born_animals.size() - self.animals.size(),
                                      replace=False)
            for c in clones:
                born_animals.erase(born_animals.begin() + c)
        else:
            clones = np.random.choice(np.arange(self.animals.size()),
                                      self.animals.size() - born_animals.size())
            for c in clones:
                born_animals.push_back(CppAnimal(self.animals[c].state()))
        self.animals = born_animals
            
    cpdef list state(self):
        cdef list state_list
        cdef int i
        state_list = []
        for i in range(self.animals.size()):
            state_list.append(self.animals[i].state())
        return state_list

In [10]:

population = CppPopulation()
[population.create_animal() for _ in range(100000)]
%timeit population.do_step()
plt.hist(population.state())
print(len(population.state()))

10 loops, best of 3: 47.7 ms per loop
100000

In [11]:

%prun -l 10 -q -T profile population.do_step()
with open("profile","r") as f:
    print(f.read())

 
*** Profile printout saved to text file u'profile'. 
         36 function calls in 0.082 seconds

   Ordered by: internal time
   List reduced from 22 to 10 due to restriction <10>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.082    0.082    0.082    0.082 _cython_magic_38e2df27a5b6471fc4ab6ddd0b6fe556.pyx:19(do_step)
        4    0.000    0.000    0.000    0.000 {method 'reduce' of 'numpy.ufunc' objects}
        1    0.000    0.000    0.000    0.000 _methods.py:53(_mean)
        1    0.000    0.000    0.000    0.000 fromnumeric.py:2388(prod)
        2    0.000    0.000    0.000    0.000 fromnumeric.py:1866(any)
        1    0.000    0.000    0.000    0.000 _methods.py:43(_count_reduce_items)
        1    0.000    0.000    0.000    0.000 fromnumeric.py:2786(mean)
        3    0.000    0.000    0.000    0.000 numeric.py:534(asanyarray)
        1    0.000    0.000    0.082    0.082 <string>:1(<module>)
        1    0.000    0.000    0.082    0.082 {method 'do_step' of '_cython_magic_38e2df27a5b6471fc4ab6ddd0b6fe556.CppPopulation' objects}

Implementing the animal class in C++ and using C++-vectors as containers gives us a nice speedup from about 370 ms to 50 ms. My Cython code is also far from perfect, since it still has some Python interactions when calling the NumPy functions. You could probably create some lightning fast code if you also implemented those yourself in Cython or even C++.

But, on the other hand, if you are able to write this code in Cython/C++ without using any Python interactions, why are you even using Python in the first place? Doing so eliminates most of the advantages that Python offers, so you might as well write your model entirely in C++, and just do the post-processing in Python. What if the design of the module was flawed from the very beginning? Let’s start from scratch.

Doing it the Python way¶

My first impulse to represent animals and populations as instances of a class might make intuitive sense, but in the end, it is an abstraction that prevents us from writing elegant code. The approach was to loop over all animals and do things one-by-one. As we have seen in example 2, this is usually a really bad idea. After all, animals can just be represented by numbers - say, their genome, or a position in space, or a lifetime fitness. So, let’s create a population that is just a structured array of numbers, with four fields:

In [12]:

def create_population(n):
    pop = np.zeros(n, dtype=[('a','<f8'),('b','<f8'),('position','<i8'),('supergene','<f8')])
    for f in pop.dtype.fields:
        pop[f] = np.random.randn(n)*5
    return pop

In [13]:

pop = create_population(10)
print(pop["position"])
print(pop["supergene"])

[ 0  4  0  0  7  7  0  3  3 -2]
[ 3.00132731  3.30461432  4.90439491  0.26981922 -3.87040995 -4.56569831
 -6.93399245  6.57216156  0.74030973  5.2755511 ]

The NumPy interface allows us to operate on the whole population at once:

In [14]:

def mutate(population):
    n = len(population)
    population["a"] += np.random.randn(n)
    population["b"] += 2.5*np.random.randn(n) + 0.25
    population["supergene"] += np.random.randn(n)
    population["position"] += np.random.randint(-1,2,size=n)
    
def fitness(population):
    fit_fun = lambda x,t: np.exp(-np.power(population[x]-t,2))
    return fit_fun("a",15) + fit_fun("b",2.5) \
             + fit_fun("supergene",10) + fit_fun("position",0)

def do_step(population):
    n = len(population)
    for _ in range(5):
        mutate(population)
    fitnesses = fitness(population)
    offspring = np.random.poisson(fitnesses / np.mean(fitnesses))
    population = np.repeat(population,offspring)
    if len(population) > n:
        population = np.random.choice(population,n,replace=False)
    elif len(population) < n:
        clones = np.random.choice(np.arange(len(population)),
                                  n-len(population),replace=False)
        clone_rep = np.ones(len(population), dtype=np.int)
        clone_rep[clones] = 2
        population = np.repeat(population,clone_rep)
    return population

Just look at the sheer length of this code, compared to the complicated implementations above. If you want, you can also put those functions into a population class that holds the animals as a member array to keep your namespace clean.

Representing the population as a structured array also allows for nice interactions with other Python packages (here Pandas and Seaborn):

In [15]:

pop = create_population(100000)
for _ in range(10):
    pop = do_step(pop)
sns.violinplot(data=pd.DataFrame(pop));

And, finally, let’s have a look at performance:

In [16]:

pop = create_population(100000)
%timeit do_step(pop)

10 loops, best of 3: 77.4 ms per loop

So, completely un-optimized Python code with four genes instead of one is not even slower by a factor of two compared to the C++/Cython implementation above. What do we learn from that?

Bottom line: don’t be stupid™¶

Don’t get carried away by unnecessary abstraction. Think in terms of clusters, not single objects.
Don’t try to apply low-level logic to high-level languages. Chances are, you will make your life miserable.
Before optimizing a solution, ask yourself whether your abstraction makes sense.
Use the tools that you are given (in this case NumPy). There are lots of developers thinking about low-level implementations so you don’t have to (that’s why we all like Python, remember).

That concludes this example, and this series. Thank you for reading, and don’t hesitate to contact me if you have questions or if you have a suggestion on how to improve my code. I’m always happy to learn.

Suck-less scientific Python

2016-11-29T00:00:00+01:00

This example shows how simple Fortran-style loops over arrays can be implemented efficiently in Python. Since many Python operations carry a significant overhead compared to a low-level language as e.g. C or Fortran, Python is often generally classified as slow. However, satisfying speeds can be achieved if the Python code makes use of packages like Cython, Numba, or NumPy. Also, it is important to keep in mind that Python is considerably easier to learn and faster to write than e.g. C. In many situations, slower runtimes can be compensated by faster development times (which is a good thing, since your time is usually more valuable than your computer’s).

As an example for tight, simple loops, we will implement a numerical solver for a two-dimensional shallow-water model using finite differences. Explicit finite differences require multiple loops over the entire domain in each time step, so the complexity of the algorithm is at least $n^2$ in two spatial dimensions.

The Model¶

The unforced shallow-water equations for geostrophic flow in 2 dimensions read:

$$\frac{\partial}{\partial t} u = -g \frac{\partial}{\partial x}h$$$$\frac{\partial}{\partial t} v = -g \frac{\partial}{\partial y}h$$$$\frac{\partial}{\partial t} h + \frac{\partial}{\partial x} (hu) + \frac{\partial}{\partial y} (hu) = 0$$

where $u$ denotes the velocity field in $x$-direction, $v$ the velocity in $y$-direction, and $h$ the layer height of the water column. $g$ is the gravitational acceleration.

These equations are discretized by

$$ u^{n+1}_{i,j} = u^n_{i,j} - g\frac{\Delta t}{\Delta x} (h_{i,j+1} - h_{i,j})$$$$ v^{n+1}_{i,j} = v^n_{i,j} - g\frac{\Delta t}{\Delta x} (h_{i+1,j} - h_{i,j})$$$$ h^{n+1}_{i,j} = h^n_{i,j} - \frac{\Delta t}{\Delta x} (f^e_{i,j} - f^w_{i,j}) - \frac{\Delta t}{\Delta y} (f^n_{i,j} - f^s_{i,j})$$

with

$$f^e_{i,j} = u^+_{i,j} h_{i,j} + u^-_{i,j} h_{i,j+1}$$$$f^n_{i,j} = v^+_{i,j} h_{i,j} + v^-_{i,j} h_{i+1,j}$$$$f^w_{i,j} = u^+_{i,j} h_{i,j-1} + u^-_{i,j} h_{i,j}$$$$f^s_{i,j} = v^+_{i,j} h_{i-1,j} + v^-_{i,j} h_{i,j}$$

and

$$u^+_{i,j} = 0.5 (u_{i,j} + |u_{i,j}|)$$$$u^-_{i,j} = 0.5 (u_{i,j} - |u_{i,j}|)$$

(analogous for $v$)

The system of three coupled partial differential equations is thus solved by explicitly discretizing the domain in each dimension and in time. The used discretization schemes are explicit Euler in time, and an upwinding scheme in space. The only boundary condition is no-normal-flow (i.e., $u=0$ on the western and easter boundaries, and $v=0$ in the north and south). To ensure stability, the layer height is artificially smoothed in every time step:

$$ h^*_{i,j} = (1-\varepsilon)~h_{i,j} + \frac{\varepsilon}{4} (h_{i-1,j} + h_{i+1,j} + h_{i,j-1} + h_{i,j+1}) $$

with a smoothing coefficient $\varepsilon$.

Set-up¶

Let’s import all the packages we’ll need:

In [1]:

%matplotlib inline
%load_ext cython
import numpy as np
import numba
import matplotlib.pyplot as plt
import seaborn as sns
sns.set("talk")

from matplotlib import animation
plt.rcParams['animation.html'] = 'html5'

Now, we create the numerical grid ($100 \times 100$ cells) and define some constants and the initial conditions (a Gaussian bump of water in the center of the domain):

In [2]:

yy, xx = np.mgrid[-50:50:100j,-100:100:100j].astype(float)
dx = xx[0,1] - xx[0,0]
dy = yy[1,0] - yy[0,0]

d = 10
g = 9.81
dt = 0.5 * min(dx,dy) / np.sqrt(g*d)
t1 = 10
eps = 0.1

h0 = d + np.exp(-(xx**2 + yy**2)/20)
u0 = np.zeros_like(xx)
v0 = np.zeros_like(xx)

Naive Python implementation¶

This implementation uses explicit loops over the domain, and checks explicitly for boundary cells:

In [3]:

def iterate_py(h,u,v):
    h_new = np.zeros_like(h)
    u_new = np.zeros_like(u)
    v_new = np.zeros_like(v)
    
    n1, n2 = h.shape
    for i in range(n1):
        for j in range(n2):
            if j == n2-1:
                u_new[i,j] = 0
            else:
                u_new[i,j] = u[i,j] - g*dt/dx * (h[i,j+1] - h[i,j])
            if i == n1-1:
                v_new[i,j] = 0
            else:
                v_new[i,j] = v[i,j] - g*dt/dy * (h[i+1,j] - h[i,j])
                
    u_plus = .5 * (u_new + np.abs(u_new))
    u_minus = .5 * (u_new - np.abs(u_new))
    v_plus = .5 * (v_new + np.abs(v_new))
    v_minus = .5 * (v_new - np.abs(v_new))
    
    for i in range(n1):
        for j in range(n2):
            if j == n2-1:
                f_e = 0
            else:
                f_e = u_plus[i,j] * h[i,j] + u_minus[i,j] * h[i,j+1]
            if i == n1-1:
                f_n = 0
            else:
                f_n = v_plus[i,j] * h[i,j] + v_minus[i,j] * h[i+1,j]
            if j == 0:
                f_w = 0
            else:
                f_w = u_plus[i,j-1] * h[i,j-1] + u_minus[i,j-1] * h[i,j]
            if i == 0:
                f_s = 0
            else:
                f_s = v_plus[i-1,j] * h[i-1,j] + v_minus[i-1,j] * h[i,j]           
            h_new[i,j] = h[i,j] - dt/dx*(f_e - f_w) - dt/dy*(f_n - f_s)
    
    h_filter = np.zeros_like(h_new)
    for i in range(n1):
        for j in range(n2):
            if j == n2-1:
                h_e = h_new[i,j-1]
            else:
                h_e = h_new[i,j+1]
            if i == n1-1:
                h_n = h_new[i-1,j]
            else:
                h_n = h_new[i+1,j]
            if j == 0:
                h_w = h_new[i,j+1]
            else:
                h_w = h_new[i,j-1]
            if i == 0:
                h_s = h_new[i+1,j]
            else:
                h_s = h_new[i-1,j]
            h_filter[i,j] = (1-eps) * h_new[i,j] + .25*eps*(h_e+h_n+h_w+h_s)
            
    return h_filter, u_new, v_new

Let’s test how long the solution takes, and animate the solution:

In [4]:

def solve_shallow_water(iterate_fun,t1=t1):
    t = 0
    h = h0
    u = u0
    v = v0
    sol = []
    while t < t1:
        h, u, v = iterate_fun(h,u,v)
        sol.append(h)
        t += dt
    return sol

In [5]:

%timeit solve_shallow_water(iterate_py)

1 loop, best of 3: 9.95 s per loop

In [6]:

def animate_shallow_water(sol):
    fig = plt.figure(figsize=(8,6))
    ax = plt.gca()
    ax.set_aspect("equal")
    ax.set_xlim((xx.min(),xx.max()))
    ax.set_ylim((yy.min(),yy.max()))
    cs = plt.pcolormesh(xx,yy,sol[0],vmin=9.9,vmax=10.1,cmap="RdBu_r")
    plt.colorbar(cs,orientation="horizontal")
    plt.close()

    def animate(i):
        ax.set_title("Layer height at t = {:.1f}s".format(dt*i), y=1.1)
        cs.set_array(sol[i][:-1,:-1].flatten())
        return (cs,)

    return animation.FuncAnimation(fig, animate, frames=len(sol), interval=20, blit=True)

sol = solve_shallow_water(iterate_py)
animate_shallow_water(sol)

Out[6]:

Your browser does not support the video tag.

The solution looks nice, but a runtime of 10 s is just too slow for such a short simulation time.

Using Numba¶

Numba is a quite recent project by Continuum Analytics (the developers of Anaconda Python. Its main module is a just-in-time compiler (jit) that aims at making math-heavy Python code more efficient by compiling it to machine instructions before execution. Using numba is very simplpe; just apply the jit decorator to the function you want to get compiled. In this case, the function code is exactly the same as before:

In [7]:

@numba.jit
def iterate_numba(h,u,v):
    h_new = np.zeros_like(h)
    u_new = np.zeros_like(u)
    v_new = np.zeros_like(v)
    
    n1, n2 = h.shape
    for i in range(n1):
        for j in range(n2):
            if j == n2-1:
                u_new[i,j] = 0
            else:
                u_new[i,j] = u[i,j] - g*dt/dx * (h[i,j+1] - h[i,j])
            if i == n1-1:
                v_new[i,j] = 0
            else:
                v_new[i,j] = v[i,j] - g*dt/dy * (h[i+1,j] - h[i,j])
                
    u_plus = .5 * (u_new + np.abs(u_new))
    u_minus = .5 * (u_new - np.abs(u_new))
    v_plus = .5 * (v_new + np.abs(v_new))
    v_minus = .5 * (v_new - np.abs(v_new))
    
    for i in range(n1):
        for j in range(n2):
            if j == n2-1:
                f_e = 0
            else:
                f_e = u_plus[i,j] * h[i,j] + u_minus[i,j] * h[i,j+1]
            if i == n1-1:
                f_n = 0
            else:
                f_n = v_plus[i,j] * h[i,j] + v_minus[i,j] * h[i+1,j]
            if j == 0:
                f_w = 0
            else:
                f_w = u_plus[i,j-1] * h[i,j-1] + u_minus[i,j-1] * h[i,j]
            if i == 0:
                f_s = 0
            else:
                f_s = v_plus[i-1,j] * h[i-1,j] + v_minus[i-1,j] * h[i,j]           
            h_new[i,j] = h[i,j] - dt/dx*(f_e - f_w) - dt/dy*(f_n - f_s)
    
    h_filter = np.zeros_like(h_new)
    for i in range(n1):
        for j in range(n2):
            if j == n2-1:
                h_e = h_new[i,j-1]
            else:
                h_e = h_new[i,j+1]
            if i == n1-1:
                h_n = h_new[i-1,j]
            else:
                h_n = h_new[i+1,j]
            if j == 0:
                h_w = h_new[i,j+1]
            else:
                h_w = h_new[i,j-1]
            if i == 0:
                h_s = h_new[i+1,j]
            else:
                h_s = h_new[i-1,j]
            h_filter[i,j] = (1-eps) * h_new[i,j] + .25*eps*(h_e+h_n+h_w+h_s)
            
    return h_filter, u_new, v_new

Let’s see how it performs:

In [8]:

%timeit solve_shallow_water(iterate_numba)

The slowest run took 29.66 times longer than the fastest. This could mean that an intermediate result is being cached.
1 loop, best of 3: 27.1 ms per loop

Wow! A speedup by a factor of about 400, just by applying a decorator to the function. Note how the first call to the function takes much longer than subsequent calls, because Numba has to compile it first.

Even though the speedup is substantial, using Numba still has a few drawbacks. First of all, in more complicated functions, I often received compilation errors - Numba still only supports a subset of the Python and NumPy capabilities - and speedups are not always that dramatic. And second of all, in order for Numba to do its magic, it is necssary to write your function in a verbose, exlicit manner. This might seem natural when coming from a C or Fortran background, but denies the access to the elegant side of Python.

Using Cython¶

Cython is a project that allows the user to add static types to Python variables, which enables efficient compilation of the typed parts of your script to C. When using Cython from a Jupyter notebook, you can make use of the convenient %%cython magic, which marks that the respective code cell should be compiled with Cython. An implementation could look like this:

In [9]:

%%cython -c=-O3
import numpy as np
cimport numpy as np
cimport cython

@cython.boundscheck(False)
cpdef iterate_cython(np.ndarray[np.float_t,ndim=2] h,
                     np.ndarray[np.float_t,ndim=2] u,
                     np.ndarray[np.float_t,ndim=2] v,
                     double g, double dx, double dy, double dt, double eps):
    cdef np.ndarray[np.float_t,ndim=2] h_new = np.zeros_like(h)
    cdef np.ndarray[np.float_t,ndim=2] u_new = np.zeros_like(u)
    cdef np.ndarray[np.float_t,ndim=2] v_new = np.zeros_like(v)
    cdef np.ndarray[np.float_t,ndim=2] u_plus, u_minus, v_plus, v_minus
    cdef np.ndarray[np.float_t,ndim=2] h_filter
    cdef np.float_t f_e, f_n, f_w, f_s
    cdef np.float_t h_e, h_n, h_w, h_s
    cdef np.int_t i, j, n1, n2
    
    n1, n2 = h.shape[0], h.shape[1]
    for i in range(n1):
        for j in range(n2):
            if j == n2-1:
                u_new[i,j] = 0
            else:
                u_new[i,j] = u[i,j] - g*dt/dx * (h[i,j+1] - h[i,j])
            if i == n1-1:
                v_new[i,j] = 0
            else:
                v_new[i,j] = v[i,j] - g*dt/dy * (h[i+1,j] - h[i,j])
                
    u_plus = .5 * (u_new + np.abs(u_new))
    u_minus = .5 * (u_new - np.abs(u_new))
    v_plus = .5 * (v_new + np.abs(v_new))
    v_minus = .5 * (v_new - np.abs(v_new))
    
    for i in range(n1):
        for j in range(n2):
            if j == n2-1:
                f_e = 0
            else:
                f_e = u_plus[i,j] * h[i,j] + u_minus[i,j] * h[i,j+1]
            if i == n1-1:
                f_n = 0
            else:
                f_n = v_plus[i,j] * h[i,j] + v_minus[i,j] * h[i+1,j]
            if j == 0:
                f_w = 0
            else:
                f_w = u_plus[i,j-1] * h[i,j-1] + u_minus[i,j-1] * h[i,j]
            if i == 0:
                f_s = 0
            else:
                f_s = v_plus[i-1,j] * h[i-1,j] + v_minus[i-1,j] * h[i,j]           
            h_new[i,j] = h[i,j] - dt/dx*(f_e - f_w) - dt/dy*(f_n - f_s)
    
    h_filter = np.zeros_like(h_new)
    for i in range(n1):
        for j in range(n2):
            if j == n2-1:
                h_e = h_new[i,j-1]
            else:
                h_e = h_new[i,j+1]
            if i == n1-1:
                h_n = h_new[i-1,j]
            else:
                h_n = h_new[i+1,j]
            if j == 0:
                h_w = h_new[i,j+1]
            else:
                h_w = h_new[i,j-1]
            if i == 0:
                h_s = h_new[i+1,j]
            else:
                h_s = h_new[i-1,j]
            h_filter[i,j] = (1-eps) * h_new[i,j] + .25*eps*(h_e+h_n+h_w+h_s)
            
    return h_filter, u_new, v_new

In [10]:

iterate_cython_wrapper = lambda h, u, v: iterate_cython(h,u,v,g,dx,dy,dt,eps)
%timeit solve_shallow_water(iterate_cython_wrapper)

10 loops, best of 3: 46.6 ms per loop

So, in this case, Cython is in fact a bit slower than Numba, and requires more changes to the code. However, Cython is generally applicable to more cases than Numba.

Using NumPy¶

In order to showcase how a more elegant and quite efficient solution could look, consider the following implementation with NumPy. It makes heavy use of array slicing and padding to enforce the boundary conditions, which is fast with NumPy. It also avoids all explicit loops, so the overall code turns out to be much shorter and more readable:

In [11]:

def iterate_numpy(h,u,v):
    h_pad = np.pad(h,1,"edge")
    
    u_new = u - g*dt/dx * (h_pad[1:-1,2:] - h)
    v_new = v - g*dt/dy * (h_pad[2:,1:-1] - h)
    
    u_new_pad = np.pad(u_new,1,"constant")
    v_new_pad = np.pad(v_new,1,"constant")
                
    u_plus = .5 * (u_new_pad + np.abs(u_new_pad))
    u_minus = .5 * (u_new_pad - np.abs(u_new_pad))
    v_plus = .5 * (v_new_pad + np.abs(v_new_pad))
    v_minus = .5 * (v_new_pad - np.abs(v_new_pad))
    
    f_e = u_plus[1:-1,1:-1] * h + u_minus[1:-1,1:-1] * h_pad[1:-1,2:]
    f_n = v_plus[1:-1,1:-1] * h + v_minus[1:-1,1:-1] * h_pad[2:,1:-1]
    f_w = u_plus[1:-1,:-2] * h_pad[1:-1,:-2] + u_minus[1:-1,:-2] * h
    f_s = v_plus[:-2,1:-1] * h_pad[:-2,1:-1] + v_minus[:-2,1:-1] * h
         
    h_new = h - dt/dx*(f_e - f_w) - dt/dy*(f_n - f_s)
    h_new_pad = np.pad(h,1,"reflect")
    h_filter = (1-eps) * h_new + .25*eps*(h_new_pad[:-2,1:-1]+h_new_pad[1:-1,:-2]+\
                                          h_new_pad[2:,1:-1]+h_new_pad[1:-1,2:])
    return h_filter, u_new, v_new

In [12]:

%timeit solve_shallow_water(iterate_numpy)

10 loops, best of 3: 105 ms per loop

So, this implementation is still about 100 times faster than pure Python, while being short and elegant, which is why we all love Python, right?

Using NumPy + Cython¶

Often, performance can be enhanced even further when using both NumPy and Cython:

In [13]:

%%cython -c=-O3
import numpy as np
cimport numpy as np

cpdef iterate_numcy(np.ndarray[np.float_t,ndim=2] h
                    , np.ndarray[np.float_t,ndim=2] u
                    , np.ndarray[np.float_t,ndim=2] v
                    , double g, double dx, double dy, double dt, double eps):
    cdef np.ndarray[np.float_t,ndim=2] h_new, u_new, v_new
    cdef np.ndarray[np.float_t,ndim=2] h_pad, u_new_pad, v_new_pad, h_new_pad
    cdef np.ndarray[np.float_t,ndim=2] h_filter
    cdef np.ndarray[np.float_t,ndim=2] f_e, f_n, f_w, f_s
    
    h_pad = np.pad(h,1,"edge")
    
    u_new = u - g*dt/dx * (h_pad[1:-1,2:] - h)
    v_new = v - g*dt/dy * (h_pad[2:,1:-1] - h)
    
    u_new_pad = np.pad(u_new,1,"constant")
    v_new_pad = np.pad(v_new,1,"constant")
                
    u_plus = .5 * (u_new_pad + np.abs(u_new_pad))
    u_minus = .5 * (u_new_pad - np.abs(u_new_pad))
    v_plus = .5 * (v_new_pad + np.abs(v_new_pad))
    v_minus = .5 * (v_new_pad - np.abs(v_new_pad))
    
    f_e = u_plus[1:-1,1:-1] * h + u_minus[1:-1,1:-1] * h_pad[1:-1,2:]
    f_n = v_plus[1:-1,1:-1] * h + v_minus[1:-1,1:-1] * h_pad[2:,1:-1]
    f_w = u_plus[1:-1,:-2] * h_pad[1:-1,:-2] + u_minus[1:-1,:-2] * h
    f_s = v_plus[:-2,1:-1] * h_pad[:-2,1:-1] + v_minus[:-2,1:-1] * h
         
    h_new = h - dt/dx*(f_e - f_w) - dt/dy*(f_n - f_s)
    h_new_pad = np.pad(h,1,"reflect")
    h_filter = (1-eps) * h_new + .25*eps*(h_new_pad[:-2,1:-1]+h_new_pad[1:-1,:-2]+\
                                          h_new_pad[2:,1:-1]+h_new_pad[1:-1,2:])
    return h_filter, u_new, v_new

In [14]:

iterate_numcy_wrapper = lambda h, u, v: iterate_numcy(h,u,v,g,dx,dy,dt,eps)
%timeit solve_shallow_water(iterate_numcy_wrapper)

10 loops, best of 3: 110 ms per loop

In this case, it seems that Cython does not grant an additional speedup compared to the NumPy implementation (I guess because all expensive computations happen inside NumPy anyway).

Conclusion¶

Comparison between all implementations:

Implementation	Runtime (s)
Python	9.95
NumPy + Cython	0.110
NumPy	0.105
Cython	0.046
Numba	0.027

Numba delivered the best performance on this problem, while still being easy to use. The (in my opinion) most elegant code is the implementation in pure NumPy (which is, by the way, also the most portable). I would thus recommend starting from there, and only use the big guns (Numba / Cython) when needed.

Oh, also, never use explicit loops over arrays in pure Python.

Before we close - here’s a longer animation of the shallow-water model, made with the Numba implementation. Enjoy!

In [15]:

sol = solve_shallow_water(iterate_numba,t1=50)
animate_shallow_water(sol)

Out[15]:

Your browser does not support the video tag.

Suck-less scientific Python

2016-11-28T00:00:00+01:00

Preface¶

This series of posts is based on a talk I held for the TS-CCES group at the Institute of Environmental Physics, Heidelberg in November 2016. I presented this Jupyter notebook directly using RISE, a Reveal.js plugin for Jupyter. This allowed me to demonstrate my code in action, and play around with it to clarify things or answer questions.

This talk was supposed to

give examples how to efficiently use Python in a scientific context; and
showcase some elegant code to convince scientists to learn Python.

To run this notebook yourself, just download the source of this post below. You will need to install a couple of packages for each example, e.g. using pip (or with conda if you’re using Anaconda Python). All code should work with either Python 2.7 or Python 3.x.

Either way, let’s start by importing some packages we’re going to need later on. Most of them are staples in the scientific community (like Matplotlib, NumPy, and Pandas). I also like Seaborn for its powerful plotting capabilities and reasonable default styles.

In [1]:

%matplotlib inline
import numpy as np
import re
import h5py
import matplotlib.pyplot as plt
import pandas as pd
import seaborn as sns
sns.set("talk")

Reading data¶

Pretty much every scientist has to process data, and many scientists process lots of data. Unfortunately, real data tends to be messy - data files often include useless headers, or obscure information with ASCII formatting. The reason for this is that some scientists prioritize human-readability over machine-readability when deciding on a data format. Thus, simply parsing these formats efficiently can prove a challenge for inexperienced programmers.

Let’s create a (slightly exaggerated) messy data file with random values:

In [2]:

with open("testdata.csv","w") as f:
    f.write("A very useless header\n")
    f.write("Hi there and welcome to my file!\n")
    f.write(" ".join([str(k+1) for k in range(4)]) + "\n")
    testdata = np.random.randn(5,4)
    for row in testdata:
        f.write("; ".join(row.astype(str)) + "\n")
    f.write("Now comes the second part of the data!\n")
    f.write("---\n")
    f.write("n = 500, eta = 15.2, foo = bar\n")
    f.write("Here it comes, for real...\n")
    f.write(" ".join(["a"*(k+1) for k in range(4)]) + "\n")
    testdata = 2*np.random.randn(10,4)
    for row in testdata:
        f.write("; ".join((2*row).astype(str)) + "\n")

The resulting file looks like this:

In [3]:

with open("testdata.csv","r") as f:
    print(f.read())

A very useless header
Hi there and welcome to my file!
1 2 3 4
1.01279224467; -1.40721121402; 0.192842751725; 0.595397342458
-1.05822481949; -0.61410582863; 0.596548028352; 0.853507967801
0.246076579597; -1.35448464373; -0.779938173578; 0.345188055691
0.262451774711; 1.35126111693; 1.49356026657; 0.453345832019
1.12903508252; -1.27305328594; 0.832493332433; -0.030581958518
Now comes the second part of the data!
---
n = 500, eta = 15.2, foo = bar
Here it comes, for real...
a aa aaa aaaa
-6.24008319723; -6.48035897782; -0.823533303002; -1.49908931092
1.98427240052; 4.70480668601; 0.0751743413074; 2.12624545828
1.06353434804; 2.70013998953; 2.30704303872; 8.6070005543
-3.74041646994; 2.82899149542; -7.90406166748; -0.342154882168
2.60588751468; 4.02536593691; 8.22260968898; 4.19770846736
0.346589152382; -6.34719243209; -1.07003043414; 4.56629825728
-0.578032055078; 0.113274915084; -0.662370168426; -1.17887797792
1.12067342885; -0.609656609035; -1.1416924058; 9.85864125608
-5.24482604427; -3.10285113881; -1.30870581842; -0.886890092603
-8.72139831232; -4.7518910358; -3.01606640974; -6.03211606132

Since there are two separate data sets in one file mixed with headers, metadata, and comments, there is no way to parse this file with built-in readers (such as e.g. NumPy’s genfromtxt). Thus, we need to write our own data parser. When extracting information from text, regular expressions have proven to be a powerful tool, with implementations in many languages. When developing regex patterns, I find it helpful to use a website such as regexr.com, which provides a cheat sheet listing each pattern, and allows for a more dynamical development.

To parse the data file above, I came up with the following patterns:

One that matches a floating number:

In [4]:

re_float = r"[-+]?[0-9]*\.?[0-9]+(?:[eE][-+]?[0-9]+)?"
re.findall(re_float, "hi there! here's a float: +15e-06")

Out[4]:

['+15e-06']

One that matches the whitespace-separated data headers:

In [5]:

re_header = r"(\w+?)(?:$|\s+)"
re.findall(re_header, "0 1 2 3 4")

Out[5]:

['0', '1', '2', '3', '4']

One that matches the data itself, consisting of semicolon-separated floating numbers:

In [6]:

re_data = r"(" + re_float + ")+?\s*(?:\;|\s*$)?"
re.findall(re_data, "1e-15; 17.2; 32")

Out[6]:

['1e-15', '17.2', '32']

And one for the metadata, in the form of comma-separated key-value pairs:

In [7]:

re_metadata = r"\s*(\w+)\s*?=\s*(.+?)\s*(?:,|$)"
meta_dict = {k: v for k,v in re.findall(re_metadata, 
                    "n = 500, hey=there,this=that; or is it?")}
meta_dict["this"]

Out[7]:

'that; or is it?'

Using these patterns, all we have to do to read the messy data file is to iterate through it line-by-line and check whether one of the patterns matches.

In [8]:

def read_data_block(f,h="<f8"):
    """
    Returns a contiguous block of data lines as structured numpy array
    """
    data = []
    for line in f:
        if not re.match(re_data,line):
            break
        data.append(tuple(map(float,re.findall(re_data,line))))
    return np.array(data, dtype=h)

def read_messy_file(filename):
    """
    Reads messy data file and returns lists of 
    found data and metadata
    """
    data = []
    meta = []
    header = []
    cur_meta = cur_header = cur_data = None
    with open(filename,"r") as f:
        for line in f:
            if re.match(re_metadata,line):
                cur_meta = {k: v for k,v in re.findall(re_metadata, line)}
                continue
            if re.match(re_header,line):
                cur_header = re.findall(re_header, line)
                continue
            if re.match(re_data,line):
                if cur_header:
                    cur_data = read_data_block(f,[(h,"<f8") for h in cur_header])
                else:
                    cur_data = read_data_block(f)
                data.append(cur_data)
                meta.append(cur_meta or {})
                header.append(cur_header or [])
                cur_meta = cur_header = cur_data = None
    return data, meta

In [9]:

data, meta = read_messy_file("testdata.csv")

All data is then contained in a structured NumPy array that can be indexed with the given column headers like a dictionary, and an actual dict containing all metadata:

In [10]:

print(data[1]["aa"])
print(meta[1])

[ 4.70480669  2.70013999  2.8289915   4.02536594 -6.34719243  0.11327492
 -0.60965661 -3.10285114 -4.75189104]
{'eta': '15.2', 'foo': 'bar', 'n': '500'}

Structured NumPy arrays can also be used to create Pandas DataFrame objects, which in turn play very well with Seaborn:

In [11]:

fig, axes = plt.subplots(1,2)
for d,ax in zip(data,axes):
    sns.violinplot(pd.DataFrame(d),ax=ax)
fig.tight_layout()

/home/dion/.virtualenvs/stats/local/lib/python2.7/site-packages/seaborn/categorical.py:2342: UserWarning: The violinplot API has been changed. Attempting to adjust your arguments for the new API (which might not work). Please update your code. See the version 0.6 release notes for more info.
  warnings.warn(msg, UserWarning)

Storing data¶

After putting so much effort into reading the data, it is important so make sure that it is saved in a more reasonable format the next time around. One great binary format for the storage of numeric arrays is HDF5. Bindings for HDF5 are available for many programming languages, such as C, C++, Fortran, Java, Ruby, and Python. This makes sure that your data will be readable, regardless of the tools you choose to process it. Since it is a binary format, you get the nice bonus effect of considerably smaller files, too.

For Python, the easiest way to work with HDF5 is h5py (available via pip). Saving the data from the messy data file to HDF5 becomes trivial:

In [12]:

with h5py.File("data.h5","w") as f:
    g1 = f.create_group("data1")
    g2 = f.create_group("data2")

    for g, d, m in zip((g1,g2), data, meta):
        dst = g.create_dataset("testdata", data=d)
        dst.attrs.update(m)
        dst.attrs["column_names"] = d.dtype.names

Reading data from a h5 file is just as easy:

In [13]:

with h5py.File("data.h5","r") as f:
    d = np.array(f["data1/testdata"])
    sns.violinplot(pd.DataFrame(d))
    print(f["data2/testdata"].attrs["column_names"])

['a' 'aa' 'aaa' 'aaaa']

Note that Pandas supports HDF5 I/O out of the box, so if you decide to use Pandas instead of NumPy, storing data in the HDF5 format becomes even easier.

Beautiful data visualization in Python with Matplotlib

2016-03-27T00:00:00+01:00

Introduction¶

In this post, I want to show you how to create the diagrams I made for my post on terrorism. I used Python 3.4 with the following packages:

IPython Notebook (Jupyter)
Matplotlib
Pandas
Seaborn

(you can install all of this via PyPi’s package manager pip)

Actually, this post itself is an IPython notebook. You can download the source code at the bottom of this page, and play around with the data and the plots yourself! Just make sure you save the filw with a .ipynb extension.

Let’s get started¶

First, we will import some packages:

In [1]:

%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd

Read data¶

I have downloaded the data I want to use from Eurostat. You can go ahead, change the contents of the data, and download it yourself, or use the data I used. Either way, let’s read it into pandas and have a look at the first 5 entries, and the data that is contained in the columns:

In [2]:

data = pd.read_csv("../downloads/hlth_cd_asdr2_1_Data.csv")
print(data[:5])

time_range = set(data.TIME)
countries = set(data.GEO)
ages = set(data.AGE)
death_causes = set(data.ICD10)

print("\nTime range: " + ", ".join(str(x) for x in time_range))
print("\nCountries: " + ", ".join(str(x) for x in countries))
print("\nAges: " + ", ".join(str(x) for x in ages))
print("\nCauses of death: " + ", ".join(str(x) for x in death_causes))

     SEX      GEO  UNIT  TIME    AGE  \
0  Total  Belgium  Rate  2011  Total   
1  Total  Belgium  Rate  2011  Total   
2  Total  Belgium  Rate  2011  Total   
3  Total  Belgium  Rate  2011  Total   
4  Total  Belgium  Rate  2011  Total   

                                               ICD10     Value  \
0    All causes of death (A00-Y89) excluding S00-T98  1,023.02   
1  Certain infectious and parasitic diseases (A00...     23.54   
2                                       Tuberculosis      0.48   
3    Viral hepatitis and sequelae of viral hepatitis      0.89   
4         Human immunodeficiency virus [HIV] disease      0.41   

  Flag and Footnotes  
0                NaN  
1                NaN  
2                NaN  
3                NaN  
4                NaN  

Time range: 2011, 2012, 2013

Countries: Germany (until 1990 former territory of the FRG), Netherlands, Italy, Portugal, Liechtenstein, Denmark, Finland, Sweden, Luxembourg, France, United Kingdom, Switzerland, Austria, Malta, Ireland, Belgium, Norway, Spain

Ages: Total, Less than 65 years

Causes of death: Malignant neoplasm of trachea, bronchus and lung, Diseases of the genitourinary system (N00-N99), Other diseases of the circulatory system (remainder of I00-I99), Diseases of the respiratory system (J00-J99), Parkinson disease, Malignant melanoma of skin, External causes of morbidity and mortality (V01-Y89), Ulcer of stomach, duodenum and jejunum, Other external causes of morbidity and mortality (remainder of V01-Y89), Diseases of kidney and ureter, Hodgkin disease and lymphomas, Other diseases of the nervous system and the sense organs (remainder of G00-H95), Acute myocardial infarction including subsequent myocardial infarction, Drug dependence, toxicomania (F11-F16, F18-F19), Certain infectious and parasitic diseases (A00-B99), Certain conditions originating in the perinatal period (P00-P96), Other diseases of the digestive system (remainder of K00-K93), Influenza (including swine flu), Accidental drowning and submersion, Chronic liver disease, Malignant neoplasm of cervix uteri, Intentional self-harm, Sudden infant death syndrome, Other lower respiratory diseases, Malignant neoplasm of colon, rectosigmoid junction, rectum, anus and anal canal, Cerebrovascular diseases, Malignant neoplasm of lip, oral cavity, pharynx, Other malignant neoplasms (remainder of C00-C97), Malignant neoplasm of breast, Assault, Symptoms, signs and abnormal clinical and laboratory findings, not elsewhere classified (R00-R99), Tuberculosis, Malignant neoplasm of thyroid gland, Diseases of the skin and subcutaneous tissue (L00-L99), Other infectious and parasitic diseases (remainder of A00-B99), Malignant neoplasm of brain and central nervous system, Transport accidents (V01-V99, Y85), Non-malignant neoplasms (benign and uncertain), Neoplasms, Other symptoms, signs and abnormal clinical and laboratory findings (remainder of R00-R99), Diabetes mellitus, Alzheimer disease, Mental and behavioural disorders due to use of alcohol, Pregnancy, childbirth and the puerperium (O00-O99), Diseases of the digestive system (K00-K93), Diseases of the musculoskeletal system and connective tissue (M00-M99), Malignant neoplasm of liver and intrahepatic bile ducts, Accidents (V01-X59, Y85, Y86), Other heart diseases, Event of undetermined intent, Malignant neoplasm of larynx, Chronic lower respiratory diseases, Malignant neoplasm of other parts of uterus, Ill-defined and unknown causes of mortality, All causes of death (A00-Y89) excluding S00-T98, Other endocrine, nutritional and metabolic diseases (remainder of E00-E90), Other diseases of the respiratory system (remainder of J00-J99), Pneumonia, Other mental and behavioural disorders (remainder of F00-F99), Mental and behavioural disorders (F00-F99), Malignant neoplasm of pancreas, Other diseases of the genitourinary system (remainder of N00-N99), Diseases of the blood and blood-forming organs and certain disorders involving the immune mechanism, Accidental poisoning by and exposure to noxious substances, Endocrine, nutritional and metabolic diseases (E00-E90), Rheumatoid arthritis and arthrosis (M05-M06,M15-M19), Malignant neoplasm of stomach, Other accidents (W20-W64, W75-X39, X50-X59, Y86), Other malignant neoplasm of lymphoid, haematopoietic and related tissue, Malignant neoplasm of bladder, Malignant neoplasm of prostate, Viral hepatitis and sequelae of viral hepatitis, Other diseases of the musculoskeletal system and connective tissue (remainder of M00-M99), Falls, Ischaemic heart diseases, Asthma and status asthmaticus, Diseases of the nervous system and the sense organs (G00-H95), Congenital malformations, deformations and chromosomal abnormalities (Q00-Q99), Leukaemia, Malignant neoplasm of kidney, except renal pelvis, Diseases of the circulatory system (I00-I99), Malignant neoplasm of oesophagus, Malignant neoplasms (C00-C97), Human immunodeficiency virus [HIV] disease, Malignant neoplasm of ovary, Dementia, Other ischaemic heart diseases

To make sure that the data is interpreted correctly, we have to specify a type for the numeric columns (TIME and Value). Also, we should specify that commas are used as a delimiter for thousands, and that missing data is replaced by a colon.

In [3]:

data = pd.read_csv("../downloads/hlth_cd_asdr2_1_Data.csv", 
                   na_values=":", thousands=",", 
                   dtype={"TIME": np.int, "Value": np.float})
print(data.Value[:5])

0    1023.02
1      23.54
2       0.48
3       0.89
4       0.41
Name: Value, dtype: float64

I want to make a diagram showing the average number of deaths in all of Western Europe from 2011-2013, so we need to average over everything except the different age classes and the causes of death. We can do that by using a multi-index, i.e. grouping all of the data by age class and cause of death, and averaging over everything that is not an index. We should also sort the data, so we get an idea of the top-killers in our data set.

In [4]:

data_mean = data.groupby(["AGE","ICD10"])["Value"] \
            .mean().sort_values(ascending=False)
print("Total:")
print(data_mean["Total"][:10])

print("\nYoung people:")
print(data_mean["Less than 65 years"][:10])

Total:
ICD10
All causes of death (A00-Y89) excluding S00-T98                           978.673333
Diseases of the circulatory system (I00-I99)                              333.203519
Neoplasms                                                                 264.851852
Malignant neoplasms (C00-C97)                                             256.885926
Ischaemic heart diseases                                                  125.679630
Diseases of the respiratory system (J00-J99)                               87.466111
Other heart diseases                                                       82.292593
Cerebrovascular diseases                                                   73.286296
Other ischaemic heart diseases                                             69.457222
Acute myocardial infarction including subsequent myocardial infarction     56.222593
Name: Value, dtype: float64

Young people:
ICD10
All causes of death (A00-Y89) excluding S00-T98         179.045370
Neoplasms                                                71.335000
Malignant neoplasms (C00-C97)                            70.279259
Diseases of the circulatory system (I00-I99)             32.671667
External causes of morbidity and mortality (V01-Y89)     25.117037
Malignant neoplasm of trachea, bronchus and lung         16.397778
Ischaemic heart diseases                                 15.391481
Accidents (V01-X59, Y85, Y86)                            12.291111
Diseases of the digestive system (K00-K93)               11.282407
Intentional self-harm                                    10.239259
Name: Value, dtype: float64

Note that the causes of death are not exclusive - for instance, Malignant neoplasms are a sub-group of Neoplasms. Let’s pick some interesting causes of death that we want to show in our diagram, and rename them into something snappier. I also convert the death rate to per 10 Million people, because the kills by terrorist are so low.

In [5]:

rename_keys = {
              "Diseases of the circulatory system (I00-I99)": 
                    "Heart disease",
              "Neoplasms": "Cancer",
              "Diseases of the respiratory system (J00-J99)": 
                    "Lung disease",
              "Diseases of the digestive system (K00-K93)": 
                    "Diseases of the\ndigestive system",
              "Diseases of the nervous system and the sense organs (G00-H95)": 
                    "Neurological diseases\n(e.g. Alzheimers, Parkinson)",
              "Mental and behavioural disorders (F00-F99)": 
                    "Mental disorders\n(e.g. Dementia)",
              "Intentional self-harm": "Suicide",
              "Accidents (V01-X59, Y85, Y86)": "Accidents",
              "Drug dependence, toxicomania (F11-F16, F18-F19)": "Drug abuse",
              "Endocrine, nutritional and metabolic diseases (E00-E90)": 
                    "Endocrine, nutritional\n and metabolic diseases\n(e.g. Diabetes)",
              "Diseases of the genitourinary system (N00-N99)": 
                    "Diseases of the\ngenitourinary system",
              "Ill-defined and unknown causes of mortality": "Unknown",
              "Accidental poisoning by and exposure to noxious substances": 
                    "Accidental poisoning",
              "Sudden infant death syndrome": "Sudden infant\ndeath syndrome",
              "Assault": "Assault"
            }
total_data, young_data = [data_mean[x][rename_keys.keys()]
                          # choose only causes of death from rename_keys dictionary
                          .rename(index=rename_keys)         
                          # do the renaming
                          .sort_values(ascending=False)     
                          # re-sort, order might change
                          * 100                             
                          # convert from death rate per 100,000 to per 10 Mio. people
                          for x in ["Total", "Less than 65 years"]]

Now we still need to add the kills by terrorists to the data. For this purpose, I took the number of people killed in terrorist attacks between 2001 and 2015 from Statista, and averaged over the 15 years. Since the data from Eurostat is in dead people / 100,000 inhabitants, we still need to convert this number a bit to get a comparable value. I estimate the population of Western Europe to be around 450 Million, of which 91.5% are aged 65 and under. Based on these numbers, I can get the death rate of people caused by terrorist attacks (assuming everyone killed is < 65).

This process might seem a bit crude, but the exact numbers really don’t matter. Even if my estimates were wrong by a factor of 2, virtually nothing would change in the resulting proportions.

In [6]:

terrorism_deaths = 567 / 15 # fatalities per year
w_e_inhabitants = 450E6
w_e_inhabitants_under_65 = w_e_inhabitants * .915

def add_terror(df,rate):
    return df.append(pd.Series([rate],index=["Terrorism"]))

total_data = add_terror(total_data, terrorism_deaths / w_e_inhabitants * 1E7)
young_data = add_terror(young_data, terrorism_deaths / w_e_inhabitants_under_65 * 1E7)

Finally, I add a ‘other’ column to the data frames, to show how much of the data we threw away when we picked only the causes of death in the rename_keys dictionary.

In [7]:

def add_other(df, total):
    return df.append(pd.Series([total - df.sum()], index=["Other"]))

total_deaths = "All causes of death (A00-Y89) excluding S00-T98"
total_data = add_other(total_data, 
                    data_mean["Total"][total_deaths] * 100)
young_data = add_other(young_data, 
                    data_mean["Less than 65 years"][total_deaths] * 100)

Plot the data¶

We can now produce a first bar plot of the data sets:

In [8]:

fig, (ax1, ax2) = plt.subplots(1,2,figsize=(12,8))

pos = np.arange(len(total_data))
ax1.bar(pos, total_data)
ax2.bar(pos, young_data);

Well, that doesn’t look so nice, yet. We need to add some labels and captions, so it becomes clear what we are plotting.

In [9]:

fig, (ax1, ax2) = plt.subplots(1,2,figsize=(12,8))
fig.text(s="The most common ways to die in the West",x=0.05,y=1,fontsize=24)
fig.text(s="Standardized death rate per 10 Mio. people "
         "in Western Europe, 2011-2013, average per year",x=0.05,y=.95)
pos = np.arange(len(total_data))

ax1.bar(pos, total_data)
ax1.set_xticks(pos)                   # set tick positions
ax1.set_xticklabels(total_data.index) # set tick labels

ax2.bar(pos, young_data)
ax2.set_xticks(pos)
ax2.set_xticklabels(young_data.index);

Well, these labels are impossible to read, because they overlap. Let’s change the orientation of the bar plot to horizontal, so the labels are getting more horizontal space.

In [10]:

fig, (ax1, ax2) = plt.subplots(1,2,figsize=(12,8))
fig.text(s="The most common ways to die in the West",x=0.05,y=1,fontsize=24)
fig.text(s="Standardized death rate per 10 Mio. people "
         "in Western Europe, 2011-2013, average per year",x=0.05,y=.95)
pos = np.arange(len(total_data),0,-1)

# align the center of the bar at the specified locations
ax1.barh(pos, total_data, align="center")  
ax1.set_yticks(pos)                        # set tick positions
ax1.set_yticklabels(total_data.index)      # set tick labels

ax2.barh(pos, young_data, align="center")
ax2.set_yticks(pos)
ax2.set_yticklabels(young_data.index, va="center");

Now the labels are readable, but the two diagrams stil overlap. Actually, there is no real reason to print the labels twice, since both plots show the same thing anyway. We can just combine the two bar plots into one subplot.

In [11]:

fig, ax = plt.subplots(figsize=(10,14))
fig.text(s="The most common ways to die in the West",x=0.05,y=1,fontsize=24)
fig.text(s="Standardized death rate per 10 Mio. people "
         "in Western Europe, 2011-2013, average per year",x=0.05,y=.95)

# location of total data bars, we need more space now
pos1 = np.arange(3*len(total_data),0,-3) 
ax.barh(pos1, total_data, align="center", color="red", 
        label="All Western Europeans")

pos2 = pos1 - 1 # location of young data bars
ax.barh(pos2, young_data[total_data.index], align="center", 
        label="Western Europeans aged 65 and under")

ax.set_yticks((pos1 + pos2)/2)  # set tick positions between pos1 and pos2 
ax.set_yticklabels(total_data.index)
plt.legend(loc="lower right");  # add a legend

Alright! We have produced a plot that is ugly, but at least readable.

Make it pretty¶

My first step towards a pretty plot is to import seaborn. Just by importing this package, calling its set function, and choosing a different color palette, the plots already become much prettier.

In [12]:

import seaborn as sns
sns.set("talk",style="white")
bar_palette = sns.color_palette("Paired",12,desat=.8)

In [13]:

# same code as before, now with seaborn and a color palette!
fig, ax = plt.subplots(figsize=(7,12))
fig.text(s="The most common ways to die in the West",x=0.05,y=1,fontsize=24)
fig.text(s="Standardized death rate per 10 Mio. people "
         "in Western Europe, 2011-2013, average per year",x=0.05,y=.95)

pos1 = np.arange(3*len(total_data),0,-3)
ax.barh(pos1, total_data, align="center", color=bar_palette[1::2], 
        label="All Western Europeans")

pos2 = pos1 - 1 
ax.barh(pos2, young_data[total_data.index], align="center", 
        label="Western Europeans aged 65 and under",
        color=bar_palette[::2])

ax.set_yticks((pos1 + pos2)/2) 
ax.set_yticklabels(total_data.index)
plt.legend(loc="lower right"); 

Seaborn provides an easy way to remove the grid spines at the top and right of the axes (which are not really needed here). I chose to lighten the remaining spines, which is often a good idea if you have big, black lines in your diagram.

In [14]:

fig, ax = plt.subplots(figsize=(7,12))
fig.text(s="The most common ways to die in the West",x=0.05,y=1,fontsize=24)
fig.text(s="Standardized death rate per 10 Mio. people "
         "in Western Europe, 2011-2013, average per year",x=0.05,y=.95)

pos1 = np.arange(3*len(total_data),0,-3)
ax.barh(pos1, total_data, align="center", color=bar_palette[1::2], 
        label="All Western Europeans")

pos2 = pos1 - 1 
ax.barh(pos2, young_data[total_data.index], align="center", 
        label="Western Europeans aged 65 and under", color=bar_palette[::2])

ax.set_yticks((pos1 + pos2)/2) 
ax.set_yticklabels(total_data.index)
plt.legend(loc="lower right")
sns.despine()
ax.spines['left'].set_edgecolor("#AAAAAA")
ax.spines['bottom'].set_edgecolor("#AAAAAA")

Actually, one of the golden rule of making pretty diagrams is to throw away everything that is unnecessary and might distract the viewer from the data. It would be much nicer to label the bars directly, and remove the bottom axis completely. This can be done via a simple code snippet:

In [15]:

def autolabel(rects):
    """ Attach some text labels to a horizontal bar plot """
    # determine maximum width of the bars, and offset labels by 2% of that
    offset = max([rect.get_width() for rect in rects]) * .02
    for rect in rects:
        width = rect.get_width()
        ax.text(width + offset, rect.get_y() + rect.get_height()/2.,
                "{:.0f}".format(round(width)), # display rounded bar width
                ha='left', va='center',fontsize="x-small")

So, let’s despine the figure completely. I have also removed the outline of the bars, giving the diagram a flatter look, and fixed the limits of the y-axis to remove unnecessary whitespace at the top.

In [16]:

fig, ax = plt.subplots(figsize=(7,12))
fig.text(s="The most common ways to die in the West",x=0.05,y=1,fontsize=24)
fig.text(s="Standardized death rate per 10 Mio. people"
         "in Western Europe, 2011-2013, average per year",x=0.05,y=.95)

pos1 = np.arange(3*len(total_data),0,-3)
bar1 = ax.barh(pos1, total_data, align="center", color=bar_palette[1::2], 
               label="All Western Europeans", linewidth=0)

pos2 = pos1 - 1 
bar2 = ax.barh(pos2, young_data[total_data.index], align="center", 
        label="Western Europeans aged 65 and under",
        color=bar_palette[::2], linewidth=0)

ax.set_yticks((pos1 + pos2)/2) 
ax.set_yticklabels(total_data.index)
plt.legend(loc="lower right")
sns.despine()
ax.spines['left'].set_edgecolor("#AAAAAA")
ax.spines['bottom'].set_visible(False)
ax.xaxis.set_visible(False)
ax.set_ylim((0,pos1[0]+2))

autolabel([rect for rect in bar1] + [rect for rect in bar2])

Some last cosmetics:

slightly adjust the spacing of the bars
calling tight_layout() to neatly arrange everything inside the figure
moving the figure title all the way to the left border, and printing it bold
adding a data source label
using subplots_adjust to make some extra room for the figure title

In [17]:

fig, ax = plt.subplots(figsize=(8,12))
fig.text(s="The most common ways to die in the West",x=0.01,y=1,
         fontsize=24,weight="bold",ha="left")
fig.text(s="Standardized death rate per 10 Mio. people, 2011-2013, average per year",
         x=0.01,y=.97,ha="left")

pos1 = np.arange(2.8*len(total_data),0,-2.8)
bar1 = ax.barh(pos1, total_data, align="center", color=bar_palette[1::2], 
               label="All Western Europeans", linewidth=0, height=1)

pos2 = pos1 - 1.2
bar2 = ax.barh(pos2, young_data[total_data.index], align="center", 
                label="Western Europeans\naged 65 and under",
                color=bar_palette[::2], linewidth=0, height=1)

ax.set_yticks((pos1 + pos2)/2) 
ax.set_yticklabels(total_data.index)
plt.legend(loc="lower right")
sns.despine()
ax.spines['left'].set_edgecolor("#AAAAAA")
ax.spines['bottom'].set_visible(False)
ax.xaxis.set_visible(False)
ax.set_ylim((0,pos1[0]+1))

autolabel([rect for rect in bar1] + [rect for rect in bar2])
plt.annotate("Data: Eurostat", (.01,.03), xycoords = "figure fraction",
             fontsize="small", color="#444444")
fig.tight_layout()
fig.subplots_adjust(top=.94)
# create a figure that measures about 500px in width
fig.savefig("deaths-bar.png",bbox_inches="tight",transparent=True,dpi=500/8)

Ta-da! Looks nice, doesn’t it? If you want, you can play around a bit - it could be nice to use a different color scheme, or to have a light grid in the background, so it becomes easier to compare values. Also, I’m not entirely happy with the different scales on the two data sets - this way, the difference in relative importance of e.g. cancer and suicide isn’t really striking the viewer.

Why you should not be afraid of terrorism

2016-03-26T00:00:00+01:00

Introduction¶

In the wake of the recent terror attacks in Western Europe (Jan 2015 and Nov 2015 in Paris; Mar 2016 in Brussels), I started observing the dynamics of the European society’s reaction to events of this kind. I find it remarkable how impactful these events seem on both the general public, including political decisions, the rise and fall of political parties, and mass-media coverage, and people in my personal social circles. One of the oldest, most important emotions in the whole animal kingdom seems to be the prime reason behind the vehement, sometimes even violent, reactions: fear.

In this blog post, I will examine where this fear comes from and why it is absolutely understandable - but most importantly, I will show my reasoning as to why it is both irrational and dangerous to be afraid of terrorism in Western Europe.

General human risk-assessment mechanisms¶

Human intuition is without a doubt one of the greatest achievements in the history of evolution. Paired with some strategical thinking, it helps us to quickly make decisions without having to consider every possible action. Intuition allows us to play the game of Go nearly as well as a computer network consisting of 1202 processors and 176 graphics cards, to learn and speak languages fluently (even those with a highly complicated underlying grammatical structure), or to evaluate risks the second they manifest. However, despite its usefulness, human intuition is definitely not flawless, especially in situations that are relatively new to mankind (i.e., that have not been around for thousands of years). Luckily, we have reason to deal with these cases, but this requires some more work.

One particular thing that humans are generally pretty bad at is making decisions that may have a life-changing impact when small probabilities are involved. When using simple intuition, the actual probability (whether given or estimated from a frequency of events) becomes less and less important the more extreme it gets, and the outcome of the decision becomes dominated by all kinds of biases.

One example where (biased) intuition is present is gambling. Any commercial gambling operator has to be sustainable and thus make money, so every gamble of pure chance will be designed in a way that favors the operator. The most popular lottery in Germany, Lotto 6 aus 49 (comparable to the US-American Powerball), with around 21 million weekly players, offers a chance of 139,838,160 to 1 to win the jackpot of averagely 9 million Euro. One lottery ticket is currently priced at 1€, but the mean winning money (including all winning categories) lies at only 50ct. To make this clear: instead of buying a lot of lottery tickets, you might as well shove half of your money out of the window (with the possible side effect of making some homeless people’s day)¹. So, why are people even playing the lottery (or even worse, slot machines and alike)? The answer is biases:

Outcome bias: the notion to only take into account the possible outcomes of the decision (i.e., buying a lottery ticket), while disregarding or misjudging their corresponding likelihood. Either you win, or you lose, or also: winning the jackpot makes my life much better, while spending 1€ on the ticket barely affects me.
Some sort of selection bias: Someone winning the lottery will be reported on the news (or will at least be talked about in their social environment), and winning the lottery has a huge emotional impact. This is why we tend to ignore the millions of losers while focusing on a few winners, and put ourselves in their shoes.
(Non-)zero-risk bias: If one doesn’t play the lottery, there is no way they can win - buying a ticket will give them a chance at the jackpot, how slim it may be (even though the money could be spent better). Someone is going to win the lottery, so it could as well be me.

Both biases are naturally strongest when considering low-probability, high-impact events.

The reasons behind a fear of terrorism¶

I have chosen playing the lottery as an example for a simple intuitive risk-assessment because most people can relate to it; but of course, the exact same analysis can be applied to events that are very unlikely, but have a huge negative impact, such as a terroristic attack. All that is needed is to re-label some terms:

winning the lottery becomes being killed or injured in an attack (low probability)
not winning the lottery becomes not being involved in an attack (high probability)
buying a lottery ticket becomes leaving the house (assuming you are safe at home, and putting yourself at risk in public spaces)

Applying the exact same analysis as before:

Outcome bias: Going outside can get you killed, while staying at home seems safe, whatever the chances are.
Selection bias: Without a doubt, terrorist attacks and their victims receive massive media attention, triggering human empathy, and making terrorist attacks seem like something that could happen to anyone, anytime.
Zero-risk bias: People would rather reduce the risk of being killed in an attack to zero (by staying at home) than taking this slim risk and, e.g., quit smoking (which increases life expectancy by a much larger amount, but does not eliminate the risk of falling ill with lung cancer entirely).

Of course, these cognitive biases are by far not the whole truth why terrorist attacks are scary. One very important aspect is also the lack of feeling in charge or being familiar with the situation. This is also the main reason why people tend to be much more scared of crashes with an airplane compared to crashes with a car. In an airplane, or a terrorist attack, there is nothing much you can do apart from praying or running for your life, while in a dangerous situation involving a car you have the *illusion of being in charge (obviously, there are many car accidents that no one saw coming, or were able to prevent).

It is very human to not be afraid in everyday situations where you have some way to react, or where one has the feeling that fate lies in their own hands (*I am not going to get cancer, since I’ve never smoked in my life). This feeling of safety is obviously false, but it is what gets us out of bed every day, and there is no evolutionary need to be afraid.

A more rational risk analysis²¶

Well, let’s have a look at some numbers. The statistics portal Statista has created a really nice diagram (using the global terrorism database), putting terrorism into perspective:

As you can see, the death toll from terrorism in Europe has been way higher in the 70s and 80s than today. Another very interesting diagram is the deaths through terrorism outside the EU:

Yep, that’s 420 deaths in Western Europe vs. over 100,000 in the rest of the world. So, obviously, terrorism is a much bigger threat in the rest of the world, and terrorism death tolls have been way higher in the past. But these are only relative statements; it doesn’t tell us how large the absolute danger of terrorism is today. To put things into perspective, I have made the following diagram with data from Eurostat:

This diagram shows how irrational a fear of terrorism really is - it causes an absolute minor contribution to the total deaths in Western Europe, of about 1 in 10 Million people. In comparison, lung cancer accounts for over 5000 times as many fatalities. Smoking increases the risk to fall ill with lung cancer by about 1000%. If everyone stopped smoking, about 16% of all deaths could be prevented - in contrast to the 0.001% of deaths that could be prevented by if no terroristic attacks were taking place.

This huge disparity shows the power of cognitive biases at work. Because terroristic attacks receive immense media coverage, because they create so much empathy with their victims, and because none of us is familiar with this kind of threat, they are perceived as unproportionally dangerous to each and every one of us, while their real threat is far less severe.

Wrapup: Don’t be afraid, or you are supporting the terrorists¶

I’m sorry, but I just have to quote Star Wars here:

“Fear is the path to the dark side. Fear leads to anger. Anger leads to hate. Hate leads to suffering.” – Yoda

Fear of terrorism is not only dangerous because you could be missing out on some fun if you’re staying at home. It causes overreactions, irrational behavior, blaming innocent people as scapegoats. It blurs people’s view, de-empathizes, and radicalizes them. It is also the main reason why these terroristic crimes are committed in the first place.

A terrorist probably has the immediate goal to kill as many people has possible. The actual goals with this action are diverse; on one hand, they may want to fulfill some kind of sacred mission and become a hero by killing non-believers. On the other hand, they want to create awareness for their cause and provoke the western society into making mistakes. The more violent the West’s reaction on a terroristic attack is, the more people the terrorists will be able to recruit. The main goal behind a terroristic attack is to cause fear, chaos, and mayhem. Don’t let them win.

¹ Of course, there are indeed situations where playing the lottery can be a quite a good idea. For example, if you happen to live by the credo “get rich or die trying”, i.e., you consider your life a complete failure in case you don’t have six figures on your account by the time you die. In this case, playing the lottery might be an appropriate - though a little desperate - strategy.

² I won’t call my analysis unbiased, since there is no such thing when humans are involved.

Creating a science conference poster with Inkscape

2015-07-17T00:00:00+02:00

Update (Sep 2021): There is now an updated version of this blog post with more recent advice on how I would design a poster today. Make sure to check it out if this post appeals to you!

First things first¶

This poster design is heavily based on the poster template provided by Felix Breuer. Many thanks to him for creating such a beautiful and unusual poster from scratch, and even publishing the template for others to use. Accordingly, this modification is published under the Creative Commons Attribution-ShareAlike 3.0 Unported License as well, which means that you are free to use and modify this template in any fashion you would like, as long as you give credit to the original authors and publish your modification as a template, too.

Our poster¶

This poster was created during a MSc level physics course on climate at Heidelberg University as final project. We were given a paper (that we are in no way affiliated with) from climate research and told to create a poster that would be well-suited to represent the contents of the paper at a conference. Our goal was to create a poster that stands out in the mass of standard, blocky posters that physicists tend to create. Additionally, we wanted the poster layout to tell a story, so you would not have to read the whole poster to understand what was going on - we wanted the plots and figures and some short summary to convey the most important ideas at a glance.

Inkscape seemed to be the perfect tool for this task, since it is free, has powerful vector tools, and is based on visuals rather than code. So, this is what we came up with:

(you can download the SVG version of the poster below)

Design choices¶

We took over most of the basic ideas from Felix Breuer’s poster. However, his poster was clearly a math poster: little text, many formulae, rather sketches and drawings than plots. The design he chose fitted the ‘math style’ exceptionally well, but did not really seem suitable for a poster from environmental physics, since we had completely different requirements. So we made the following changes to the original design:

The color scheme was changed. The original blue-and-grey scheme was pretty, but we wanted to use somewhat lighter colors, and match the color scheme used by our faculty (red and white). However, the original idea of applying a color gradient to any surface remains unchanged, and leads to a nice illusion of depth.
Our poster would be much more text-heavy, so we needed more space for the text boxes around the edges of the poster. We chose a larger font than Felix Breuer did (28 instead of 24) to make the wall of text seem less intimidating (and to prevent ourselves from writing too much text).
Probably the most notable difference is the different tile structure in the middle of the poster. A circular pattern did not seem reasonable in our case: We would have to include quite a few rectangular figures, so a circle would take away too much usable space. Instead, we decided on a flattened U-shape, which still had the original spirit of Felix Breuer’s poster (explain everything with the plots in the center).
As the poster’s structure is highly unusual, many readers were confused as of where to begin to read, and where to continue. We tried to make this intuitively clear by adding as many arrows and visual hints as possible. This way, only very few people tried to read downwards from the block at the upper-left corner, and pretty much everyone found their way towards the right hand side of the poster.
The summary block was moved from the middle to the top of the poster. This spot is definitely not perfect, since it distorts the reading flow of a reader that starts at the upper-left of the poster as the arrows indicate. After reading the first block, the reader will arrive at the summary block instead of the block at the upper-right. Other ideas would be to move the summary to the middle of the poster again, or put it as a first block right above the title of the poster (or omit it entirely).
We added a QR code created here (which features some short text/advertisement while redirecting when using their free plan - we did not find this particularly annoying, but if it bothers you, just pay for it, create the code yourself, or create a code without a background image) linking to the paper our work was based on. We found this a nice gimmick, especially with the logo in the background, since one could immediately tell where they would end up after scanning the code. However, I doubt many people would actually scan it on a conference, and you could just omit it.
We were experimenting with different effects on the poster’s title and our names. At first, we used a much larger drop shadow, but we found the contrast at these places to be quite low. This certainly got better by using a sharper shadow, but you could consider removing it entirely.
We typeset all texts directly in inkscape and created hidden boxes that we flowed the text into. There might be a more elegant way to do this, but it did the trick. They are all in their own hidden layer, so if you want to change the shape of the flowing text, just show that layer and edit the boxes.

Printing¶

Following Felix Breuer’s suggestion, we exported the poster as a 500-dpi png (i.e., raster image), and printing worked like a charm, although the resulting image was about 40 MB large. Later, we also tried to export the poster as pdf using inkscape’s export function, and this seemed to work well, too. Text turned out to be crisp, and all vector plots printed very well - only the QR code and two of the plots, which we did not have in vectorized form, turned out a little blurry, so make sure you get your hands on vectorized versions of anything you want to include.

Conclusion¶

Creating our poster in inkscape was highly rewarding and a lot of fun, even though we had never used the program before. I honestly don’t think that the poster creation would have been much faster using LaTeX or any other common tool, and the result would definitely have been more boring. We are pretty happy with the result, and it received positive criticism during our private poster session - everyone agreed, that they were immediately curious to read the poster, just by having a glance at the unusual design.

Download the poster template¶

Inkscape SVG file

Make sure to use right click - Save Link As.. and open the file in inkscape instead of viewing it in your browser, where it will probably not render correctly.