Cybercat Institute

Autodiff through function types: Categorical semantics the ultimate backpropagator

2026-02-20T00:00:00+00:00

In this post we will see how to extend reverse mode automatic differentiation to a language with first class function types, function application and $\lambda$-abstraction. This method is not new, but we will give a new derivation of it by showing how it arises universally from noticing that the category of “additive lenses” is cartesian closed. In the end we will see that this idea sounds like it should revolutionise machine learning, but then doesn’t. Everything in this post is joint work with Bruno Gavranović.

Additive containers and additive lenses

We need to begin with some background categorical cybernetics, from the paper Categorical foundations of gradient-based learning and Bruno’s PhD thesis.

The categorical semantics of types in a differentiable language are additive containers, which is what we call set-indexed families of commutative monoids. (The reason we use an unnecessarily fancy word is because set-indexing is stronger than we sometimes want or need; categorical cybernetics has lots of machinery for working with related objects so we use terminology that unifies them.) The idea is the semantics of a type is (1) a set of values that the type can have in the forwards pass, and (2) for each of those values, a commutative monoid of derivatives it can have at that value in the backwards pass.

The semantics of a function $A \to B$ is an additive lens between the additive containers $(A, A’)$ and $(B, B’)$. This consists of two things: (1) an ordinary function between sets $f : A \to B$ describing the semantics of the forwards pass, and (2) an $A$-indexed family of monoid homomorphisms $f’ (x, -)$ from the commutative monoid $B’ (f (x))$ to $A’ (x)$. That is to say, these are dependent lenses whose backwards passes are additive in their second input. The chain rule for lens composition exactly describes the denotational semantics of how the forwards and backwards passes interact in autodiff.

In the category of ordinary containers and lenses, without the comonoid structure, we have both a cartesian product and a tensor product. The cartesian product of containers (ie. it satisfies the universal property) is given by product in the forwards pass and coproduct in the backwards pass: $(A, A’) \times (B, B’) = (A \times B, A’ + B’)$. (To be more specific, the backwards part is the $(A \times B)$-indexed family of sets $(A’ + B’) (x, y) = A’ (x) + B’ (y)$.) This happens essentially because the backwards pass is backwards, so the universal property of a product turns into a coproduct. There is also a tensor product given by product in the forwards and backwards passes: $(A, A’) \otimes (B, B’) = (A \times B, A’ \times B’)$. For the majority of applications in categorical cybernetics the tensor product is the useful one, but it doesn’t satisfy a straightforward universal property.¹

But in additive containers something different happens: the thing that satisfies the universal property of a cartesian product switches to $(A, A’) \times (B, B’) = (A \times B, A’ \times B’)$. The structural reason for this is that the category $\mathbf{CMon}$ of commutative monoids has finite biproducts: the monoid of pairs $A \times B$ satisfies the universal properties of both a product and a coproduct. It’s worth thinking for a moment about why this is: given maps $f : A \to C$ and $g : B \to C$, we get a copairing $[f, g] : A \times B \to C$ given by $[f, g] (x, y) = f (x) + g (y)$; and injections are given by setting one of the values to be zero.

This gives the correct semantics to copying in autodiff: if we use a forwards pass value multiple times, we get a different derivative coming from each use site and the right thing to do is to add those derivatives.

Additive containers are cartesian closed

The category of ordinary containers and ordinary lenses is monoidal closed for the tensor product.² Given containers $(A, A’)$ and $(B, B’)$, the hom-container has as its forwards set the set of all lenses $(A, A’) \to (B, B’)$, and as the backwards set indexed by a specific lens $(f, f’) : (A, A’) \to (B, B’)$ it has the set $\Sigma (x : A) . B’ (f (x))$, whose elements are pairs $(x, y’)$ where $x : A$ and $y’ : B’ (f (x))$. Writing this whole container as $(A, A’) \multimap (B, B’)$, this gives us the universal property that lenses $(A, A’) \otimes (B, B’) \to (C, C’)$ are in natural bijection with lenses $(A, A’) \to (B, B’) \multimap (C, C’)$.

Meanwhile in the world of additive containers, the thing that was previously just the tensor product is now the cartesian product, so we might wonder whether additive containers are cartesian closed. And they are! The forwards set of $(A, A’) \multimap (B, B’)$ is, as expected, the set of additive lenses of that type. (I am going to continue using the symbol $\multimap$ for the internal hom of additive containers, to carefully distinguish it from functions in the underlying category of sets.) But there is a very interesting twist in the backwards part.

The backwards monoid is still the thing that we would write $\Sigma (x : A) . B’ (f (x))$. It is the coproduct in the category $\mathbf{CMon}$ of the $A$-indexed family of monoids $B’ (f (x))$. So we need to know what coproducts of commutative monoids are like. The category $\mathbf{CMon}$ has finite biproducts, but for infinite families products and coproducts no longer coincide. The product remains the obvious thing, the monoid whose elements are dependent functions - this is called a direct product. The coproduct is a slightly different construction called a direct sum: an element of $\Sigma (x : A) . B’ (f (x))$ is a dependent function $(x : X) \to B’ (f (x))$ satisfying the property that all except finitely many values are zero. That is, it is a finite support dependent function. (The monoid structure is pointwise addition of functions.) The underlying set of $\Sigma (x : X) . B’ (f (x))$ is something I would write as $\Pi_{\mathrm{fs}} (x : A) . B’ (f (x))$.

It’s worth also spending a moment thinking about why the coproduct has to be something like this. Suppose we have an infinite family of comonoids $A_i$, and for each of them we have a map out $f_i : A_i \to B$. We need to be able to take their “cotupling” $[f_i]_i : \Sigma_i A_i \to B$, which we know from the binary case should be to evaluate each $f_i (a_i)$ and then add the results in $B$. But $B$ doesn’t admit any kind of infinitary addition in general, so we can only do this if we know that only finitely many of the $a_i$ are nonzero.

To summarise: the category of additive containers and additive lenses is cartesian closed, where the hom-container $(A, A’) \multimap (B, B’)$ has as its forwards set the set of additive lenses, and where the backwards commutative monoid over an additive lens $(f, f’)$ is the coproduct of commutative monoids $\Sigma (x : A) . B’ (f (X))$, whose underlying set is the set of finite support functions $(x : A) \to B’ (f (x))$.

Representing the direct sum

How do we represent the set of finite support functions in an implementation? The simplest idea would be to just use a function with an informal datatype invariant. But it turns out that we need the copairing operation to be implementable, which is not possible using this representation. The thing we are actually going to use is a finite list of pairs of inputs and outputs: we represent the set $\Pi_{\mathrm{fs}} (x : A) . B’ (f (x))$ as $(\Sigma (x : A) . B’ (f (x)))^*$.

Of course lists of pairs are not actually the same as finite support functions. For one thing the order doesn’t matter, so we need to quotient out list permutation. Our next thought might be that we should have a datatype invariant that each input value $x$ can only appear once in the list, so it is never ambiguous what the output should be. But there turns out to be a better way: we say that the value of $f (x)$ is the sum of all of the $y$ such that $(x, y)$ appears in the list. (And if $x$ never appears then the sum is $f (x) = 0$.) This amounts to defining a quotient type generated by 3 classes of identifications:

$(x_1, y_1), (x_2, y_2), xs = (x_2, y_2), (x_1, y_1), xs$
$(x, 0), xs = xs$
$(x, y_1), (x, y_2), xs = (x, y_1 + y_2), xs$

If we happened to be working in a language with higher inductive types we could tell the machine this, but as it transpires that would be overkill: it turns out that the only operation we will ever need to define out of a direct sum is to add together all of its entries, and that plainly respects all of these equations. (In the case of the first one, that’s exactly where we use the fact that all of our monoids are commutative.)

I wrote earlier than the commutative monoid structure on finite support functions is pointwise, implicitly relying on the fact that pointwise addition of finite support functions is finite support, and the unit of pointwise addition is the constant zero function which is also finite support. The reason that we want a representation where $f (x)$ is the sum of all $y$ with $(x, y)$ in the list, rather than the arguably more obvious approach of requiring that each $x$ appears in the list at most once, is that it allows us to do a very slick magic trick: pointwise addition of functions is represented by list concatenation. Normally list concatenation gives results in noncommutative monoids, but it is commutative modulo our first quotient equation. This is a slight subtlety: our data representation uses monoids that are technically not commutative, but everything is “heredetarily commutative” in the appropriate sense that whenever we evaluate down to actual numbers everything will work out. There is a sense in which all of our additions are “suspended”, and we will see later that they are “resumed” when the backwards pass gets to the point where we created the function, namely $\lambda$-abstraction.

The 3 steps from function types to direct sums to lists of pairs is a combination of ideas that I have not seen together in one place. But every 2 out of 3 appear somewhere, and the connection between all 3 is no doubt folklore known to some, although we haven’t been able to find actual evidence of that. So this post might be the first universal derivation of the list-of-pairs method, in the sense that everything follows from structural principles and essentially no choices are made.

The connection between function types and lists of pairs seems to be credited to the paper Lambda the ultimate backpropagator, but I find that paper quite impenetrable, so I don’t know exactly how much of it appears there. It also appears clearly for example in The differentiable curry. By far the closest I have seen is the paper CHAD for expressive total languages, which derives cartesian closure for additive lenses appears up to the infinitary direct sum. (Literally the same category: they refer to the category of additive lenses by its less catchy standard name, $\mathbf{Fam} (\mathbf{CMon}^\mathrm{op})$.) But then they never take the final step to lists of pairs, or more importantly recognise that they may have rederived a known method but with a universal derivation. However, the paper has accompanying Haskell code and the list of pairs construction does appear there if you dig deeply enough.

Abstraction and application

Let’s go through the semantics of $\lambda$-abstraction and function application, which introduce and eliminate function types respectively. To do this for real we need the multicategorical version which tracks how a term in a programming language is a function of all of the variables it refers to; but to keep things simple let’s just do the version for categories, which is essentially the same.

Function application is an additive lens of type $\mathrm{ap} : (A \multimap B) \times A \to B$. Its forwards pass takes an additive lens $(f, f’) : (A, A’) \to (B, B’)$ and a value $a : A$ to the value $f (a) : B$. Its backwards pass

\[\mathrm{ap} : ((f, f') : (A, A') \to (B, B')) \times (a : A) \times B' (f (a)) \to ((A, A') \multimap (B, B'))' (f, f') \times A' (a)\]

takes an additive lens $(f, f’) : A \to B$, a value $a : A$ and a gradient $b’ : B’ (f (a))$, and we must produce a gradient in $A’ (a)$ and a function gradient in $((A, A’) \multimap (B, B’))’ (f, f’)$. Of course the $A’ (a)$ gradient is $f’ (a, b’)$.

The interesting part is the function type gradient, which is a list of pairs $(a : A, B’ (f (a)))^*$. The thing of that type that we can make is the singleton list $(a, b’)$. That list represents the function $(a : A) \to B’ (f (a))$ that takes $a$ to $b’$ and every other value to $0$. This might sound useless, but usually the reason we would have a function is because we want to call it many times, and if we do that then the semantics of copying the function value will, in the backwards pass, concatenate all of these singletons together. This will end up tracking the total gradient from every call to each possible input of the function; since a function can only be called finitely many times in any particular run of a program this will always be a finite amount of data.

That covers function application. Function types are made using $\lambda$-abstraction, an operation that takes an additive lens of type $f = A \times B \to C$ (where $A$, $B$ and $C$ are additive containers), and turns it into an additive lens $\lambda (f) : A \to B \multimap C$, as through we are abstracting a variable of type $A$. So the data we have to work with is a fowards pass $f : A \times B \to C$ and a backwards pass $f’ : (a : A) \times (b : B) \times C’ \to A’ (a) \times B’ (b)$, which we will split into $f’_1 : (a : A) \times B \times C’ \to A’ (a)$ and $f’_2 : A \times (b : B) \times C’ \to B’ (b)$.

The forwards pass of $\lambda (f)$ inputs a value $a : A$ and must output an additive lens $\lambda (f) (a) : (B, B’) \to (C, C’)$. The forwards pass of that is obviously given by $\lambda (f) (a) (b) = f (a, b)$, and its backwards pass is $(\lambda (f) (a))’ (b, c’) = f’_2 (a, b, c’)$. (The fact that our forwards pass has a backwards pass is, of course, because we’re doing something higher-order.)

The backwards pass of $\lambda (f)$ takes an input $a : A$ and a list of pairs $(b : B, C’ (f (a, b)))^*$, and it must produce a gradient in $A’ (a)$. What we need to do is to take all of the values $f’_1 (a, b, c’)$, which all live in the commuative monoid $A’ (a)$, and add them together. This is an example of cotupling for a direct sum.

In ordinary first-order autodiff, copying values leads to adding gradients. There is a sense that function types cause reverse copying to be suspended, carrying around a list of all gradients instead of adding them. And then $\lambda$-abstraction is where the suspended operation is resumed and the values are actually added.

There is no free lunch

Several years ago, I was talking with Mario Alvarez-Picallo about the question of differentiating function types, and he told me a piece of wisdom. He explained that if differentiating function types worked the best possible way, then for any supervised learning problem to learn a function $A \to B$, we could choose our parameter space to be the hom-type $A \multimap B$ and our architecture to be $\mathrm{ap} : A \times (A \multimap B) \to B$. That is, the application lens is a universal architecture and the hom-type is a universal parameter space, and in this way the entire science and art of designing deep learning architectures would be entirely obsolete. This would not be so much a free lunch as winning free lunches every day for the rest of our life. But of course, this is not what happens.

The catch is that we have only talked about backprop but not about gradient descent. A deep learning architecture consists of a lens of the shape $f : A \times P \to B$, where $P$ is the parameter space. A simplified view of training (ignoring batching and various other complications) works like this: we take an example pair $(a, b)$ from our dataset, and the current parameters $p$. We have a loss function that takes the predicted output $f (a, p)$ and the target output $b$ and gives us a gradient $b’$, which we then inject into the backwards pass $f’ (a, p, b’)$ which gives us an input gradient $a’$ and a parameter gradient $p’$. The input gradient is thrown away (its work is done by the time it reaches us), but the parameter $p’$ is sent to the optimiser. The most straightforward optimiser is gradient descent: it mutates the current parameters $p$ to $p - \varepsilon p’$, where $\varepsilon$ is a learning rate.³

This makes sense when our parameter space is some $\mathbb R^n$, whose tangent space is also $\mathbb R^n$, so it forms the additive container $(\mathbb R^n, \mathbb R^n)$. (Technically, the first $\mathbb R^n$ is the set of points, and the second $\mathbb R^n$ is the constant function that takes every point in the set to the commutative monoid of vectors with addition.) What if our parameter space is instead a space of functions $\mathbb R^m \to \mathbb R^n$, whose tangent space is the set of finite support functions from points in $\mathbb R^m$ to gradients in $\mathbb R^n$?

It is technically possible to do gradient descent on these things directly, if the current value of our parameters is always a finite support function then we can add the gradients pointwise. This more or less amounts to an architecture that by design will memorise its training data - it can only overfit.

There is something much smarter we can do, which is to replace our optimiser with an entire other supervised learning architecture. We parameterise the function space $\mathbb R^m \to \mathbb R^n$ using a choice of architecture and parameter space $\mathbb R^p$, and when we receive a function gradient - that is a list of pairs $(a, b’)$ - we do our own run of supervised learning on it to optimise our own parameters $p$. And so our free lunch has evaporated in front of our eyes, and we are back to doing ordinary deep learning with extra steps.

I find it plausible that this idea could still have useful applications, in the same way that function types themselves are still useful even though we could have just worked in a Turing-complete language without functions instead. For example, it would allow us to make use of all of the basic functional programming design patterns like mapping and filtering when implementing our architectures, letting the underlying autodiff framework figure out the right thing to do instead of having to unroll everything manually.

How practical this actually is right now depends heavily on how easy it would be to hack this method on top of an existing autodiff framework, which is something we haven’t thought much about yet. My best guess is that making it performant would really require a whole new compute kernel that is able to handle reducing (ie. recursively folding over) lists of suspended gradients, and that this is possible but best left to somebody who knows what they are doing with kernel programming and takes on the task of developing a serious 21st-century autodiff framework that isn’t balanced on top of ancient Fortran code and a programming paradigm that is 50 years out of date.

The thing that currently excites me the most about this is that when we have function types we can simulate continuation passing style using a continuation monad, and so a language with function types can be used as an intermediate compiler target for a language with complex control flow features like throw/catch. What would autodiff through throw/catch look like? I have no idea, but it could be interesting to find out.

That is not to say the tensor product can’t be given any universal property at all. My favourite is to characterise it as the fibrewise opposite of a cartesian product. ↩
It happens to also be cartesian closed, in an extremely non-obvious way, see the paper Higher order containers. That structure plays no role for us. ↩
There is some interesting differential geometry going on here that I only partially understand. ↩

Dependent optics II: Optics via forcing costates

2025-10-16T00:00:00+00:00

I’ve been putting this series of posts off for a long time, although nowhere near as long as we’ve been putting off the corresponding paper.

For anyone who knows anything about my research between 2021 and 2024, dependent optics need no introduction. A team consisting of (in alphabetical order) Dylan Braithwaite, Matteo Capucci, Bruno Gavranović, Eigil Rischel and me put everything we had into the problem of unifying dependent lenses and monoidal optics, for 2 years. We had so many false solutions that “we solved dependent optics!” became a meme, and then we solved it. But by then, time had caught up with us and between us we did not collectively have the ability, energy and motivation to write the paper.

The construction became obscure folklore because we talked about it in various places, for example in this seminar I gave early this year. I am writing this series of posts to upgrade it from folklore to merely grey literature. The reason I put off starting the series for so long is one of the same reasons blocking the writing of the paper: some of the introductory material is some of the most difficult to write. It has been such a long time that I no longer know how to adequately explain why the problem is so difficult. Additionally, one of the current blockers to a paper is figuring out how our construction relates to the one in Vertechi’s paper. So, I have made the outrageous choice to start this blog series with part II, and put off writing part I until the end.

The goal of this post is to reconstruct simple optics in the way that turned out to generalise correctly.

A refresher on optics

So that we are all starting from the same place I will give the standard construction of optics, from this paper and this paper.

If we have a symmetric monoidal category $\mathcal C$, its category of optics $\mathbf{Optic} (\mathcal C)$ has as objects pairs of objects of $\mathcal C$, and morphisms given by elements of the coend $\mathbf{Optic} (\mathcal C) \left( \binom{X'}{X}, \binom{Y'}{Y} \right) = \int^{M : \mathcal C} \mathcal C (X, M \otimes Y) \times \mathcal C (M \otimes Y', X')$

A couple of notes on this:

If the assumption that $\mathcal C$ is symmetric is dropped this construction still does something, but it does the wrong thing (I may write a future blog post on this point). Symmetry turns out to be necessary for what follows anyway, so I am assuming it here.
I am writing my objects with the backwards pass over the forwards pass, for consistency with the fibrational notation for dependent lenses.

If $\mathcal C$ is cartesian monoidal, then $\mathbf{Optic} (\mathcal C)$ is isomorphic to the category $\mathbf{Lens} (\mathcal C)$ of simply-typed lenses. This fact will be important in later posts.

More generally, if we have a span of strong symmetric monoidal functors $\mathcal C \overset{L}\leftarrow \mathcal M \overset{R}\rightarrow \mathcal D$ (or equivalently symmetric monoidal actions of $\mathcal M$ on $\mathcal C$ and $\mathcal D$ - I may also write a future post on this point) then the category of mixed optics $\mathbf{Optic}_\mathcal M (\mathcal C, \mathcal D)$ has as objects pairs of objects of $\mathcal C$ and $\mathcal D$, and morphisms given by elements of the coend $\mathbf{Optic}_\mathcal M (\mathcal C, \mathcal D) \left( \binom{X'}{X}, \binom{Y'}{Y} \right) = \int^{M : \mathcal M} \mathcal C (X, L (M) \otimes Y) \times \mathcal D (R (M) \otimes Y', X')$

In both cases, the category of optics is itself a symmteric monoidal category, given on objects by pairwise monoidal product of the underlying categories.

The bicategory of optics

Categories of optics arise naturally as quotients of bicategories, and this gives a more refined view that will be crucial. With the same setup for mixed optics, we build a symmetric monoidal bicategory $\mathbf{2Optic}_\mathcal M (\mathcal C, \mathcal D)$. Its 0-cells are again pairs of objects of $\mathcal C$ and $\mathcal D$, its 1-cells $\binom{X’}{X} \to \binom{Y’}{Y}$ are triples $\left( M : \mathcal M, f : X \to L (M) \otimes Y, f' : R (M) \otimes Y' \to X' \right)$ and its 2-cells are morphisms $M \to M’$ of $\mathcal M$ making the resulting diagrams commute.

There are 2 universal ways to turn a bicategory into a category. By far the more common way is to quotient out invertible 2-cells, which turns out to be the same thing (at least for an appropriately weak notion of enrichment) as change of enrichment basis $U^* : \mathbf{Cat}-\mathbf{Cat} \to \mathbf{Set}-\mathbf{Cat}$ along the functor $U : \mathbf{Cat} \to \mathbf{Set}$ that takes each category to its set of isomorphism classes.

There is another functor $\pi_0 : \mathbf{Cat} \to \mathbf{Set}$ that instead takes each category to its set of connected components, quotienting out reachability, or equivalently adding all formal inverses before quotienting out isomorphism. (They are related by a string of adjunctions between $\mathbf{Set}$ and $\mathbf{Cat}$, but I don’t remember the details of that right now.) This is not often seen because most categories encountered in practice - for example any category with either an initial or a terminal object - are connected and sent to the 1-element set by $\pi_0$. But the hom-categories of $\mathbf{2Optic}_\mathcal M (\mathcal C, \mathcal D)$ are not connected, and in fact:

Lemma: There is an isomorphism of symmetric monoidal categories $\pi_0^* \left( \mathbf{2Optic} (\mathcal C) \right) \cong \mathbf{Optic}_\mathcal M (\mathcal C, \mathcal D)$.

Proving that all of this actually coheres with the symmetric monoidal structure is a very minor part of what still needs to be done, that will probably take weeks of effort.

Parametrisation

Given a strong symmetric monoidal functor $F : \mathcal M \to \mathcal C$ (or more generally an action of $\mathcal M$ on $\mathcal C$), we can form a symmetric monoidal bicategory $\mathbf{Para}_\mathcal M (\mathcal C)$ of $\mathcal M$-parametrised morphisms of $\mathcal C$. Its 0-cells are the 0-cells of $\mathcal C$, its 1-cells $X \to Y$ are pairs $(M : \mathcal M, f : F (M) \otimes X \to Y)$ and its 2-cells are morphisms $M \to M’$ making the resulting diagram commute.

$\mathbf{Para}_{\mathcal M} (\mathcal C)$ is locally fibred over $\mathcal M$: every hom-category is equipped with a fibration $\mathbf{Para}_\mathcal M (\mathcal C) (X, Y) \to \mathcal M$ compatible with the other structure.

There is a dual construction $\mathbf{Copara}_\mathcal M (\mathcal C)$ whose 1-cells are pairs of $M$ and $X \to F (M) \otimes Y$. This has a local opfibration structure $\mathbf{Copara}_\mathcal M (\mathcal C) (X, Y) \to \mathcal M$.

Here is a construction of optics that is a major contributor to the feeling of how close-knit the foundations of categorical cybernetics are, but is not the construction we are looking for: the hom-category $\mathbf{2Optic}_\mathcal M (\mathcal C, \mathcal D) \left( \binom{X’}{X}, \binom{Y’}{Y} \right)$ is the pullback in $\mathbf{Cat}$ of this opfibration and fibration: $\mathbf{Copara}_\mathcal M (\mathcal C) (X, Y) \rightarrow \mathcal M \leftarrow \mathbf{Para}_\mathcal M (\mathcal D) (Y', X')$

Forcing costates

Suppose we have a symmetric monoidal category $\mathcal C$ and a lax symmetric monoidal presheaf $K : \mathcal C^\mathrm{op} \to \mathbf{Set}$ to the cartesian monoidal category of sets. The lax monoidal structure is given by a unit $e : K (I)$ and a laxator $\nabla : K (X) \times K (Y) \to K (X \otimes Y)$ We want to formally adjoin, or force, all of the elements $x : K (X)$ to $\mathcal C$ as costates $x : X \to I$, modulo all of the evident equations arising from the structure of $K$.

We take the category of elements $\int K$, whose objects are pairs $(X : \mathcal C, x : K (X))$ and whose morphisms are morphisms $X \to Y$ making the resulting diagram commute. $\int K$ is a symmetric monoidal category with the monoidal unit $(I, e)$ and the monoidal product $(X, x) \otimes (Y, y) = (X \otimes Y, x \nabla y)$, and is equipped with a strong symmetric monoidal fibration $\int K \to \mathcal C$.

We form the symmetric monoidal bicategory $\mathbf{Copara}_{\int K} (\mathcal C)$ for this functor. Its 0-cells are objects of $\mathcal C$, its 1-cells $X \to Y$ are triples $(M : \mathcal C, m : K (M), f : X \to M \otimes Y)$ and its 2-cells are morphisms $M \to M’$ of $\mathcal M$ making the resulting diagrams commute.

The hom-categories $\mathbf{Copara}_{\int K} (\mathcal C) (X, Y)$ are once again not connected. Here is the sledgehammer theorem:

Theorem (Hermida & Tennent, dualised and reformulated). $\mathcal C [K] = \pi_0^* \left( \mathbf{Copara}_{\int K} (\mathcal C) \right)$ is the smallest symmetric monoidal category that contains $\mathcal C$ and contains a morphism $x : X \to I$ for each element $x : K (X)$ modulo the evident equations.

I’ve talked about this construction before in its original dual form for forcing states from a copresheaf rather than costates from a presheaf, in this post, to add iteration to categories of optics. The pictures in that post are helpful for understanding how this construction works.

From adaptors to optics

We will now apply this theorem to construct categories of optics.

Given a symmetric monoidal category $\mathcal C$, we can form the symmetric monoidal category $\mathbf{Adt} (\mathcal C) = \mathcal C \times \mathcal C^\mathrm{op}$ with the pairwise monoidal product. We call the morphisms of this category adaptors, and think of them as optics whose backwards pass does not use its forwards input. There is an embedding $\mathbf{Adt} (\mathcal C) \to \mathbf{Optic} (\mathcal C)$ that is identity on objects and always chooses $M = I$.

The hom functor of $\mathcal C$ is a presheaf $\hom : \mathbf{Adt} (\mathcal C)^\mathrm{op} = \mathcal C^\mathrm{op} \times \mathcal C \to \mathbf{Set}$, and it is lax symmetric monoidal. (It is strong monoidal if and only if $\mathcal C$ is cartesian monodial.) It turns out that if we take $\mathrm{Adt} (\mathcal C)$ and adjoin a morphism $\binom{X’}{X} \to \binom{I}{I}$ for every underlying morphism $X \to X’$, we get exactly the category of optics:

Lemma: There is an isomorphism of symmetric monoidal bicategories $\mathbf{Copara}_{\int \hom} (\mathbf{Adt} (\mathcal C)) \cong \mathbf{2Optic} (\mathcal C)$, and therefore an isomorphism of symmetric monoidal categories $\mathbf{Adt} (\mathcal C) [\hom] \cong \mathbf{Optic} (\mathcal C)$.

This also works for general mixed optics. Suppose we have a span of strong symmetric monoidal functors $\mathcal C \overset{L}\leftarrow \mathcal M \overset{R}\rightarrow \mathcal D$. We form a lax symmetric monoidal functor $K : \mathcal C^\mathrm{op} \times \mathcal D \to \mathbf{Set}$ by $K \binom{X'}{X} = \int^{M : \mathcal M} \mathcal C (X, L (M)) \times \mathcal D (R (M), Y)$ Then:

Lemma: There is an isomorphism of symmetric monoidal bicategories $\mathbf{Copara}_{\int K} (\mathcal C \times \mathcal D^\mathrm{op}) \cong \mathbf{2Optic}_\mathcal M (\mathcal C, \mathcal D)$, and therefore an isomorphism of symmetric monoidal categories $(\mathcal C \times \mathcal D^\mathrm{op}) [K] \cong \mathbf{Optic}_\mathcal M (\mathcal C, \mathcal D)$.

If $\mathcal C$ is cartseian monoidal then of course we also have an isomorphism $\mathbf{Lens} (\mathcal C) \cong \mathbf{Adt} (\mathcal C) [\hom]$. It will be useful to write a direct proof of this fact instead of cutting the equations $\mathbf{Adt} (\mathcal C)[\hom] \cong \mathbf{Optic} (\mathcal C) \cong \mathbf{Lens} (\mathcal C)$, but I will put this off until a later post as a practice run for the main theorem of dependent optics.

Now we can reformulate the dependent optics question: given appropriate structure on $\mathcal C$ we need to construct a symmetric monoidal category $\mathbf{DAdt} (\mathcal C)$ of dependent adaptors and a lax symmetric monoidal presheaf $K : \mathbf{DAdt} (\mathcal C)^\mathrm{op} \to \mathbf{Set}$ with the property that $\mathbf{DAdt} (\mathcal C) [K] \cong \mathbf{DLens} (\mathcal C)$ is the category of dependent lenses in $\mathcal C$. (Of course we have to rule out trivial solutions like taking $K$ to be the terminal presheaf.) This is what we will do over this series of posts.

From Equilibrium Checking to Learning with the Open Game Engine

2025-06-26T00:00:00+00:00

Compositional game theory, like game theory in general, is not just for toy models. Game theory is the standard tool for modelling in a wide range of applications in microeconomics, and many of those benefit from compositionality. One of these applications is pricing, and especially its modern version, dynamic pricing. We have been working on a research project on how to do dynamic pricing in a strategic, competitor-aware way, using compositional game theory and its natural connections to optimal control and reinforcement learning, and building on the Open Game Engine.

In this post I’m going to discuss how we adapted the Open Game Engine, which is fundamentally designed as an equilibrium checker, to do multi-agent learning instead. The project we did also involved a lot of economics, which Nicolas wrote about in this post.

From equilibrium checking to dynamic programming

The idea that open games can be adapted to do dynamic programming has been known since 2019, when I replicated a dynamic social dilemma environmental economics model from Barfuss’ PhD thesis using the Open Game Engine (although that early implementation was very naive and almost melted my CPU before converging). A much more robust implementation, which used the Open Game Engine together with Python’s RLlib library, was built by Philipp and Nicolas (who I have been collaborating with on this project) for their paper on algorithmic collusion in pricing games.

At the time there was no theory explaining why this worked, that came slowly over the next few years. First there was Towards Foundations of Categorical Cybernetics (ie. the paper that really founded this field) revealing the full scope of open game-like constructions, and theoretically grounding all of the ways in which I was already creatively mis-using the Open Game Engine to do things it was not originally designed to do.

Then came the paper Value Iteration is Optic Composition, which fully explained the trick that I had used to do dynamic programming in the open game engine. In short, what that paper showed is that Bellman operators are representable as optics (both in the intuitive sense but also in the technical sense of representable functors), and that Bellman backup (ie. the application of a Bellman operator to a value function) can be justified as just precomposition with that optic. This operation, together with juggling the necessary indexing, is one of the main tasks that the computational backend of the Open Game Engine was built to do, since it is also what is needed for compositional Nash equilibrium checking.

The intuitive idea of how these things fit together can be explained with a diagram taken from Towards Foundations, the picture of a generic parametrised bidirectional process.

For a typical game-theoretic application the left and right boundaries have game states flowing forwards and payoff vectors flowing backwards, and they compose like bidirectional process, by backprop. The way we use this dimension for compositional modelling is unmodified from how we have always done it, so I will say nothing more about it here. For us, the interesting part is the top boundary.

The input on the top boundary is strategy profiles (which also goes by terms such as “parameters”, “policy” etc. depending on the application). The output on the top boundary, which I sometimes call “costrategies”, is the one that we need to focus on. On paper this output is a boolean value, and the different correctness lemmas for various classes of open games show that when the left and right boundaries are trivial, this value is true exactly when the input strategy profile belongs to some class of equilibria.

On one level, going from equilibrium checking to dynamic programming is as simple as changing this boolean to a real-valued payoff, and then attaching to the top boundary an optimiser that does policy improvement, ie. each loop iteration adjusting the policy in a way that increases that value. Since value improvement (ie. estimating the value of the current policy) is handled by the magic of optic composition in the horizontal direction, this is enough to give us dynamic programming.

We can also go beyond this, from dynamic programming to reinforcement learning. This is only a small step in theory but required a lot of details to be figured in practice, which appear in the paper Reinforcement Learning in Categorical Cybernetics.

Now let’s look at how all this looks in code.

A look inside the Open Game Engine

Let’s start with the most important definition in the Open Game Engine code base, the definition of open games themselves, and dissect it. The definition can be found here and looks like this:

data OpenGame o c a b x s y r = OpenGame
  { play :: a -> o x s y r,
    evaluate :: a -> c x s y r -> b
  }

(I have made one change from the linked code, which uses heterogenous lists instead of types a and b, an implementation detail that can be ignored in this post.)

There are several different minor variations of the definition of open games and the differences between them are understandably confusing. This version predates both this paper and this paper which significantly changed how we think about open games, but even so this version of the definition has stood the test of time extremely well.

The last 4 parameters - x, s, y, r - are the 4 legs of an open game. This post is not an introduction to how open games work, so if you want to understand this part you can go back to the original paper.

Next let’s talk about o and c, which from their use sites can be seen to take x, s, y, r as parameters. They range over typeclasses for optics and contexts respectively. The most basic example of such an o is o = Lens, which we use for game theory with deterministic (aka. pure) strategies, which could be imported from a library such as Control.Lens or simply defined concretely as Lens x s y r = x -> (y, r -> s). (The source code still uses the letters that I first wrote down in early 2015, years before I knew about lenses or the famous s t a b convention.)

The corresponding instance of contexts c that we use for basic game theory is c x s y r = (x, y -> r), which is also called Context in Control.Lens here. The structure that c has to satisfy is what I call a Tambara comodule, which I blogged about in this post.

For most serious game-theoretic modelling we use the definition of optics and contexts for Bayesian open games developed in this paper, which use existential types and can be found in the codebase here.

Now we come to the last two parameters, a and b, which are the important ones for this post. a is the type of strategy profiles of the open game. At this point we can read the second line of the definition, play :: a -> o x s y r, and say that an open game has an optic indexed by strategy profiles.

b is probably the point where what we do in the Open Game Engine differs most significantly from any version that has been written down on paper. On paper this is a boolean value that records whether the given strategy profile was a Nash equilibrium or not. What we use there in practice is a record type called DiagnosticInfo, which captures all available information about the decision, including the move that was played, that payoff received, the optimal move and payoff, the information that was visible and the information that was not visible. By looping the evaluate function over parameters and looking at the information contained in the returned record, a lot of microeconomic analysis is possible.

When modifying this output to be simply values, the basic decision operator that is the main generating element of real models - which is very complicated in a game-theoretic setting - becomes very simple and does nothing but shuffle values around, in particular forwarding the relevant payoff out of the top boundary to the optimiser. The code for that can be seen here. The reason for doing it this way is that the optimiser then lives outside of the open games framework, which means it can use unboxed arrays and other tricks in order to squeeze enough speed out of Haskell.

Learning in practice

For the past 6 months we have been working with a major airline on a pilot research project ¹ on strategic, competitor-aware dynamic pricing using this technology.

The game part of our model is a market, and the reason that compositional game theory is such a good fit is that the market for airline tickets has significant natural compositional structure due for example to connecting flights. This is precisely the point where we expect that our methodology will outperform traditional modelling. There is at least an entire blog post of things to say about this point alone, but that post will come later.

There are two very different ways we can proceed from there. The first way we can use it is to fix the strategy of one competitor (in our model a strategy is a policy for how a price varies dynamically over a time period) and then learn an optimal response for the other competitor. This puts us in a traditional dynamic programming setting, which is important because we get very strong convergence guarantees.

The other way to use it is to simultaneously learn both policies by multiagent RL, starting from some initial policies. This is another point where there is an entire blog post worth of things to say, but in short, there are few convergence guarantees but our experience is that the method usually works very well in practice. Fortunately, the complex issues we face here are not made any more difficult by using compositional game theory, so there is a lot of existing knowledge on multiagent learning that we can draw on.

All of this is to say that what I’ve been writing in this post is only one part of a much bigger research topic that includes significant amounts of computer science and economics (and I haven’t even mentioned the data science part). This very much fits into my overall vision for applied category theory as a field, where category theory makes its one contribution (ie. bringing compositionality to a new domain) and then exits the picture.

Pun unintentional but unavoidable ↩

The Untapped Potential of Game Theory in Revenue Management

2025-05-16T00:00:00+00:00

Revenue management has evolved significantly since its origins in 1980s airline yield management. Travel, logistics, and hospitality industries employ it to maximize occupancies; e-commerce platforms use it to schedule promotion campaigns. It is the go-to tool across industries to match resource availability with consumer demand.

But most current approaches still rely on relatively simple demand forecasting and optimization. The behaviour of other players in the market (for example competitors) is rarely explicitly accounted for, even though their behaviour implicitly shows up in the data used to estimate the models. Game theory — which has emerged as the dominant tool for modeling strategic dynamics — offers powerful potential to change this.

Revenue Management Today

Today’s revenue management typically translates historical demand patterns into future forecasts or price elasticity models. In other words, the dominant design is competitor-agnostic. This despite the fact that real-time data availability increasingly allows for more rapid price adjustments, which require customer acceptance and this acceptance is inevitably linked to competitor pricing.

“Competitor-aware pricing” is generally in its infancy and considered an area of cutting-edge yield management research. Digital e-commerce platforms, for example, use highly sophisticated algorithms to dynamically adjust prices — usually by timing promotional campaigns — but systems generally respond to immediate market conditions and demand projections rather than predicting and incorporating the expected responses of suppliers or competitors. Hospitality and retail industries are following the lead of the airline industry, but tend to use traditional rules-based systems and basic environment-agnostic forecasting.

In short: strategic analysis typically happens offline in a manual way, far away from the possibility frontier. This leaves a lot of money on the table.

The Game Theory Opportunity

Game theory has completely transformed business models in Silicon Valley. Hal Varian, Google’s Chief Economist, famously played a key role in designing Google’s ad auctions — still to date the dominant source of revenue for the company. Game theory similarly has the power to transform revenue management by offering the possibility to:

Forecast how competitors might adjust their strategies. What would a “smart” competitor do? And what is the likely reaction of suppliers or competitors to different promotional strategies?
Estimate the long-term effects of strategic decisions and evaluate trade-offs between short-term revenue and long-term market position. Standard revenue management estimates are often notoriously short-sighted.
Optimize the timing of price changes to maximize long-term value and strategically time promotions.
Analyze and forecast strategic customer behavior, anticipation effects, and dynamic customer learning and adaptation strategies.

It should be noted that the possibilities game theory offers are well-known to revenue management professionals. Every business and economics student has learned about simple games of market entry and studied the relevant strategic effects. But applying game theory in practice beyond simple ‘2 players, 2 strategies’ textbook toy examples requires much deeper and more specialized skills, combined with the right framework and technology.

Modern enterprise-scale game theory engines are highly complex, connecting many interlocking parts, mathematically sophisticated, and need to be designed with care.

Beyond “Traditional” Game Theory

We approached this challenge with a new mathematical lens — particularly through a structure called compositional game theory — to model complex strategic interactions with unprecedented fidelity. This mathematical foundation allows us to:

Model interactions between multiple economic agents (competitors, customers, suppliers) simultaneously with true compositionality.
Integrate statistical estimates from real-world data on customer demand or company strategy — and feed results back, building a “Digital Twin”.
Incorporate machine learning algorithms to find optimal strategies, predict competitor responses, and optimize pricing.
Capture complex feedback loops of multi-market interactions and supply chain cascades in a simple, modular way.

Compositional game theory provides a framework where strategic interactions can be assembled, broken down, analyzed, and reconstructed while preserving their essential characteristics. This makes it possible to model real-world market dynamics far more accurately than with standard economic models.

Case Example: Competitor-Aware Dynamic Pricing

To demonstrate the power of compositional game theory in revenue management, we recently completed a six-month research pilot with a major airline focused on competitor-aware dynamic pricing. While traditional approaches generally do not react to competitor price changes, our approach explicitly models competitor behavior, allowing for more strategic pricing decisions.

We can model the complex structure of airline bookings, including connecting flights or multi-segment journeys, learn optimal pricing policies in response to competitor strategies, and even simultaneously optimize multiple agents’ pricing strategies within the same market.

What makes this approach particularly powerful is its ability to handle the natural compositional structure of airline markets. Airline revenue management perfectly fits the modular structure of our framework, since each flight can be thought of as its own optimization problem. But because competing airlines are active across multiple of these, the overarching strategic interaction operates across various different flights. So the pricing decisions for individual flights interact in complex ways that are difficult to capture with traditional methods. For those interested, see the companion blog post explaining how we adapted the Open Game Engine for this specific application.

Looking Ahead

This pilot project represents just one example of how game theory can transform revenue management practices. By explicitly modeling strategic interactions between market participants, companies can move beyond reactive pricing and toward truly strategic revenue optimization — avoiding markets from spiraling down, for example, and strategically considering the longer-term effects of today’s decisions.

The limitations of current revenue management approaches are becoming increasingly apparent as markets grow more complex and interconnected and data becomes abundant. Our approach allows us to unlock this value through sophisticated game-theoretic models that integrate seamlessly with existing systems.

For revenue management and dynamic pricing experts interested in exploring these advanced approaches, we invite you to connect with us to discuss the opportunities modern algorithmic game theory offers to revenue management practice.

An Invitation to Neural Picture Alchemy

2025-05-07T00:00:00+00:00

TL;DR for busy people:

Premiss:

Neural networks are variable maps

Objectives are equations governing such variables

Perspective: Deep learning is just representation theory, except:

Instead of groups, we have function-composition algebras

Instead of linear maps, the target is continuous maps

Payoff:

Characterise behaviour as equations, let backprop find you an implementation

Bonus: it’s pictures.

Introduction

String diagrams? in my deep learning? It’s more likely than you think.

If you’re reading this, you’re probably already comfortable with string diagrams and category theory, so I won’t belabour the basics. Usually when people draw string diagrams for deep learning, they’re depicting architectures, or something that’s compositional with respect to architectures such as backpropagation. Unfortunately this tensor-plumbery is a bit rizzless, since autograd and einops means IRL chad DL-engineers already don’t break a sweat over these aspects.

So instead of the syntax of DL, let’s turn towards the semantics: the relationship between the objective functions and the behaviours of the trained models. Meaty and messy stuff. Yes, these are “dark arts”, and everyone knows or at least suspects that DL is a sort of scruffy alchemy. But I think it might be a mistake marching in with our epistemic buttholes puckered to wrestle DL into a hard science: what if DL wants to be more like Architecture and Gastronomy than Physics and Chemistry? I would prefer to embrace the prescientific and preparadigmatic vibe: i.e. let’s just make shit up that sounds good like the Greeks did, and see where that gets us.

Ok, so I’m going to use diagrams to do dumb things fast with our good friend deep learning. We’re moving loosey-goosey because we’ve got nothing to lose, so be cool. If you want slower for the first part, then see here.

String Diagrams for Objective Functions

deeplearns? Is it in the room with us right now?

You should be ashamed not to know in the year of our lord 2025 that neural networks are parameterised functions, and backprop is how we change the parameters by pouring a stream of data through the network. We’ve got enough compute around that we can just pretend (be cool) that the Universal Approximation Theorem is a synthetic axiom: a neural network can — with appropriate parameterisation — be any function (of matching input-output type). In easyspeak, let’s pretend learners are variable functions.

Whenever we have variables around we probably want some equations too, to make the variables do or become interesting things. So let’s just pretend (be cool) that objective functions are these equations. It’s not an insane leap: here’s an objective function for an autoencoder:

\[\text{argmin}_{\theta_e, \theta_d} \left( \mathbb{E}_{x \sim \mathcal{X}} \left[ \mathbf{D} \left( \text{dec}_{\theta_d}(\text{enc}_{\theta_e}(x)), x \right) \right] \right)\]

Now let’s squint away the details.

\[\text{blabla}_{bla, bla} ( \mathbb{B}_{l \sim \mathcal{A}} [ \mathbf{b} ( \underbrace{\text{dec}_{\theta_d}(\text{enc}_{\theta_e}(x))}_{\text{expression 1}}, \underbrace{x}_{\text{expression 2}} ) ] )\]

It’s a bunch of stuff that cares about a pair of expressions basically, the rest is too complicated to be essential. So, two expressions with infix notation.

\[\text{dec}_{\theta_d}(\text{enc}_{\theta_e}(x)) \rightleftharpoons x\]

Let’s draw out those expressions as string diagrams. We will read them from left-to-right. When doing this, we actually have to add some information that’s missing from the blabla, namely the typing of the various learners. Let’s say that encoders go from some representation space $X$ into a latent space $L$, and the decoder does the reverse. $x : X$ is just the identity on $X$.

And let’s decorate the diagram a little more to recover some of the detail we squinted away. We’ll need to put in a source of data $\mathcal{X}$ that lives over the space $X$, so let’s just draw that as a state (be cool). Since we dgaf about the particular parameter spaces as we’re dealing with universal approximators, we’ll just say that enc and dec are variable functions and note down that they’re on the hook for doing something.

All the rest of the blabla is saying is: “give me a statistical divergence $\mathbf{D}$ (i.e. a spicy distance), and a bunch of data, and we want to twiddle the parameters so that the distance between left-expression applied to the data and right-expression applied to the data is small, on average.” Of course, this is needlessly complicated; the learners are just going to do whatever they can with whatever data they have to backpropagate over to make this bla-bla an equality. They want to kiss, and data+compute will help them do it. Easy-peasy; autoencoders are just an encoder and decoder tryharding to become split-idempotents, where the decoder is the retract and the encoder is the section.

Specialisations

If NNs can be anything, then we can do a directed rewrite (be cool) to make them a particular thing. The visual metaphor is we treat the variable function as a hole to plug something else into. What does that mean formally? I personally think it’s something like an operad on the homsets of an SMC (someone who actually knows category theory should check me on this).

“Why is this diagram-perversion useful?” This gives us a way to compare different architectures; sometimes we can squint at a thing and see how it’s really just another thing but sparkling, or we may proactively play god and mutate things for fun and profit. Take variational autoencoders for instance. The usual presentation involves variational lower bounds, KL-divergences, and a lot of greek. But diagrammatically? It’s just a spicy autoencoder. Let’s reinvent VAEs together.

What if instead of (de/en)coding an input as deterministic points, we used probability distributions for nondeterministic (de/en)codings? We can do this in three steps: first we can make the latent space $L$ a space of (mean,variance) tuples for Gaussians:

Then we can specialise the decoder to first sample from the distribution, and then decode. The whole setup still ought to behave like an autoencoder, as all we’re doing is declaring additional constraints.

This is pretty good, now let’s re-examine our wants. We want the behaviour to be actually nondeterministic. It pays now to be paranoid: what if the encoder and decoder collude against our wishes? Maybe the encoder will always choose 0-variance encodings, so that the overall behaviour degenerates into being deterministic. Ok, so we need to strongarm the encoder into nontrivial distributions. But wait, what if the encoder still wants to cheat us by choosing means that are so far apart that the output distributions for different inputs may as well be separate points? Alright, let’s pressure the encoder to encode near the origin-centred unit-variance Gaussian then, that will sort them. We’ll add in another equational constraint.

As I understand it, the term of art for such an extra rule intended to rule out unwanted behaviours is “regularisation”. Ok, kind of neat that we can in principle figure out how to regularise things with a little imagination. You know what’s extra neat? Turns out that this presentation of VAEs is equivalent to the usual very gnarly one. The consequences will leave thinkbros in shambles: we can just declare what sort of behaviours we want satisfied as equations instead of having to do math to figure it out from the ground up; we’re really embracing the idea that learners are just behaviouralist black-boxes that will do whatever they need to in order to objectivemaxx.

If this is true enough, then what would be enabling is a collection of behavioural building blocks that are already captured equationally. What kinds of behaviours do we have equations for? Good question, nephew. Let’s look at some patterns.

Patterns: Well-Understood Tasks

In this framework, we can identify certain “patterns” - common diagrammatic constraints that correspond to well-understood ML tasks. Here for example is all of supervised learning, both classification and regression:

We’re solving an equation just like $1 + x = 2$, except with functions. We’re looking for a function $foo: \text{Data} \rightarrow \text{Label}$ such that for a dataset of labelled pairs $(d,l) \sim \mathcal{D}$, $foo(d) = l$. If backprop finds us such function or one that’s basically good enough, the term of art is that $foo$ is a classifier if the target labels are discrete, and $foo$ is a regressor is the target labels are continuous. From our smoothbrained perspective though, such a distinction is unnecessary.

Autoencoders we’ve already done, those are good for creating representations. If you have everywhere positive measures of “energy” for representations that you want to minimise, then you get a framework to express a bunch of unsupervised learning techniques like $k$-means and PCA. Here it is as a single equation, with energy functions $\mathbf{E}_d$ and $\mathbf{E}_e$ and some positive hyperparameter $\gamma$.

Here’s a pattern I think is nice but also mysterious, and I don’t fully grok it yet. Let’s say we’re interested in generating instances of a “well-formed” subtype of a broader type, but we can’t spell out what “well-formed” means using math. For example, we might be interested in a type of “images of flowers” out of a broader type of “images”. It’s a fool’s errand to try writing out what characterises images of flowers equationally, so what else could we do? Looking at our toolkit so far, a simple option is to have a powerful classifier that recognises flowers that postselects flowers out of random images, but this is impractical, like searching the Library of Babel. We want a way for backprop to do the hard work for us of identifying what that good subspace of images is, and we can achieve this by setting up a simple dynamical system of learners. We want a generator that effectively samples the subspace of flowers, and a discriminator that distinguishes flowers from non-flowers. We want the generator produce images of flowers good enough to fool the discriminator, and we want the discriminator to pass real flowers and punish non-flowers and fake-flowers.

This of course is a generative adversarial network. The hope is that there is a virtuous arms-race between the generator and discriminator such that the generator produces flowers indistinguishable from the true flower-space that discriminator hones in on. The adversariality comes from the fact that by design, we cannot simultaneously have that both generator and discriminator are perfect. So we can never find a perfect representation of the GAN equations as concrete functions, and yet this unsatisfiable setup does seem to work at shrinkwrapping around otherwise ineffable subtypes. Very strange.

New Patterns?

Lenses are a nice candidate from BX of a behaviour characterised in an equational, point-free way. If you’ve spent time in the bidirectional transformations crowd, you know these are just get/put pairs that obey some laws like GetPut and PutGet. If you haven’t, no worries — just know they’re a formal way of looking at and changing one attribute of a complex object while preserving the rest. So why not turn that into task-equations like we’ve already been using? If we want to manipulate attributes of data without screwing up everything else, that’s exactly what we need. Seriously, why aren’t we using these more in DL? Here’s a working recipe. We have some State data of type $S$, which carries some value data of type $V$, and we want a read-method $get: S \rightarrow V$ to look at the value of a state, and a write-method $put: S \times V \rightarrow S$ that takes a state and a new value, and returns a new state that carries the new value. First let’s ask for $get$ to behave like a classifier that learns how to extract value-features from states, trained on a labelled-dataset:

Now we want to spell out what makes the $put$ a write-method and not just some random function. It had better be the case that if we write a new value and then take a look, we can extract that new value. Let’s say that $\bar{\mathcal{X}}$ is just the state part of our data, and $\mathcal{A}$ is a random sampling of attribute values, and we’ll use these jointly independently distributed.

It had also better be the case that if we find that a state has a certain value, writing that same value back into the state ought to do nothing.

There are stronger conditions than this next one we could ask for, but we’ll just demand that we can undo changes: if we write a value in, and then change our mind and write back the old value, then we should end up with the state we started with originally.

Now, let’s say that the State is something like someone’s face, and the Value is whether that person is smiling. We want to add or remove smiles without changing other aspects like identity, hair color, or background.

TL;DR: it works. Now, a fair objection: pics or it didn’t happen.

The model learns to do exactly what we want. There’s a secret extra cool thing about this example, which is that the only data the model has ever seen is faces with a binary label indicating “smile” or “no-smile”; so how is it interpolating between smiling and not-smiling? How would you do it?

Dumber is better: if we specialise the putter to be a learned vector offset we add on to the latent of an autoencoder, then in order for the put and get to satisfy our requirements, embeddings must separate and there has to be a steering vector we can interpolate along for the different classes. No adversarial training needed, no complex masking schemes — just a task that says “change this one thing and nothing else, and make the edit a vector addition.”

Hey, here’s an idea: if you’re doing mechanistic interpretability and trying to understand things by digging in their innards, how about you stop that and just characterise what behaviours you want in a point-free way (surprise surprise that’s what category theorists are good at) and smack that DL-go-brrr. Let gradient descent figure out the implementation details. Why bother with the how when you can just declaratively specify the what?

Case Study: Computing Nash Equilibria with Gradient Descent

Check this out: I’m an idiot. I’ve been brain-damaged by string diagrams so I can’t math anymore, and also I suck at coding. My formal game theory knowledge consists of precisely two facts: Mixed Strategy Nash Equilibria exist, and they’re a pain to compute.

Have you heard the bad news? Sam Altman called, he said it’s the age of AI! You don’t have to understand phenomena in order to produce them! Verum et factum convertuntain’t, cybernetics crying in the club.

But my guy, you say, where is this going? I’m saying that I (dumbass) used string diagram objective function alchemy to bootstrap my way out of stupid and do a thing with vibe-coding, because I had the recipe for the exact behaviour I wanted drawn up with crayon ready to go.

Is this yap cap? The answer may surprise you.

First, I sketched out a diagram for finding Mixed Strategy Nash Equilibria (MSNE) using gradient descent. If you know game theory, you’re probably thinking “that’s not novel.” True! But I didn’t need to know anything beyond “MSNE means neither player can improve by changing strategy unilaterally.”

In a 2-player game with size-$K$ finite strategy sets $X$ and $Y$, mixed strategies (i.e. probability distributions over strategies) can be viewed as points in the $K$-simplices $\Delta^{|X|}$ and $\Delta^{|Y|}$. Source: trust me bro, that’s how triangles work. So a mixed strategy for 2-players is a choice of point inside some hyperpyramid for each player. Here’s a drawing.

A 2-player MSNE then is a pair of points where neither player can improve their expected payoff by unilaterally changing their strategy. In other words, each player’s strategy is optimal given what the other player is doing.

🎤 Live diagrampilled reaction:
🔺 Mixed strategies are places in a triangle
🚶🏽‍♂️ Players are moving around in their own triangles
🚷 MSNE is when players stop moving
🤑 Stop moving = locally-payoffmaxxing
🤬 payoffmaxxing = losscels seethe
⬇️ losscels = descentcels
🤔 descentcels = downwardcels
☝️ downwardcels seethe because upwardchads
🧗🏻 upwardchads ascentmaxx
🧠 Thus ascentmaxxing = locally-payoffmaxxing
⬆️ Thus playerchads must ascend
😎 Cool fact: $\infty$ is the tallest number

The trick is splitting the paths during backprop. Each player treats the other’s strategy as fixed when computing how to update their own strategy. This corresponds exactly to the “unilateral” part of the Nash equilibrium definition.

I showed these drawings and some words to Gemini 2.5 and asked it “make this like a computer person speak”. And then I took that and demanded “implement this and use the most DL-go-brrr”. And then I cleaned up the syntax errors in cursor by asking “😭” a few times. Tbh, the real value-add here was getting a GUI working so that it was relatively easy to change starting configurations and visualise how the strategies evolved:

The resulting code with a more detailed math spec is all here. It works, and while this approach doesn’t scale well to more than like 7 players and 7 actions, we can always buy a new datacenter and drink another river. What’s the point of all this? It’s not that I’ve revolutionised game theory computation, it’s that I (understander of sweet fuckall) was able to direct the automatic implementation of working DL solutions just by drawing pictures and grovelling at an LLM, and that means so can you.

Conclusion

String diagrams aren’t just elegant brainrot, they’re a practical tool for cheesing deep learning. By expressing objectives as equational constraints between diagrams, we gain:

A unified language for expressing diverse DL paradigms
A way to build up complex objectives declaratively
The ability to bullshit our way through domains we barely understand

The future of DL might not be about designing complex architectures, but about specifying ever more precise behaviours. You deserve a break, thinkbro: join the dark side and let the universal approximation theorem do the heavy lifting. Next time you’re designing a learning system, consider sketching its intended behaviour as a string diagram first. Or when you’re approaching a domain you know nothing about, try drawing it out diagrammatically and see if DL-powered representation theory can save you from having to learn things.

Pipelines Part 2: Categorical Pipelines

2025-03-13T00:00:00+00:00

In the previous post we saw how to implement pipelines and their benefits:

Separate the specification of the pipeline from its implementation
Provide multiple runtimes for the same pipeline

We are now going to generalise the pipeline architecture to categories so that Run and RunM will be one and the same. In addition to handling effects, this makes the pipeline able to compute any sequence of morphisms in any category, and we are going to see how to use the same infrastructure to build a pipeline in the category of dependent lenses. Finally, the pipeline architecture can be further generalised to graded categories providing the same benefit but for graded morphisms, and we are going to use this functionality to automatically combine errors in effectful programs.

Abstracting to categories

You might have noticed that Run and RunM share many similarities:

Run [x, y] f = f
Run (x :: y :: z :: xs) (f, cont) = Run (y :: z :: xs) cont . f

RunM [x, y] f = f
RunM (s :: t :: x :: xs) (f, cont) = RunM (y :: z :: xs) cont <=< f

The base case is the same and the inductive case differs only in how we compose functions, in the simple case we use function composition (.) and in the monadic case we use kleisli composition <=<. Can we abstract over what composition operator we use? We sure can. In fact, doing so means that we’re abstracting over categories. That is, we are going to abstract over a generic composition operator (.) : arr b c -> arr a b -> arr a c with a generic “arrow type”arr : o -> o -> Type . This is the basic definition of a category as an interface in idris:

-- "| arr" means that the 'arr' argument is uniquely defined
-- for each instance of this interface, but `obj` isn't.
interface Category (0 arr : obj -> obj -> Type) | arr where
  id  : arr a a
  (.) : arr b c -> arr a b -> arr a c

To use this in our generalisation, we need to change our definition of pipeline to not be about types but to be about objects in a category. We represent this using parameter blocks:

parameters (o : Type) (arr : o -> o -> Type)

We simplify our definition of ImplCat using higher order functions instead of pattern matching.

ImplCat : Vect (2 + n) o -> Type
ImplCat = All (uncurry arr) . pairUp

We implement RunCat in the same way as before except (.) now refers to the composition of morphism of the Category interface.

RunCat : Category arr =>
    (p : Vect (2 + n) o) ->
    ImplCat p -> arr (head p) (last p)
RunCat [x, y] [f] = f
RunCat (x :: y :: z :: xs) (f :: cont) = RunCat (y :: z :: xs) cont . f

With this setup we can now use the same code for both pure and effecful pipelines! The only difference between the two is what category is used to run the pipeline. In the pure case, we use plain functions where objects are Type and arrows are Fn. In the effectful case we use Kleisli Morphisms where the objects are type and the morphisms are given by Kleisli m where m : Type -> Type is a monad, it defines the class of functions a -> m b.

Remember that we had two compiler definitions, a pure one without side effects:

lex : List Char -> List Token
parse : List Token -> Tree
typecheck : Tree -> Sema
codegen : Sema -> Bytecode

And a monadic one where errors are handled using the Maybe monad.

lexM : List Char -> Maybe (List Token)
parseM : List Token -> Maybe Tree
typecheckM : Tree -> Maybe Sema
codegenM : Sema -> Maybe Bytecode

Using the same RunCat we can execute the pipeline CompilerPipeline in those two different categories. First one where objects are Type and morphisms are functions, and the second one where objects are also Type and the morphisms are kleisli morphisms, represented by the KleisliMorphism type.

CompilerPipeline : Vect ? Type
CompilerPipeline = [String, List Char, List Token, Tree, Sema, Bytecode]

runCompiler' : String -> Bytecode
runCompiler' =
  RunCat Type Fn CompilerPipeline
    [unpack, lex, parse, typecheck, codegen]

Here you can see that the first two arguments are the objects and morphisms of the category we are working on and the pipeline and implementation are the same as before.

-- equivalent to String -> Maybe Bytecode
runCompilerM' : Kleislimorphism Maybe String Bytecode
runCompilerM' =
  RunCat Type (Kleislimorphism Maybe) CompilerPipeline
         [Kleisli (pure . unpack),
          Kleisli lexM,
          Kleisli parseM,
          Kleisli typecheckM,
          Kleisli codegenM]

Likewise, the first two arguments are the object and morphisms: in this case, the morphisms are Kleislimorphism Maybe which is equivalent to a newtype around x -> Maybe y. Because Kleislimorphism is a newtype, we need to wrap each value around its constructor Kleisli.

Pipelines of lenses

Because lenses form a category, we can define a pipeline as a list of input-output pairs where the morphisms between each layer are lenses. In our case we can do even better and use dependent lenses/container morphisms as our category, and build entire applications with them.

For example, the following program describes a command-line program written using containers and stages between them to communicate with a database. The first stage converts from an effectful CLI interface that receives and sends strings into an abstract type of messages that the app can deal with. The internal API of the app exposing those messages is given by the AppOp container. After distributing the monads around, each of those messages is converted into a database query. Database queries are then executed by the database and the result is returned to the client.

This entire pipeline is represented as a list of containers, where FM is the effectful comonad on container, MaybeAll the Maybe monad on containers and CUnit the monoidal unit.

0 AppPipeline : Vect 6 Container
AppPipeline = [ FM CLI
              , FM (MaybeAll AppOp)
              , MaybeAll (FM AppOp)
              , MaybeAll (FM DB_API)
              , MaybeAll CUnit
              , CUnit
              ]

The implementation is given by a sequence of morphisms with the right domain and codomain.

appImpl : DB => ImplCat Container (=%>) AppPipeline
appImpl = [ parser
          , distribMaybeAF {a = AppOp}
          , map_MaybeAll (mapLift' OpToDB)
          , map_MaybeAll DBCostate
          , maybeAUnit
          ]

This pipeline can also be represented graphically:

Unfortunately we cannot use kleisli morphisms with this pipeline because we are not using the same monad across the entire pipeline.

The problem of dealing with different monads appears much earlier, if we write a pipeline where each morphism is a different coproduct monad. We need to translate each morphism into a common coproduct monad. For example in the following program, we need to convert each error from each stage into a common error type such that every morphism lives in the same monad.

lexE : List Char -> Either LexerError (List Token)
parseE : List Token -> Either ParseError Tree
typecheckE : Tree -> Either TypeError Sema

data CompilerErr
  = LexE LexerError
  | ParseE ParseError
  | TypeE TypeError

runCompilerErr : Kleislimorphism (Either CompilerErr) String Bytecode
runCompilerErr =
  RunCat Type (Kleislimorphism (Either CompilerErr)) CompilerPipeline
         [Kleisli (pure . unpack),
          Kleisli (mapFst LexE . lexE),
          Kleisli (mapFst ParseE . parseE),
          Kleisli (mapFst TypeE . typecheckE),
          Kleisli (pure . codegen)]

While writing mapFst with the appropriate error-conversion function is not the end of the world, it feels like something we should be able to deal with automatically. This is the topic of the next section.

Abstracting to graded categories

To ease the handling of errors, we are going to further generalise our pipelines to graded categories. A graded category is like a category but carries an additional monoid that will annotate each morphism. We define graded categories as :

A set of objects $C$.
A monoid $G$.
For all $x, y \in C$ and $g \in G$ we have a set of graded morphisms x -[g]> y.
An identity morphism x -[u]> x where u is the monoidal unit in $G$.
Graded morphism composition x -[g]> y, y -[h]> z to x -[g <+> h]> z where <+> is the multiplication of the monoid.
And the usual proofs of associativity and identity.

We can implement graded categories as an interface, and ignore the proofs for simplicity.

export infixr 1 #>

public export
interface GradedCat (mon : Monoid gr) (0 arr : gr -> obj -> obj -> Type) | arr where
  constructor MkGradedCat
  identity : forall x. arr (Prelude.Interfaces.neutral @{mon}) x x
  (#>) : forall x, y, z.  {g1, g2 : _} -> arr g1 x y -> arr g2 y z -> arr (g1 <+> g2) x z

With this approach, we can write a graded pipeline where we first give a list of layers for it, then additionally, we provide a list of morphisms and a list of grades for each morphism.

-- We parameterise everything by
-- the grade, the objects and the graded morphisms
parameters (0 gr : Type)  (0 obj : Type) (0 arr : gr -> obj -> obj -> Type)

  -- An implementation needs both the layers of the pipeline, and the grades of each morphism.
  -- For each triple of `source`, `target` and `grade` we build the graded morphism `source -[grade]> target`
  -- as the type of the implementation of the corresponding stage
  ImplGr : forall n. Vect (2 + n) obj -> Vect (S n) gr -> Type
  ImplGr layers grades = All (\arg => arr (fst arg) (fst (snd arg)) (snd (snd arg))) (zip grades (pairUp layers))

This way, given a list of layers [a, b, c, d] and a list of grades [g1, g2, g3] we obtain a list of graded morphisms [a -[g1]> b, b -[g2]> c, c -[g3]> d.

To run this list of morphisms we compose each of them using graded morphism composition: because we have a list of grades [g1, g2, g3] the resulting morphism will have the grade g1 <+> g2 <+> g3. The final morphism will have type a -[g1 <+> g2 <+> g3]> d.

  RunGrCat : forall n. (mon : Monoid gr) => (cat : GradedCat mon arr) ->
                 (p : Vect (2 + n) obj) -> (grades : Vect (S n) gr) ->
                 ImplGr p grades -> arr (foldGrades grades) (Vect.head p) (Vect.last p)
  RunGrCat cat [x, y] [gr] [f] = f
  RunGrCat cat (x :: y :: z :: xs) (gr1 :: (g :: gs)) (f :: c :: cs) =
    f #> RunGrCat cat (y :: z :: xs) (g :: gs) (c :: cs)

Implementing errors with Either

With this infrastructure, we can build the graded pipeline of compiler stages with errors. Whenever a stage returns no error we use Either Void to indicate that the Left choice is impossible. First we need to define an appropriate monoid, in this case, we want a monoid on Type where each value represent a possible error, we combine them with Either to say that only one of the two possible given errors is going to occur

[CoprodSemi] Semigroup Type where
  (<+>) = Either

[CoprodMon] Monoid Type using CoprodSemi where
  neutral = Void

GradedEither : Type -> Type -> Type -> Type
GradedEither g a b = a -> Either g b

With this monoid, we define the graded category where morphisms are graded kleisli arrows a -> Either g b

[EitherGrCat] GradedCat CoprodMon GradedEither using CoprodSemi where
  identity = pure
  (#>) f g x =
    case f x of
         Left err => Left (Left err)
         Right v =>
           case g v of
                 Left err => Left (Right err)
                 Right v => pure v

Finally, we implement a compiler with errors by using our monoid on types and our graded category. The implementation is as expected and the types compose automatically.

lexErr : List Char -> Either LexerError (List Token)
parseErr : List Token -> Either ParseError Tree
typecheckErr : Tree -> Either TypeError Sema

CompilerEither : String -> Either ? Bytecode
CompilerEither = RunGrCat @{CoprodMon} Type Type (\g, x, y => x -> Either g y)
    EitherGrCat CompilerPipeline [Void, LexerError, ParseError, TypeError, Void]
    [pure . unpack, lexErr, parseErr, typecheckErr, pure . codegen]

The full type of this function is quite large, and contains some redundancy with Either Void:

CompilerEither : String -> Either (Either Void (Either LexerError (Either ParseError (Either TypeError Void)))) Bytecode

While we managed to handle errors automatically, it would be nice if our implementation did not suffer from those artefacts. Thankfully, this is easily done by changing the monoid we are working with.

Implementing errors with `Any`

To address the problem of redundant neutral values from the previous example, we use the List monoid rather than Type and Either. With [] as the neutral element and concatenation (++) as the monoidal operations, lists naturally ignore their neutral elements when combined. A program that returns no error therefore returns an empty list of errors, and programs with errors return a singleton list containing the error they emit.

Now that we know that we want to grade our arrows by lists, we have yet to define what is an appropriate morphism graded by such a list. For this we use the OneOf : List Type -> Type type, a sort of iterated coproduct where each possible choice is given by the types in the list. We write it as a special case of Any: a predicate on lists asserting that exactly one element in a list of a fullfils the predicate p : a -> Type.

data Any : (p : a -> Type) -> List a -> Type where
  Here : {0 p : a -> Type} -> p x -> Any p (x :: xs)
  There : Any p xs -> Any p (x :: xs)

OneOf : List Type -> Type
OneOf = Any id

Using OneOf we define a graded category where the monoid is given by List and the morphisms by functions a -> Either (OneOf g) b. We write them as a -[g]> b.

GrListMor : List Type -> Type -> Type -> Type
GrListMor g x y = x -> Either (OneOf g) y

[OneOfCat] GradedCat %search GrListMor where
  -- Elided for brevity

Using some idris notation trickery and this graded category, we can define our list of compiler stages with grades where some stages do not emit any errors and some have their own bespoke errors.

namespace GradedCompilerAny
  split : String -[]> List Char
  lex : List Char -[LexerError]> List Token
  parse : List Token -[ParseError]> Tree
  typecheck : Tree -[TypeError]> Sema
  codegen : Sema -[]> Bytecode

  CompilerErrors : Vect 5 (List Type)
  CompilerErrors = [[], [LexerError], [ParseError], [TypeError], []]

  runGradedCompiler : String -[LexerError, ParseError, TypeError]> Bytecode
  runGradedCompiler = RunGrCat (List Type) Type (\g, a, b => a -> Either (OneOf g) b)
      OneOfCat CompilerPipeline CompilerErrors
      [split, lex, parse, typecheck, codegen]

The result is an elegant function String -[LexerError, ParseError, TypeError]> Bytecode containing only the errors we care about and the implementation is essentially the same as the version using Either.

Conclusion

The same pipeline architecture can run all sorts of programs by generalising to categories. The pattern works not only for pure and monadic programs, but also for lenses and dependent lenses. Generalising further to graded categories yields a way to dynamically combine errors without additional ceremony. Although we haven’t had the time to talk about it here, the ability to use graded morphisms also comes in handy for composing graded dependent lenses which we can obtain via the para-construction on dependent lenses for example.

What Is the CyberCat Institute?

2025-03-03T00:00:00+00:00

CyberCat is a network of researchers established in 2022 with the common goal to develop leading-edge knowledge in categorical cybernetics and applied category theory, and to show its economic usefulness.

The Institute for Categorical Cybernetics was incorporated as a non-profit in 2024 to provide an institutional backbone for the network’s activities.

Its purpose is to support CyberCat’s whole innovation pipeline from basic research to spinning off high-tech ventures. At the basic research level, it coordinates research activities among its members including funding and joint implementation of research projects, as well as hosting informal and formal events. At the other end of the funnel, the Institute acts as an incubator for ventures with the express goal to discover and commercialize economically useful technologies.

The first incubated technology of our research network, the Open Game Engine, was successfully turned into a for-profit company in 2023, 20squares. This startup is fully operational, independent, and profitable.

This initial success convinced us to found the Institute, seeded with funding from the startup.

In 2024, the CyberCat network saw two more technologies crossing into startup territory:

A dynamic pricing engine
A novel approach to quantum cryptography

The former acquired the first major corporation, an international airline, as a client. The latter received significant venture funding to commercialize its technology.

Other technologies currently in incubation include multiple forms of compositional machine learning (deep, reinforcement and variational), and applications to difficult microeconomic problems such as mechanism design. Our plan is to spin off at least one to two technologies per year into standalone entities.

This setup and modus operandi was selected to avoid common pitfalls in the academic technology transfer world, and to keep administrative overhead as light as possible.

CyberCat was designed around the shared belief that the set of highly abstract mathematical principles we caption as categorical cybernetics (and applied category theory more broadly) can have immediate social and economic value, that an informal exchange among like-minded researchers, practitioners and founders is the best way to discover promising technologies, and that for-profit spin-offs are the best way to demonstrate economic feasibility.

The CyberCat Institute is currently the vehicle for activities that require an incorporated non-profit entity such as application for research funding or contracting for applied research, especially in the open source ecosystem.

In the near future, we envision the Institute to become a fully staffed organization, to also organize regular events (meetups, conferences, summer schools) and publish both scientific and popular periodicals. By design the spin-off activities take precedence, and the Institute remains a separate legal entity.

As such it is supposed not to replace, but to support the informal “invisible college” at the center of CyberCat.

Bidirectional Typechecking with Dependent Lenses

2025-02-24T00:00:00+00:00

In this post, I will reproduce Jules’ implementation of bidirectional type checking using dependent lenses instead of van Laarhoven (VLH) lenses. We are going to see how to build and adapt a program build from the ground up using lenses and how to manipulate the types involved to achieve our goal. The previous blog post did that and uncovered monadic traversals. We are going through a similar journey, showing off different monads and functors on containers along the way.

Finding the right types

The first step is to reproduce the same language structure in Idris, we define the types for the lambda calculus, and its two classes of terms checkable terms and synthesizable terms.

data Ty = TVar String | Unit | Function Ty Ty

%runElab derive "Ty" [Eq]

data Mode = Synthesizable | Checkable

data Term : Mode -> Type where
  Var : String -> Term Synthesizable
  App : (fn : Term Synthesizable) -> (arg : Term Checkable) -> Term Synthesizable
  Down : Ty -> Term Checkable -> Term Synthesizable
  Lambda : (name : String) -> Term Checkable -> Term Checkable
  Up : Term Synthesizable -> Term Checkable

A context is a list of names and their associated types.

Context : Type
Context = List (String, Ty)

Questions are either “can we synthesize the type of this term” or “can we check that this term has this type”.

data Question =
  Syn Context (Term Synthesizable) | Check Context Ty (Term Checkable)

An answer is either a synthesized type, or a confirmation that the check went through.

data Answer = Synthesized Ty | Checked

The boundary of each lens is a question-answer pair, we represent this as a container with a constant fibre. The (:-) operator is a smart constructor for such containers.

TC : Container
TC = Question :- Answer

The first attempt is to model our type checker with a morphism from question-answer pairs to other question-answer pairs: TC =%> TC.

The $Up$ Rule

With it we can start implementing each rule, let’s start with the $Up$ rule. We’re going to use closed lenses to more accurately represent the continuation nature of lenses.

upRule : Closed TC TC
upRule = MkClosed $ \case
  (Check ctx ty (Up term)) =>
     (Syn ctx term) ##
       (\case (Synthesized ty') =>
                if ty == ty' then Checked else ?error1
              _ => ?error2)

Looking at error1 we see that we are required to return an Answer even when it does not make sense to do so. In that case, whenever the types do not match, we should not return an answer but an error instead.

   term : TermSyn
   ty : Ty
   ctx : List (String, Ty)
   ty' : Ty
------------------------------
error1 : Answer

Because the error occurs only when returning answers, we can use the lift operation on containers to wrap responses in a monad.

The corrected type is

upRuleOk : Closed (Either String • TC) TC
upRuleOk = MkClosed $ \case
  (Check ctx ty (Up term)) =>
     (Syn ctx term) ##
       (\case (Synthesized ty') =>
                if ty == ty' then pure Checked else Left "type mismatch"
              _ => ?error)

This is a small win in expressivity because we do not need to pollute our Answer type with a dummy error value.

We’re not done however, the error hole requires us to return an answer or an error, but really, neither should happen. This branch will only be taken if we receive an answer that is not a Synthesized answer, but because we asked a Syn question, we should only ever obtain Synthesized answer. We could fill the hole with a value Left "should never happen" but surely we can do better.

   term : TermSyn
   ty : Ty
   ctx : List (String, Ty)
------------------------------
error : Either String Answer

To address this, we change the type of Answer to be indexed over the type of questions such that we always get the answer we expect from the question we asked.

data SafeAnswer : Question -> Type where

  -- For any synthesized type there is a type and the details of its syn question
  SynAnswer : Ty -> SafeAnswer (Syn ctx term)

  -- For any checked answer there is a `Check` question
  CheckAnswer : SafeAnswer (Check ctx ty term)

Our new container of question-answers makes use of safe answers to avoid the additional ambiguity introduced in the backward part of each lens.

Typecheck : Container
Typecheck = (q : Question) !> SafeAnswer q

With this we can review our up-rule and remove the need for throwing an error when the answer is not the one expected.

We also define TCErr = Either String to simplify our type signature.

upRuleFinal : Closed (TCErr • Typecheck) Typecheck
upRuleFinal = MkClosed $ \case
  (Check ctx ty (Up term)) =>
     (Syn ctx term) ##
       (\case (SynAnswer ty') =>
                if ty == ty' then Right CheckAnswer else Left "type mismatch")

The $Var$ Rule

Moving on to the $Var$ rule, it looks up the type in context and returns it. If the variable is not found, an error is emitted.

varRule : Closed (TCErr • Typecheck) Typecheck
varRule = MkClosed $ \case
  (Syn ctx (Var str)) =>
     ?noQuestion ## \_ => case lookup str ctx of
                   Just ty => pure (SynAnswer ty)
                   Nothing => Left "unbound variable \{str}"

The problem with this rule is that it does not ask any more questions, it ends immediately and returns the answer. If we look at the hole noQuestion, we see that it requires a question but there is no way to say “we’re done here, no more questions”

   str : String
   ctx : List (String, Ty)
------------------------------
noQuestion : Question

To express the fact that we might run out of question, the type will need to be a Maybe monad on container such that if there is a question, then there will also be an answer. Otherwise there is no answer comming back and we’re expected to produce one ourselves, which is exactly what the $var$ rule does.

The maybe monad on containers is written MaybeAll and so the function becomes:

varRule2 : Closed (TCErr • Typecheck) (MaybeAll Typecheck)
varRule2 = MkClosed $ \case
  (Syn ctx (Var str)) =>
     Nothing ## \_ => case lookup str ctx of
                   Just ty => pure (SynAnswer ty)
                   Nothing => Left "unbound variable \{str}"

The argument of the continuation is a unit value so we can safely ignore it.

The $App$ Rule

Next we implement the $App$ rule, for it to work, we need to synthesize the type of the function, then check that its domain matches the type of the argument, and return the type of the codomain.

appRule : Closed (TCErr • Typecheck) (MaybeAll Typecheck)
appRule = MkClosed $ \case
   (Syn ctx (App fn arg)) =>
     Just (Syn ctx fn) ##
       (\case { (Aye (SynAnswer (Function a b))) => ?checkTypes
              ; (Aye arg) => Left "not a function"
              })

However, we get stuck on checkType. We’re asked to produce a value of type Either String (SafeAnswer (Syn ctx (App fn arg))) but this can only be produced by another call to the type checker.

   arg : TermChk
   fn : TermSyn
   ctx : List (String, Ty)
   b : Ty
   a : Ty
------------------------------
checkTypes : Either String (SafeAnswer (Syn ctx (App fn arg)))

To represent the fact that rules can sequence multiple question-answers we use the kleene star on containers. This construction is equivalently known as the free monoid on $\circ$ where $\circ$ is the composition of containers. The Kleene star is called Star : Container -> Container here, and with it, the app rule can be written as follows.

appRule2 : Closed (TCErr • Typecheck) (Star Typecheck)
appRule2 = MkClosed $ \case
   (Syn ctx (App fn arg)) =>
     More (Syn ctx fn)
       (\case { (SynAnswer (Function a b)) =>
                 More (Check ctx a arg)
                      (const Done)
              ; y => Done
              })
       ## \case (StarM (SynAnswer (Function a b)) (StarM CheckAnswer StarU)) => pure (SynAnswer b)
                (StarM (SynAnswer _) _) => Left "not a function"

The question part of this lens is the most interesting so we’ll go through it line by line:

More (Syn ctx fn): First ask a question Syn ctx fn to synthezise the type of the function.
(\case { (SynAnswer (Function a b)) =>: Match on the answer and ensure we got a function back
- More (Check ctx a arg): Knowing that it is a function, check the type of the argument agains the type a.
- (const Done): After that, we’re done, no more questsions.
; y => Done: In case the synthesized type wasn’t a function, no more questions.

When handling responses, we merely decompose the response, ensure that we get a function and return an error if we do not.

Putting It All Together

Other rules pose no additional challenge, therefore the final rules of the simply typed lambda calculus can be written using a lens TCErr • Typecheck =%> Star Typecheck.

rules : TCErr • Typecheck =%> Star Typecheck
rules = fromClosed $ MkClosed $ lamRules
  where
    lamRules : (x : Question) -> Σ (StarShp Typecheck) (\y => StarPos Typecheck y -> Either String (SafeAnswer x))
    -- Var rule
    lamRules (Syn ctx (Var str))
      = case lookup str ctx of
             Just ty => Done ## (const (pure (SynAnswer ty)))
             Nothing => Done ## (const (Left "Undeclared variable \{str}"))

    -- App rule
    lamRules (Syn ctx (App fn arg))
      = More (Syn ctx fn)
          (\case { (SynAnswer (Function a b)) =>
                    More (Check ctx a arg)
                         (const Done)
                 ; y => Done
                 })
          ## \case (StarM (SynAnswer (Function a b)) (StarM CheckAnswer StarU)) =>
                       pure (SynAnswer b)
                   (StarM (SynAnswer t) _) =>
                       Left "Expecting \{show fn} to be a function, instead it has type \{show t}"

    -- Up rule
    lamRules (Check ctx ty (Up term))
     = singleton (Syn ctx term) ##
       (\case (StarM (SynAnswer ty') _)
              => if ty == ty'
                    then pure CheckAnswer
                    else Left """
                      Expecting \{show term} to have the type \{show ty}
                      instead I found it has type \{show ty'}
                      """)
    -- Down rule
    lamRules (Syn ctx (Down ty val))
      = singleton (Check ctx ty val) ## \(StarM CheckAnswer StarU) => pure (SynAnswer ty)
    -- Lambda rule
    lamRules (Check ctx (Function a b) (Lambda nm term))
      = singleton (Check ((nm, a) :: ctx) b term) ## \(StarM CheckAnswer StarU) => pure CheckAnswer
    lamRules _ = Done ## const (Left "oops")

To run the typechecker we need to make sure it has the shape a =%> Star a. We do this by first defining an appropriate type alias TCF = TCErr • Typecheck which stands for “TypeCheckFail”. Then we use the fact that the • operator forms a comonad and use the comultiplication to build the lens TCF =%> Star TCF. Additionally, TCErr • _ commutes with Star so by going from TCF to TCErr • TCF via a comultiplication, and then mapping across our typechecker to obtain TCErr • Star Typecheck and then distributing around TCErr we end up with Star (TCErr • Typecheck) which is the same as Star TCF.

TCF : Container
TCF = TCErr • Typecheck

typecheck' : TCF =%> Star TCF
typecheck' = comultM {a = Typecheck}
    |%> mapLift' {a = TCF, b = Star Typecheck} rules
    |%> commuteEitherStar

With this, we can loop our typechecker using the loop : a =%> Star a -> Star a =%> CUnit combinator. This function takes a lens a =%> Star a and will run itself as long as it is fed some input, as soon as the input is spent, it will stop with the final result.

covering
typechecker : TCF =%> CUnit
typechecker = typecheck' |%> loop typecheck'

Finally, this lens gives rise to a function (q : Question) -> Either String (SafeAnswer q) by means of a costate.

covering
runTypechecker : (q : Question) -> Either String (SafeAnswer q)
runTypechecker = runCostate typechecker

covering
printTypechecker : Question -> String
printTypechecker q = show (runTypechecker q)

printTypechecker makes the output readable, and with it we can start typechecking programs. Here is $(\lambda y. y : a \to a) x$ with [x : a] in the context.

program1 : Question
program1 = Syn [("x", TVar "a")]
             $ App (Down (Function (TVar "a") (TVar "a")) (Lambda "y" (Up (Var "y"))))
                   (Up (Var "x"))

We call the typechecker in the repl and obtain the reponse immediately

$ :exec putStrLn (printTypechecker program1)
> Right (SynAnswer a)

Checking an incorrect program $(x\ x)$ produces the appropriate error.

program2 : Question
program2 = Syn [("x", TVar "a")]
             $ App (Var "x")
                   (Up (Var "x"))

$ :exec putStrLn (printTypechecker program2)
> Left "Expecting x to be a function, instead it has type a"

Conclusion

Just like the non-dependent version, we need to worry about how to sequence operations in our typechecker. Instead of monadic traversals we use the Kleene star on container to achieve the same thing. This suggests that there is an additional relationship in the correspondance between VLH lenses and dependent lenses:

VLH lenses	Dependent lenses
Traversal	List monad
Affine Traversal	Maybe monad
Monadic Traversal	Kleene star

Generalized Transformers from Applicative Functors

2025-02-12T00:00:00+00:00

Cross-posted on the GLAIVe blog

Transformers are a machine-learning model at the foundation of many state-of-the-art systems in modern AI, originally proposed in [arXiv:1706.03762]. In this post, we are going to build a generalization of Transformer models that can operate on (almost) arbitrary structures such as functions, graphs, probability distributions, not just matrices and vectors.

We will do this using the language of applicative functors, and indeed many of the constructions here have similar ideas to those presented in the original paper introducing applicative functors by McBride and Paterson, the only difference is that we interpret them in the context of machine learning, rather than in the context of functional programming with effects.

Although I’m not aware of this particular construction appearing elsewhere in the wild, it is related to and inspired by various other models in the literature, in particular neural operators [arXiv:2108.08481], which define models very similar to the Funcformer model that we will present later.

This work is part of a series of similar ideas exploring machine learning through abstract diagrammatical means. If you think this is interesting, I would recommend reading other posts and papers in the same series, such as:

On the anatomy of attention [arXiv:2407.02423] (the ‘tube’ notation in Part 4 is equivalent to the ‘SIMD’ notation in that paper)
A pattern language for machine learning tasks [arXiv:2407.02424]

These ideas were developed collaboratively in conversation with many colleagues, but in particular: Vincent Wang-Maścianica, Nikhil Khatri, Jono Liu, Ben Rodatz, Ian Fan, Neil Ortega, and Blake Wilson.

At its core, the Transformer is a method for mapping sequences of vectors to sequences of vectors. We will skip any conceptual explanation and go straight to the mathematics, since this has been explained at length elsewhere (for instance, by Grant Sanderson). We have as input a sequences of $n$ vectors $x_i \in \mathbb{R}^d$, and the model outputs a sequence $y_i \in \mathbb{R}^d$ also of length $n$. Both of these are arranged as matrices $X, Y \in \mathbb{R}^{n \times d}$. The basic operation of a Transformer relies on interleaving two different operations: multi-layer perceptrons (MLPs) and self-attention. Given learnable matrices $Q, K \in \mathbb{R}^{d \times d_k}$, $V \in \mathbb{R}^{d \times d}$, $W_1 \in \mathbb{R}^{d \times d_{ff}}$, and $W_2 \in \mathbb{R}^{d_{ff} \times d}$, and vectors $b_1 \in \mathbb{R}^{d_{ff}}$ and $b_2 \in \mathbb{R}^{d}$, these operations can be expressed as follows:

\[\begin{aligned} &\mathrm{MLP} : \mathbb{R}^{n \times d} \to \mathbb{R}^{n \times d} &~~~~ &\mathrm{SelfAtt : \mathbb{R}^{n \times d} \to \mathbb{R}^{n \times d}}\\ &\mathrm{MLP}(X) = \sigma(XW_1 + 1_nb_1^T)W_2 + 1_nb_2^T & &\mathrm{SelfAtt(X) = \rho(XQ(XK)^T)XV}\\ \end{aligned}\]

where $1_n \in \mathbb{R}^n$ is the all-ones vector, $\sigma : \mathbb{R} \to \mathbb{R}$ is an activation function applied element-wise that is usually taken as $\sigma(x) = \mathrm{ReLU}(x) = \max(x, 0)$, and $\rho : \mathbb{R}^n \to \mathbb{R}^n$ is a normalization function applied to each column that is usually taken as the scaled softmax:

\[\rho(x)_i = \frac{e^{x_i}}{\sqrt{d_k}\sum_i e^{x_i}}\]

A (single-headed) Transformer can be built by iterating these functions (each with separate learnable weights) along with some other components like residuals and layer-norms that we will omit here for simplicity.

Part 1: A basic model

Suppose we wanted to implement a machine-learning model like the Transformer in a functional programming language - in this post we will use Haskell. We can start with a very naive type-level model, taking vectors as lists and matrices as lists of row-vectors. It is easy to build up the basic components like dot products, vector-matrix multiplication, and matrix-vector multiplication:

-- Dot product of two vectors
dot :: [Float] -> [Float] -> Float
dot a b = sum $ zipWith (*) a b

-- Multiply a matrix by a vector from the right
mulMV :: [[Float]] -> [Float] -> [Float]
mulMV m v = map (`dot` v) m

-- Sum a list of vectors
vectorSum :: [[Float]] -> [Float]
vectorSum = foldl (zipWith (+)) (repeat 0.0)

-- Multiply a matrix by a vector from the left
mulVM :: [Float] -> [[Float]] -> [Float]
mulVM v m = vectorSum $ zipWith (map . (*)) v m 

-- Multiply a matrix by another matrix
mulMM :: [[Float]] -> [[Float]] -> [[Float]]
mulMM a b = map (`mulVM` b) a

-- Multiply a matrix by the transpose of another matrix
mulMMT :: [[Float]] -> [[Float]] -> [[Float]]
mulMMT a b = map (a `mulMV`) b

And from here we can build the MLP and self-attention operations. In this case we build self-attention from a generic attention operation, where $XQ$, $XK$, and $XV$ are provided directly:

scaleSoftmax :: Int -> [Float] -> [Float]
scaleSoftmax dk v = map (/ n) e 
    where e = map exp v
          n = sqrt (fromIntegral dk) * sum e

-- A generic attention function, before the inputs are specialized
attention :: [[Float]] -> [[Float]] -> [[Float]] -> [[Float]]
attention queries keys values = attMatrix `mulMM` values
    where dk = length $ head queries
          -- The 'attention matrix'
          attMatrix = map (scaleSoftmax dk) (queries `mulMMT` keys)

selfAttention :: [[Float]] -> [[Float]] -> [[Float]] -> [[Float]] -> [[Float]]
selfAttention Q K V X = attention (X `mulMM` Q) (X `mulMM` Q) (X `mulMM` V)

-- Apply a linear function f(X) = XW^T + 1b^T to the input
linear :: [[Float]] -> [Float] -> [[Float]] -> [[Float]]
linear weights bias input = map (zipWith (+) bias) (input `mulMMT` weights)

mlp :: [[Float]] -> [Float] -> [[Float]] -> [Float] -> [[Float]] -> [[Float]]
mlp W1 b1 W2 b2 X = linear W2 b2 $ map (map relu) $ linear W1 b1 X
    where relu x = max 0.0 x

There are two major problems with this code:

Since the dimensions of the problem are not encoded in the types, it would be easy to pass a malformed input to these functions. For example, accidentally transposing the weights of an MLP would neither be caught at compile-time nor result in a runtime error.
It would seem this code is too specific - all these operations ought to be possible over more generic structures than just [[Float]], wouldn’t it be better if the code could handle any appropriate structure?

Part 2: Fixing the problems

We will solve both of these at the same time. As a first guess, since the list type is a monad, is it it possible to generalize this code to any monad? The components we would need are:

map :: (a -> b) -> [a] -> [b]
zipWith :: (a -> b -> c) -> [a] -> [b] -> [c]
vectorSum :: [[Float]] -> [Float]
sum :: [Float] -> Float

This initially looks promising, since we could substitute the first three with

fmap :: Functor f => (a -> b) -> f a -> f b

liftM2 :: Monad m => (a -> b -> c) -> m a -> m b -> m c
liftM2 f xs ys = do
    x <- xs
    y <- ys
    return $ f x y

join :: Monad m => m (m a) -> m a
join = (>>= id)

and the type signatures would match! However, substituting the list monad into these functions gives the wrong answer - while fmap is correct, liftM2 and join behave like cartesian products rather than like zips:

liftM2 (,) [1, 2] [1, 2] = [(1, 1), (1, 2), (2, 1), (2, 2)] 
                        != [(1, 1), (2, 2)] 
                         = zip [1, 2] [1, 2]
join [[1, 2], [1, 2]] = [1, 2, 1, 2]

Indeed, this is also related to our first problem. To solve that, let’s assume that the dimension of each vector was encoded in its type as Vector N a where N is a type-level natural number. As before, we will take matrices to be vectors of vectors. In this setting, no function that has this cartesian-product behaviour could exist, because the length of the vector would not be preserved, so it would not type-check. It seems even that Vector N cannot be a monad in a non-trivial way, because any join-like function would have to ‘throw away some information’ in general: since there are no non-trivial generic functions a -> a -> a, then given $N^2$ values of a generic type a that we must compress to only $N$ values, we have to discard almost all of them.

For the same reasons, such a generic version of vectorSum or sum that applies to any interior type a is too generic - we would end up throwing away information. The solution for this is to just abstract over these definitions. Let’s define a class that encapsulates these operations:

class Counit f a where
    counit :: f a -> a

So for instance we would have Monad m => Counit m (m a) given by join, and for comonads we could have Comonad m => Counit m a given by extract. In this way, we don’t have to define Counit generically, but only for types a which have a meaningful and non-trivial counit operation.

To solve the problem with liftM2, we can be careful to define Vector N so that fmap behaves the same as map and liftM2 does behave the same as zipWith. However, as we determined, Vector N shouldn’t be a monad. But it can be an applicative functor! Indeed, liftM2 is the monad specialization of liftA2:

liftA2 :: Applicative f => (a -> b -> c) -> f a -> f b -> f c
liftA2 f a b = (fmap f a) <*> b

Applicative functors are a superset of monads that are defined by the Applicative class. It is usually defined by a pair of functions

pure :: Applicative f => a -> f a
f <*> a :: Applicative f => f (a -> b) -> f a -> f b

where if we think of applicative functors as collections, pure can be interpreted as a ‘unit’ mapping any value into a collection, and <*> can be interpreted as taking a collection of functions and a collection of arguments and returning a collection of results. However, there is a more concrete definition in terms of pair:

pair :: Applicative f => f a -> f b -> f (a, b)

which can be thought of as pairing up elements of collections. These definitions are equivalent, since we have:

pair a b = (fmap (,) a) <*> b
f <*> a = fmap (uncurry ($)) $ pair f a 

We will work in terms of pair, since this is closer to the category-theoretic definition of an applicative functor as a ‘monoidal functor with tensorial strength’.

With this in mind, we can define (ignoring some Haskell specifics regarding type-level naturals)

data Vector (N :: Natural) a = Vector [a] 

instance Functor (Vector N a) where
    fmap f (Vector xs) = Vector $ map f xs

instance Applicative (Vector N a) where
    pure a = Vector $ take N $ repeat a
    pair (Vector a) (Vector b) = Vector $ zip a b

and we can see that now pair matches exactly with zip, as we wanted! See page 4 of McBride and Paterson for a similar construction.

Part 3: Applicative matrix operations

To recreate our original code in these new terms, we are still missing a few elements specific to Floats that we don’t have in the generic picture. We will ignore the scaled-softmax and ReLU operations for now as they can in principle be replaced with any normalization and activation functions, and come back to them later. However, we still need a definitions of multiplication and addition. We will just abstract over these as a typeclass:

class Ring a where
    zero :: a
    (~+~) :: a -> a -> a
    (~*~) :: a -> a -> a

We can then recover our definitions of sum and vectorSum in terms of Vector N by constructing the appropriate Counit instances:

instance Ring a => Counit (Vector N) a where
    counit (Vector xs) = foldl (~+~) zero xs

instance (Ring a, Applicative f) => Counit (Vector N) (f a) where
    counit (Vector xs) = foldl ((.) (fmap $ uncurry (~+~)) . pair) (pure zero) xs

With these definitions, we can build a fully generic version of our code that also encode dimensionality information in the type-signatures. We need only operations from the typeclasses we’ve defined, and substituting in Vector as our functor will yield something equivalent to our original code, and hence the original Transformer!

The matrix operations can be given as:

dot :: (Applicative f, Ring a, Counit f a) => f a -> f a -> a
dot a b = counit $ uncurry (~*~) <$> pair a b

mulMV :: (Applicative f, Applicative g, Ring a, Counit g a) => f (g a) -> g a -> f a
mulMV m v = uncurry dot <$> pair m (pure v) 

mulVM :: (Applicative f, Applicative g, Ring a, Counit f (g a)) => f a -> f (g a) -> g a
mulVM v m = counit $ fmap (uncurry (~*~)) . uncurry pair <$> pair (pure <$> v) m

mulMMT :: (
    Functor h, Applicative f, Applicative g, Ring a, Counit g a
) => f (g a) -> h (g a) -> h (f a)
mulMMT = fmap . mulMV

mulMM :: (
    Applicative h, Functor f, Applicative g, Ring a, Counit g (h a)
) => f (g a) -> g (h a) -> f (h a)
mulMM = flip $ fmap . flip mulVM

So we see that instead of defining a matrix as [[Float]] and a vector as [Float], these have become f (g a) and f a respectively for arbitrary type constructors f and g and arbitrary ‘scalar’ types a. If f and g are different instances of Vector N for example, then this will prevent many bugs of the type we identified earlier with stronger compile-time checks.

By abstracting over the scaled-softmax and ReLU functions as arbitrary functions that can be passed in, we can define the generic attention and linear function operations as before:

attention :: (
    Applicative f, Applicative g, Ring a, Counit g a, Counit f (g a)
) => (f a -> f a) -> f (g a) -> f (g a) -> f (g a) -> f (g a)
attention softmax queries keys values = attMatrix `mulMM` values
    where attMatrix = fmap softmax (queries `mulMMT` keys)

linear :: (
    Applicative g1, Applicative g2, Ring a, Counit g1 a
) => g2 (g1 a) -> g2 a -> g1 a -> g2 a
linear weights bias input = fmap (uncurry (~+~)) $ pair bias $ mulMV weights input

And we can put it all together to define self-attention and MLP operations. We can make it a bit nicer than before by packaging all the matrices together as data-structures:

data SelfAttention s d a = SelfAttention {
    -- This abstracts scaled-softmax
    softmax :: s a -> s a,
    queryMat :: d (d a),
    keyMat :: d (d a),
    valueMat :: d (d a)
}

runSelfAttention :: (
    Ring a, Applicative s, Applicative d, Counit d a, Counit s (d a)
) => SelfAttention s d a -> s (d a) -> s (d a)
runSelfAttention SelfAttention { softmax, queryMat, keyMat, valueMat } input = 
    attention softmax queries keys values
    where queries = mulMMT queryMat input
          keys = mulMMT keyMat input
          values = mulMMT valueMat input

data MLP din dff dout a = MLP {
    -- This abstracts ReLU
    activation :: a -> a,
    weights1 :: dff (din a),
    bias1 :: dff a,
    weights2 :: dout (dff a),
    bias2 :: dout a
}

runMLP :: (
    Ring a, Applicative dff, Applicative dout, 
    Counit dff a, Applicative din, Counit din a
) => MLP din dff dout a -> din a -> dout a
runMLP MLP { weights1, bias1, weights2, bias2, activation } = 
    linear weights2 bias2 . fmap activation . linear weights1 bias1

Finally, we can combine these blocks to make a whole Transformer. We are ignoring the layer-norms, and using only a single attention head, but otherwise this is identical to the original proposal from Vaswani et al:

-- One transformer layer is a self-attention block and an MLP block with residuals
data TransformerLayer s d dff a = TransformerLayer {
    mlp :: MLP d dff d a,
    satt :: SelfAttention s d a
}

runTransformerLayer :: (
    Ring b, Applicative dff, Applicative d, Applicative s, 
    Counit dff b, Counit d b, Counit s (d b)
) => TransformerLayer s d dff b -> s (d b) -> s (d b)
runTransformerLayer TransformerLayer { mlp, satt } = 
    residual (runMLP mlp <$>) . residual (runSelfAttention satt)
    where add x y = fmap (uncurry (~+~)) . uncurry pair <$> pair x y
          residual f x = add (f x) x 

-- A transformer is a linear embedding matrix, followed by a 
-- stack of transformer layers composed sequentially, 
-- followed by a linear unembedding matrix
data Transformer s f d dff a = Transformer {
    layers :: [TransformerLayer s d dff a],
    embed :: d (f a),
    unembed :: f (d a)
}

runTransformer :: (
    Ring a, Applicative f, Applicative d, Applicative dff, Applicative s, 
    Counit d a, Counit dff a, Counit s (d a), Counit f a
) => Transformer s f d dff a -> s (f a) -> s (f a)
runTransformer Transformer { layers, embed, unembed } = 
    mulMMT unembed 
    . flip (foldl (flip runTransformerLayer)) layers 
    . mulMMT embed

Part 4: Diagrammatics

(Many thanks to Vincent Wang-Maścianica for his help with this part)

All this uncurrying we needed to do to write the code above using pair is pretty miserable. That’s because we are trying to do fundamentally 2D things in 1D and it sucks. Luckily there is a formal diagram system for this.

We build on the tube-notation for monoidal monads introduced by Joe Moeller in his blogpost, which is an evident extension of functor box notation, by stretching out functor-boxes as “windows” along the length of wires to become “tubes”. So if we have an underlying symmetric monoidal category $(\mathcal{C},\otimes,I)$ along with a symmetric monoidal endofunctor $\mathbf{T}: \mathcal{C} \rightarrow \mathcal{C}$, we would depict the object $\mathbf{T}X$ of $\mathcal{C}$ as the wire $X$ wrapped by a $\mathbf{T}$-tube:

As is sort-of known, applicative functors are equivalently lax-monoidal functors with a tensorial strength. In tube-notation, the laxator natural transformation $\tau_{-,=} : \mathbf{T}(-) \otimes \mathbf{T}(=) \rightarrow \mathbf{T}(- \ \otimes =)$ are depicted as pants that merge two parallel tubes:

This is exactly the pair operation we have in the code above. The visual tube metaphor captures the naturality conditions we want string-diagrammatically, as deformation of string diagrams freely within the confines of tubes. For example, this pretty little monster is the naturality condition that allows the laxator to cohere with a simplified form of interchange, given two morphisms $f: A \rightarrow X$ and $g: B \rightarrow Y$:

But all we’re really asking for here in string-diagrammatic terms is that you can slide morphisms around in pants that merge tubes.

The rest of the lax monoidal structure goes as one would expect. For example, we want the monoidal laxator to be appropriately associative (we won’t label wires with types anymore, they can be any object in the category.)

There’s also another laxator $\nu: I \rightarrow \mathbf{T}I$ for the monoidal unit (depicted here with a dashed wire)

And these laxators should behave as we expect, namely they should be appropriately unital. Below we get the middle equalities (depicting the left and right unitor isomorphisms of $\mathcal{C}$) for free, as we can assume that we’re working with a strict symmetric monoidal category.

And of course, we would like the laxators to cohere sensibly with braidings.

Now the tensorial-strength, which is a natural transformation $\beta: - \otimes \mathbf{T}(=) \rightarrow \mathbf{T}(- \ \otimes =)$. In tube-notation, this corresponds to taking a wire and shoving it inside an adjacent tube.

There are a couple of coherence conditions we expect, namely that it doesn’t matter whether we shove wires in one-by-one or all at once, that shoving-in the monoidal unit does nothing, and that shoving-in can be done from either side in a way that coheres with the braiding.

And we’re done with the tricky bit. In terms of the code above, this is

beta :: Applicative f => a -> f b -> f (a, b)
beta a b = pair (pure a) b

nu :: Applicative f => () -> f ()
nu () = pure ()

which along with $\tau$ (equivalently pair) and the right unitor $\rho$ from above is also enough to define pure:

rho :: Applicative f => f (a, ()) -> f a
rho = fmap fst

pure :: Applicative f => a -> f a
pure a = rho $ beta a $ nu ()

completing the equivalence between the diagrams and the code.

The other ingredients we need are garden-varietal diagrammatic gadgets. We will need a copy-gadget, which we can have in any Cartesian (or Markov) category. Copy comonoids are traditionally depicted as dots that gather the various copied branches. Since we will be copying tubes, which will make dots too big, we will use a fork-notation for copying instead.

We will need learnable parameters, such as weights and biases. These are just elements of a given object (for instance, $\mathbf{TT}X$ or $\mathbf{T}X$) and to be depicted as triangles, though to indicate that it is a learner with some parameter space, for historical reasons ¹ we will colour them red.

We will need an algebra $\alpha_X \mathbf{T}X \rightarrow X$ (this is Counit) which extracts wires from tubes. We’ll depict these as triangles that cap off tubes.

We will need some magma $X \otimes X \rightarrow X$ (this is ~*~ from Ring). In practice, the multiplication of any off-the-shelf monoid will do.

Optionally, we may have a “normalisation” $\sigma: \mathbf{TT}X \rightarrow \mathbf{TT}X$ (which generalizes softmax).

Now we can assemble these ingredients. The pants, magma, and the algebra allow us to define an abstract inner product of type $\mathbf{T}X \otimes \mathbf{T}X \rightarrow X$.

We can also define an abstract outer product of type $\mathbf{T}X \otimes \mathbf{T}X \rightarrow \mathbf{T}\mathbf{T}X$, using the magma and two instances of the tensorial strength of the applicative.

Now we’re ready for the abstract attention mechanism. The idea is to use the outer product gadget to create a doubly-nested tube, and then to use the inner product gadget along with the pants and shoving of the applicative to bring that back down to a singly-nested tube, while liberally sprinkling in learners. Since the outer product requires two $\mathbf{T}X$ inputs, and we require at least one extra copy of $\mathbf{T}X$ apart from those to use the inner product, it makes sense to start with three copies of the initial input $\mathbf{T}X$, and a little thought will yield this:

For the sake of convention, we put learnable linear transformations on the three inputs and call them “queries”, “keys”, and “values”. This is exactly the same as runSelfAttention. If we pick $X$ to be $\mathbb{R}$, our applicative functor to be the $k$-tupling endofunctor $X \mapsto X^k$, where $k$ is some positive integer “context-length” for the outer tubes and a “model dimension” for the inner tubes, and if we choose our algebra to be summation, our monoid to be multiplication, and our normalisation to be softmax, then we have precisely a classic attention mechanism as in the matrix-based selfAttention above. The $\mathbf{TT}X$ wire in this case is the abstract analog of the attention matrix; the synthetic inner and outer products we define here are special cases of tensor contraction, where each index of a tensor corresponds to a layer of tubing.

Part 5: Funcformer

At this point, we are done in theory! We have a fully generalized construction of a Transformer model. However, what does this buy us in practice? Can we construct anything meaningfully different than the standard model? Indeed there is, and we shall look at one such construction now, which we call Funcformer. We start with this not necessarily because it is the most useful generalization we can make, but because it is the easiest to implement.

One potential generalization of the Vector N type comes from viewing any vector $v \in X^{n}$ over some type $X$ as a function from indices to components of the vector - i.e $v : {1, 2, \dots, n} \to X$ given by $v(i) = v_i$. There is no fundamental reason why this index set ought to be finite, or even discrete. To that end, let’s define a ‘continuous vector’ CVector which has ‘components’ parameterized by a real number. We can define this in code as

newtype CVector a = CVector { (~!!~) :: Float -> a }

where the ~!!~ operator is analogous to the indexing operator on lists, !! : [a] -> Int -> a. This type is indeed an applicative functor (in fact, it is a specialization of the Reader monad):

instance Functor CVector where
    fmap f x = CVector { (~!!~) = f . (x ~!!~) } 

instance Applicative CVector where
    pure x = CVector { (~!!~) = const x }
    pair x y = CVector { (~!!~) = \t -> (x ~!!~ t, y ~!!~ t) }

Mathematically, we can think of v : CVector a as a function $v \in \mathbb{R} \to a$, and we will notate it as such (i.e $v(x)$). Then the operations defined here can be interpreted as

\[\begin{align*} \mathrm{fmap} &: f, x(t) ~~\to~~ v(t) = f(x(t)) \\ \mathrm{pure} &: x ~~\to~~ v(t) = x \\ \mathrm{pair} &: x(t), ~y(t) ~~\to ~~v(t) = (v(t), y(t)) \end{align*}\]

We also need instances of Counit for this type. Here, we can draw some inspiration from the original Transformer, where these corresponded to sums over the components of vectors, or sums of the rows of a matrix. Analagous to a sum over a vector, we can consider the integral of a function over its domain - of course, this is not straightforward to define in code, but let’s pretend we have a magical black box:

integrate :: (Float -> Float) -> Float

Using this we can define:

instance Counit CVector Float where
    counit f = integrate (f ~!!~)

instance Counit CVector (CVector Float) where
    counit f = CVector { (~!!~) = \y -> integrate (\x -> f ~!!~ x ~!!~ y) }

Note here that our equivalent of a matrix, v : CVector (CVector a) is equivalent to a function with type $v : \mathbb{R} \to \mathbb{R} \to a$. By currying, this is the same as a two-argument function $v : \mathbb{R}^2 \to a$, and so we will notate it as $v(x, y)$. We can again interpret these operations mathematically, as:

\[\begin{align*} \mathrm{counit_{a}} &: x(t) ~~\to~~ \int_{-\infty}^\infty x(t) ~\mathrm{dt} \\ \mathrm{counit_{\mathbb{R} \to a}} &: x(t_1, t_2) ~~\to~~ v(t_2) = \int_{-\infty}^\infty x(t_1, t_2) ~\mathrm{dt_2} \end{align*}\]

This is actually all we need to translate the whole Transformer into CVectors! Working through the code, we can have a look at what the building blocks operations become, mathematically:

\[\begin{align*} \mathrm{dot} &: x(t), ~y(t) ~~\to~~ \int_{-\infty}^\infty x(t)y(t) ~\mathrm{dt} \\ \mathrm{mulMV} &: m(t_1, t_2), ~v(t_2) ~~\to~~ u(t_1) = \int_{-\infty}^\infty m(t_1, t_2)v(t_2)~\mathrm{dt_2} \\ \mathrm{mulMM} &: m_1(t_1, t_2), ~m_2(t_2, t_3) ~~\to~~ m_3(t_1, t_3) = \int_{-\infty}^\infty m_1(t_1, t_2)m_2(t_2, t_3)~\mathrm{dt_2} \end{align*}\]

It is pleasing to see that dot products over vectors translate directly into the $L^2$ dot product on functions, and that the multiplication operations generalize matrix multiplication exactly as one would expect! (I’ve skipped the transposed versions here because they are symmetric.) Given $Q, K, V, W_1, W_2 \in \mathbb{R}^2 \to \mathbb{R}$, and $b_1, b_2 \in \mathbb{R} \to \mathbb{R}$, the MLP operation becomes

\[\begin{align*} \mathrm{MLP} &: x(t_1, t_2) ~~\to~~ v(t_1, t_2) = \\ &\int_{-\infty}^\infty\sigma\left(\int_{-\infty}^{\infty} x(t_1, t_3)W_1(t_3, t_4) ~\mathrm{dt_3} ~+~ b_1(t_4)\right)W_2(t_4, t_2) ~\mathrm{dt_4} + b_2(t_2) \end{align*}\]

where $\sigma \in \mathbb{R} \to \mathbb{R}$ is some arbitrary activation function. If we take the obvious continuous generalization of the scaled-softmax operator to be given by

\[\rho : x(t) ~~\to~~ v(t) = \frac{e^{x(t)}}{\int_{-\infty}^\infty e^{x(t)} ~\mathrm{dt}}\]

then we can write the self-attention operation as:

\[\begin{align*} \mathrm{SelfAtt} : ~~&x(t_1, t_2) ~~\to~~ v(t_1, t_2) = \\ &\int_{\mathbb{R}^2}\frac{e^{\int_{\mathbb{R}^3} x(t_1, t_3)Q(t_3, t_4)K(t_4, t_5)x(t_5, t_6) ~\mathrm{dt_3}\mathrm{dt_4}\mathrm{dt_5}}}{\int_{-\infty}^\infty e^{\int_{\mathbb{R}^3} x(t_1, t_3)Q(t_3, t_4)K(t_4, t_5)x(t_5, t_6) ~\mathrm{dt_3}\mathrm{dt_4}\mathrm{dt_5}}~\mathrm{dt_1}} x(t_6, t_7)V(t_7, t_2)~\mathrm{dt_6}\mathrm{dt_7} \end{align*}\]

This is pretty gnarly! Obviously, there are very few functions where you could expect this to be exactly solvable, and really I’m only writing it down here to show that it does in fact exist.

To regroup, let’s just remind ourselves what we just derived: we replaced Vectors in the original Transformer with CVectors and so now we have a model that instead of mapping matrices to matrices, maps two-argument functions to two-argument functions! In the usual interpretation of the Transformer as it is applied to natural language processing, we consider one dimension of the matrix to be the ‘sequence’ dimension (that is, it represents your position in the text), and the other to be the ‘feature’ dimension (that is, it represents something about the meaning of the words at each position). So in our model, we can think perhaps that we have a continuum of text positions, and a continuum of features at each position, rather than finitely many of each.

So how can we make this idea practical? We need a way to represent functions and integration efficiently, and hence in a finite-dimensional way. This could be done with a table of values for each function and numerical integration, but it’s both more efficient and more interesting to choose a different method. Since any function can be approximated as a polynomial, and it is easy to integrate polynomials, we can choose a set of polynomials that is closed under addition as our finite-dimensional representation. In particular, we make the following (arbitrary) choice, representing functions as 1D or 2D Chebyshev series:

\[f(x) = \sum_{i = 1}^N f_i T_i(x) ~~~~~f(x, y) = \sum_{i = 1}^N \sum_{j = 1}^N f_{ij}T_i(x)T_j(y)\]

where $T_n(x)$ represents the $n$-th Chebyshev polynomial of the first kind. Moreover, rather than considering domain of our functions to be the whole real line, we will just consider the range $[-1, 1]$. This choice of polynomials has the advantage that it is more numerically stable when interpolating data points than the standard monomial $a + bx + cx^2 + \cdots$ basis.

For $n, m, |n - m| \geq 1$ (and the other cases follow similarly), we can use a known result about Chebyshev polynmomials, that

\[\int_{-1}^1 T_n(x)T_m(x)~\mathrm{dx} = \frac{(1 + (-1)^{n + m})(1 - n^2 - m^2)}{(1 - (n + m)^2)(1 - (m - n)^2)} = W_{n, m}\]

which we define to be the components of the weight matrix $W_{n, m}$. Suppose then that we have two functions $f(x)$ and $g(x)$ represented as series with coefficients $f_i$ and $g_i$, then we can write their dot product defined above as

\[\begin{align*} \mathrm{dot}(f, g) &= \int_{-1}^{-1} f(x)g(x) ~\mathrm{dx} = \int_{-1}^1 \sum_{i = 0}^N\sum_{j = 0}^N f_i g_j T_i(x)T_j(x)~\mathrm{dx} \\ &= \sum_{i = 0}^N\sum_{j = 0}^N f_i W_{ij} g_i = f_*^TWg_* \end{align*}\]

if $f_* $ and $g_* $ are vectors representing the coefficients of the series. So we can see that a dot product of functions has become simply a dot product of vectors, albeit with respect to a different bilinear form than the usual dot product!

Indeed, this is also true for the (abstract) matrix-vector and matrix-matrix multiplication operations defined above. In the same way as above, let us denote $m_*$ as the matrix of coefficients $m_{ij}$ corresponding to a two-argument function $m(x, y)$. We have that:

\[\begin{align*} \mathrm{mulMV} &: m_*, ~v_* ~~\to~~ u_* = m_* W v_* \\ \mathrm{mulMM} &: a_*, ~b_* ~~\to~~ c_* = a_* W b_* \end{align*}\]

This is almost all the parts we need to build a Transformer. The only thing that is left is activation function for the MLP and the softmax operation. While in the applicative-based code above, these were specified as element-wise and ‘row’-wise operations, respectively, in practice they could be almost anything.

For the activation function, we chose to keep very close to the original definition - you can approximately apply an elementwise operation to a function represented as a Chebyshev series by first evaluating the polynomial on a grid of points, applying the operation to each of those points, and then interpolating a series from the new values. This is particularly efficient for Chebyshev polynomials, as transforming between the series and point-grid representations can be done using a variant of the fast Fourier transform called the discrete Chebyshev transform, see for instance here for details. We used this technique to apply the ReLU activation function.

For the softmax, we chose a slightly less faithful implementation, by applying the softmax operation row-wise directly to the matrix of coefficients. This is much more efficient than trying to apply it to the function values themselves, and seems to work well in practice. We note that the original purpose of the softmax in the Transformer is to ensure that each row of the ‘attention matrix’ $XQ(XK)^T$ has nicely bounded values, and in the continuous case this is also achieved by applying the softmax to a coefficient vector, since bounded coefficients imply a bounded value of the overall function in a given domain.

At this point it is worth noting that this model is very similar to those constructed in the literature of neural operators [arXiv:2108.08481] [arXiv:2003.03485]. In particular a model called the Fourier neural operator [arXiv:2010.08895] is a quite similar idea, replacing Chebyshev series with Fourier series. It is a good sanity check to see that our generic approach has yielded a model that has been previously considered and is regarded to be useful in practice, and it suggests that continuing this line of thought may lead to more useful models!

With these details worked out, it is possible to build this model explicitly in a machine learning framework such as PyTorch - we have a very basic implementation available on GitHub. As a very simple test, we can try the following task - given a function $f(x, y)$ we can define Poisson’s equation:

\[\nabla^2 g(x, y) = \left(\frac{\partial g}{\partial x}\right)^2 + \left(\frac{\partial g}{\partial y}\right)^2 = f(x, y)\]

This is a partial differential equation that can be solved numerically via a variety of strategies (for instance, finite difference or collocation approaches). The right hand side $f(x, y)$ is conventionally known as the ‘source term’, and the solution $g(x, y)$ is sometimes called the ‘potential’. Since the process of solving this equation can be thought of as a map from source term to potential, then rather than solving this equation numerically, we will train a Funcformer model to do it for us! Note that this is not training a model to approximate the solution to a particular instance of Poisson’s equation but a more general map from source terms to potential, as in the theory of operator learning [arXiv:1910.03193].

We trained the model on random smooth functions (i.e the highest-order Chebyshev coefficients were fixed to zero) with fixed boundary conditions of $f(x, \pm 1) = f(\pm 1, y) = 0$, and it appears to converge well. See below for an example solution, and see YouTube for a video of how this example evolves throughout training.

Part 6: A generalized Transformer zoo

However, CVector is not the only alternative applicative functor we could use - let’s run through some more that might be interesting! Before we start, it’s worth pointing out that while we are using functional programming as a way to express these ideas, in practice these models can and should be implemented in an imperative language using a machine learning framework, as for any other model.

Expression trees and DAGs

Suppose we have a binary tree datatype defined like

data Tree l = Node (Tree l) (Tree l) | Leaf l

then this is indeed an applicative functor:

instance Functor Tree where
    fmap f (Node left right) = Node (fmap f left) (fmap f right)
    fmap f (Leaf x) = Leaf $ f x

instance Applicative (Tree op) where
    pure x = Leaf x
    pair (Node la ra) (Node lb rb) = Node (pair la lb) (pair ra rb)
    pair (Node l r) (Leaf x) = Node (fmap (,x) l) (fmap (,x) r)
    pair (Leaf x) (Node l r) = Node (fmap (x,) l) (fmap (x,) r)
    pair (Leaf x) (Leaf y) = Leaf (x, y)

We can think of the pair operation here as mapping two trees to their common refinement (i.e the smallest tree containing them both) by expanding leaves to subtrees where necessary, and then zipping the two trees together. Given any method of combining two leaves (for example, addition), we can define a counit for trees by combining leaves from the bottom-up:

instance Ring a => Counit Tree a where
    counit (Node l r) = counit l ~+~ counit r
    counit (Leaf x) = x

instance (Ring a, Applicative f) => Counit Tree (f a) where
    counit (Node l r) = uncurry (~+~) <$> pair (counit l) (counit r)
    counit (Leaf x) = x

We have previously tried to implement this in practice, but as you would expect it is very hard to do efficiently and especially in a GPU-accelerated way!

It is important to note that with this definition, all the structure of the trees has to be present in the input data, in some form - the generalized Transformer model cannot be used to, for example, infer a tree structure from data (at least not directly), because the output tree will always be the common refinement of the inputs - what the model is learning to modify is the values of the leaves.

There are two obvious extensions to this: general expression trees, where the operations used to combine each node may be stored in the node itself, and directed-acylic graphs, where the node combination procedes from sources to sinks. Various applications of this come to mind, for example, code synthesis and comprehension, or structured data retrieval.

Probability distributions

It is well known that you can form a monad from probability distributions (indeed, finite discrete distributions are essentially a variant of the list monad discussed earlier), and since all monads are applicative functors, this is sufficient for our purposes. We can write

data Distribution a = Distribution [(a, Float)]

to represent a finite discrete distribution, and we assume that it is constrained so that the probabilities can only take positive values, and sum to one. This is indeed a functor as expected:

instance Functor Distribution where
    fmap f (Distribution dist) = Distribution $ map (\(x, p) -> (f x, p))

instance Applicative Distribution where
    pure x = Distribution [(x, 1.0)]
    pair (Distribution a) (Distribution b) = 
        Distribution $ zipWith (\(x, px) (y, py) -> ((x, y), px * py)) a b

This would be very inefficient, and in practice, you should perform some kind step to combine equal outcomes in the distribution (but this is not permitted by the type signature of Applicative since it would require an equality typeclass constraint or similar). Indeed, this restriction is only due to the confines of expressing these operations in Haskell (where all functors must be endofunctors on $\mathrm{Hask}$), certainly continuous probability distributions also form a monad, and thus an applicative functor. For instance, if $X \sim D$ is a random variable, then the random variable $f(X) \sim f(D)$ should be well-defined if the codomain of $f$ is measurable. We would like to write this

data Distribution a = Distribution { pdf :: a -> Float }

instance Functor Distribution where
    fmap f dist = Distribution { pdf = ... }

but it is not possible to implement Functor here in a sensible way. This is possible with more advanced types, see for instance the monad-bayes package. Counit could be defined in many ways, but taking the expectation value of the distribution would appear to be a somewhat canonical choice:

instance Counit Distribution Float where
    counit (Distribution dist) = sum $ map (uncurry (*)) dist

To make this practical, we could imagine parameterizing the space of all probability distributions either with finite discrete distributions as we defined above, or by some finite-dimensional representation such as a mixture of Gaussians. Furthermore, probability distributions over probability distributions (in the sense of Distribution (Distribution Float)) could be modeled using Gaussian processes or similar. The natural use-case would seem to be something like Bayesian inference or regression problems with uncertainties.

Finally, we can mention briefly that a generalization of Distribution to

data Quantum a = Quantum [(a, Complex Float)]

permits the expression of quantum machine learning models in this framework, as they can be viewed as a kind of generalized probabilistic theory with complex-valued probabilities called amplitudes. Functor and Applicative would be defined exactly as before, but the counit is changed to be the expected value of the distribution given by Born’s rule (that is, each probability is the squared magnitude of the corresponding amplitude):

instance Counit Quantum Float where
    counit (Quantum dist) = sum $ map (\(x, a) -> x * (magnitude a)^2) dist

Note that the generalized Transformer models defined in this way probably cannot be implemented efficiently on quantum computers.

Cross-modal models

Our applicative definition of attention is given by

attention :: (
    Applicative f, Applicative g, Ring a, Counit g a, Counit f (g a)
) => (f a -> f a) -> f (g a) -> f (g a) -> f (g a) -> f (g a)
attention softmax queries keys values = attMatrix `mulMM` values
    where attMatrix = fmap softmax (queries `mulMMT` keys)

which does not constrain the two ‘dimensions’ of the input (f and g) beyond that they must be applicative functors - thus they don’t have to be the same! So our model easily generalizes then to various combinations of the functors we have already defined - for instance, we could consider inputs that are typed like vectors of functions, trees of probability distributions, and so on.

However, we can go further - note that we can actually make the type-signature of attention significantly less restrictive without changing its definition:

attention :: (
    Applicative f, Applicative g, Applicative h, Applicative i, 
    Ring a, Counit h a, Counit f (g a)
) => (f a -> f a) -> f (h a) -> i (h a) -> f (g a) -> i (g a)
attention softmax queries keys values = attMatrix `mulMM` values
    where attMatrix = fmap softmax (queries `mulMMT` keys)

In this definition the queries and keys may come from an entirely different source than the values, and have a completely different type! This is similar in spirit to the cross-attention operation used in encoder-decoder architectures, where the keys and values may come from text in one language (for example) and the queries come from a different stream that represents text in another language. This is also used for multi-modal models that incorporate both text and image inputs. We can imagine building generalized multi-modal Transformers that can incoporate not just different streams of similarly typed data, but also distinctly typed data, for instance transforming text to trees (e.g parsing) or images to functions (e.g resolution-independent image upscaling).

Part 7: Conclusion

To summarize, we have observed that the Transformer, which is a model that maps sequences of vectors to sequences of vectors, can be generalized to map (almost) arbitrary applicative functors to applicative functors. It can also be described in a purely diagrammatic way based on tube diagrams. Then, we showed how a model based on the CVector applicative functor (that is related to the Reader monad) can be implemented in practice, and yields a model similar to those considered in the neural operator literature. Finally, we suggested some other functors that may yield interesting models, and considered how multi-modal models come about as a natural consequence of our framework.

Regarding future work: although in this post we’ve talked mainly about transformers, all the pieces are there to build many other architectures – for instance, recurrent neural networks. Additionally, while interesting in theory, serious engineering effort (i.e custom GPU kernels) would be required to develop any of these models beyond the toy-scale, because the foundational building block of performant machine learning on modern GPUs - linear algebra, and especially matrix multiplication - is what we are generalizing! Unfortunately, until that work is done, it isn’t obvious that these models perform better in practice than the normal Transformer. However, if a set of efficient kernels were developed for each applicative functor of interest (say trees, vectors, and functions), then you can imagine being able to plug them together to generate efficient models for any combination of types.

In discussions we called them “black-boxes” among ourselves, but we coloured them red because we worked on blackboards, and so could not have black. So now red is the new black. This continues an Oxonian string-diagram tradition of picking bad colour conventions on the basis of what writing instruments were readily available. ↩

Bidirectional Typechecking is Bidirectional

2025-01-28T00:00:00+00:00

For a while I’ve had an intuition that typechecking should be optical, with subterms going forwards and types (and type errors) for subterms going backwards. In this post I’ll make sense of this idea, by implementing a bidirectional typechecker for simply typed lambda calculus as an optic, using Haskell’s Control.Lens.

There is basically nothing here that is specialised to typechecking, bidirectional or otherwise, it is merely a neat and well-specified example of a tree search problem and makes for a funny title. This is a hammer I’ve been weighing up for quite a long time for a wide variety of applications. One fun thing I want to work out as a proof of concept is scripting NPC AI for videogames, or as I put it for the meme, “Metal Gear Solid is lenses” (stealth games being an example where the AI behaving realistically is more important than usual).

Bidirectionally typed STLC

The starting point of bidirectional typechecking is to partition the terms of the language into two classes, the synthesisable terms (whose type can be algorithmically inferred) and checkable terms (whose type can be algorithmically checked). In STLC, a term that is a variable $x$ is synthesisable, an application $tu$ is synthesisable when $t$ is synthesisable and $u$ is checkable, and an abstraction $\lambda x . t$ is checkable when $t$ is checkable (note that the abstracted variable $x$ does not carry a type annotation, unlike in some presentations of STLC). Also, any synthesisable term can be considered as checkable, and any checkable term can be considered as synthesisable if paired with a type annotation.

In Haskell this is nothing fancy:

data Type = TVar String | Unit | Function Type Type
  deriving (Eq, Show)

data TermSyn = Var String | App TermSyn TermChk | Down Type TermChk
  deriving (Show)

data TermChk = Lambda String TermChk | Up TermSyn
  deriving (Show)

Let’s write the rules of bidirectional STLC in traditional programming languages notation. If $t$ is a synthesisable term and synthesising its type returns the result $a$ then we write $t \in a$. If $t$ is a checkable term and we successfully check that $t$ has type $a$ then we write $a \ni t$. (The symbol $\ni$ is pronounced “ni”, and also written that way in LaTeX/MathJax, hence the paper The Types Who Say Ni by Conor Mc Bride.) We also say “the type $a$ accepts the term $t$”.

$\frac{}{\displaystyle \Gamma, x : a \vdash x \in a}\text{(Var)}$

To synthesise the type of a variable $x$ we simply look it up in the context and return what the context says its type is

$\frac{\displaystyle \Gamma \vdash t \in a \to b \qquad \Gamma \vdash a \ni u}{\displaystyle \Gamma \vdash tu \in b}\text{(App)}$

To synthesise the type of an application $tu$, we synthesise the type of $t$, find that the result is a function type $a \to b$, check that $u$ has type $a$, and return $b$

$\frac{\displaystyle \Gamma, x : a \vdash b \ni t}{\displaystyle \Gamma \vdash a \to b \ni \lambda x . t}\text{(Lam)}$

To check that $\lambda x . t$ has type $a \to b$, we check that $t$ has type $b$ after extending the context with $x : a$

$\frac{\displaystyle \Gamma \vdash a \ni t}{\displaystyle \Gamma \vdash \downarrow t : a \in a}\text{(Down)}$

To synthesise the type of a coerced checkable term $t$ paired with a type annotation $a$, we check that $t$ has type $a$, then return $a$

$\frac{\displaystyle \Gamma \vdash t \in b \qquad a = b}{\displaystyle \Gamma \vdash a \ni \uparrow t}\text{(Up)}$

To check that a coerced synthesisable term $t$ has type $a$, we synthesise the type of $t$ and make sure that the result is equal to $a$

Questions and answers

To arrange all of this into an optic, we need to go back to one of the earliest ideas for using lens-like things in strange ways, in this classic paper by Blass, which is to have a boundary made of questions and answers. We ask an optic a question, it can in turn ask questions on its opposite boundary, it receives answers, and then finally it answers to us. The questions we can ask here are “what is the type of this synthesisable term?” and “does this checkable term have this type?”:

type Context = [(String, Type)]

data Question = Synthesise Context TermSyn | Check Context Type TermChk
  deriving (Show)

data Answer = Synthesised Type | Checked | TypeError
  deriving (Show)

For now I have here chosen that “type error” carries no additional information, but the question of how best to handle the sad path is one we’ll come back to later because it creates a tension between theory and implementation.

Here is an example of how such a “question-and-answer protocol” is implemented using a functor lens in Haskell, illustrated using the (Lam) rule:

lam :: forall f. (Functor f) => (Question -> f Answer) -> Question -> f Answer
lam k (Check ctx (Function a b) (Lambda x t)) = k (Check ((x, a) : ctx) b t)

I find the CPS-like van Laarhoven representation helpful here (for once!) by reflecting the protocol directly. We ask the lens the question Check ctx (Function a b) (Lambda x t), it asks its continuation the question Check ((x, a) : ctx) b t, it gets back the answer, and then it returns the answer to us unchanged.

There is a very obscure but very useful bit of intuition that I learned from Bob Atkey in this seminar, which is that superclasses of Lens differ in the quantity that they have access to their continuation. A lens must call its continuation exactly once, an affine traversal can call its continuation at most once, and a traversal can call its continuation any finite number of times. In the van Laarhoven encoding, a lens must call its continuation at least once because there is no other way to get a value in the unknown functor f, and there is also no way to combine the result of two different calls so it can only return one of them. For affine traversals we use pointed functors, which allow us to get into the functor without calling the continuation but still do not allow us to combine the result from two different calls. For traversals we use applicatives, which do both.

All this is to say, I expected the (App) rule, which has 2 hypotheses, to be a Traversal, ie. something with type

app :: forall f. (Applicative f) => (Question -> f Answer) -> Question -> f Answer

But when I came to write it, I reached an unexpected subtlety. Traversals can call their continuation any number of times, but those calls must be independent. The (App) rule, which must answer the question “what is the type of $tu$?”, first asks the question “what is the type of $t$?”, and then upon receiving the answer “$a$”, it asks the followup question “does $u$ have type $a$?”. That is, the second question depends on the first answer, and this is something that is not possible with a Traversal.

Monadic traversals

The type signature needed to express the (App) rule is something that, at first sight, looks like an extremely obvious superclass of Traversals:

app :: forall f. (Monad f) => (Question -> f Answer) -> Question -> f Answer
app k (Synthesise ctx (App t u)) = do
  Synthesised (Function a b) <- k (Synthesise ctx t)
  Checked <- k (Check ctx a u)
  return (Synthesised b)

The appearance of a on the second line of the do block is exactly the thing that can’t be done with an Applicative.

This class of optics apparently has no name, although I few people have suggested the idea. I already use the terms “monadic lens” and “monadic optic” already (respectively for lenses whose backward pass is in a kleisli category and monoidal optics entirely in a kleisli category) - I like Zanzi’s suggestion to call this a monadic traversal. Apparently there are no nontrivial examples of lawful monadic traversals¹, that is to say, every monadic traversal satisfying the traversal laws is already an ordinary traversal. Fortunately, my entire career has been spent spitting in the face of the optic laws. More to the point, I’m fairly sure these question-and-answer optics can never be meaningfully lawful even in the most generalised sense, because an answer does not determine a canonical followup question, but they are still extremely important for many applications.

I conjecture that the optic

type MonadicTraversal s t a b = forall f. (Monad f) => (a -> f b) -> s -> f t

is isomorphic to this concrete implementation:

type MonadicTraversal s t a b = s -> Interaction a b t

data Interaction a b t = Done t
                       | More a (b -> Interaction a b t)

An Interaction a b t can either terminate with an answer of type t, or it can ask a question of type a, and for each answer (of type b) it has a “continuation interaction”. This is the equivalent for monadic traversals of what for ordinary traversals is called a FunList (also known as a Bazaar in its functor encoding).

Putting it together

With all this setup, here is the full implementation of a typechecker as a monadic traversal:

rules :: MonadicTraversal Question Answer Question Answer
rules _ (Synthesise ctx (Var x)) = case lookup x ctx of
    Just a -> return (Synthesised a)
    Nothing -> return TypeError
rules k (Synthesise ctx (App t1 t2)) = k (Synthesise ctx t1) >>= \case
    Synthesised (Function a b) -> k (Check ctx a t2) >>= \case
        Checked -> return (Synthesised b)
        _ -> return TypeError
    _ -> return TypeError
rules k (Synthesise ctx (Down a t)) = k (Check ctx a t) >>= \case
    Checked -> return (Synthesised a)
    _ -> return TypeError
rules k (Check ctx (Function a b) (Lambda x t)) = k (Check ((x, a) : ctx) b t)
rules _ (Check _ _ (Lambda _ _)) = return TypeError
rules k (Check ctx a (Up t)) = k (Synthesise ctx t) >>= \case
    Synthesised b -> if a == b then return Checked else return TypeError
    _ -> return TypeError

This optic implements typechecking provided it is post-composed with another optic that implements typechecking for subterms. Any actual lambda term has subterms nested finitely but arbitrarily deep, so what we need to do is to compose this optic with itself and take a fixpoint, remembering that van Laarhoven encoded optics compose by reverse function composition:

stlc :: MonadicTraversal Question Answer Question Answer
stlc = rules . stlc

Personally I find this infinite composition chain surprisingly similar to the way we implemented Bellman iteration as optic composition.

Now we can write a typechecker by calling this lenses using standard Control.Lens combinators:

typecheck :: Question -> Answer
typecheck q = q & stlc .~ TypeError

And this works! Here is what it looks like running on the lambda term $(\lambda y . y) x$ in the context $x : a$:

ghci> typecheck $ Synthesise [("x", TVar "a")] 
                $ App (Down (Function (TVar "a") (TVar "a")) (Lambda "y" (Up (Var "y")))) 
                      (Up (Var "x"))
Synthesised (TVar "a")

The backward input TypeError in the function typecheck, which comes at the end of the infinite composition chain, is never actually used: we only go as far down the chain as the subterm depth of our lambda term before returning. It can be changed to undefined and everything still works correctly. This is almost exactly what happens with Bellman iteration with lenses, where the backward input is an arbitrary real number that gets infinitely discounted to zero.

Handling the sad path

In theory we are done, but in practice we are handling type errors in a very bad way: about half of the implementation is handling type errors, and despite this we are still collapsing all type errors to a single value instead of returning any useful information to the caller.

Just for fun, here is an idea I had that fixes the first problem while ignoring the second, by heavily abusing how Haskell desugars do-notation. There is a class in Prelude called MonadFail that exists for legacy reasons, and adds to the monad signature a function fail :: String -> m a. When Haskell encounters certain syntactic forms of possibly-failing pattern match in a do-block, it desugars it to MonadFail, filling the string with information about the failed match.

Any reader who was unhappy about monadic traversals should probably skip the next block of code, because I am about to deploy an optic for MonadFail. It’s fair to say that this is one of the more horrifying things I have ever done in Haskell.

The caveat is that this “feature” appears to be only half finished, or maybe more likely has only been half removed: pattern matches on the left of a <- desugar to fail, but pattern matches in a let form instead desugar to Prelude.error. So we have to transform let x = y into x <- return y for this to work.

rules :: forall f. (MonadFail f) => (Question -> f Answer) -> Question -> f Answer
rules _ (Synthesise ctx (Var x)) = do
    Just a <- return (lookup x ctx) 
    return (Synthesised a)
rules k (Synthesise ctx (App t u)) = do
    Synthesised (Function a b) <- k (Synthesise ctx t)
    Checked <- k (Check ctx a u)
    return (Synthesised b)
rules k (Synthesise ctx (Down a t)) = do
    Checked <- k (Check ctx a t)
    return (Synthesised a)
rules k (Check ctx a (Lambda x t)) = do
    Function b c <- return a
    k (Check ((x, b) : ctx) c t)
rules k (Check ctx a (Up t)) = do
    Synthesised b <- k (Synthesise ctx t)
    if a == b then return Checked 
              else fail $ unwords ["Could not match", show a, "and", show b]

stlc :: forall f. (MonadFail f) => (Question -> f Answer) -> Question -> f Answer
stlc = rules . stlc

typecheck :: Question -> Maybe Answer
typecheck = stlc (const Nothing)

The last part of this is using the fact that Maybe has a builtin instance of MonadFail (which ignores the error string).

Back in the world of actual engineering instead of memeing, my best idea for the right way to deal with the unhappy path is to define a TypeError type, which can be as informative as we want, and use an optic for the class Control.Monad.Except.MonadError TypeError from mtl. And, obviously, we do the work of explicitly detecting and throwing informative type errors instead of the preceeding silliness. I haven’t actually built it because I’m not entirely convinced that optics for a class from mtl is actually a good idea, but so far I do not have any better ideas.

Pipelines

Recently André wrote a post about compiler pipelines, and is working on a followup post doing the same thing with lenses. For example, typechecking is one stage of a compiler pipeline that is preceeded by scopechecking and is followed by code generation.

What I have done in this post throws a spanner into that, because I am using optics that are infinite composition chains that never actually reach their codomain boundary, so it doesn’t make sense to compose them end-to-end.

My gut feeling is that there should be some (asymmetric) monoidal product on traversals that is the one true way of sequencing things defined in this way. But I haven’t found it yet, so instead I’m going to show something that works but I don’t think is right.

Suppose as well as typechecking we have another monadic traversal of this type:

data ScopeCheckQuestion = ScopeCheckQuestion
data ScopeCheckAnswer = WellScoped | ScopeError

scopecheck :: MonadicTraversal ScopeCheckQuestion ScopeCheckAnswer 
                               ScopeCheckQuestion ScopeCheckAnswer

We need to take the coproduct of the scopechecking and typechecking traversals. Coproducts of optics require dependent types to write correctly, but here is the closest we can get in Haskell, with possibly-failing pattern matches that the type is unable to rule out:

plus :: (Functor f)
     => ((a -> f b) -> s -> f t) -> ((a' -> f b') -> s' -> f t')
     -> (Either a a' -> f (Either b b') -> Either s s' -> f (Either t t'))
plus l _ k (Left s) = Left <$> l (\a -> fmap (\(Left s) -> s) (k (Left a))) s
plus _ l' k (Right s') = Right <$> l' (\a' -> fmap (\(Right s') -> s') (k (Right a'))) s' 

Now we can precompose this with a monadic traversal that calls the two parts in the right order:

data OverallAnswer = ScopeError' | TypeError' | Type Type

pipeline :: MonadicTraversal (Context, TermSyn) 
                             OverallAnswer
                             (Either ScopeCheckQuestion Question) 
                             (Either ScopeCheckAnswer Answer)
pipeline k (ctx, t) = k (Left ScopeCheckQuestion) >>= \case
  Left ScopeError -> return ScopeError'
  Left WellScoped -> k (Right (Synthesise ctx t)) >>= \case
    Right TypeError -> return TypeError'
    Right (Synthesised a) -> return (Type a)

I’m not quite happy with this, even after putting the missing dependent types back in, but it does work. I’m going to leave it for the future to find a more canonical way to sequence pipeline steps.

Thanks to @effectfully on twitter for this link ↩

Optics for UI 1: Deconstructing React with Parametrised Lenses

2025-01-21T00:00:00+00:00

Lenses model a new paradigm of programming based on bidirectional processses. Recently André Videla has figured out how use them to implement RESTful servers, and as we will see in this blog post, they can also capture UIs.

In this blog post we will look at UI frameworks like React and The Elm Architecture and see how to express them using parametrised lenses.

This is not the first time that lenses were applied to UIs, however there’s been a lot more work on understanding the category theory of lenses since then, especially parametrised lenses, and we will see how this new understanding gives us even more expressive power than before.

Bidirectional programming for component-based architectures follows a simple design pattern - information flows from parent to child components, and actions flow back from child to parent.

This is already different to how older top-down software architectures worked, but is in line with how modern React-based UI frameworks mediate between parent and child components.

Stateless components

In React terminology the signals flowing from parent to child are called “props”, and we can use callbacks to pass the response from a child to parent.

In addition to props, React components have a notion of internal state, which remains hidden from other components. The key idea is that rather than exposing the state itself, a component will only expose props derived from the state, which is what the child components will use in their logic.

If we take a stateless view, our picture would look somewhat like this diagram of lenses:

$\binom{\displaystyle \text{WebApp props}}{\displaystyle \text{WebApp action}} \leftrightharpoons \binom{\displaystyle \text{List props}}{\displaystyle \text{List action}} \leftrightharpoons \binom{\displaystyle \text{List item props}}{\displaystyle \text{List item action}}$

Looking at it this way, we can see that this describes a bidirectional process. A web-app component will have access to the top-level prop, it will then pass a list to the list component, and the list component will pass each individual item to its own child component. Whenever an item is clicked, it will then pass on an action upwards up the hierarchy.

The forwards pass is determined by the flow of information from the parent prop, and the backwards pass is determined by the child responding to the parent.

The Elm Architecture

Of course in a functional world, with neither state nor effects, this becomes rather trivial. Props are immutable signals, so a child component can’t ask the parent to actually change a prop. We would either need to make the information contained in the props mutable, or add some kind of notion of effect.

Because this picture is rather limiting, languages like Elm take an entirely different approach and take state as primitive. A basic component in Elm would have a view function and an update function, and it’s not hard that the two form a lens:

view : Model -> Html Msg 
update : Model -> Msg -> Model 

In fact, because the update function takes a model to a model, we can see that this is actually a special kind of a lens, known as a Moore machine.

$\text{Moore} : \binom{\displaystyle \text{state}}{\displaystyle \text{state}} \leftrightharpoons \binom{\displaystyle \text{output}}{\displaystyle \text{input}}$

view : state -> output 
update : state -> input -> state

Moore machines give you a lot of expressivity, but they also have a significant limitation - they don’t compose by lens composition. This means that if we have two separate components it’s easy to put them side-by-side using parallel composition, but there’s no way to talk about sub-components of a bigger component.

(Moore machines can compose with lenses to the left- and right- though, something we’ll talk about in a future blog post on reparametrisation).

Stateful components

So taking immutable props as primitive causes the framework to trivialise, and taking encapsulated state as primitive causes it to lose compositionality. Is there any way out?

Early experiments with optic-based UI frameworks tried to answer this question by weakening the requirement that state must remain encapsulated within a component. Instead, a parent components would expose parts of its state to its children, which would then be able to act on it.

This means that we would get a picture much like before, but now the state will be mutable.

$\binom{\displaystyle \text{WebApp state}}{\displaystyle \text{WebApp action}} \leftrightharpoons \binom{\displaystyle \text{List state}}{\displaystyle \text{List action}} \leftrightharpoons \binom{\displaystyle \text{List item state}}{\displaystyle \text{List item action}}$

The nice thing about this approach is that components can now be composed using lens composition.

Let’s say we have the following interfaces:

AppComponent = (AppState, AppUpdate) 
ListComponent = (ListState, ListUpdate)
ItemComponent = (ItemState, ItemUpdate)

The key to translating from this into the language of lenses is to start by inverting the control. The component-based view takes components as primary and treats the information flow between them as implicit. What lenses do is to make this information flow into a first-class citizen. This is why instead we name the lenses and treat the components as implicit:

$\text{ListLens} : \text{AppComponent} \leftrightharpoons \text{ListComponent}$

$\text{ItemLens} : \text{ListComponent} \leftrightharpoons \text{ItemComponent}$

with the corresponding getters and setters

ListLens : Lens AppComponent ListComponent
fwd1 : AppState -> ListState 
bwd1 : AppState -> ListUpdate -> AppUpdate 

ItemLens : Lens ListComponent ItemComponent
fwd2 : ListState -> ItemState 
bwd2 : ListState -> ItemUpdate -> ListUpdate 

Their composition will give you the lens

ListLens . ItemLens : Lens AppComponent ItemComponent

In other words, knowing how to propagate information from one parent to a child, we get the propagation along a hierarchy.

Unlike the Elm model, this gives you hierarchical composition of components. But now we’ve lost one of the core principles of software development by exposing state across the entire application.

Is there a way to get the best of both worlds, to retain hierarchical composition of components while encapsulating state?

As we’ve seen so far, we want components to compose like lenses, but retain internal state. Is there a notion of “Lens with an internal state”? This is exactly what the Para construction does.

It turns out that there is, using the Para(Optic) construction that we’ve seen in a previous blog post. What’s more surprising, is that this construction gives you back exactly the React model.

React was right all along

How should state be organized? This is one of the fundamental questions of UI development. React takes this question further and asks “How should information flow within an application be structured?”

So far the models we’ve looked at have either worked with immutable props or mutable state. But what if we work with both?

React’s big insight was that a component has access to its own props and state, and the information it passes to its children is immutably derived from these:

fwd : (ParentProp, ParentState) -> ChildProp

On the other hand, when a component receives an update from a child, it now does two things: it can pass along a request to its own parent, or it can update its internal state.

bwd : (ParentProp, ParentState, ChildAction) -> GrandParentAction 

update : (ParentProp, ParentState, ChildAction) -> ParentState 

If we combine backwards and update into a single function,

bwd' : (ParentProp, ParentState, ChildAction) -> (GrandParentAction, ParentState)

we will see that this is exactly the same as the Para construction over lenses, which gives us the morphisms:

fwd : (a, p) -> b 
bwd : (a, dp, db) -> (da, dp)

More importantly, composition of parametric lenses gives us composition of stateful components:

(These diagrams were made with Tangle).

Conclusion

We’ve looked at how we can use lenses to model several popular UI frameworks, as well as seen how the lens-based approach has evolved over the years. Our main takeaway is that lenses by themselves have always had something missing that kept them from being a truly powerful approach to UIs. And the claim that will be made in the subsequent posts in this series is that Para is exactly the structure that we’ve been missing. In the next few posts we will also see that Para is not only good for modelling state boundaries, but can be applied to modeling the boundaries of user input/output as well as async calls.

Programming Pipelines Using Dependent Types

2025-01-13T00:00:00+00:00

Sometimes, writing a large program is conceptually as simple as translating from a big unstructured input into a more and more structured output. A good example is a single-pass compiler:

First we perform lexical analysis on a big string, this gives us a list of tokens.
Then we parse the tokens into a tree.
Then we do some semantic analysis on the tree, usually this involves some typechecking and error reporting
Once we know what the program is, we generate code for it.

Introducing the pipeline

This structure is purely linear, and functional programmers will be delighted to point out that you can implement an entire compiler pipeline using nothing but function composition:

codegen :: Sema -> String
typecheck :: Tree -> Sema
parsing :: [Token] -> Tree
lexing :: String -> [Token]

compiler :: String -> String
compiler = codegen . typecheck . parsing . lexing

Of course this is pure fantasy, there is no error reporting, no way to debug it, and no command line tool. But the core idea is compelling: what if there was a way to implement a compiler as a single pipeline of operation that clearly describes what each step achieves? For this, we’re going to define pipelined programs using Idris, because it uses dependent types, and it is reasonably fast and production ready. We define a pipeline as a list of Types, each type represents an intermediate layer.

Pipeline : Nat -> Type
Pipeline length = Vect length Type

CompilerPipeline : Pipeline ?
CompilerPipeline =
  [ String
  , List Token
  , Tree
  , Sema
  , String
  ]

With it, we define a compiler pipeline as the list of types String, List Token, Tree, Sema, and String, representing all the stages of our compiler.

The pipeline is indexed by it length, to ensure that our pipeline is valid this length needs to be at least 2, since we need a type from which we start, and a type to which we arrive. Using this fact we can define the implementation for a pipeline.

-- Well formed pipelines have at least 2 elements, the start and the end type
Impl : Pipeline (2 + n) -> Type
Impl (x :: y :: []) = x -> y
Impl (x :: y :: z :: xs) = Pair (x -> y) (Impl (y :: z :: xs))

We can test our implementation by asking in the REPL what is a valid implementation of out compiler pipeline.

> Impl CompilerPipeline
< (String -> List Token, (List Token -> Tree, (Tree -> Sema, Sema -> String)))

This is a 4-tuple of functions String -> List Token, List Token -> Tree, Tree -> Sema and Sema -> String, exactly what we wanted, each of those function represents a stage in our compiler.

Now that Impl correctly gives us the type of an appropriate implementation for a pipeline, we still need to run the pipeline. Running the pipeline should amount to using function composition on each of the functions. Unfortunately it’s not that easy because each intermediate step uses a different type and we can’t just blindly apply a function without knowing its type.

Like before, to run a Pipeline, it needs at least two elements. Running a pipeline should result in a simple function from the first layer of the pipeline to the last one. To write this we use the function head and last from Data.Vect those functions do exactly what you expect from their name but here we call them to define the type of the function that will result from running the pipeline.

-- When we run the pipeline `p` we expect to get out a function `head p -> last `p`,
-- that is, the first stage of the pipeline as the argument of the function and the
-- last stage as the result of it.
Run : (p : Pipeline (2 + n)) -> Impl p -> head p -> last p
-- In the base case, the pipeline contains only one stage x -> y so we return it
Run [x, y] f = f
-- In the inductive case, we compose `f` with the rest of the pipeline
Run (s :: t :: x :: xs) (f, cont) = Run (t :: x :: xs) cont . f

The base case runs the single stage we have with the argument we’re given. The inductive case runs the remainder of the pipeline after running the current stage represented by the function f.

We can test this idea by assuming we have functions lex, parse, typecheck and codegen like in the haskell example and see what happens:

lex : String -> List Token
parse : List Token -> Tree
typecheck : Tree -> Sema
codegen : Sema -> String

runCompiler : String -> String
runCompiler = Run CompilerPipeline (lex, parse, typecheck, codegen)

And it typechecks! We cannot run this code because we don’t have an implementation for lex, parse, etc, but we’ve at least reached our goal: to build a datastructure that help us keep track of what are the stages in our pipeline.

Handling effects

This is quite cool, but it’s not the end. Remember that compiler need to perform a lot of side effects, returning errors, sometimes print out intermediate trees for debugging, how does the pipeline help for that?

Well the great benefit from the pipeline is that we’ve separated the information about the stages from the information about the runtime implementation of the stages. This information is created by Impl which creates the type of an implementation given a pipeline, but this current version only places the types end-to-end without changes. What if we add an extra Monad around the types such that instead of functions a -> b, each stage is now an effectful function a -> m b.

First we define m : Type -> Type, our monad, we use a parameter block because we will use it for running our programs too.

parameters {0 m : Type -> Type}

Then, in the implementation, we use the same type signature, but in the base-case, instead of returning x -> y, we return x -> m y. And similarly with the inductive case.

  0
  ImplM : Pipeline (2 + n) -> Type
  ImplM [x, y] = x -> m y
  ImplM (x :: y :: z :: xs) = Pair (x -> m y) (ImplM (y :: z :: xs))

Like before, we now test ImplM in the repl with a monad Maybe and we get:

> ImplM {m = Maybe} CompilerPipeline
< (String -> Maybe (List Token), (List Token -> Maybe Tree, (Tree -> Maybe Sema, Sema -> Maybe String)))

Now every stage runs in a Maybe monad, and we could replace this monad by anything else and obtain all sorts of effects. It remains to run a pipeline with effects using our effectful implementation. We implement a function just like Run but we make use of kleisli composition instead of function composition.

  RunM : (mon : Monad m) => (p : Pipeline (2 + n)) -> ImplM p -> Vect.head p -> m (Vect.last p)
  RunM [x, y]  f = f
  RunM (x :: y :: z :: xs) (f, c) = f >=> RunM (y :: z :: xs) c

To run the compiler, each stage needs to perform an effect, here we’re using Either String to represent errors:

namespace EffectfulCompiler
  lex : String -> Either String (List Token)
  parse : List Token -> Either String Tree
  typecheck : Tree -> Either String Sema
  codegen : Sema -> Either String String

  runCompiler : String -> Either String String
  runCompiler = RunM CompilerPipeline (lex, parse, typecheck, codegen)

Crucially, we’ve not changed the pipeline at all! We’ve only changed how to interpret it as a runtime program.

Debugging programs

Finally, using the same datastructure we can run our pipeline in debug mode. This mode will print in the terminal all the intermediate steps so that you can see what is happening at each stage. For this to work, we need to make sure that, for all layers in the pipeline, we have a way to print the result of the computation, we achieve this with the All : (ty -> Type) -> Vect n ty -> Type type which takes a list of elements and a predicate over that list, and describes a list of proofs that ensure the predicate hold for every value of ty given. In our case, the predicate given is the Show interface, and so for each type there must exist a Show instance.

The rest of the function is the same as before, except we compose each f with pure . traceValBy show to print the value we just computed.

RunTraceM : Monad m => (p : Pipeline (2 + n)) -> (print : All Show p) => ImplM {m} p -> Vect.head p -> m (Vect.last p)
RunTraceM [x, y] {print = [p1, p2]} f = f >=> pure . traceValBy show
RunTraceM (x :: y :: z :: xs) {print = p :: q :: ps} (f, c) =
  f >=> pure . traceValBy show >=> RunTraceM (y :: z :: xs) c

Calling our pipeline with RunTraceM will print out every intermediate value produced by the pipeline, It won’t print the first argument given to it but since we already have it, we can print it when we call the pipeline.

Conclusion

There is much more to say about this but is already more than we can usually do without dependent types. We can take the pipeline and build multiple runtimes for it, and run it in multiple modes as well. I’ve not shown it here but we can also implement operations on the pipeline like concatenation, splicing, etc. Those operations should reflect what happens at runtime: Concatenating two pipeline should compose two programs that run them. This framework can be extended in many other ways but that will serve as a solid base for now, see you in the next one.

The code is available as a library: https://gitlab.com/glaive-research/pipelines.

Foundations of Bidirectional Programming IV: Running Forwards and Running Backwards

2025-01-03T00:00:00+00:00

See parts I, II and III

We are now at the point where we can write an interpreter to run some programs. To be clear, this is mainly for demonstration and debugging purposes, and the eventual goal is to write an optimising compiler. By the end of this post we will be able to demonstrate interpreting some very simple straight-line differentiable programs, and also demonstrate just how horrendous it is to actually write programs in our kernel syntax. This will motivate several follow-up posts where we slowly work upwards towards a human-usable surface language.

Continuing our well-typed-by-construction methodology we will build a well typed interpreter. This means that we first interpret types of our language into Idris types, and then these give the type of the corresponding interpreted program. For languages less weird than ours, such as one of the example languages from the first post, we would interpret terms as Idris functions, resulting in signatures like this:

EvalType : Ty -> Type
eval : Term xs x -> All EvalType xs -> EvalType x

Instead of interpreting to Idris functions, terms of Aptwe will be interpreted as lenses. For this we need to give 2 different interpretations of each type, one covariant and the other contravariant:

Cov : Ty -> Type
Con : Ty -> Type
eval : Term xs x -> Lens (All Cov xs) (All Con xs) (Cov x) (Con x)

Base types

We need to start by defining Cov and Con, which is completely straightforward. This is the point at which we upgrade our type language to have a proper mechanism for base types (source code is here and here):

data BaseTy : Kind -> Type where
  Nat : BaseTy (True, False)
  Bool : BaseTy (True, True)
  Real : BaseTy (True, True)

data Ty : Kind -> Type where
  BaseTy : BaseTy (cov, con) -> Ty (cov, con)

This gives us 3 base types to play with: naturals (aka. ints), booleans, and reals (aka. doubles). I have made the choice that naturals will be purely covariant, whereas booleans and reals will be bivariant. As we will discover, interpreting a bivariant type amounts to choosing a canonical commutative monoid structure on it, which becomes baked into that type’s scoping rules: every time we do something that looks like shadowing, instead the monoid operation will be applied.

These choices come from key applications: for reals it will be addition which is used in autodiff (see the last section of this post), and for booleans it will be conjunction which is used in compositional game theory (see a future post). Of course addition is also an obvious choice for naturals, but since I don’t know any application for it in bidirectional programming, I have instead made naturals purely covariant.

Base types are also responsible for the only non-obvious part of our interpreter for types:

data Echo : Type where
  X : Echo

mutual
  Cov : Ty k -> Type
  Cov Unit = Unit
  Cov (BaseTy x) = EvalBaseTy x
  Cov (Not x) = Con x
  Cov (Tensor x y) = (Cov x, Cov y)

  Con : Ty k -> Type
  Con Unit = Unit
  Con (BaseTy x) = Echo
  Con (Not x) = Cov x
  Con (Tensor x y) = (Con x, Con y)

The type Echo is an isomorphic copy of Unit. The choice I have made is that base types have a nontrivial interpretation only in the covariant direction. The name echo refers to the indescribable thing that is observed when looking at something travelling through time in the opposite direction to the observer. Its value X ends up proliferating through interpreted terms, but this is only something we’ll have to put up with while using this prototype interpreter.

Interpreting structure

The next thing we need to do is to interpret structure, that is, the data structure here that is used in the Rename case of terms. In principle a renaming could be interpreted as a lens, but it transpires that renamings only ever mean lenses whose backwards pass does not use its forwards input (I call these adaptors). This means that we actually need to write 2 functions:

structureCov : Structure xs ys -> IxAll Cov xs -> IxAll Cov ys
structureCon : Structure xs ys -> IxAll Con ys -> IxAll Con xs

All of the source code from this section can be found in this file.

The hard part of this is to interpret the 4 operations of delete, copy, spawn and merge. Interpreting delete and copy of covariant values is easy, since it ends up being delete and copy of Idris values. Similarly interpreting spawn and merge of contravariant values also ends up being delete and copy of Idris values. So what we need to do is to spawn and merge covariant values, and to delete and copy contravariant values, and this is where we need to make a choice of a monoid structure for each base type:

unit : {x : BaseTy (cov, True)} -> Cov (BaseTy x)
unit {x = Bool} = True
unit {x = Real} = 0

multiply : {x : BaseTy (cov, True)} 
        -> Cov (BaseTy x) -> Cov (BaseTy x) -> Cov (BaseTy x)
multiply {x = Bool} p q = p && q
multiply {x = Real} p q = p + q

Now we can delete and spawn because unit takes care of the missing base case for base types:

mutual
  spawnCov : {x : Ty (cov, True)} -> Cov x
  spawnCov {x = Unit} = ()
  spawnCov {x = BaseTy x} = unit
  spawnCov {x = Not x} = deleteCon

  deleteCon : {x : Ty (True, con)} -> Con x
  deleteCon {x = Unit} = ()
  deleteCon {x = BaseTy x} = X
  deleteCon {x = Not x} = spawnCov

The case for spawning or deleting a tensor product should be

  spawnCov {x = Tensor x y} = (spawnCov, spawnCov)

But instead this case takes us into a very unfortunate corner case of Idris syntax, which I’m not going to attempt to explain because I only partially understand what’s going on:

  spawnCov {x = Tensor {con = (True ** and)} x y} with (and)
    spawnCov {x = Tensor {con = (True ** and)} x y} | True 
      = (spawnCov, spawnCov)

The cases for copy/merge are almost identical. Now we can write the functions we need, for which the interesting cases are as follows:

structureCov : Structure xs ys -> IxAll Cov xs -> IxAll Cov ys
structureCov (Delete f) (x :: xs) = structureCov f xs
structureCov (Copy e f) xs = ixSelect e xs :: structureCov f xs
structureCov (Spawn f) xs = spawnCov :: structureCov f xs
structureCov (Merge e f) (x :: xs) = applyAt e (mergeCov x) (structureCov f xs)

structureCon : Structure xs ys -> IxAll Con ys -> IxAll Con xs
structureCon (Delete f) ys = deleteCon :: structureCon f ys
structureCon (Copy e f) (y :: ys) = applyAt e (copyCon y) (structureCon f ys)
structureCon (Spawn f) (y :: ys) = structureCon f ys
structureCon (Merge e f) ys = ixSelect e ys :: structureCon f ys

Observe that the covariant delete case and the contravariant spawn case become a deletion in Idris, meaning that the head of the list does not appear on the right hand side. Similarly the cases for covariant copy and contravariant delete use the helper function ixSelect, which in the end is copying a value from the middle of the list to its head. This leaves the remaining cases to call our 4 helper functions: covariant spawning and merging, and contravariant deleting and copying.

Writing the interpreter

Now we come to the main thing: writing the function

eval : Term xs y -> IxAll Cov xs -> (Cov y, Con y -> IxAll Con xs)

The complete implementation is here. Let’s start with a couple of very easy cases to warm up:

eval (BaseTerm t) xs = evalBaseTerm t xs
eval Var [x] = (x, \y' => [y'])
eval UnitIntro [] = ((), \() => [])

The case for renaming is not much harder, since we spent the whole of the previous section writing the helper functions it needs:

eval (Rename f t) xs = let (y, k) = eval t (structureCov f xs)
                        in (y, \y' => structureCon f (k y'))

Probably the most instructive case is the one for Let. When writing an ordinary interpreter of terms into functions Let becomes function composition, or slightly more precisely substitution into one input of a many-input function. For us, Let becomes lens composition. The one complication, which also happens in most of the remaining cases, is that we need to use the simplex carried by the proof rule to tell us how to pull apart the input list into two, and then stitch it back together.

eval (Let {cs = (_ ** _ ** s)} t1 t2) xs
  = let (y, k1) = eval t1 (ixUncatL s xs)
        (z, k2) = eval t2 (y :: ixUncatR s xs)
     in (z, \z' => let y' :: ys' = k2 z'
                    in ixConcat s (k1 y') ys')

The subtlety of this case is just the subtlety of lens composition: the first output of t1 in the forwards direction becomes an input of t2 in the forwards direction, and then the continuations are unwound in reverse order for the backwards direction. The helper functions

ixConcat : IxSimplex as bs cs -> IxAll q as -> IxAll q bs -> IxAll q cs
ixUncatL : IxSimplex as bs cs -> IxAll q cs -> IxAll q as
ixUncatR : IxSimplex as bs cs -> IxAll q cs -> IxAll q bs

which can be found in IxUtils, respectively stitch or unstitch a pair of indexed lists as directed by a simplex.

Next let’s look at the introduction and elimination rules for Tensor:

eval (TensorIntro {cs = (_ ** _ ** s)} t1 t2) xs 
  = let (y1, k1) = eval t1 (ixUncatL s xs)
        (y2, k2) = eval t2 (ixUncatR s xs)
     in ((y1, y2), \(y1', y2') => ixConcat s (k1 y1') (k2 y2'))
eval (TensorElim {cs = (_ ** _ ** s)} t1 t2) xs 
  = let ((y1, y2), k1) = eval t1 (ixUncatL s xs)
        (y2, k2) = eval t2 (y1 :: y2 :: ixUncatR s xs)
     in (y2, \y' => let x1' :: x2' :: xs' = k2 y'
                     in ixConcat s (k1 (x1', x2')) xs')

The case for TensorIntro is nothing but the tensor product of two lenses, and the elimination rule is extremely similar to Let but binds two variables at once, namely the two halves of the product being eliminated.

This marks the dividing line between the cases I was able to understand before writing them, and the ones for which I really relied on the Idris type checker for help. What remains is the unit elimination rule, and the rules for negation. In an ordinary language the tensor elimination rule is very easy: any term that returns a unit can be completely discarded. But for us, a term that returns a unit can still produce output backwards:

eval (UnitElim {cs = (_ ** _ ** s)} t1 t2) xs 
  = let ((), k1) = eval t1 (ixUncatL s xs)
        (y, k2) = eval t2 (ixUncatR s xs)
     in (y, \y' => ixConcat s (k1 ()) (k2 y'))

Arguably, the most important rule in the entire language is negation elimination, because it is the only rule that directly allows communication from the forwards pass to the backwards pass. In traditional differentiable programming terminology, the implementation of this rule is to write to the tape. Here is its implementation:

eval (NotElim {cs = (_ ** _ ** s)} t1 t2) xs 
  = let (y1, k1) = eval t1 (ixUncatL s xs)
        (y2, k2) = eval t2 (ixUncatR s xs)
     in ((), \() => ixConcat s (k1 y2) (k2 y1))

This leaves the 2 negation introdution rules that, as I wrote in the previous post, I discovered while working on these cases for the interpreter. They are truly sus, and while writing this post I changed my mind several times about whether they should be in the language or whether they are mistakes. Currently the deciding factor is that one of them, the covariant one, is actually required in practice: we will use it repeatedly in the next section when implementing differentiable functions.

In a sense the negation introduction rules are scoping rules rather than computational rules: they use the helper functions we developed for the interpretation of Rename, but themselves cannot be expressed in terms of Rename. My provisional conclusion is that these rules need to be in the language because I don’t have any alternative, but they are very much on thin ice.

Here is the code I wrote:

eval (NotIntroCov t) xs 
  = (deleteCon, \x' => let ((), k) = eval t (x' :: xs)
                           y' :: ys = k ()
                        in ys)
eval (NotIntroCon t) xs 
  = let ((), k) = eval t (spawnCov :: xs) 
        y :: ys = k ()
     in (y, \y' => ys)

It is worth meditating on these definitions. Both of them delete something “and then” spawn a monoid unit to replace it (in category theory this is called a zero morphism), and they use the lack of causal flow from the input to the output to move the scopes around in a way that is impossible to do with the other rules.

Our first program

The code for this section and the next can be found here.

From the start of this blog series until now we have been pretty far down the abstraction ladder, far enough that it is hard to remember what the point is. Now we can finally run programs, let’s finally return to near the surface by implementing a baby example of automatic differentiation.

There are essentially two main aspects to automatic differentiation. The first is the reverse chain rule, which is the name in calculus for the fundamental computational model of Aptwe. The other is the purely syntactic procedure that associates each primitive element of a program with its reverse derivative, the starting point from which the reverse chain rule differentiates the entire program compositionally. This second thing is not something we can do yet, and in my opinion it is not a feature that belongs in a kernel language, instead belonging in a surface language specialised to differentiable programming. So, we will write a program where each primitive function is “decorated” with its reverse derivative and then the reverse chain rule takes care of the rest.

For now, I have defined base terms to be terms that carry around an Idris lens, and the corresponding interpreter cases simply apply the forward and backward passes of that lens as functions:

data BaseTerm : All Ty ks -> Ty k -> Type where
  Builtin : (IxAll Cov xs -> (Cov y, Con y -> IxAll Con xs)) -> BaseTerm xs y

This is temporary for as long as we are using the prototype interpreter; eventually something much more subtle will be needed here. Using this, we can for example lift functions between doubles into Aptwe terms:

sin : Term [BaseTy Real] (BaseTy Real)
sin = BaseTerm $ Builtin $ \[x] => (sin x, \X => [X])

cos : Term [BaseTy Real] (BaseTy Real)
cos = BaseTerm $ Builtin $ \[x] => (cos x, \X => [X])

These functions respectively apply sin and cos in the forward pass, and are trivial in the backward pass. Similarly, we can multiply two doubles in the forward pass with the function

times : Term [BaseTy Real, BaseTy Real] (BaseTy Real)
times = BaseTerm $ Builtin $ \[x, y] => (x * y, \X => [X, X])

It will be useful to have a shorthand for types of “monomorphic” lenses, ie. whose forwards and backwards types are the same, since autodiff functions all have this form:

Mono : Ty (True, True) -> Ty (True, True)
Mono a = Tensor a (Not a)

Our main goal is to implement a combinator diff which takes a term representing a function $f$ and its ordinary derivative $f’$ and combines them as follows. The forward pass is $f$ itself, ie. it takes the forward input $x$ to the forward output $f (x)$. Now we take the equation $\frac{dy}{dx} = f' (x)$ and rearrange it to $\frac{1}{dx} = f' (x) \cdot \frac{1}{dy}$. The backward pass implements this equation as a function, ie. it takes the forward input $x$ and backward input $\frac{1}{dy}$ to the backward output $f' (x) \cdot \frac{1}{dy}$.

The problem with the following code is that it’s incomprehensible for (at least) 2 different reasons:

diff : Term [BaseTy Real] (BaseTy Real) -> Term [BaseTy Real] (BaseTy Real)
    -> Term [Mono (BaseTy Real)] (Mono (BaseTy Real))
diff f df = TensorElim Var
          $ Rename (Copy Z $ Insert Id Z $ Insert Id Z $ Empty)
          $ TensorIntro f
          $ NotIntroCov
          $ Rename (Insert Id (S (S Z)) $ Insert Id (S Z) $ Insert Id Z $ Empty)
          $ NotElim Var
          $ Let df
          $ times

The first reason is that variables are referred to by their position rather than by name. The second reason is that programming with elimination forms like TensorElim and (especially) NotElim is just very unintuitive. To be honest, I don’t expect anybody to understand it because I don’t exactly understand it myself, but it does work. Some highlights are the Copy on line 4 which copies the forward input $x$ because it is used by both the forward and backward passes, and the last line which is the multiplication of $f' (x)$ by the backward input $\frac{1}{dy}$.

It was suffering through writing this function (and debugging it, since on my first try I got the 2 inputs to the backward pass the wrong way round) that made me decide to push towards a human-understandable language faster than I originally intended to.

Now, to write a differentiable sin function we just have to say what its derivative is:

dsin : Term [Mono (BaseTy Real)] (Mono (BaseTy Real))
dsin = diff sin cos

To run dsin we need to call eval on it, but also pack and unpack some boxes:

dtest : Term [Mono (BaseTy Real)] (Mono (BaseTy Real)) 
     -> Double -> (Double, Double -> Double)
dtest t x = let ((y, X), k) = eval t [(x, X)]
             in (y, \dy => let [(X, dx)] = k (X, dy)
                            in dx)

Autodiff

Before we stop we should write a slightly less trivial function, just to make sure that we are doing autodiff correctly when we compose things together. Let’s write the function $x \sin x^2$. Although Aptwe has the chain rule built in, we need to write a differentiable multiplication that contains the essence of the product rule. If you thought diff was painful, this one is worse:

dtimes : Term [Mono (BaseTy Real), Mono (BaseTy Real)] (Mono (BaseTy Real))
dtimes = TensorElim Var
       $ Rename (Insert Id (S (S Z)) $ Copy Z $ Insert Id Z $ Insert Id Z $ Empty)
       $ TensorElim Var
       $ Rename (Insert Id (S (S Z)) $ Copy Z $ Insert Id (S (S (S Z))) 
               $ Insert Id Z $ Insert Id Z $ Insert Id Z $ Empty)
       $ TensorIntro times
       $ NotIntroCov
       $ Rename (Insert Id (S Z) $ Insert Id (S (S Z)) $ Copy Z 
               $ Insert Id (S (S Z)) $ Insert Id Z $ Insert Id Z $ Empty)
       $ NotElim Var
       $ UnitElim (NotElim Var $ times)
       $ times

This calls times (our base term for ordinary, non-autodiff multiplication) 3 different times, once in the forward pass for the actual multiplication, and twice more in the backward pass: remember the product rule contains 2 instances of multiplication $(f \cdot g)' (x) = f (x) \cdot g' (x) + f' (x) \cdot g (x)$. But notice, there is no corresponding subterm for addition. Instead, there are 3 instances of Copy: in order of appearance, 2 to copy each of the forward inputs which are used once each in the forward and backward passes, and 1 more which is actually merging a negative variable. This last Copy is actually the addition in the product rule.

There are two pieces of good news. The first is that this function works (I even got this one right first try, although it took me about an hour), and the second is that we are over the hill, everything after this point should be actually understandable.

We can implement squaring in terms of multiplcation by copying the input:

dsquare : Term [Mono (BaseTy Real)] (Mono (BaseTy Real))
dsquare = Rename (Copy Z $ Insert Id Z $ Empty)
         $ dtimes

Something important is happening here. This Copy is being applied to a variable of type Mono (BaseTy Real), which is the tensor product of the forward input and the backward output. Copying a tensor product is copying each part, but copying the negative part is actually a merge, which is addition. So this single Copy is a true autodiff copy, which is simultaneously copying in the forward pass and adding in the backward pass. So we never need to specify that the deriative of $x^2$ is $2x$, and instead we get it as a consequence of the product rule.

Now we have all the pieces for our slightly more complicated example of $x \sin x^2$, whose only difficulty is writing without variable names:

example : Term [Mono (BaseTy Real)] (Mono (BaseTy Real))
example = Rename (Copy Z $ Insert Id Z $ Empty)
        $ Let (Let dsquare dsin)
        $ dtimes

And it works! We can use our testing function from before to verify that the backward pass gives the correct reverse derivative, which is $\frac{1}{dx} = (\sin x^2 + 2 x^2 \cos x^2) \cdot \frac{1}{dy}$.

In conclusion, everything works but is absolutely horrendous to program in. In principle this is fine because Aptwe is explicitly intended to be a kernel language and not to be written by humans. But after this experience I have decided to prioritise working towards a prototype frontend language, for demonstration purposes and for the sake of my own sanity. My plan is to build a series of elaboration passes in reverse order of the compiler pipeline:

Replace elimination forms with patterns (this is particularly important because Aptwe is substructural, so we can’t take the easy option of adding primitives for projection out of products)
Type inference (which is very straightforward right now because Aptwe is still simply typed)
Scope checking and kind inference (uniquely to Aptwe, I expect these to be interconnected in an interesting way)
Parsing a concrete syntax

On the other side of the scope checking pass, we will have a language with names rather than positional variables, and this is the point at which programming should become humanly possible.

Separately to this I also want to add some more primitives to the kernel language, particularly linear function types and linear coproducts (the par operator of linear logic). The eventual goal is to support algebraic datatypes, but there is still basic research to be done on the theory of datatypes in categories of lenses. Of course we will also need to build in some non-algebraic datatypes, the most obvious example being tensors. My plan is to work on these features in parallel with working on the frontend language, so expect the next few blog posts to alternate between these topics.

Mathematics for Governance Design

2024-10-28T00:00:00+00:00

Recently we held a workshop in Edinburgh titled Mathematics for Governance Design, consisting of a roughly 50/50 split between social scientists and category theorists.

The workshop was organised by Philipp and myself from the CyberCat Institute together with Seth Frey, Saba Siddiki and Josh Tan from (in an overlapping way) Metagov, the Institutional Grammar Research Initiative and the Computational Institutional Science lab. It was funded and hosted by the International Centre for Mathematical Sciences as part of their Mathematics for Humanity programme of events.

We designed the workshop to have as little scheduled time as possible and as much unstructured working group time as possible - inspired by our past experience running a workshop at Wytham Abbey and originally inspired by Dagstuhl. And it was a resounding success: it felt like the theme of the week was seeing famous people who we would never expect to interact with each other interacting. The danger of running a workshop like this is that the two different groups would form cliques and only interact with each other under duress, but the exact opposite happened.

Probably my personal highlight was being able to meet Matilde Marcolli and talk about our shared interest in the very hard question of how to surpass the well-known scalability barrier for human self-organisation (for example written about extensively by Elinor Ostrom). We agreed that compositional game theory and related categorical cybernetics methods could plausibly have a role to play for building models of social situations consisting for example of groups of groups arranged in an approximate hierarchy. (When taken as revolutionary this is an aspect of anarcho-communism, although my personal interest is a bit too theoretical to call it that.) In fact I should write a blog post on the general topic of hierarchies of lenses, which is something I’ve talked extensively about with several people, most notably Toby Smithe in the context of modelling the human cortex. I talked quite a bit it in high level terms in this blog post, but a more technical post might be in order.

Another highlight was being able to finally engage with institutional grammar and think seriously about how it could relate to open games. There is some past work (this and this, plus a paper I wrote earlier this year with Vincent Wang-Maścianica that isn’t released yet) on connections between open games and natural language, but to me the limiting factor has always been that the open game semantics of individual words must be hand-crafted, which will not scale beyond the tiniest of toy examples, and there has never been a plan for it besides “maybe hand-craft enough examples to fine-tune an LLM and hope for the best”. My immediate thought now is that the type of natural language texts we would like to apply this to will often factor through an institutional grammar representation, in a way that is likely to be extremely useful for game-theoretic analysis. I learned last week that institutional grammar has already been connected to agent-based models, and if it is possible to go to an agent-based model then it should be not significantly harder to go to a game-theoretic model. The benefit, of course, is that Nash equilibrium is a very standard perspective for thinking about the theoretical nature of institutions.

Of course we understand as well as anybody that interdisplinary research is extremely hard and we should not expect immediate technical results after bringing together two such different groups. But we felt real excitement in the room, and we have every reason to expect multiple new collaborations to be formed, and we are already planning a successor workshop next year.

On Hopfield Networks and Boltzmann Machines

2024-10-14T00:00:00+00:00

“All of this will lead to theories [of computation] which are much less rigidly of an all-or-none nature than past and present logic. They will be of a much less combinatorial, and much more analytical, character.

In fact, there are numerous indications to make us believe that this new system of formal logic will move closer to another discipline which has been little linked in the past with logic.

This is thermodynamics, primarily in the form it was received from Boltzmann, and is in part theoretical physics which comes nearest in some of its aspects to manipulating and measuring information.”

– John Von Neumann, The General and Logical Theory of Automata, 1948.

Allow me a quick excursion from the regular programming to celebrate the 2024 physics Nobel Prize awarded to John Hopfield, inventor of the eponymous Hopfield network, and Geoffrey Hinton, co-inventor (with Terrence Sejnowski) of the Boltzmann machine.

Since this is an economic design series, the question why a physics Nobel, and especially a Nobel Prize awarded for a contribution to machine learning, should be of interest is a fair one.

The long answer is that, having spent a few long years translating the underlying mechanisms of both networks into economic game theory, and in turn into the emergence of consensus (or its opposite, partisanship) in social groups, I think I can offer a fairly unique perspective to discuss the impact of this prize on economics.

The short answer is that these two networks also shape the whole economic outlook presented in EconPatterns.

To recapitulate.

In the first post, I established the economy as a network of a small set of fundamental activities: stocks, flows, transformations, which have to be orchestrated to produce desirable outputs.

This orchestration requires agreement on beliefs among participants, first that these activities do indeed lead to these outcomes, and second that these outcomes are indeed desirable.

This framing mapped a network of economic activities onto a belief network, with the underlying assumption that unless all participants have perfectly homogenous beliefs, goal conflict within the network becomes inevitable as the network becomes larger, until ultimately the network has to crumble into smaller subnetworks (aka clusters) that can hold shared beliefs.

Expressed in the first law of organization: the objective of organization is to resolve the conflict between moving forward (orchestrate activities that produce a desirable output) and staying together (hold the shared belief that these activities do indeed lead to the proposed desirable output).

Where this conflict cannot be resolved within an organization, competition emerges.

Competition is the starting point of economic inquiry, and it typically treats it as exogenous. In other words, competition has to happen, and by virtue of simply happening (and by drawing attention to surpluses and scarcity in the economic network via price signals) it helps steer the economy in the right direction.

What it skips is the question where exactly the beliefs diverge sufficiently that orchestration within the same organization is no longer possible so that rifts open up and competition emerges.¹

This is a question that economics of organization tries to tackle in the form of the make-or-buy decision, but finding an appropriate formalization has been elusive. And this is where Hopfield, Hinton, and Sejnowski come in.

Hopfield networks and belief clustering

To make that leap we first have to divest ourselves from any expectation that our formalization expresses any kind of tangible economic activity, and accept that we go down to bare-bones expressions of individual beliefs, and the main activity is to both be influenced by and trying to influence the beliefs of ours peers.

In other words, for any proposition, participants express their beliefs in the simplest possible way as subjective expectations: in simple Boolean logic, zero for “I believe it’s false”, one for “I believe it’s true”, or in a stochastic setting, any value between zero and one as the expression how probable they consider the proposition to be true. (Alternatively we can consider a wider range from −1 to +1 to express opposing beliefs, which is especially useful in political settings.)

Hopfield’s first paper, published in the midst of the first “AI winter” in 1982, astounds in its brevity. It is only five pages long.

Up to this juncture, including the emerging connectionist revolution that lead to Rumelhart & McClelland’s famous two-volume work in 1986 (which also included Hinton & Sejnowski’s paper), neural networks where almost exclusively conceived as feedforward networks (information flows from input to output) with backpropagation (feedback flows from output to input) as learning mechanism.

Hopfield’s recognition was to fold the network unto itself: all network nodes can send and receive signals to and from all others, and the designation as input or output nodes is arbitrary.

In isolation this wouldn’t be particularly interesting, but the marvel of neural networks in general, and Hopfield networks in particular, is that the behaviors of individual nodes are connected, and that this connectivity can be expressed in a weight (or covariate) matrix, where high positive weights translate as “shared beliefs” and high positive weights as “opposing beliefs”.

Neural networks function in two modes: training mode (weights are flexible) and execution mode (weights are fixed). Training in this case translates into finding out which nodes hold correlating beliefs, and setting the weights accordingly.

Hopfield’s question is what happens when a connected network with a given set of (symmetric) weights plus a vector of isolated beliefs (aka biases) per node is allowed to converge from a given starting state (the input) to a stable state (the output), when each node tries to agree with all connected neighboring nodes with shared beliefs and disagree with neighboring nodes with opposing beliefs.

Hopfield’s first paper from 1982 tackles this question with a Boolean choice of zero and one for all nodes, and a second paper from 1984, also five pages long, expands this to allow uncertainty in the form of probabilistic belief values between zero and one, plus a sigmoid function to connect inputs and outputs.

His conclusion, in the shortest possible form, is that the network exhibits memory, in a form that makes it a “content-addressable memory”.

In other words, the network converges from an input pattern to the nearest pattern it has been trained on — an important feature in pattern recognition with an obvious early application in detecting handwriting. If the input pattern is something that vaguely looks like a 7, the output ideally should identify this as a 7 and not a 3.

Under the right conditions, if the training data set contains a number of shapes that are vaguely 7-ish looking, the network should memorize this as a distinct pattern and when activated, recognize this.

In somewhat more technical language, the network should contain local optima representing the trained-on patterns and basins of attraction that capture all the trained variants (and their interpolations).

Boltzmann machines and the rationality of erratic behavior

As a physics-inspired mathematical construct, this is extremely neat and its translation into belief-driven collective action expands beyond the metaphorical similarity. Implemented as a feedback network, it has quite a few drawbacks which curbed its widespread adoption in favor of less finicky backpropagation architectures.

One major drawback, in an analogy to what I like to call the “bicycle repair cooperative on Shattuck Ave”, is that it doesn’t scale particularly well.

Shattuck Avenue is in downtown Berkeley and the bicycle cooperative prided itself on its strong collectivist ethos, where all topics are discussed and decided together. This might work if the collective is small and beliefs are highly aligned, but runs into trouble when the collective gets bigger (adding one new member adds N new connections) and beliefs diverge.

Which is why “fully connected consensus” never translates as a template for large companies.

The other problem is that it produces a whole lot of local optima which don’t map to trained patterns, so the network is always at risk of producing meaningless output — a problem that also increases with network size.

One remedy for this comes from Hinton & Sejnowski’s Boltzmann machine, which introduces “vanishing noise” as a means to avoid local minima.

Vanishing noise just means that as a node is called upon to update its belief (aka state), we introduce a small likelihood that the node accepts a new state even if it is unfavorable, and that this likelihood becomes smaller over time.

This is very much an analogy for shaking up the system, implemented as “distributed simulated annealing” — annealing being the metallurgical procedure to add short spurts of heat in a cooling process to avoid getting trapped in imperfect lattice structures.

The connection to thermal annealing not only creates a connection to physics proper, it also opens another batch of neat features.

For one, we suddenly have a system that even if behavior happens on the individual level — each node updates its belief individually and only according to its own interests — we can still express the behavior of the whole system in a single macro equation.

The economic equivalent of this is a potential game (introduced by Dov Monderer and another Nobelist, Lloyd Shapley) where changes in individual utilities can be captured in a single equation for the whole game.

The other intriguing feature is that under vanishing noise, we can characterize the equilibria the system reaches using the eponymous Boltzmann distribution, which tightly connects the model to statistical mechanics, and in turn to entropy, free energy, and — importantly for us — surprise.

This might answer the question why the physics Nobel committee deemed their work worthy of recognition.

In the early 1980s, at a time when artificial intelligence in general and perceptrons in particular were seemingly going nowhere, a bunch of researchers centered around Princeton and Carnegie Mellon put computation on a physical footing, just as John Von Neumann had predicted.

Economics of influence, economics of attention

But the goal of this post is not to resolve the befuddlement that befell some physicists at the news that the physics prize was seemingly awarded to a discovery in computer science, but to map out why this matters to economics, and in particular to economic design.

Starting out in this endeavor, I had at best a vague notion of statistical mechanics and mostly considered this kind of feedback network a bare-bones metaphorical model of what would happen in a social group that faces a simple binary choice and influences each other in their choices, couched in the at the time headlines-making problem of technology standard competition, with the major claim to novelty being that if you consider heterogeneous network effects, you’ll get more interesting results — especially that you can get interesting partisan dynamics that go against the then agreed-upon paradigm that positive network effects inevitably lead to monopolization.

The findings were mostly met with indifference at the time, but economics has evolved significantly since then. Graph theory has become a recognized tool in large part because of the emergence of social networks and the “network science” revolution in sociology. Machine learning has arrived in economics some ten years ago and is currently in a state that can only be described as feeding frenzy.

There is a recognition that the landscape of products is unsurveyable for the consumer, leading to attention dynamics and herding behavior (then of little interest outside finance), and to the introduction of two novel economic engines that found widespread adoption in the internet age: recommender engines and reputation engines.

There is an emerging understanding that preferences are not inscrutable and outside the scope of economic inquiry, but that they are tractable and that they evolve in predictable ways.

There is now far less discomfort dealing with scenarios that have more than one equilibrium, so existence proofs have gradually given way to convergence and evolutionary dynamics.

Bayesian inference, including Bayesian games of incomplete information, is slowly making inroads, giving us a richer toolset to thing about belief propagation and the evolution of norms.

And most importantly, very rich interaction data has become available, pushing the “not very interesting” findings of ideological polarization as an outcome of network heterogeneity to the forefront of the academic (and non-academic) debate.

My own position also evolved over time.

For one, I no longer consider it merely a metaphorical model of human behavior, useful as an illustrative but empirically intractable shorthand for what happens in a social group when traditional preferences and peer influence intersect.

The evidence that statistical mechanics plays a role not only in human behavior, that thing we call rationality (or even bounded rationality), but also in the goal-directed behavior of organisms we don’t generally consider rational, keeps mounting, as I mapped out in the post on surprises for eyeballs as the fundamental exchange of (not only) the attention economy.

Mostly outside of economics, much progress has been made at the intersection of statistical mechanics, Bayesian inference, and information theory, and it slowly trickles into economics proper via the translation into (evolutionary) game theory.

Cognitive effort is a scarce resource which needs to be allocated towards the activities that promise the highest return. The mechanism by which this allocation happens is attention, a term that, other than a common acceptance that we now live in the attention economy, has gained little traction in economics, even if the administration of scarce resources is at the core of economic theory.

In economic design this is somewhat different, since it ultimately deals with conceiving structures that facilitate mutually beneficial exchange. So the orthodoxy that befits theory is of little help, as the primary goal is to make the engines work, to make sure they fulfill their designed function.

This inevitably requires a wider vocabulary, including learning agents, including agents that learn from each other, including agents that form belief clusters as the subset of peers they’re willing to learn from. That includes machine learning as a facsimile for learning agents.

This also includes investigating rationally erratic behavior, mutation, or “physiological” annealing, deviation from self-interested behavior that gets more frequent as the temperature increases.

It includes developing an understanding of the emergence of norms as decentralized constraints of individual behavior.

Once we start incorporating these tools into our design handbook, it quickly becomes apparent that they go a long way towards explaining common behaviors, including behavior we don’t necessarily consider “economic”.

It also fuels the prospect that we will conclude in due time that the mechanism Hopfield, Hinton, and their peers described is embedded in far more biological systems than we thought, and maybe humans are indeed lightning calculators of pleasure and pain, but not in the way we assumed they were.

Literature

John J. Hopfield, “Neural networks and physical systems with emergent collective computational abilities”, PNAS, 1982.
John J. Hopfield, “Neurons with graded response have collective computational properties like those of two-state neurons”, PNAS, 1984.
Geoffrey E. Hinton, Terrence J. Sejnowski, David H. Ackley, “Boltzmann machines: constraint satisfaction networks that learn”, Tech report, CMU, 1984.
Geoffrey E. Hinton, Terrence J. Sejnowski, “Learning and relearning in Boltzmann machines”, in Rumelhart & McClelland, 1986.

This is of course extremely truncated. The famous economic calculation debate centered on the question of central planning, where “central” was defined as the administrative state also orchestrating economic activity. The perspective EconPatterns takes is of course a different one, as laid out in the newsletters on organizations as tectonic plates and the blurry boundary between economics and operations research. ↩

Foundations of Bidirectional Programming III: The Logic of Lenses

2024-09-12T00:00:00+00:00

See parts I and II of this series. There is also now a source repo!

In this post we will make probably the single most important step from a generic type theory to one specialised to bidirecional programming.

In a bidirectional language there are 2 kinds of variables: ones travelling forwards in time, and ones travelling backwards in time. I will refer to the types of these variables covariant and contravariant for short (or at least shorter). Variables of covariant type are the ones we are used to: they behave according to ordinary cartesian scoping rules, which means they can be implicitly deleted (referred to zero times) and implicitly copied (referred to more than once).

Variables of cocartesian type are the weird ones. They can also be implicitly copied and deleted, but because they are going backwards, from the forwards perspective it looks like they can be implicitly merged and spawned. By spawned I mean they can be bound zero times and still referred to; and by merged I mean they can be bound twice without the later binding shadowing the earlier binding. On the other hand they can not be implicitly copied or deleted. Their scoping rules are cocartesian.

I can now reveal that my party trick in the last section of my first post where I built a cocartesian language was secretly not in fact a party trick after all, but was all along in preparation for this exact moment.

Now suppose we take a tensor product of a covariant type and a contravariant type. Now we have a variable referring to a bundle of a cartesian value, which can be deleted and copied but not spawned or merged, and a cocartesian value, which can be spawned and merged but not deleted or copied. Since doing any of these operations to a pair means doing it to both, our variable cannot be deleted or copied or spawned or merged, which means it is a linear variable. I will refer to such a tensor product type as invariant. Secretly, all of the types in my second post were invariant, since that language was linear.

There is a secret fourth thing, which is variables with bicartesian scoping rules, so they can be deleted and copied and spawned and merged. I will refer to these types as bivariant. It would be entirely reasonable to not include these, but I have a secret plan for them that will be revealed in the next post. For now, the only bivariant type will be the monoidal unit.

The 4 kinds of things

Just as terms are classified by types, types are classified by kinds, which means I have been talking about kinds the whole time. But these are not kinds as we know them from languages like Haskell. We have the added complication that kinds control the scoping rules of variables of the types they classify; but on the other hand we have the added simplification that there are exactly 4 kinds, rather than an infinite supply of them. I have to thank Michael Arntzenius, who visited Zanzi and me a few weeks ago, for planting the idea of a 4-element lattice of kinds.

Let’s start with the easy parts:

Kind : Type
Kind = (Bool, Bool)

data Ty : Kind -> Type where
  Unit : Ty (True, True)
  Ground : Ty (True, False)
  Not : Ty (cov, con) -> Ty (con, cov)

I implement the 4-element lattice as the product (Bool, Bool), where the first flag tracks whether the type is covariant and the second whether it is contravariant. I anticipate that much later, probably when I add type polymorphism, it will become necessary to take seriously that there are subkind relationships here, but I will cross that particular bridge when it is in front of me. Unit is a bivariant type, Ground is a covariant type (it is still a placeholder, and in the next post will be replaced with a proper system for base types). The Not operator acts on the underlying kind by interchanging the covariant and contravariant capabilities; so strictly covariant types become strictly contravariant and vice versa, whereas the bivariant and invariant kinds are both stable under negation.

There is an alternative way of implementing it, but it is much more tedious: instead of representing kinds explicitly, have 4 different versions of Ty for each of the 4 kinds. That leads to a proliferation of type operators - 4 operators for Not and 16 for Tensor (which we’re coming to next), 1 for each combination of kinds of the 2 inputs. I really hope that won’t be necessary, but it’s something I’m keeping in my back pocket just in case I run into an insurmountable problem with this encoding.

Now we come to the hard part: tensor products. A tensor product is covariant if both parts are covariant, and is contravariant if both parts are contravariant. In a world with a sufficiently smart typechecker we would simply write this:

  Tensor : Ty (covx, conx) -> Ty (covy, cony)
        -> Ty (covx && covy, conx && cony)

But we already know how to handle this situation, it’s the same idea as for list concatenation in the first post. We define the relation corresponding to the boolean conjunction function, and a section of it:

data And : Bool -> Bool -> Bool -> Type where
  True  : And True b b
  False : And False b False

(&&) : (a : Bool) -> (b : Bool) -> (c : Bool ** And a b c)
True  && b = (b ** True)
False && b = (False ** False)

(Here’s I’m having fun with Idris’ ability to disambiguate names based on typing information, something I really wish I could do in Haskell.)

Now we can write the actual definition of tensor products, which is much more complicated looking than it should be, but this is just what we have to deal with until the day when somebody builds a typechecker that can handle trivial equational reasoning:

  Tensor : {covx, covy, conx, cony : Bool}
        -> {default (covx && covy) cov : _} -> {default (conx && cony) con : _}
        -> Ty (covx, conx) -> Ty (covy, cony) -> Ty (cov.fst, con.fst)

The scoping rules

Now we come to the main topic of this post: how kinds influence scoping rules. In the first post we saw how to implement context morphisms for planar, linear, cartesian and cocartesian languages. Those definitions were all polymorphic over arbitrary lists. Then in the second post we defined context morphisms that could introduce and elimination double negations, which was no longer polymorphic but specialised to a particular language of types, and this will continue.

Now that types are indexed by kinds, everything else becomes more complicated: everything we do from now on becomes additionally indexed by kinds, starting with this.

(Note, I decided to reuse the name Structure, since this should be the last one we ever need.)

Let’s start with the linear rules:

data Structure : All Ty kas -> All Ty kbs -> Type where
  Empty  : Structure [] []
  Insert : {a, b : Ty (cov, con)} -> {as : All Ty kas} -> {bs : All Ty kbs}
        -> Parity a b -> IxInsertion a as as' -> Structure as bs -> Structure as' (b :: bs)

Here IxInsertion is an indexed version of the Insertion datatype from the previous post, relationally defining insertions into an All type indexed by an underlying list:

data IxInsertion : {0 x : a} -> p x 
                -> {0 xs : List a} -> All p xs 
                -> {0 ys : List a} -> All p ys -> Type where
  Z : IxInsertion {x} a {xs} as {ys = x :: xs} (a :: as)
  S : IxInsertion {x} a {xs} as {ys} bs 
   -> IxInsertion {x = x} a {xs = y :: xs} (b :: as) {ys = y :: ys} (b :: bs)

I’m not going to explain all of the syntax and semantics of Idris happening here because it would take us too far afield, and despite appearances this is not intended to be a tutorial series on Idris programming. Suffice to say, defining indexed versions of standard datatypes is significantly harder than defining the originals.

The definition of Parity, which handles double negations, is unchanged from the previous post besides adding the kind indexes:

data Parity : Ty (cov, con) -> Ty (cov, con) -> Type where
  Id    : Parity a a
  Raise : Parity a b -> Parity a (Not (Not b))
  Lower : Parity a b -> Parity (Not (Not a)) b

The Idris typechecker is, at least, smart enough to figure out that double negation leaves the kind unchanged. This is one reason that I write out kinds as pairs of booleans everywhere rather than defining a function swap : Kind -> Kind, since that causes Idris to get stuck here.

One thing to note is that I reversed the polarity of the Insert constructor from the previous post, so that it now conses on the codomain and inserts on the domain rather than the other way round. The previous post was “supply driven”, saying where everything in the domain goes in the codomain, whereas this version is “demand driven”, saying where everything in the codomain came from in the domain. This was a late change after I experienced manually defining terms in this core syntax and found that this way makes manual proof search much easier. If it later turns out that the original supply driven version makes writing elaborators easier, it’s an easy change to turn it back.

Next we have the rules for delete and copy, which can be applied only to covariant types. We don’t care whether the type is also contravariant, ie. we can use it for both strictly covariant types and bivariant types. This is the other reason I define kinds to be pairs of booleans rather than their own dedicated language, because otherwise we would have twice as many constructors here.

  Delete : {a : Ty (True, con)} -> Structure as bs -> Structure (a :: as) bs
  Copy   : {a : Ty (True, con)} -> {as : All Ty xs} 
        -> IxElem a as -> Structure as bs -> Structure as (a :: bs)

Here IxElem is, of course, an indexed version of the Elem datatype, which I won’t bother to write here but was also painful to define. (If you want to you can find it in the IxUtils module of the source repo.)

With that, defining the constructors for spawn and merge is easy, and completes our definition of Structure once and for all.

  Spawn  : {b : Ty (cov, True)} -> Structure as bs -> Structure as (b :: bs)
  Merge  : {b : Ty (cov, True)} -> {bs : All Ty ys} 
        -> IxElem b bs -> Structure as bs -> Structure (b :: as) bs

The full logic of lenses

We can now put the pieces together and make the first real definition of our kernel language. I hope that nothing here will change, only be added to in the future.

Of course everything is kind indexed:

data Term : All Ty ks -> Ty k -> Type where
  Var : Term [x] x
  Rename : Structure as bs -> Term bs x -> Term as x

I renamed Act to Rename since the last post, since somebody pointed out that’s what it is.

Unit introduction and elimination rules work as they did before:

  UnitIntro : Term [] Unit
  UnitElim : {as : All Ty kas} -> {bs : All Ty kbs} -> {default (ixSimplex as bs) cs : _}
          -> Term as Unit -> Term bs x -> Term (cs.snd.fst) x

Here ixSimplex is the tactic corresponding to a datatype IxSimplex which is the relational version of the indexed version of ++, operating on All rather than List. This one is worth writing down because it returns a twice-iterated Sigma type, so we have to extract the part we need with .snd.fst:

data IxSimplex : {0 xs : List a} -> All p xs 
              -> {0 ys : List a} -> All p ys 
              -> {0 zs : List a} -> All p zs -> Type where
  Z : IxSimplex [] bs bs
  S : IxSimplex {xs} as {ys} bs {zs} cs 
   -> IxSimplex {xs = x :: xs} (a :: as) {ys = ys} bs {zs = x :: zs} (a :: cs)

ixSimplex : {xs : List a} -> (as : All p xs)
         -> {ys : List a} -> (bs : All p ys)
         -> (zs : List a ** cs : All p zs ** IxSimplex as bs cs)
ixSimplex {xs = []} [] {ys} bs = (ys ** bs ** Z)
ixSimplex {xs = x :: xs} (a :: as) {ys} bs 
  = let (zs ** cs ** n) = ixSimplex {xs = xs} as {ys = ys} bs 
     in (x :: zs ** a :: cs ** S n)

Now we come to the negation rules, and the final twist of this post. In the language we reached at the end of the previous post, which amounts to the fragment of this language for only invariant types, there was no not-introduction rule - correctly so. As I speculated there, there are indeed some valid instances of not-introduction, but they are not sufficient to prove general double negation introduction or elimination - and they can’t even be expressed without the kind system we developed in this post.

It turns out that we have two not-introduction rules, one that is valid for covariant types and one that is valid for contravariant types. This introduces a sort of incompatibility between negation and tensor, since the tensor product of two variables with a valid not-introduction rule can fail to have one.

  NotIntroCov : {a : Ty (True, con)} -> Term (a :: as) Unit -> Term as (Not a)
  NotIntroCon : {a : Ty (cov, True)} -> Term (a :: as) Unit -> Term as (Not a)
  NotElim : {as : All Ty kas} -> {bs : All Ty kbs} -> {default (ixSimplex as bs) cs : _}
         -> Term as (Not x) -> Term bs x -> Term (cs.snd.fst) Unit

I would say that this is the first case where my well typed by construction methodology enabled me to do something that I think I would have failed at otherwise. I figured out these rules at the same time as the corresponding cases of the well typed evaluator (which will be the topic of the next post). Although they look like they are among the simplest rules in the language, they are the ones I understand by far the least.

If we have a bivariant type then both of these rules can be applied to produce the same result. It seems tempting to try to roll these two rules into one, which can be applied when either cov or con is True. This is a bit tricky to do, but it turns out to also be a bad idea: although these rules are logically equivalent, they are not type-theoretically equivalent: they have completely different operational semantics! And by that I don’t mean something like different evaluation strategies, I mean that they actually output different results.

Now there’s only the tensor rules, and for once there is nothing to say about them, they are the indexed versions of the tensor rules from the last 2 posts.

  TensorIntro : {as : All Ty kas} -> {bs : All Ty kbs} -> {default (ixSimplex as bs) cs : _}
             -> Term as x -> Term bs y -> Term (cs.snd.fst) (Tensor x y)
  TensorElim : {as : All Ty kas} -> {bs : All Ty kbs} -> {default (ixSimplex as bs) cs : _}
            -> Term as (Tensor x y) -> Term (x :: y :: bs) z -> Term (cs.snd.fst) z

And that’s it!

Like I said, in the next post we will build a well typed evaluator, which means we will also write and run our first programs - and we can already do some interesting things, like basic automatic differentiation. The only small thing that will need to be added to the language from this post is a mechanism for adding basic types and basic terms, such as arithmetic operators between doubles.

Foundations of Bidirectional Programming II: Negative Types

2024-09-05T00:00:00+00:00

See part I of this series

In this post we’ll begin designing a kernel language in which all programs are optics. What I mean by a “kernel language” is that it will serve as a compiler intermediate representation, with a surface language compiling down to it. I intend the surface language to be imperative style like the current Open Game Engine (with an approximately Python-like syntax), but the kernel language will reflect the category theory as closely as possible. I plan the kernel language to be well typed by construction, something that seems like overkill until I think about the problem of figuring out how pattern matching should work in a bidirectional language.

My first design choice is that object language types denote pairs of metalanguage types, with one denoting the forward part (sometimes I might call it the covariant denotation) and the other denoting the backward part (the contravariant denotation).

Cov : Ty -> Type
Con : Ty -> Type

The interpretation of a term will be a lens:

eval : Term xs y -> All Cov xs -> (Cov y, Con y -> All Con xs)

Here All is an Idris standard library function that combines a type-level map with cartesian product:

data All : (a -> Type) -> List a -> Type where
  Nil : All p []
  (::) : p x -> All p xs -> All p (x :: xs)

(Idris has what in Haskell would be called rebindable syntax switched on by default, so we can use the usual syntactic sugar for lists to refer to elements of All.)

eval is a well typed interpreter, and is a placeholder while we prototype: much later, this is where the backend of the compiler will begin.

The logic of lenses

My second design choice is that I want the basic product former to be interpreted as the tensor product of lenses, which is pairwise cartesian product on the covariant and contravariant parts. This is an uncontroversial choice, but importantly this product is symmetric monoidal but not cartesian, so it means that we are designing some kind of linear type theory.

My third design choice is that I want negation to be interpreted as swapping the covariant and contravariant parts. This sounds uncontroversial at first - many well known semantic categories do the same thing - until we notice that it is not functorial. That is to say, if we have a lens $(X, X’) \to (Y, Y’)$ then in general we can’t make a lens between $(X’, X)$ and $(Y’, Y)$ in either direction. Years ago I wrote a paper called Coherence for lenses and open games in which this non-functorial pair swapping featured heavily, and I still stake everything on the claim that it is the right way to go.

At this point we have enough to build a term language and its interpretation. I will add a single ground type which will be interpreted as purely covariant.

data Ty : Type where
  Unit : Ty
  Ground : Ty
  Not : Ty -> Ty
  Tensor : Ty -> Ty -> Ty

mutual
  Cov : Ty -> Type
  Cov Unit = Unit
  Cov Ground = Nat
  Cov (Not x) = Con x
  Cov (Tensor x y) = (Cov x, Cov y)

  Con : Ty -> Type
  Con Unit = Unit
  Con Ground = Unit
  Con (Not x) = Cov x
  Con (Tensor x y) = (Con x, Con y)

Let’s think through the consequences of these choices. We think of Tensor as linear conjunction, so its neutral element Unit is linear truth. The interpretation of Unit is the pair $(1, 1)$, and so Not Unit - which we would think of as linear falsity - has the same interpretation. So we have a linear logic where falsity and truth coincide semantically. Similarly, the de Morgan dual of Tensor, which we would call linear disjunction, coincides with it semantically. So we have an inconsistent interpretation of linear logic. This is nowhere near as bad as it sounds, since many reasonable semantic categories do the same, but we need to keep it in mind.

Since Tensor is a perfectly cromulent symmetric monoidal product, its introduction and elimination rules will be exactly the same as the ones in my previous post. But the negation rules are going to be quite a puzzle.

Our interpretation of negation is strictly involutive - swapping twice is a no-op - something we can call a classical linear negation. This means our semantics validates the principles of double negation introduction and double negation elimination: both of them are interpreted as an identity lens.

The principle of explosion says that $p$ and $\neg p$ together entail falsity, for every proposition $p$. Since falsity and truth coincide, for us the principle of explosion says that $p$ and $\neg p$ together entail truth. This is a valid principle in our semantics. Suppose $p$ is interpreted as $(X, X’)$, then $p \otimes \neg p$ is interpreted as $(X \times X’, X’ \times X)$. There is indeed a canonical lens $(X \times X’, X’ \times X) \to (1, 1)$, namely the “turnaround” lens, which I normally call a counit.

In Idris, suppose we have

explosion : {a : Ty} -> Term [a, Not a] Unit

Then we must have

eval explosion : {a : Ty} -> All Cov [a, Not a] -> (Unit, Unit -> All Con [a, Not a])

which up to isomorphism reduces to

eval explosion : {a : Ty} -> (Cov a, Con a) -> (Con a, Cov a)

Of course, we want to implement eval so that this gives us the swap function.

The de Morgan dual of the principle of explosion is the principle of excluded middle, which says that truth entails $p$ or $\neg p$. Remembering that our conjunction and disjunction coincide, if $p$ has interpretation $(X, X’)$ then excluded middle would denote a lens $(1, 1) \to (X \times X’, X’ \times X)$. In general there is no lens of this type, so our semantics does not validate excluded middle.

In Idris, suppose we had

lem : {a : Ty} -> Term [] (Tensor a (Not a))

Then we must have

eval lem : {a : Ty} -> All Cov [] -> ((Cov a, Con a), (Con a, Cov a) -> All Con [])

which up to isomorphism reduces to

eval lem : {a : Ty} -> (Cov a, Con a)

which is impossible as soon as we introduce any types that are not pointed.

A logic puzzle

This suggests a logic puzzle: can we design a proof system for negation that validates double negation introduction, double negation elimination and the principle of explosion, but does not validate excluded middle?

After some tinkering I did indeed invent a system with these properties. Sadly it turned out to be a red herring, since it ended up proving these principles that are valid for lenses in terms of more primitive principles that are not valid for lenses. But I still think it’s an interesting enough sideline to report here.

The system I designed was a 2-sided hybrid of a natural deduction calculus and a sequent calculus, with general right-elimination, and both left-elimination and right-introduction restricted to empty sequents on the right. In standard proof theory syntax I would write it like this:

\[\frac{\Gamma, \varphi \vdash}{\Gamma \vdash \neg \varphi} (RI) \qquad \frac{\Gamma, \neg \varphi \vdash}{\Gamma \vdash \varphi} (LE) \qquad \frac{\Gamma \vdash \varphi, \Delta \qquad \Gamma' \vdash \neg \varphi, \Delta'}{\Gamma, \Gamma' \vdash \Delta, \Delta'} (RE)\]

In Idris:

data Term : List Ty -> List Ty -> Type where
  Var : Term [x] [x]
  LAct : Symmetric xs' xs -> Term xs ys -> Term xs' ys
  RAct : Term xs ys -> Symmetric ys ys' -> Term xs ys'
  NotIntroR : Term (x :: xs) [] -> Term xs [Not x]
  NotElimL : Term (Not x :: xs) [] -> Term xs [x]
  NotElimR : Simplex xs1 xs2 xs3 -> Simplex ys1 ys2 ys3
          -> Term xs1 (y :: ys1) -> Term xs2 (Not y :: ys2) -> Term xs3 ys3

Symmetric is the structure for permutations that I introduced in the previous post.

Here are what our principles look like, together with some non-proofs that are ruled out by the restrictions on right-introduction and left-elimination:

dni : {a : Ty} -> Term [a] [Not (Not a)]
dni = NotIntroR (LAct (Insert (There Here) (Insert Here Empty)) 
                      (NotElimR (Left Right) Right Var Var))

dne : {a : Ty} -> Term [Not (Not a)] [a]
dne = NotElimL (NotElimR (Left Right) Right Var Var)
-- ruled out by NotIntroR restriction
-- dne = NotElimR Right (Left Right) (NotIntroR Var) Var

explosion : {a : Ty} -> Term [a, Not a] []
explosion = NotElimR (Left Right) Right Var Var

lem : {a : Ty} -> Term [] [Not a, a]
-- ruled out by NotIntroR restriction
-- lem = NotIntroR Var
-- ruled out by NotElimL restriction
-- lem = NotElimL (NotElimL (NotElimR (Left Right) Right Var Var))

Unfortunately, although my restricted left-elimination and right-introduction rules can be used to prove the semantically valid principles of double negation introduction and elimination, they are themselves not semantically valid. The problems start to appear once we add back in the rules for tensor, which in this 2-sided calculus are

  TensorIntro : Simplex xs1 xs2 xs3 -> Simplex ys1 ys2 ys3
             -> Term xs1 (y1 :: ys1) -> Term xs2 (y2 :: ys2) 
             -> Term xs3 (Tensor y1 y2 :: ys3)
  TensorElim : Simplex xs1 xs2 xs3 -> Simplex ys1 ys2 ys3
            -> Term xs1 (Tensor x y :: ys1) -> Term (x :: y :: xs2) ys2 
            -> Term xs3 ys3

Now we can write a bad term:

bad : {a : Ty} -> Term [] [Not (Tensor a (Not a))]
bad = NotIntroR (TensorElim (Left Right) Right Var explosion)

Although these rules don’t seem to be strong enough to prove the distributive law between tensor and negation, semantically this is the same shape as excluded middle. I think it would be possible to restrict left-elimination and right-introduction differently to rule out this kind of thing, but only at the expense of leaving us with unprovable instances of double negation introduction and elimination.

Structurally involutive negation

Although I would love to come up with a calculus that fulfills my requirements using pure logic, I currently believe that it’s impossible. So instead I will bring out the big guns, and use a Structure. The methodology I introduced in the previous post yields a clean conceptual separation into syntax and logic. If we want to say that two things are syntactically identical, for example permutations of contexts, we use a Structure to encode that. So what we are about to do is to encode a principle that $p$ and $\neg \neg p$ are not merely logically equivalent but syntactically identical.

This is how we do it:

data Parity : Ty -> Ty -> Type where
  Id : Parity x x
  Raise : Parity x y -> Parity x (Not (Not y))
  Lower : Parity x y -> Parity (Not (Not x)) y

data Involutive : Structure Ty where
  Empty : Involutive [] []
  Insert : Parity x y -> Insertion y ys zs -> Involutive xs ys -> Involutive (x :: xs) zs

An element of Involutive xs ys is a witness that ys is a permutation of xs but with an arbitrary number of double negatives inserted or removed.

With double negation introduction and elimination taken care of, all we have to do is to make a logic that validates the principle of explosion and not excluded middle, which is easy: it’s an ordinary 1-sided natural deduction calculus with the negation introduction rule omitted.

data Term : List Ty -> Ty -> Type where
  Var : Term [x] x
  Act : Involutive xs ys -> Term ys t -> Term xs t
  NotElim : {xs, ys : List Ty} -> {default (simplex xs ys) prf : _}
         -> Term xs t -> Term ys (Not t) -> Term prf.fst Unit
  TensorIntro : {xs, ys : List Ty} -> {default (simplex xs ys) prf : _}
             -> Term xs t -> Term ys t' -> Term prf.fst (Tensor t t')
  TensorElim : {xs, ys : List Ty} -> {default (simplex xs ys) prf : _}
            -> Term xs (Tensor x y) -> Term (x :: y :: ys) z -> Term prf.fst z

We can write exactly the terms that we want to:

dni : {a : Ty} -> Term [a] (Not (Not a))
dni = Act (Insert (Raise Id) Here Empty) Var

dne : {a : Ty} -> Term [Not (Not a)] a
dne = Act (Insert (Lower Id) Here Empty) Var

explosion : {a : Ty} -> Term [a, Not a] Unit
explosion = NotElim Var Var

lem : {a : Ty} -> Term [] (Tensor a (Not a))
-- impossible

And it’s now possible to write a well typed interpreter for this definition of terms, although I’ll skip it here because it involves several pages of mostly tedious boilerplate code. In the next post we’ll add the missing scoping rules to our language, so that by the time we come back to the well typed interpreter in post number 4, we’ll be able to use it to do a little bit of differentiable programming.

On Modelling

2024-09-02T00:00:00+00:00

Cross-posted from Oliver’s EconPatterns blog

Despite its bad reputation, “flat earth” is a perfectly scientific, mathematically grounded, and highly useful model of reality.

Indeed, it might be the perfect example to illustrate George Box’s famous aphorism about how all models are wrong, but some are useful.

We confirm its usefulness every time we open a map, on paper or on a screen, and manage to get from A to B thanks to its guidance, nevermind that it (and we) completely ignored the curvature of the earth all along the way.

Of course we could’ve consulted a globe, but for most everyday pairings of A and B, a travel-sized globe won’t help us much in navigating the route, and a globe that’s big enough to provide us with sufficient detail about our particular route would be much too big to carry around.

Indeed, if we push this forward, of the hierarchy of simplifying abstractions:

earth is flat
earth is spherical (a ball)
earth is a rotational oblate ellipsoid (a pill)
earth is a rotational oblate ellipsoid with a variance in surface elevation never exceeding 0.2% of its diameter. the last one — hills and valleys — matters much more in everyday life than the knowledge that earth is spherical or even an ellipsoid.

That doesn’t mean it’s entirely useless knowledge. Nobelist Ken Arrow’s very first academic publication, about optimizing flight paths just after World War 2, pushed the envelope, so to speak, from a planar world to a spherical world.

“All the literature assumed that the world was flat, that everything was on a plane, which may be germane if you’re flying a hundred miles. But we were already flying planes across the Atlantic, from Newfoundland to Scotland. It turned out to be an interesting mathematical problem to change these results to be applicable to the sphere — and that was my contribution.” — Kenneth Arrow, 1995.

Indeed today, Arrow’s contribution is used in any long-distance flight planning software, and its effect is visible every time we fly from Frankfurt to New York and are surprised looking out the window to find ourselves above Greenland.

But we shouldn’t be led into thinking that before Arrow scientists believed that the earth is flat. They just recognized that for their task it didn’t matter that it wasn’t, and at a time when “computers” were still human professionals rather than electronic devices, simplifying the calculations mattered.

Unobservables and counterfactuals

One reason why “flat earth” is such a great example for proper modeling is that it gets the point about scope across, simply because modeling scope matches geographic scope: for short hops, flat earth is perfectly fine, but for transatlantic flights, you’re bound to run into trouble. Somewhere inbetween is a fuzzy boundary where the simple model gradually fails and complication becomes necessary.

Another reason is that it’s a perfect little ploy to expose a particular type of academic conceit, simply because it goes against the Pavlovian reflex by certain academics to roll out “flat earth” as the synecdoche for conclusively disproven pseudoscience.

But there is a critical difference between claiming earth is flat (an empirical hypothesis without support) and proposing a deliberate counterfactual of a flat earth (a modeling design choice), and it strikes at the heart of George Box’s aphorism, which we could amend to say that models are useful because they are wrong.

A map is useful because it’s not the territory.

This distinction is crucial, because so many people, including and maybe especially academics get it wrong: a counterfactual is not the same as an unobservable.

Unobservables are hidden truths. Counterfactuals are openly expressed falsehoods.

In model design, counterfactuals — things we invoke even if we know they’re wrong — play an important role in the form of assumptions, which is why making assumptions explicit is a crucial but oft-ignored exercise in model design.

From flat to round

Graduate students in econometrics often get the advice (or at least used to) to pick up Peter Kennedy’s A Guide to Econometrics in addition to whichever weighty and forbidding tome is assigned as the official textbook (typically Greene’s doorstop), with the undertone that Kennedy’s book might provide a cheat code to unlock the arcane secrets encloistered in its more voluminous peer.

Kennedy succeeds in doing this not because he dilutes the mathematical heft of the official textbooks, but because he offers a very succinct exposé of how we should approach ordinary least squares (OLS), the workhorse of econometric modeling. Step by step:

This is the original OLS setup
Here are the five major assumptions that undergird it
Because of these assumptions, the scope of OLS in its original form is quite limited
But we can, one by one, test if each assumption is violated and implement a fix
And if everything fails, here are a few alternatives…

This is so lucid that it’s surprising (and somewhat disheartening) to see it rarely ever expressed in this succinct — and compositional — form.

Assumptions are counterfactuals that limit the scope, and to expand the scope we have to investigate and potentially drop some of the assumptions, but this always involves a tradeoff between parsimony and universality.

Models are by design complexity-reducing conceits. But for this to be successful the modeler has to be willing to start by ruthlessly reducing complexity to expose the underlying mechanism, and academia isn’t always an environment where Occam’s razor is sharp.

Stereotypically speaking, academia is incentivized to err on the side of giving convoluted answers, including giving convoluted answers to simple questions, or even the worst of all words: convoluted wrong answers to simple questions.

Pretty much everyone in academia who laughed about large language models getting very simple questions horribly wrong (“9.11>9.9”, “how to ferry a goat”, “countries with ‘mania’”) should’ve felt a pang of recognition. Trying to get the big questions right often comes at the cost of getting the simple questions wrong. That might be an acceptable tradeoff in academia, in design it can be fatal.

“Since all models are wrong the scientist cannot obtain a “correct” one by excessive elaboration. On the contrary following William of Occam he should seek an economical description of natural phenomena. Just as the ability to devise simple but evocative models is the signature of the great scientist so overelaboration and overparameterization is often the mark of mediocrity.” — George Box, 1976.

The point Box tries to drive home is not only that there are decreasing returns to upping model complexity and pursuing correctness is an elusive goal, but that returns are indeed quite often negative.

Models and the cybernetic economy

The scene-setting assumption of EconPatterns, expressed in the very first post, is that we operate in a cybernetic economy were macroeconomic aggregates are often deceptive.

Economic engines — take for example a dynamic pricing model for an airline or an energy provider — are elaborate beasts by necessity. If we want to capture time, portfolio, geography, bundling, and other consumer preferences in price finding, we are quickly confronted with a staggering number of variables we have to juggle to produce anything coherent, nevermind accurate, which seems to go counter to Box’s remonstrances about beauty in simplicity.

But the seeming contradiction is easily resolved. Even the newsletter emphasized that “economics usually skips this operational layer for the sake of expositional expediency, and for the most part it does ok doing so.”

The skill in modeling rests first and foremost in the ability to ruthlessly pursue parsimony, but also in the ability to recognize when and where parsimony fails us.

Translated into a design strategy, this means both to have a mental model of the ultimate end product when starting with the first sketches, but also to recognize the underlying fundamental mechanisms — the primitives — in the end product.

The only way to resolve this is to modularize the design process, not only to make it composable (we can assemble the system from the modules), but also compositional (we can infer system behavior from module behavior).

What if models are wrong?

Anyone who has ever spent any time in the engine rooms of capitalism knows how ubiquitous quantitative modeling is to predict anything at all. Even smalltown bakeries predict sales to decide how much flour to buy and how many loaves of bread to bake. Insurances run massive predictive operations staffed by actuaries. Even the microchips in our smart phones use prediction to allocate tasks.

We are surrounded by model-based predictions in our everyday lives, indeed one might claim our livelihoods depend on them. We just choose to ignore them because they’re mostly operating quite well until we’re confronted with the consequences of them breaking down — the negative surprise that gets our attention.

Undergrad microeconomic classes at business schools teach expected value of imperfect information (EVII) as a simple “managerial” framework to explain Bayesian updating from a decision-theoretic perspective.

If you as a decision maker have a 20% chance of being right in your predictions, how much would you pay someone who has a track record of being right 30% of the time for a particular prediction? Not much, you’d think.

Narrowly speaking, that’s a pretty good mental model about how the consulting industry works, but a bit more philosophically speaking, the idea that models have to be perfect to be useful is a (very common) fallacy — usually expressed as “we don’t have enough data to model anything” or “the model made a wrong prediction so the model must be wrong”. Indeed models can often be especially useful even if they are far from accurate.

Strictly economically speaking, models are useful if they shift the probability of being right about the future upward, even if it’s only by a small delta. We only have to compare the salaries of baseball players who hit a .200 batting average (the proverbial Mendoza line) with those who hit at a .300 clip. Getting it wrong 70 percent of the time can be a pretty lucrative skill under the right circumstances.

The purpose of modeling is to reduce the propensity of negative surprise, which is why we usually only notice when they do the opposite.

To update Max Planck’s famous dictum that science progresses one funeral at a time, formal modeling helps us to speed up science so that it progresses one public embarrassment at a time — which happens every time a confidently made prediction is stumped by reality.¹

To up the ante, managerially speaking, models are important decision tools even if they don’t improve chances of being right at all, simply because they act as tiebreaker tools to overcome decision paralysis, especially in scenarios where “we don’t have enough data to model anything”.

It’s a simple explanation why soothsaying and bird divining existed throughout history, and why they’re still around. Sometimes people need a device — any device — to prune the branches of their personal decision trees, to overcome the form of decision paralysis we know as “procrastination”.

Good models, beyond improving predictive accuracy, also help simply by providing a formal grid to map out the structure of the decision scenario. This is what makes game (and decision) theory relevant: it’s a formal tool to map out the interrelatedness of scenarios around the decisions the participants face.

How to decide if models are wrong

Popular opinions about modeling range from “a modeled prediction establishes a scientific fact” to “models can’t predict anything at all since (prominent example where a model went wrong)”. Strangely enough, both mental models seem to be especially popular in the natural sciences, sometimes even proposed by the same person.

Neither of these extreme positions have any grounding in reality, and their popularity is likely more the result of ambiguity intolerance and conceptual problems with the idea that improvement can come in the form of an upshift in likelihoods.

As Milton Friedman put it in the context of positive economics as a scientific endeavor:

“Its task is to provide a system of generalizations that can be used to make correct predictions about the consequences of any change in circumstances. Its performance is to be judged by the precision, scope, and conformity with experience of the predictions it yields.” — Milton Friedman, 1953.

But this is only one side of a coin. We can devise models that are purely predictive (the internal causal mechanism remains opaque to us) or models that are purely causal (they make no claim about predictive accuracy or might even be deliberately wrong in the expectation that how and where they go wrong reveals something about the internal causal mechanics) — like we do with pretty much every financial plan ever.

Most models end up somewhere in-between on that spectrum. The important thing is to be upfront about what the design objective is.

I’ve written about the role of modeling in science, the social sciences, and economics before, but it remains a contested issue, so it felt like devoting a whole post to it might be worth the effort.

My take is ultimately shaped by my own experience in industry, and in turn shapes what I am trying to achieve with EconPatterns.

The short version is that formal modeling is a relevant part of economic practice, especially the unobserved part (the “engine room”) of economic practice, that a sound understanding of formal modeling tools is necessary for anyone within economics (even if the need for mathematical rigor varies widely between fields).

The economy is also a data-rich environment, and we have enough experience to know that certain things in the economic realm are bound to follow discernible and generalizable patterns.

But formal modeling has to rest on a sound conceptual understanding, and economic endeavors, especially those that include economic design, should spend enough time on the conceptual architecture to not end up building complicated models that fail at the simple answers.

On the same topic, see also Philipp Zahn’s perspective.

It should be noted here that Planck’s dictum itself is an astute observation about belief systems and the glacial progress of belief propagation in academia. This might be worth its own newsletter. ↩

Foundations of Bidirectional Programming I: Well-Typed Substructural Languages

2024-08-26T00:00:00+00:00

Cross-posted from Zanzi’s blog

This is the first post in a new series documenting my work developing a bidirectional programming language, in which all programs are interpreted as optics. This is something I’ve been thinking about for a long time, and eventually I became convinced that there were enough subtle issues that I should take things extremely slowly and actually learn some programming language theory. As a result, this post will not be about categorical cybernetics at all, but is a foundation to a huge tower of categorical cybernetics machinery that I will build later.

The first issue is that because the tensor product of optics is not cartesian, a language for optics is necessarily substructural. So this post is about a way to implement substructural languages in a way that avoids hidden pitfalls. The code in this post is Idris2, a dependently typed language.

We begin with a tiny language for types, with monoidal products (a neutral name because later we will be making it behave like different kinds of product), a unit type to be the neutral element of the monoidal product, and a “ground” type that is intended to contain some nontrivial values.

data Ty : Type where
  Unit : Ty
  Ground : Ty
  Tensor : Ty -> Ty -> Ty

Cartesian terms

Although we have used the name “tensor”, suppose we want to make an ordinary cartesian language where variables can be implicitly copied and discarded. Here is a standard way to do it: it is an intuitionistic natural deduction calculus.

data Term : List Ty -> Ty -> Type where
  -- A variable is a term if we can point to it in scope
  Var : Elem x xs -> Term xs x
  -- Unit is always a term in every scope
  UnitIntro : Term xs Unit
  -- Pattern matching on Unit, redunant here but kept for comparison to later
  UnitElim : Term xs Unit -> Term xs x -> Term xs x
  -- A pair is a term if each side is a term
  TensorIntro : Term xs x -> Term xs y -> Term xs (Tensor x y)
  -- Pattern matching on a pair, adding both sides to the scope
  TensorElim : Term xs (Tensor x y) -> Term (x :: y :: xs) z -> Term xs z

The constructor for Var uses Elem, a standard library type that defines pointers into a list:

data Elem : a -> List a -> Type where
  Here : Elem x (x :: xs)
  There : Elem x xs -> Elem x (x' :: xs)

Here are some examples of programs we can write in this language:

-- \x => ()
delete : CartesianTerm [a] Unit
delete = UnitIntro

-- \(x, y) => x
prjl : CartesianTerm [a, b] a
prjl = Var Here

-- \(x, y) => y
prjr : CartesianTerm [a, b] b
prjr = Var (There Here)

-- \x => (x, x)
copy : CartesianTerm [a] (Tensor a a)
copy = TensorIntro (Var Here) (Var Here)

-- \(x, y) => (y, x)
swap : CartesianTerm [a, b] (Tensor b a)
swap = TensorIntro (Var (There Here)) (Var Here)

The thing that makes this language cartesian and allows us to write these 3 terms is the way that the context xs gets shared by the inputs of the different term constructors. In the next section we will define terms a different way, and then none of these examples will typecheck.

Planar terms

Next let’s go to the opposite extreme and build a fully substructural language, in which we cannot delete or copy or swap. I learned how to do this from Conor Mc Bride and Zanzi. Here is the idea:

data Term : List Ty -> Ty -> Type where
  -- A variable is a term only if it is the only thing in scope
  Var : Term [x] x
  -- Unit is a term only in the empty scope
  UnitIntro : Term [] Unit
  -- Pattern matching on Unit consumes its scope
  UnitElim : Term xs Unit -> Term ys y -> Term (xs ++ ys) y
  -- Constructing a pair consumes the scopes of both sides
  TensorIntro : Term xs x -> Term ys y -> Term (xs ++ ys) (Tensor x y)
  -- Pattern matching on a pair consumes its scope
  TensorElim : Term xs (Tensor x y) -> Term (x :: y :: ys) z -> Term (xs ++ ys) z

This is a semantically correct definition of planar terms and it would work if we had a sufficiently smart typechecker, but for the current generation of dependent typecheckers we can’t use this definition because it suffers from what’s called green slime. The problem is that we have types containing terms that involve the recursive function ++, and the typechecker will get stuck when this function tries to pattern match on a free variable. (I have no idea how you learn this if you don’t happen to drink in the same pubs as Conor. Dependently typed programming has a catastrophic lack of books that teach it.)

The fix is that we need to define a datatype that witnesses that the concatenation of two lists is equal to a third list - a witness that the composition of two things is equal to a third thing is called a simplex. The key idea is that this datatype exactly reflects the recursive structure of ++, but as a relation rather than a function:

data Simplex : List a -> List a -> List a -> Type where
  Right : Simplex [] ys ys
  Left : Simplex xs ys zs -> Simplex (x :: xs) ys (x :: zs)

Now we can write a definition of planar terms that we can work with:

data Term : List Ty -> Ty -> Type where
  Var : Term [x] x
  UnitIntro : Term [] Unit
  UnitElim : Simplex xs ys zs 
          -> Term xs Unit -> Term ys y -> Term zs y
  TensorIntro : Simplex xs ys zs 
             -> Term xs x -> Term ys y -> Term zs (Tensor x y)
  TensorElim : Simplex xs ys zs 
            -> Term xs (Tensor x y) -> Term (x :: y :: ys) z -> Term zs z

This language is so restricted that it’s hard to show it doing anything, but here is one example of a term we can write:

-- \(x, y) => (x, y)
foo : Term [a, b] (Tensor a b)
foo = TensorIntro (Left Right) Var Var

Manually defining simplicies, which cut a context into two halves, is very good as a learning exercise but eventually gets irritating. We can direct Idris to search for the simplex automatically:

data Term : List Ty -> Ty -> Type where
  Var : Term [x] x
  UnitIntro : Term [] Unit
  UnitElim : {auto prf : Simplex xs ys zs} 
          -> Term xs Unit -> Term ys y -> Term zs y
  TensorIntro : {auto prf : Simplex xs ys zs} 
             -> Term xs x -> Term ys y -> Term zs (Tensor x y)
  TensorElim : {auto prf : Simplex xs ys zs} 
            -> Term xs (Tensor x y) -> Term (x :: y :: ys) z -> Term zs z

foo : Term [a, b] (Tensor a b)
foo = TensorIntro Var Var

This works, but I find that the proof search gets confused easily (although it works fine for the baby examples in this post), so let’s pull out the big guns and write a tactic:

concat : (xs, ys : List a) -> (zs : List a ** Simplex xs ys zs)
concat [] ys = (ys ** Right)
concat (x :: xs) ys = let (zs ** s) = concat xs ys in (x :: zs ** Left s)

This function takes two lists and returns their concatenation together with a simplex that witnesses this fact. Here ** is Idris syntax for both the type former and term former for dependent pair (aka. Sigma) types.

data Term : List Ty -> Ty -> Type where
  Var : Term [x] x
  UnitIntro : Term [] Unit
  UnitElim : {xs, ys : List Ty} 
          -> {default (concat xs ys) prf : (zs : List Ty ** Simplex xs ys zs)}
          -> Term xs Unit -> Term ys y -> Term prf.fst y
  TensorIntro : {xs, ys : List Ty} 
              -> {default (concat xs ys) prf : (zs : List Ty ** Simplex xs ys zs)}
              -> Term xs x -> Term ys y -> Term prf.fst (Tensor x y)
  TensorElim : {xs, ys : List Ty} 
            -> {default (concat xs ys) prf : (zs : List Ty ** Simplex xs ys zs)} 
            -> Term xs (Tensor x y) -> Term (x :: y :: ys) z -> Term prf.fst z

foo : {a, b : Ty} -> Term [a, b] (Tensor a b)
foo = TensorIntro Var Var

I find Idris’ default syntax to be a bit awkward, but it feels to me like a potentially very powerful tool, and something I wish Haskell had for scripting instance search.

Context morphisms

Unfortunately, going from a planar language to a linear one - that is, ruling out copy and delete but allowing swaps - is much harder. I figured out a technique for doing this that turns out to be very powerful and give very fine control over the scoping rules of a language.

The idea is to isolate a category of context morphisms (technically a coloured pro, that is a strict monoidal category whose monoid of objects is free). Then we will parametrise a planar language by an action of this category. The good news is that this is the final iteration of the definition of Term, and we’ll be working with it for the rest of this blog post.

Structure : Type -> Type
Structure a = List a -> List a -> Type

data Term : Structure Ty -> List Ty -> Ty -> Type where
  Var : Term hom [x] x
  Act : hom xs ys -> Term hom ys x -> Term hom xs x
  UnitIntro : Term hom [] Unit
  UnitElim : {xs, ys : List Ty} 
          -> {default (concat xs ys) prf : (zs : List Ty ** Simplex xs ys zs)}
          -> Term hom xs Unit -> Term hom ys y -> Term hom prf.fst y
  TensorIntro : {xs, ys : List Ty}
             -> {default (concat xs ys) prf : (zs : List Ty ** Simplex xs ys zs)}
             -> Term hom xs x -> Term hom ys y -> Term hom prf.fst (Tensor x y)
  TensorElim : {xs, ys : List Ty} 
            -> {default (concat xs ys) prf : (zs : List Ty ** Simplex xs ys zs)} 
            -> Term hom xs (Tensor x y) -> Term hom (x :: y :: ys) z 
            -> Term hom prf.fst z

First, let’s recover planar terms. To do this, we want to define a Structure where hom xs ys is a proof that xs = ys:

data Planar : Structure a where
  Empty : Planar [] []
  Whisker : Planar xs ys -> Planar (x :: xs) (x :: ys)

Now let’s deal with linear terms. For that, we want to define a Structure where hom xs ys is a proof that ys is a permutation of xs. We can do this in two steps:

-- Insertion x xs ys is a witness that ys consists of xs with x inserted somewhere
data Insertion : a -> List a -> List a -> Type where
  -- The insertion is at the head of the list
  Here : Insertion x xs (x :: xs)
  -- The insertion is somewhere in the tail of the list
  There : Insertion x xs ys -> Insertion x (y :: xs) (y :: ys)

data Symmetric : Structure a where
  -- The empty list has a unique permutation to itself
  Empty : Symmetric [] []
  -- Extend a permutation by inserting the head element into the permuted tail
  Insert : Insertion x ys zs -> Symmetric xs ys -> Symmetric (x :: xs) zs

(Incidentally, this is the point where I realised that although Idris looks like Haskell, programming in it feels a lot closer to programming in Prolog.)

Now we write swap as term:

swap : {a, b : Ty} -> Term Symmetric [a, b] (Tensor b a)
swap = Act (Insert (There Here) (Insert Here Empty)) (TensorIntro Var Var)

Explicitly cartesian terms

Now we can come full circle and redefine cartesian terms in a way that uniformly matches the way we do substructural terms.

data Cartesian : Structure a where
  -- Delete everything in scope
  Delete : Cartesian xs []
  -- Point to a variable in scope and make a copy on top
  Copy : Elem y xs -> Cartesian xs ys -> Cartesian xs (y :: ys)

With this, we can rewrite all the terms we started with:

delete : {a : Ty} -> Term Cartesian [a] Unit
delete = Act Delete UnitIntro

prjl : {a, b : Ty} -> Term Cartesian [a, b] a
prjl = Act (Copy Here Delete) Var

prjr : {a, b : Ty} -> Term Cartesian [a, b] b
prjr = Act (Copy (There Here) Delete) Var

copy : {a : Ty} -> Term Cartesian [a] (Tensor a a)
copy = Act (Copy Here (Copy Here Delete)) (TensorIntro Var Var)

swap : {a, b : Ty} -> Term Cartesian [a, b] (Tensor b a)
swap = Act (Copy (There Here) (Copy Here Delete)) (TensorIntro Var Var)

Let’s end with a party trick. What would a cocartesian language look like - one where we can’t delete or copy, but we can spawn and merge?

Co : Structure a -> Structure a
Co hom xs ys = hom ys xs

-- spawn : Void -> a
-- spawn = \case {}
spawn : {a : Ty} -> Term (Co Cartesian) [] a
spawn = Act Delete Var

-- merge : Either a a -> a
-- merge = \case {Left x => x; Right x => x}
merge : {a : Ty} -> Term (Co Cartesian) [a, a] a
merge = Act (Copy Here (Copy Here Delete)) Var

-- injl : a -> Either a b
-- injl = \x => Left x
injl : {a, b : Ty} -> Term (Co Cartesian) [a] (Tensor a b)
injl = Act (Copy Here Delete) (TensorIntro Var Var)

-- injr : b -> Either a b
-- injr = \y => Right y
injr : {a, b : Ty} -> Term (Co Cartesian) [b] (Tensor a b)
injr = Act (Copy (There Here) Delete) (TensorIntro Var Var)

Since at the very beginning we added a single generating type Ground, and the category generated by one object and finite coproducts is finite sets and functions, this language can define exactly the functions between finite sets. For example, there are exactly 4 functions from booleans to booleans:

id, false, true, not : Term (Co Cartesian) [Ground, Ground] (Tensor Ground Ground)
id = Act (Copy Here (Copy (There Here) Delete)) (TensorIntro Var Var)
false = Act (Copy Here (Copy Here Delete)) (TensorIntro Var Var)
true = Act (Copy (There Here) (Copy (There Here) Delete)) (TensorIntro Var Var)
not = Act (Copy (There Here) (Copy Here Delete)) (TensorIntro Var Var)

That’s enough for today, but next time I will continue using this style of term language to start dealing with the difficult issues of building a programming language for optics.

Beliefs, Belief Propagation and Belief Clusters

2024-08-15T00:00:00+00:00

Cross-posted from Oliver’s EconPatterns blog

There’s a non-zero chance that sometime in the not-so-far future we will think of the “Bayesian Revolution” in the same way we think of the “Marginal Revolution”.

Bayesian beliefs give us the same opportunity to think of expectations as an attribute linked to the observer rather than to the observed object in the same way utilities gave us an opportunity to accept that value is not an attribute intrinsic to an object, but exists only in the eye of the beholder.

Much of modern economics rests on the recognition that differences in valuation create opportunities for mutually beneficial — and thus voluntary — exchange.

We’re still a few steps away from translating that recognition to differences in expectations, in no small part because most of the current effort seems to go into trying to shoehorn Bayesian statistics into the domains dominated by trad statistics (aka frequentism), but one dimension on which we’re inching towards a better understanding of subjective probabilities is that it’s slowly dawning on us that there might be perfectly legitimate reasons why different individuals might attach different likelihoods to the same event — and even more, why their estimates might diverge over time.

The operative word being “legitimate” here, since if we start from the idea that whichever event we’re only partially observing was suddenly revealed to us in full, a persistent divergence in beliefs about this event surely means that at least one party must be dead wrong.

And even if we’re allowing that there might be multiple paths to the ultimate objective truth, if one party is wrong, it must surely be shown up in the future.

Economists love to use sports metaphors when teaching statistics, simply because sport is a realm where the outcome is recorded right at the conclusion of the contest, from a single authoritative source, without ambiguity.

As a teaching device this is perfectly understandable in that it takes away a distraction which seems peripheral to the topic, but ultimately every practitioner will run into the problem that such an occurrence of an unfailing, immediate, and impartial single source of truth is quite rare in the real world.

(Indeed once we’re getting into the nitty-gritty of how the outcomes of sports events are tallied, it’s quickly becoming obvious that the “unfailing, immediate, and impartial” umpire is mostly a figment of our imagination. From photo finishes to disqualifications, not to mention accusations of favoritism, the idea that the true value of a random variable can suddenly be disclosed with finality is a popular ploy in economics that doesn’t even have a realistic foil in sports. Economic theory might get away with such a foreshortening of reality, but economic design doesn’t.)

We can go one step further and claim that for most uncertain events, meaning all events that are only partially observable, there might not be an underlying true value, no ground truth, at all. We’re eternally in limbo about what the underlying “true value” is, simply because there is no moment of truth.

In most scenarios the truth typically remains elusive, at the end of a lengthy, costly, meandering, and conflicted discovery process, and we tend to swap in “truth” in the computer science meaning of “(single) source of truth”: that which is held to be true at any stage of the discovery process, as a proxy for the “ground truth”: that which would be held to be true at the end of a fully exhaustive discovery process.

In either form, whether there’s no unimpeachable source of truth or no underlying ground truth, once we make that leap to accepting that the truth is elusive, we suddenly gain access to a far richer world of behaviors, especially collective behaviors.

As a pattern, it means to start from the assumption that observed values are never perfectly true, true values are never perfectly observed, and truthfinding is an asymptotic reconciliation process between conflicting beliefs that’s not guaranteed to terminate in finite time.

Organizations as belief clusters

I have previously called organizations tectonic plates shaped around shared beliefs broken up by fault lines where beliefs — especially beliefs about the future and the likely outcomes of actions taken — are no longer reconcilable.

In this series I am going to be a bit more precise in working out the question how belief convergence or belief divergence shape the coordinated sequence of activities within and across organizations we call a value chain. In particular, how these concepts can be applied across domains: social, political, cultural, religious, with the economic realm just a special case.

One of the first lessons of organization theory is that organizations are paradoxes: they can, under best conditions, produce more collectively than the sum of individual efforts of all members, but this requires that individual efforts have to be constrained away from what the individuals would do if left to their own devices. In other words, to achieve an organizational goal (almost) everyone will have to compromise.

The other lesson for organizations is that they take two steps: first, the aggregation of efforts, and second, the disaggregation of gains. Both have to work in the eyes of the beholders making up the organization for it to work, or otherwise — especially in the case of efforts being exerted in the present but rewards only collected and distributed in the future.

If this doesn’t hold true: every participant holds the shared belief that individual contributions are being rewarded beyond what they could expect outside the organization, it will either disintegrate or has to be propped up by force.

Nothing up to this point has limited what type of organization we’re talking about here. Simply put, we’ll find this type of pattern in any kind of organization: economic, religious, or political, flat or hierarchical, although we expect the kinds of rewards to differ.

Along with the three underlying sources of goal conflict (moving forward vs staying together, moving forward vs staying put, moving in which direction), this gives us a grid to think about the sources of conflict between the individual and the group, and later between ingroup and outgroup, to think about why organizations are shaped the way they are (flat or hierarchical, open or closed…) and why which forms of organizations are prevalent in which environment: why firms are organized differently than political parties, sports clubs, or religious congregations.

Belief propagation and filter bubbles

So far the discussion has looked at shared beliefs from the vantage point of the (formally incorporated) organization. In my previous post, I looked at the question from the perspective of value chains. But what about starting with the individual?

Rather than asking how organizations fail due to a breakdown of shared beliefs, we might as well ask how organizations emerge as a consequence of shared beliefs.

Luckily, most people have heard of filter bubbles, the subgroups on social media channels created (and algorithmically amplified) by positive interaction with like-minded people, and negative interaction with other-minded people.

This is a fundamental economic and social mechanism (formalized in my dissertation before social media or filter bubbles were a thing), not only to socialize with like-minded people, especially to filter incoming information based on whether the sender holds shared beliefs on prior issues, but also to adjust the credibility we assign a sender based on how much the newly transmitted information matches our priors.

If you take this filtering-by-affinity mechanism and impose it on the network structure of the cybernetic economy, you get an information flow pattern known as a “gossip network”.

It is such an ubiquitous mechanism that we find it everywhere, on all levels of social and economic organization, but it also comes across as slightly unevolutionary. If we want to fully understand the hazards of our environment, we should take in all information from all sides and not just amplify the information that confirms what we think we already know. But we don’t. We filter — which makes sense — but we filter out what we don’t want to hear, not what we shouldn’t hear — which doesn’t make sense.

The interesting thing is that we know much more about these mechanisms now thanks to the internet, and especially thanks to recommender systems: Amazon uses them to guess what else we want to buy. Spotify offers new music to discover, Tinder believers matching social preferences translate into a good romantic match.

The reason why recommender systems emerged in the early days of the internet is also tightly connected to Amazon, and later to the resurgence of machine learning thanks to the Netflix prize: the recognition that there is an unsurveyably large number of choice alternatives, and the inevitable corollary that our notion of consumers having well-informed preferences between them — an axiom that undergirds modern microeconomics — is no longer tenable.

Recommender systems have become one of the fundamental economic design templates, and in the process have reshaped economics (and in the case of matchmaking platforms, also our social lives), not only because they provide the raw data for much more fine-tuned consumer choice, but also because they give us a deeper insight into the evolution of preferences.

But there is also a deeper insight which comes from the information good that is being transferred. Preferences are in our economic understanding unimpeachable. They express individual tastes and are as such not rankable by social desirability, as much as we might want to impose our own, undoubtedly (in our minds) more sophisticated, tastes on others, or construct an argument why some tastes are more in tune with a more or less well-defined social good than others.

On the other extreme of the spectrum are mutually agreed-upon ground truths, better known as facts. In-between these extremes preferences (no truth value can be assigned) and facts (undisputed common truth value aligned with ground truth) lies the wide world of counterfactuals: states of the world to which we assign (and, as new knowledge emerges, adjust) truth values between zero and one. This world of counterfactuals inevitably involves the future.

This is a pattern we find everywhere, not only in the economic realm and its various subrealms like entrepreneurship (high individual belief in the success of a venture opportunity countering widespread low beliefs), but also in the political and social realms. We can connect this to Thomas Kuhn’s model of scientific inquiry and Ludwik Fleck’s harmony of illusions, or to the geographic expansion of religion, language, pottery design, agriculture, or any idea based on shared counterfactuals.

The search pattern: avoidance of negative surprise, I’ve already discussed in a previous newsletter. The canonical conflict resolution mechanisms: markets or hierarchies in the economic realm, wars or elections in the political realm, are typically tied to the fundamental pattern of exchange in that realm, as I will discuss in a future newsletter.

Cybercat Institute

Autodiff through function types: Categorical semantics the ultimate backpropagator

Additive containers and additive lenses

Additive containers are cartesian closed

Representing the direct sum

Abstraction and application

There is no free lunch

Dependent optics II: Optics via forcing costates

A refresher on optics

The bicategory of optics

Parametrisation

Forcing costates

From adaptors to optics

From Equilibrium Checking to Learning with the Open Game Engine

From equilibrium checking to dynamic programming

A look inside the Open Game Engine

Learning in practice

The Untapped Potential of Game Theory in Revenue Management

Revenue Management Today

The Game Theory Opportunity

Beyond “Traditional” Game Theory

Case Example: Competitor-Aware Dynamic Pricing

Looking Ahead

An Invitation to Neural Picture Alchemy

Introduction

String diagrams? in my deep learning? It’s more likely than you think.

String Diagrams for Objective Functions

deeplearns? Is it in the room with us right now?

Specialisations

Patterns: Well-Understood Tasks

New Patterns?

Case Study: Computing Nash Equilibria with Gradient Descent

Is this yap cap? The answer may surprise you.

Conclusion

Pipelines Part 2: Categorical Pipelines

Abstracting to categories

Pipelines of lenses

Abstracting to graded categories

Implementing errors with Either

Implementing errors with Any

Conclusion

What Is the CyberCat Institute?

Bidirectional Typechecking with Dependent Lenses

Finding the right types

The $Up$ Rule

The $Var$ Rule

The $App$ Rule

Putting It All Together

Conclusion

Generalized Transformers from Applicative Functors

Part 1: A basic model

Part 2: Fixing the problems

Part 3: Applicative matrix operations

Part 4: Diagrammatics

Part 5: Funcformer

Part 6: A generalized Transformer zoo

Expression trees and DAGs

Probability distributions

Cross-modal models

Part 7: Conclusion

Bidirectional Typechecking is Bidirectional

Bidirectionally typed STLC

Questions and answers

Monadic traversals

Putting it together

Handling the sad path

Pipelines

Optics for UI 1: Deconstructing React with Parametrised Lenses

Stateless components

The Elm Architecture

Stateful components

React was right all along

Conclusion

Programming Pipelines Using Dependent Types

Introducing the pipeline

Handling effects

Debugging programs

Conclusion

Foundations of Bidirectional Programming IV: Running Forwards and Running Backwards

Base types

Implementing errors with `Any`