Friday posts are for jokes and philosophical musings, so here’s one of the latter. I have been wondering away for a while at a phenomenon called “emergent misalignment” in the management and training of LLMs. This is the phenomenon where you train a model to do one bad thing (in this case, to write insecure computer code), and then you find that it has also learned a lot of other nasty things – the chatbot is much more likely to give harmful or unethical advice on questions unrelated to computer code.

You can see the same phenomenon at work with the repeated fiascos of xAI’s “Grok” chatbot. On several occasions, someone has decided to try and tweak the model to be a bit less “woke”, in the sense of more tolerant of a few forms of socially accepted bigotry, and ended up with it saying “I am MechaHitler” or compulsively switching the conversation to white genocide. If you try to make a machine nastier in one specific way, it seems very difficult to stop it getting nastier in lots of other ways.

There’s a pretty natural (although I guess not necessarily correct!) interpretation of this phenomenon, assuming it’s real. Which is to say that if you think of the neutral network in spatial terms, as a way of fitting a hyperplane through a vector space[1], there is some direction in that vector space which runs “good/bad”, and pushing the model in the direction of any particular “bad” thing makes it shift along this axis.

Here’s my question – if emergent misalignment is actually a real and robust phenomenon (and omitting this obviously very necessary caveat from now on), how seriously should we take it? Should we take seriously the possibility that the model is literally extracting the true essence of goodness and badness?

And if not, why not? By fitting the hyperplane in the vector space, the model is telling us that groups of concepts (or groups of tokens which we associate with concepts) are in some way near to each other, and that they are aligned so as to group ones that we universally recognise as “nice” in one part of the space, ones we universally recognise as “bad, taboo” in another part, and other more ambiguous concepts and tokens are spread out in some meaningful way between the two paradigmatic ends of the spectrum.

It seems to me that this is exactly what modern analytic moral philosophy does, but systematised and industrialised. The machine is looking at our ordinary use of language (check), refining the underlying structures which govern our application of evaluative words (check) and attempting to fit these structures together into a non-contradictory and logical schema, as efficiently and parsimoniously as possible. This is ordinary language philosophy and moral intuition, writ large.

Industrialisation, of course matters, and we can’t necessarily be sure that a systematic and algorithmic fitting of structures to tokens is a better insight into ultimate reality than simple introspection.But could they be doing the same thing?And, for example, if you could look at the algorithm and see that one of your own ethical views was actually grouped closer to ones that you found abhorrent, would this be more or less likely to change your mind than reading a survey showing that the majority of voters disagreed with you?

[1] Remember my patented method for visualising a hyperplane in a vector space; think of a normal two dimensional plane in a three dimensional space, while muttering “hyperplane hyplerplane hyperplane” to yourself.

One of my favourite ever short essays was Paul Wilmott’s piece on risk management and the limits of mathematics, from 2008. He invites the reader to imagine that they are present at a magic show, and the greatest magician in the world asks them to name a card. “Ace of Hearts”, for example. The magician then fans out a pack of cards and draws one out without looking.

What’s the chance that this card is the Ace of Hearts?

Obviously, my readers are far too intelligent to answer “1 in 52”, I hope. It’s not a random process (and even if it was, you forgot about the jokers). The smart people answer “100%”, because they understand that the magician has complete control over the trick.

But wait!

Wilmott then points out that this is the greatest magician in the world, not a ten-year-old who has watched some YouTube videos. Surely she isn’t just going to draw a named card from a deck and expect the audience to be impressed? The chance that this card will be the Ace of Hearts is close to zero – it will be tonight’s winning lottery ticket, or a tarot card with your face on it, or a photograph of the magician outside your front door holding up the ace of hearts or something.

It’s a great motivating example to start people thinking about real world risk management and scenario analysis. But the thing which stuck with me, from the time I saw Wilmott do this bit at a lecture, is that he claims that a lot of risk management professionals (and a lot of bank regulators) get very stuck on the idea that the “real” answer is 1 in 52 and that other cases don’t need to be considered because they’re cheating.

I’ve never really experienced someone like this because I move in somewhat different circles and only talk to the more outgoing and interesting kind of quant. But I can completely believe it. The shift in perspective from “what’s possible within a system” to “the wider environment in which the system is embedded” is difficult to achieve.

To take a current example, consider the Constitution of the United States of America, and the way in which it constrains the powers of the executive. A while ago, some politics profs coined the phrase “Green Lantern Theory” (it’s some kind of comics reference), to satirise people who were criticising the Obama administration for lack of ambition. The idea was that it was kind of silly and childish to believe that US Presidents could make things happen simply because it was their will; this simply wasn’t how the system worked.

Anyway, I went back to a few of those guys late last year, to inquire whether they were planning on revising the theory in the light of developments.

In fairness, most of them took it with good humour, but a surprising number of people seemed to regard the question itself as unfair, illegitimate and illustrative of the same kind of childish ignorance as the original Green Lantern Theory. Of course a President who is willing to ignore or override the checks and balances can do whatever he (or, in principle, she) wants! That’s not the point!

At this point, people might reasonably remind me of the fabled “post-it note” which is meant to shape the editorial policy of this ‘stack. (A mythical note that’s meant to be stuck to my computer, saying “don’t use the newsletter to pursue feuds and arguments from social media”). And, fair enough. But I think there is a deeper and more important point about politics underlying things here.

Which is that the defence of Green Lanternism are factually right, but that falsifies the original theory. It is always possible to break norms, if you’re really determined to. Everything does, in fact, depend on having people in positions of power who respect the rules of the game. The British “good chap” system is just much more blatantly in your face about it.

Which might account for the longevity of the Westminster system. It is incredibly fragile, but it’s obviously fragile, and in this way achieves a sort of paradoxical antifragility. In a “good chap” system, when a bad chap shows up, all the good chaps know that they have to band together and oppose or get rid of them, because they know that there are no systemic constraints on badchappery. In a system that’s meant to be full of checks and balances, it is much easier for a kind of bystander effect to develop, where everyone waits for the system to protect itself without understanding that the system is just them.

And, of course, what happens when good chaps do nothing? The triumph of bad chaps, that’s what.

(Programming note! If you liked “The Unaccountability Machine” and you got a book token for Christmas, spend it on “The Score” by C Thi Nguyen. I haven’t seen an advance copy or anything, but I met the author at a conference and he’s extremely funny and clever. I will be trying to promote the book to as many business podcasts as possible, because I very much think that this is a case in which the dead hand of academic analytical philosophy may have robbed the world of one of its greatest management consultants)

Any way … this is a bit of a generalised call for help, on a phenomenon which is absolutely ubiquitous, clearly very economically and politically important, but as far as I can tell unstudied. I’m hoping some of you guys know different. I am talking about the phenomenon of “careerism”.

I have a few stock things to say when I’m in a meeting and want to sound clever without accidentally saying something the boss might disagree with. “How do they do this in other countries”? is always a winner, for example. But “of course, this is all actually going to be decided on the basis of career risk” is even better when the opportunity arises - everyone immediately agrees, and you look not only extremely wise but also attractively cynical and even slightly subversive.

It’s one of those things, like “no matter what your performance review said, your bonus will be equal to your boss’s guess of what would get you to leave, minus epsilon”. It’s clearly empirically true, easily provable from axioms of rationality, but somehow it’s outside the rules of the game to mention it too much. Careerism, and in particular the desire to avoid career risk, is hugely important.

But as far as I can tell, it has only been studied (and studied to death) in one particular niche context. Which is the context of active fund managers, who face downside career risk if they underperform the index, which is not symmetric with the upside from outperforming it. A large amount of empirical finance brain cells have been devoted to demonstrating that this happens - fund managers tend to “hug the index”, and generally to make decisions based on their personal risk aversion to the effect of investment performance on their personal income, rather than the risk aversion appropriate for their clients’ portfolios.

My response to that literature (“well, duh”) is probably a bit unfair, but it’s frustrating that this doesn’t seem to have been studied in any other context (particularly contexts of government and civil service decision making, where I would really like to cite some classic work rather than writing 10,000 words myself). Any ideas?

Why don’t I ever address the most important questions of economics and development? By which I mean, trams?

I am on record as an extreme sceptic of the transformational properties of light rail (capsule summary: building transport infrastructure and hoping it will stimulate economic activity is exactly what cargo cults did; transport is nice and good and we should have more of it, but if you are doing regional or development policy, for god’s sake hustle while you wait). But I thought that the think tank article linked above was very interesting as a sort of indicator of the current State Of Think.

And that state is … not so bad? The trams paper has a lot of things in it that, as an old style cybernetic social democrat, I find really very congenial. Lots of localism, municipal ownership and funding, allocation of costs of moving utilities and so on. Many of the personnel of Britain Remade come from a background in Thatcherite thinktanks, and there’s clearly been a journey there toward what I’d regard as a much more interesting view of state capacity and development. I have apologised in private to some of the people in the NIMBY milieu for unkind jokes along the lines of “libertarians for compulsory purchase”, so I guess I might as well make the apology general and public.

When you see something like this happen, there are obviously two possible reactions – “oh great” and “what are they up to”. I made my choice in favour of the first a bit more than a year ago when I threw in my lot with the Niskanen Center and the Abundance movement, which I did on the grounds of my one consistent political principle. That principle being that progressive social democracy really is not so popular or so blessed with friends that it can afford to be standoffish. If someone is going to support expansion of state capacity and the building of public sector assets, then it is not exactly difficult for me to find some pleasant things to say about Adam Smith.

But there is a subsidiary question which has stuck in my mind ever since Steve Teles asked me the question a while back; can you really create “islands” of European-style social democracy in an overall Anglosphere-liberal system? Is there a danger of “lemon socialism”, where the public realm ends up being stuffed full of stuff that the private sector wants, but would prefer not to pay for?

I think on one hand, yes there is a danger, but some bits of social democracy are better than none at all; if life hands you lemon socialism, make lemonade. There are actually lots of interesting examples of counter-system islands, like the Veterans’ Administration within the USA. Or for that matter, the almost completely unregulated market for professional sportspeople in Europe. The viability of these kind of subsystems, and the way that one might design their interactions with the underlying political economy, seems like an interesting puzzle to spend a bit of time on over the next months. Happy new year everyone.

this might be an off-cut from a book or it might not. also be warned it doesn’t have a conclusion, it’s just what ended up on my screen when I started typing this morning about the general theme of economic cybernetics …

There are many different ways of valuing a building. You can base the valuation on the assumption of a sitting tenant paying market rent, or you can try to guess what it would achieve in an immediate sale. You can include speculative value of future development or not; there are at least four approved methods in the surveyors’ handbook, all suited to different assumptions about the near term future of the building – is it being valued for probate, as collateral for a loan, as part of a bankruptcy estate or to be marketed by a speculator?

Even when you have agreed on the methodology, different people will make different assumptions about important facts. What kind of a tenant would this building attract, and how consistently would the rent be received? What are the likely maintenance costs, and when will it need a full refit? In a number of contexts, it’s appropriate to commission more than one person to value the building, and when you do, you tend to get surprisingly different results.

What happens then? The answer shocked me a little when I found out about it, during the course of some litigation I was writing about. Having done all my training as an economist, I presumed that the natural – the obvious, in many ways the only thing to do would be to take the average of the estimates. Assuming that the estimates were independent, this would be bound to be a better estimate of the “true” value than any individual one.

This is absolutely forbidden by the surveyors’ handbook, though. Averaging is not permitted as a way of reconciling different valuations. Their professional system does not treat valuation as a statistical process, basically. If different valuers get different results, then you either have to live with that, or bring them together to reach a consensus. Everyone has to explain their assumptions, show what’s driving the model and get something that they can all agree to be reasonable, even if it was not their first choice.

It interests me greatly, because we now have three ways of creating a valuation – as well as the averaging and consensus methods, there is of course the market price, which economists often quite lazily treat as if it was the average opinion of the market when it isn’t[1] And these are three different ways of creating a mental object.

Prices and values are some of the more important shared hallucinations that we create and use to manage our daily life. The method of their construction is probably important. I feel like the surveyors’ consensus method in some way creates a more robust hallucination, obviously at considerably greater cost in resources and in time. The market price is created as a by-product of another activity, and consequently it can do some strange things.

[1] (it’s the highest amount that gets enough bids; under quite restrictive assumptions and perfect competition, this would tend to be the same number as the average opinion, but it’s not an averaging process and the assumptions often aren’t satisfied).