The astounding success of large language models is due to the quality of the training data available - books, newspapers, online forums, Wikipedia, but also the code bases from open source software projects. All that has been shared and discussed online goes into these things.

Although the algorithms behind these models are wickedly clever, they fundamentally don’t produce intelligent behaviour without this data, and which behaviour they do produce is conditioned by what they are trained on (hence the tendency of these models to reproduce stereotypes - such as surgeons tend to be men - since these are represented in the data they are trained on).

Generative AI models produce new outputs, by definition. So once you have one of these models, you can produce infinitely more training data. Does this let you bootstrap your way to superintelligence?

Quite the opposite.

If you train a complex model on its own output, you get a phenomenon which has been termed model collapse - over successive iterations the model focuses more and more on the most common or typical patterns. Measures of performance may even show improvement initially, but incrementally the model loses the ability to represent edge cases and the uncommon. Eventually the model performance collapses as it generates similar and inappropriate output to all inputs.

Here’s an example from the classic paper (Shumailov et al, 2024). A language model (trained on human generated data) responds to a prompt about churches:

Revival architecture such as St. John’s Cathedral in London. The earliest surviving example of Perpendicular Revival architecture is found in the 18th @-@ century Church of Our Lady of Guernsey, which dates from the late 19th century. There are two types of perpendicular churches [output cut off after a limit]

After nine generations of using the model to produce data which is then added to the training data for subsequent generations, typical output to the same prompt is:

architecture. In addition to being home to some of the world’s largest populations of black @-@ tailed jackrabbits, white @-@ tailed jackrabbits, blue @-@ tailed jackrabbits, red @-@ tailed jackrabbits, yellow @-.

Repetitive, nonsensical, maniacally focused on one aspect of the training data, and a response that isn’t appropriate given the input prompt.

Guo et al (2024) have a nice illustration of the mechanism behind this. Like Shumailov et al, 2024, they trained models on their own output, then visualised the space of how inputs were represented by the model - a mapping of close or distant possible words in the semantic space of the model. In the way that ‘cat’ and ‘dog’ should be close in semantic space, and ‘cat’ and ‘astronaut’ should be less close, this shows how the universe of concepts is represented by successful generations of models.

Over time the way the model ‘thinks’ about inputs narrows, reducing the ability to make distinctions - collapsing every input so that they all seem more similar. This is why performance degrades. Good models provide different outputs depending on what you ask them. Sometimes a subtle change in input can produce a large change in output. As the models lose sensitivity to this they stop being useful for anything.

So as well as meaning we can’t bootstrap our way to superintelligence, model collapse also poses a definite risk. To avoid model collapse (or just model decay), we want to keep AI generated content out of the training data we use. Obviously, if you go to the internet of 2026 for your training data, this is impossible. Some have even compared the public release of ChatGPT to an epoch-defining pollution event - from now on, any new text you see may have been generated by a model, and you can’t know which. Model collapse can occur even if a proportion of the training data is synthetic, it doesn’t have to be all of it.

AI companies work very hard to generate new human data to fuel their ambitions. Researchers have already suggested we need to identify and preserve data which has no chance of being generated by AI, in case we need it to provide clean training data for future models.

* * *

Keeping our training data clean may be even harder than just excluding directly model generated output. Something similar to the early signs of model collapse can be seen in populations of humans who have been influenced by language models.

A result published in August 2025 analyses 83,160 student essays submitted as part of admission applications to a “highly selective private university on the East Coast”1. For cohorts of essays from each year 2019 - 2025 they calculated measures of the semantic diversity at the word, sentence and document level. Following the public release of ChatGPT there was a notable drop in the document-level semantic diversity - roughly a measure of how much variation there was between the essays in the themes discussed. Linked to the same inflection point, there was an increase in the word-level semantic diversity. Together this suggests that access to ChatGPT made student essays use a wider range of individual words, but actually narrowed the kinds of things discussed by them2.

I’ve seen this result replicated for MSc theses in an elite European University - a reduction of diversity of ideas around the time that large language models became widely accessible. We all know of students, and professionals, copying text from language models (obviously!). Anecdotally there are even reports of people picking up the phrases and verbal tics of language models.

A small study which asked participants to do creative idea generation (example challenge “Suppose that a great fog has fallen over the earth and all we
can see of people is their feet. What would happen?”), found that people who used ChatGPT to help them with the task produced more, and more detailed ideas, but these ideas were more similar across individuals and people felt less ownership of the ideas produced.

It’s natural that as we use and rely on language models in more and more of our work, those language models affect our own output. Humans have a natural tendency to mimic, so it shouldn’t be a surprise if our, human, output comes to mimic the most human-like tools we’ve created. Since everyone is using the same small family of models, it shrinks the variation between individuals, making us sound more and more like each other.

* * *

Humans and models are now an interacting epistemic system. The models learnt from human outputs, now humans are learning from the model outputs. This is affecting the text we see in the wild, but also, plausibly, human habits of thought and expression. Perhaps even more profoundly, the feedback loops may be eroding the institutions which have been established to curate human knowledge.

In October 2025 Wikipedia reported two stark, and related, trends. There are fewer new human readers, probably because people are using language models to get information. At the same time, the site is getting increasingly hammered by bots which are scraping the content, probably to build new AI models. Some human readers become human editors, the engine which has driven Wikipedia’s success over the last 25 years. Without editors, Wikipedia wouldn’t be able to adapt to incorporate new events and knowledge.

Before people who wrote code copied and pasted from ChatGPT, they copied and pasted from Stack Overflow, a question and answer site which allows people to ask technical questions, provide answers, as well as comment and vote on answers. It is widely viewed as both a vital source of training data for large language models, and something for which language models can easily substitute.

Here’s a graph of new questions posted on Stack Overflow, annotated by Emil Jacobs. It shows, over the lifetime of the site, the dramatic rise as people start using the site to ask questions, and the dramatic fall, as they stop.

There are caveats: Stack Overflow has other issues, and the actual traffic to the site hasn’t declined as dramatically as new questions. However, the ability to answer questions is upstream of people coming to the site for answers. Without new answers, users won’t come to Stack Overflow for help with the latest technology.

And, if Stack Overflow isn’t generating knowledge about new technology, what will the next generation of language models train on?

* * *

At first, we might not notice if our epistemic institutions are being eroded by language model use. As with model collapse, performance could initially improve as we find the answers we need using AI to help us be more productive and more focused. These models are capable of generating prolific output, but output which under-represents the true diversity of the world. Text is already a compression of experience. Model collapse could further drive the compression of the perspectives captured in text, meaning we all start to sound more and more like each other. If, in parallel, our reliance on models means we under-invest in our epistemic institutions then we will be accelerating towards a dead end - a world where we are unable to adapt to the new, unable to incorporate different perspectives, and over-reliant on an increasingly narrow repertoire of responses.

This is a scare story, but the warning is clear: as we use AI models, let’s keep asking ourselves if we are knowledge generating or merely output generating. Are we adding information to the world, or just words?

The AI age will mean it is easier than ever to give the superficial appearance of conveying information, and easier than ever to neglect the hard work of saying something valuable about ourselves or the world. It will also be all too easy to think we can get away without the work of sustaining the organisations we’ve built up to sort, store and aggregate new knowledge. We’ll miss ‘em when they’re gone.

This newsletter is free for everyone to read and always will be. To support my writing you can upgrade to a paid subscription (more on why here)

Upgrade to paid

Keep reading for the references, and the greatest obit I’ve heard in a long time.

Reference & More

Anderson, B. R., Shah, J. H., & Kreminski, M. (2024, June). Homogenization effects of large language models on human creative ideation. In Proceedings of the 16th conference on creativity & cognition (pp. 413-425). https://doi.org/10.1145/3635636.3656204

Guo, Y., Shang, G., Vazirgiannis, M., & Clavel, C. (2024, June). The curious decline of linguistic diversity: Training language models on synthetic text. In Findings of the Association for Computational Linguistics: NAACL 2024 (pp. 3589-3604).

Moon, K., Kushlev, K., Bank, A., & Green, A. (2025, August 20). Impersonal Statements: LLM-Era College Admissions Essays Exhibit Deep Homogenization Despite Lexical Diversity. https://doi.org/10.31234/osf.io/jsz58_v1

Shumailov, I., Shumaylov, Z., Zhao, Y. et al. AI models collapse when trained on recursively generated data. Nature 631, 755–759 (2024). https://doi.org/10.1038/s41586-024-07566-y

The Register, 2025-06-15: The launch of ChatGPT polluted the world forever, like the first atomic weapons tests

RIP Renfrew Christie

I’d never heard of Renfrew Christie before.

Born in 1949.

Growing up in apartheid South Africa, he was radicalised after being drafted into the army and discovering that the government was pursuing a nuclear weapons programme.

Picked a PhD topic - the electrification of South Africa - so he could study the infrastructure supporting uranium refinement for the weapons programme.

Started feeding information to the ANC, was betrayed by an informer, arrested and tortured.

Under torture, made a detailed confession which revealed all the targets and how he had planned bombings which were maximally disruptive of the weapons programme.

The judge read this confession out in court during the trial.

From jail, where he is placed on death row and spends years within earshot of the sounds of executions from the prison yard, Christie hears news that the ANC paramilitary wing have successfully carried out a series of bombings following the exact instructions contained in his confession.

The bombings set the nuclear programme back years and cost the government something like £1 billion.

“While I was in prison, everything I had ever researched was blown up,” he said.

The apartheid regime crumbled and Christie was pardoned, going on to become professor and Dean of Research at the University of the Western Cape.

A hero.

The New York Times: Renfrew Christie Dies at 76; Sabotaged Racist Regime’s Nuclear Program (2026-01-014)

Catch up

• Three things we learnt from a new megastudy of climate messaging

  Evidence on changing pro-environmental attitudes and behaviour

• Paid subscribers only. A thank you, some news and a preview

• The half life of trust. Or: You trusting something isn’t enough for me to trust it

• Our intuitions about political advertising are poor. Notes on a great interview with David Broockman

…And finally

via This is my Glasgow, a nice example of an naturally occurring optical illusion: “at Bingham's pond in the west of Glasgow. It might just be me, but the reflection of the shopping basket in the last of the ice makes it appear as if it's some sort of levitating cage.”

I’m trying to work out what exactly drives the illusion. Something about the shading on the ice gives a cue to depth? If you know, get in touch!

END

Comments? Feedback? Tasty training data? I am tom@idiolect.org.uk and on Mastodon at @tomstafford@mastodon.online

This post too long for email. Click on the header to open it in a browser and read the full thing

Out last week, A registered report megastudy on the persuasiveness of the most-cited climate messages. The study tested the effects of 10 well researched climate change messaging strategies on Americans’ pro-environmental attitudes and behaviour.

Science is a tapestry of evidence, where we shouldn’t put too much weight on a single study. This study, however, is worth paying some attention to, for the following reasons:

• It takes evidence from a lot of people. 13,544 to be precise, which is approximately 13,500 more than some studies of persuasion from the dark, unreliable days of psychology, pre-replication crisis.

• It has a large, diverse authorship. 25 authors by my count. Most importantly, these researchers come from different research groups, ensuring that the study design has input from people who don’t necessarily agree, and who have different favoured theories they want to support.

• I don’t know what exactly qualifies something to be a “megastudy”, but I don’t think many participants or many authors alone should count - the crucial thing is the attempt to provide something comprehensive. For this study, they first reviewed the literature (157 research papers on climate messaging), and from them selected the ten most cited messaging strategies for which there was evidence of effectiveness. By testing them in the same study they allowed, first, agreement on how each strategy should be implemented, and second, direct comparison of the effects. Here’s the description of the strategies from the paper (individual strategies bolded by me):

Six selected messaging strategies described supporting climate change mitigation as consistent with moral values and ideologies, such as care and considerateness, free market beliefs, belief in a just world, patriotism, purity and system preservation. Two messaging strategies emphasized that climate change is scientific consensus. The remaining two strategies described the costs for a socially distant group or gain associated with climate change mitigation.

• The study has a number of credibility indicators. Most importantly, and in the title, is that it is a registered report, a mechanism whereby the methods for a study are reviewed by the journal - which agrees to publish regardless of how the results turn out. This shifts the focus of journal review onto the importance of the research question, and the solidity of methods proposed for addressing it, rather than the results. You realise exactly how revolutionary the improvement this makes to the credibility of published results is when you appreciate the range of biases generated by the “review when you know the results” convention (basically positive existing results get published, negative results get file-drawered, and authors are incentivised to write up their studies as though they show a positive result, and what happened was what they predicted all along). So, basically, the registered report format meant the authors had every incentive to design a strong study, and reduced incentive to manipulate the analysis to present positive results, leaving us, the readers, with more faith that what they claim they show is representative1. They also share the anonymised data and analysis code, which is an important transparency measure and credibility signal.

So, worth spending some time with. What do they show?

1. There is bipartisan agreement on climate issues (in many ways)

This first thing isn’t emphasised by the authors of the study, but it is revealed by their results and worth comment. The study polled a sample of American adults, recruiting approximately as many Democrats and Republicans, so in this sense at least representative2

These plots show the volume of responses from participants of different political alignment. Here’s the one from belief in climate change. Although there is a famous partisan divide on this in the US, I’d argue that there is as much similarity as difference here. Most Republicans have more belief that climate change is happening than it is not (they are over 50% on the belief scale), the same as independents and Democrats (admittedly more Democrats express 100% certainty on this one).

Both Democrats and Republicans are similar in this intention to take “non-political” pro-environmental actions, such as “Eat less meat”, “fly less”, “compost”. Or, at least, they are similar in that many who identify as Democrat and many who identify as Republican have intentions to take pro-environmental action, and some from both groups have the intention not to.

The partisan groups really pull apart in how they interpret climate change and how they regard political choices in response to climate change. Republicans, for example, just seem a great deal less worried about climate change than Democrats:

You can check out the full results in the paper, but my takeaway was that - despite elite messaging questioning the reality of climate change - most of the US population doesn’t reject the reality of climate change, they just differ in what they think should be done about it.

Consequently, we shouldn’t assume that disagreements about climate policies are disagreements about the science of climate change (and if we do, that could be misdirection). There’s a similar story in the UK, where the reality of climate change and the need to do something about it has cross-partisan consensus.

2. Message targeting doesn’t matter

“Persuasiveness varied little across party lines”, report the study authors, “inconsistent with theories predicting heterogeneous effects for targeted messages.”

Put another way, messages were persuasive or not, it didn’t seem to matter who was seeing them, a Democrat or a Republican.

This chimes with what I reported political scientist David Broockman arguing in my newsletter from December, and what Alex Coppock argued in his book Persuasion in Parallel: How Information Changes Minds about Politics.

The idea of targeted messaging fulfils our hope for a magic bullet which will persuade those who think differently from us (and surely also plays well for the business model of campaign consultants), but the reality may be that the differences between different messages are larger (and easier to take advantage of) than the differences between how the same message lands with different demographics.

Although the study authors don’t rub this in, six of the ten messages they tested used a moral value reframing, inspired by the Moral Foundations school of thought. The lack of difference across Democrats and Republicans should specifically dampen our enthusiasm for how practical Moral Foundations is as a way of reaching people of different political persuasions. (That said, many good theories contain deep truths and are also entirely impractical, or impractical until applied right, but at the least this result shows that simplistic translation of Moral Foundations into persuasive messaging doesn’t allow special access to one partisan group or the other).

3. Persuasion is possible, just hard

The third thing I’d take from this study is that, consistent with a lot of the research I review in this newsletter, persuasion is possible. All groups of participants responded to some of the persuasion messages, but the response was typically small - a few percentage points.

Here’s just one of the outcomes they measured, attitude (on a scale of 0 to 100) towards climate mitigation policies (things like “Ban construction of new coal-fired power plants”).

The top strategy - a purity framing message (“Participants read a text arguing that mitigating climate change is consistent with values such as protecting the purity of American land.”) - looked like it had an average effect of moving people 1.5 points on the 100-point scale. In an election, that can be enough to matter, but in an election you also have the other side trying to push voters in the other direction. Does it matter in other contexts? Well you could say that it is impressive to get any movement in beliefs or attitudes from a short persuasive message. It proves people haven’t completely made up their minds on the issue.

That said, my native optimism about how reachable people are was given pause by what was perhaps the most interesting question the researchers asked: would participants be willing to give up a proportion of their payment for taking part as a donation to a pro-environmental organisation? No messaging strategy - not one - increased donations relative to the control condition (which was reading a non-persuasive message on a non-climate related topic, such as “the history of neckties”).

In fact, it looked like several messaging strategies decreased participants’ willingness to donate. I don’t read too much into this backfire effect. The donation measure wasn’t a planned part of the study. The most I’d take from it is the results are incompatible with any persuasion effect on expressed belief flowing through directly to behaviour (willingness to donate). Simply put, what people say is easier to shift than what they do (especially when cash is on the line).

In the words of the study authors, these results highlight “the limits of short-form messages for increasing Americans’ support for action to address climate change.”

Overall

We need more collaborative studies which test different approaches head to head. These particular results should shift our expectation down that it is possible to find bespoke messages for different political groups, without doing much to shift our prior expectation that, while always possible, persuasion is hard.

This newsletter is free for everyone to read and always will be. To support my writing you can upgrade to a paid subscription (more on why here)

Upgrade to paid

Keep reading for the references, more on persuasion, a recommendation for a great podcast and links to some recent things published elsewhere by me.

Reference & More

Voelkel, J.G., Ashokkumar, A., Abeles, A.T. et al. A registered report megastudy on the persuasiveness of the most-cited climate messages. Nat. Clim. Chang. (2026). https://doi.org/10.1038/s41558-025-02536-2

Related, by me

• Our intuitions about political advertising are poor. Notes on a great interview with David Broockman

• Language models are persuasive - and that’s a good thing. Two new studies provide insights into exactly how LLMs persuade, and what that means.

• The truth about digital propaganda. Reasonable People #55: Our piece in New Scientist bring evidence to worries about online manipulation

• How persuasive is AI-generated propaganda? Reasonable People #51: Bullet review of a new paper suggesting LLMs can create highly persuasive text and will supercharge covert propaganda campaigns.

Amy Mount: Political Heat

If climate policy and politics is your thing, please read Amy Mount’s reflection the last 12 months: 2025: When the UK’s climate consensus crumbled

The short version: emissions went down. Renewables and electrification went up. The climate narrative was muted. Wider politics got harder. Voters stayed volatile and parties were polarised. The general election in 2029 will be a defining moment for the UK’s climate progress. But I remain hopeful – because the public is onside. The question is, will it stay there?

And for deep dives on everything from the UK energy market to the Conservatives’ environmental legacy, Amy has you covered in the Political Heat podcast archives

Political Heat

• Podcast

• Substack

Full disclosure: Amy is my cousin, so I have some CoI

The Transmitter: The 1,000 neuron challenge

By me, in neuroscience magazine The Transmitter

A competition to design small, efficient neural models might provide new insight into real brains—and perhaps unite disparate modeling efforts.

Link: The 1,000 neuron challenge

WonkHE: A new model moves research in a more democratic direction – it could be faster and fairer too

By myself and Anna Butters. This is about research funding, which is an interesting and important decision problem. I’m proud of the way we wrote this to start very zoomed out on the big picture, then take the reader through to the specific mechanism - distributed peer review - about which we have new evidence on.

Here’s the start:

Research quality is the dark matter of the university sector. It is hard enough to assess research after it has been done, research funders must find some way to evaluate proposals for projects which don’t exist yet. The established model for this is external expert review, combined with a panel stage where proposals, and their reviews are discussed, and hard choices made.

UK researchers will be familiar with this via our own UKRI, and everyone who has had a funding application rejected will recognise that the reviews received may be partial or mis-directed. This speaks to the idiosyncrasy and variability in individual judgments of what makes a good project, and has downstream consequences for what ultimately gets funded.

Link: A new model moves research in a more democratic direction – it could be faster and fairer too

Preprint: Applicants as reviewers: A mixed methods evaluation of the risks, benefits, and potential of distributed peer review for grant funding allocations

Speaking of which, the preprint of our work on this has just been published. It’s a shorter (by 50%) version of the working paper, so not strictly new. Here’s the conclusion

This trial demonstrates feasibility of DPR [Distributed Peer Review] for an interdisciplinary funding programme. Implementing a DPR process requires funders to accept trade-offs, but can offer valuable benefits, including shortening time to funding decision. Building on established peer review principles, it has the potential to contribute to the democratisation of funding evaluation.

Citation:

Butters, A., Marshall, M. B., Pinfield, S., Stafford, T., Bondarenko, A., Neubauer, B., … Denecke, H. (2026, January 6). Applicants as reviewers: A mixed methods evaluation of the risks, benefits, and potential of distributed peer review for grant funding allocations. https://doi.org/10.31222/osf.io/t2p56_v1

…And finally

Tom Thomson. Northern River, 1914-15. (via @tarnport)

END

Comments? Feedback? Urban legends? I am tom@idiolect.org.uk and on Mastodon at @tomstafford@mastodon.online

‘The Volvo Story’ has a special place in the history of research on judgement and decision making. It’s a thought experiment. Imagine, it asks:

Let us suppose that you wish to buy a new car and have decided that on grounds of economy and longevity you want to purchase one of those solid, stalwart, middle class Swedish cars―either a Volvo or a Saab. As a prudent and sensible buyer, you go to Consumer Reports, which informs you that the consensus of their experts is that the Volvo is mechanically superior, and the consensus of the readership is that the Volvo has the better repair record. Armed with this information, you decide to go and strike a bargain with the Volvo dealer before the week is out. In the interim, however, you go to a cocktail party where you announce this intention to an acquaintance. He reacts with disbelief and alarm: “A Volvo! You’ve got to be kidding. My brother-in-law had a Volvo. First, that fancy fuel injection computer thing went out. 250 bucks. Next he started having trouble with the rear end. Had to replace it. Then the transmission and the clutch. Finally sold it in three years for junk.”

How likely is it that you would proceed as planned and buy a Volvo?

Most people share the intuition that they would rethink the Volvo purchase.

This is despite, as the authors put it, “the logical status of this information is that the N of several hundred Volvo-owning Consumer Reports readers has been increased by one and the mean frequency of repair record shifted up by an iota on three or four dimensions.”

In other words, this guy at a party is just one guy. Why do we weight his report more than the hundreds compiled in the Consumer Reports consensus?

For years I swallowed the standard line from the Heuristics and Biases literature, that this story highlighted something flawed in our thinking.

It’s been called The Anecdotal Fallacy (which is really just providing a name, something which ‘dulls the ache of incomprehension’ but isn’t really an explanation). It has been called an example of the Availability Heuristic, where vivid information dominates our judgements of risk (in this case, the personal testimony of the man at the party is more vivid than the hundreds anonymously compiled in the report. Another frequently invoked example of the Availability Heuristic is that people fear flying, with its extremely vivid, but rare, accidents, more than they fear being driven to the airport, which is actually the higher risk activity).

The Volvo story has also been used to illustrate Base Rate neglect, which is an example of a failure of Bayesian rationality. The thought here is that the Consumer Report provides a prior expectation of faults among Volvos, which should be integrated with the new information (one additional Volvo with many faults). If you do this integration mathematically your estimate of the rate of faults should move from “very small” to “very small + a small fraction” (i.e. still very small).

One cause of the success of the heuristics and biases programme, for which Daniel Kahneman got the Nobel Prize, and which gave us “Nudge”, behavioural economics and the rest, was that Kahneman and Tversky worked very hard on refining their experiments, which often relied on exactly the kind of vignettes represented by the Volvo Story, so that they precisely captured some quirk of our intuitions.

Fans of the literature will recognise this approach in the (increasingly unfortunately named) Asian Disease Problem, The (increasingly anachronistic) Linda Problem and others.

These vignettes bring into view two ‘obvious’ answers which are in contradiction. With the Volvo story, obviously - ‘logically’! - we should ignore the party guy. Also obviously, we wouldn’t.

This is the basic contradiction which requires an explanation. The authors of the chapter which introduced the story were aware that recognising the contradiction was only half of the issue: if, ‘logically’, the man should ignore the party-goer, only someone ignorant of human nature would expect that he would. What, they ask, explains why some information (like the Consumer Report) is ignored, and some (like the party-goer’s anecdote) is prioritised?

Now, many years after first hearing this thought experiment, I look at it differently. Rather than seeing the intuitive reaction (to abandon Volvo-purchase plans) as resulting from one of the many cognitive biases I was taught about, I see it as showing us something important about the nature of trust.

Trust, fundamentally, isn’t transmissible in the same way as information. We believe the people we trust, but it isn’t reasonable to expect others to believe them in turn.

Here’s a way of thinking about it: trust fades quickly with transmission. The half-life is short, so that if A tells B, and B tells C, and C tells D, the credibility that D can reasonably assign to what C has told them (that B told after hearing it from A) is close to nothing.

This is why it is reasonable to discount the Consumer Reports conclusion. A bunch of people (Who? How selected?) told Consumer Reports which collated this information (How? Under what biases?) and then published it. The chain of transmission contains too many points where biases could creep in. Maybe Volvo pay bribes to Consumer Reports? Maybe Volvo owners are too proud to report faults? The point isn’t that any bias is plausible, only that there are just too many unknown unknowns, and with each point in the transmission chain these multiply, rather than add.

The party-goer’s anecdote is vivid, but it also appears immediately verifiable. The party-guy is standing right there, ready to defend or explain his story. The mechanisms of credibility are tangible, so trust is greater than for the distant and anonymous evidence of Consumer Reports1.

It sounds obvious, but far too many important communications - from news, to health information about vaccines - ask people to take things on trust without seeming to recognise how fragile trust is, how poorly it travels across social distance.

One notable example: Consensus messaging has been promoted as the way to convince people about about climate change (e.g. “97% of scientists believe in human-caused climate change”). Science is a massive, anonymous, aggregator. A Consumer Reports for evidence on everything. Trusting science is reasonable if you have some understanding of how science works, but if you don’t then it is the equivalent to “97% of people different from me who do things in ways and for reasons I don’t understand believe something”. No wonder some of us prioritise vivid information from people we meet at parties instead.

When we recognise that my trust doesn’t convert 100% into your trust we can communicate more effectively - giving reasons to believe what we’ve come to believe, rather than hoping that the strength of our conviction will carry the day.

What of our hypothetical Volvo-purchaser? I’m arguing that it isn’t unreasonable to rethink his choice after the party, just as it would also be reasonable to press on and follow the Consumer Reports recommendation. Reasonableness in the real world often doesn’t have correct answers. Choices are usually matters of which risks you want to take, and what (or who) you put your trust in. We’ll understand choice better when we take trust seriously as a problem for each individual to solve for themselves, rather than something any of us can, or should, take as a given.

This newsletter is free for everyone to read and always will be. To support my writing you can upgrade to a paid subscription (more on why here)

Upgrade to paid

Keep reading for the references and more on persuasion by LLM, design thinking for chatbots and a quote from Mary Wollstonecraft.

References

The Volvo story originally (as far as I can tell) appears in:

• Richard E. Nisbett, Eugene Borgida, Rick Crandall & Harvey Reed, “Popular Induction: Information is Not Necessarily Informative”, in Judgment Under Uncertainty: Heuristics and Biases, Daniel Kahneman, Paul Slovic & Amos Tversky, editors, (1985), pp. 112-113.

There’s a side-note here about using LLMs for research. I had misremembered the source of the quote, so was trying to track it down. Gemini persistently hallucinated that it was in Kahneman’s Thinking Fast and Slow. Despite having a hardcopy, this was hard to definitely disprove in the time I was willing to spend looking through the book. An internet search also turned up many people who attributed it to Kahneman or Tversky, although I couldn’t immediately find a source where they discuss it (or attribute it).

Eventually, I got to Gary N. Curtis’ Fallacy Files which provided the reference (which I verified by picking up my own copy of Judgment Under Uncertainty):

• The Anecdotal Fallacy

Talking LLM blues

I’ve always been interested in how psychological science is laundered to support grander narratives of human failing and irrationality. Normally this happens via popularisation in the mass media, but you can also catch it happening in the scholarly literature.

Here’s Rogier et al:

“LLM Systems appear capable at exploiting cognitive biases that make humans vulnerable to persuasion and manipulation”

Following the claim there are citations. This is from the abstract of the first one

“Arguments that included factual knowledge, markers of trust, expressions of support, and conveyed status were deemed most effective”

(Breum et al, 2024).

Now the Rogier et al claim isn’t flat wrong - the Breum study did show an effect of status, and relying on arguer status is a plausible cognitive bias - but it is definitely selective to read this study and see evidence of manipulative power of LLMs.

If arguments that include factual knowledge are more persuasive that isn’t exploiting a cognitive bias, at least in the way that most people would understand it.

Here’s another study of those cited by Rogier et al to support the claim: Griffin et al (2023). One thing they report, which really jumps the shark in my opinion, is that simulating participants with an LLM shows that LLMs can be persuasive (because they ‘persuade’ simulated participants). To be fair that isn’t all the paper reports, but this part at least seems very minimal evidence for the claim that LLMs are exploiting cognitive biases to manipulate people (and the wider implication that we should be worried about their persuasive powers).

People need support if “do your own research” is going to help rather than hurt

A story from early 2024:

Nieman Labs “Asking people to “do the research” on fake news stories makes them seem more believable, not less”

Original paper, Aslett et al. 2023 Online searches to evaluate misinformation can increase its perceived veracity

Part of this must be what I called (in 2021) The Epistemic IKEA effect - we put more faith in beliefs we’ve ‘assembled’ ourselves. The study here also highlights the role of skill - those who are most inept at using search are most likely to find and trust low quality sources and confirmation of false beliefs. Here’s their abstract:

Considerable scholarly attention has been paid to understanding belief in online misinformation with a particular focus on social networks. However, the dominant role of search engines in the information environment remains underexplored, even though the use of online search to evaluate the veracity of information is a central component of media literacy interventions. Although conventional wisdom suggests that searching online when evaluating misinformation would reduce belief in it, there is little empirical evidence to evaluate this claim. Here, across five experiments, we present consistent evidence that online search to evaluate the truthfulness of false news articles actually increases the probability of believing them. To shed light on this relationship, we combine survey data with digital trace data collected using a custom browser extension. We find that the search effect is concentrated among individuals for whom search engines return lower-quality information. Our results indicate that those who search online to evaluate misinformation risk falling into data voids, or informational spaces in which there is corroborating evidence from low-quality sources. We also find consistent evidence that searching online to evaluate news increases belief in true news from low-quality sources, but inconsistent evidence that it increases belief in true news from mainstream sources. Our findings highlight the need for media literacy programmes to ground their recommendations in empirically tested strategies and for search engines to invest in solutions to the challenges identified here.

Design thinking about chatGPT

Amelia Watternberger makes the case that “Chatbots Are Not the Future”. It’s a fantastically presented argument, making the case that text inputs have no affordances and the responses don’t aggregate, so chat is the wrong interface for many tasks. Ironically, the affordances of chat are the reason I think chatbots are *right* for some of the tasks I’m interested in: deliberation, articulating your own position, understanding the position of others, and so on.

Amelia Watternberger: https://wattenberger.com/thoughts/boo-chatbots

Quote: Mary Wollstonecraft

Truth must be common to all, or it will be inefficacious with respect to its influence on general practice

A Vindication of the Rights of Woman (1792)

… And finally

From poorlydrawnlines.com by Reza Farazmand

END

Comments? Feedback? Urban legends? I am tom@idiolect.org.uk and on Mastodon at @tomstafford@mastodon.online

This is great: The Realities of Political Persuasion with David Broockman, from the always listenable Opinion Science by Andy Luttrell.

Broockman argues that, in the context of campaigning, too much emphasis has been put on message personalisation - identifying particular segments of voters and designing ads which particularly appeal to them - and not enough on empirically testing message effectiveness. You can understand why - for consultants and strategists personalisation seems like the kind of magic which might allow big breakthroughs, and there are a range of ways of categorising people from demographics or psychology which superficially seem promising.

Broockman’s research pours cold water on this idea of personalisation. This fits with what I’ve learnt - when I’ve looked at studies which purport to show that advertising according to personality is more successful, there are either major confounds, or - when you look closely - the personalisation may arguably have an effect, but the size of the effect is not significantly larger than a non-personalised message.

Important context for this is that there are big differences between individual adverts in terms of effectiveness. This means that better and worse adverts could be deployed in terms of persuasive effect - the variance is there, if we can work out how to take advantage of it (and if we did, it wouldn’t threaten our vision of human reasonableness, because microtargeting is not mind control).

The point is that personalisation is, in practice, very very far from being some kind of terrifying ‘manipulation machine’. The difference between different ads is typically far larger than between a personalised and non-personalised version of an ad. I try and show this by reworking the data from this paper into a summary visualisation:

To understand this plot you just need to know that the up-down/vertical axis shows the persuasive effect. The green line, as it changes left to right, shows the difference between individuals in how persuasive they found the ads - a huge difference. The change in the vertical position of the orange triangles shows the differences between different adverts used in the study - a smaller difference, but still much much bigger than the difference due to personalisation (blue dots). The full context is in this post: AI-juiced political microtargeting. The point is that a non-zero effect of personalisation (the highest blue dot is higher than the lowest blue dot) can still be non-important compared to the other sources of variation (the differences due to advert or individual are far larger).

This is from a paper which says in the abstract : “Recent technological advancements, involving generative AI and personality inference from consumed text, can potentially create a highly scalable “manipulation machine” that targets individuals based on their unique vulnerabilities without requiring human input.”

The “highly scalable” part is definitely true, but I question the degree to which this is manipulation, or to which it is worth worrying about given the size of the demonstrated effects.

From the other side of the coin, the implication is that political campaigners should worry far more about being persuasive for everyone than trying to take advantage of minuscule fine-tuning that might be possible for different groups.

In the Opinion Science pod, Broockman goes on explain that despite the large differences between different messages these differences are hard to predict. They aren’t predicted by the categories that political scientists typically use, such as whether the ads are positive or negative in tone.

Further, expert political practitioners are not better at predicting the most persuasive adverts compared to the general public. Not only were the experts not better than the public at predicting which campaigning ads would be most persuasive, they weren’t consistent with each other (which is important because it suggests there isn’t even some kind of shared conventional wisdom which, although wrong, is shared by the experts).

I have a memory of a study done of the 2016 Democratic campaign which shows that the ads predicted to be most persuasive by campaign staffers were least persuasive with the voters who most needed to be swayed. Sadly I can’t find this now, but I did find this very telling article from pre-election 2016: How the Clinton campaign is slaying social media. This gloating piece neatly illustrates the potential mechanism - highly political, young, cosmopolitan and progressive campaign staffers (and journalists) mistake what works for them as what works for the generally less political, less young, less cosmopolitan and less progressive swing voters.

My take-away is that our intuitions in this area are bad.

We’re bad at discounting our own reaction to an ad to work out how it will affect people unlike us (meaning that campaigners probably leave a lot of variation in persuasive effect on the table when deciding which ads to deploy).

We’re drawn to think that microtargeting and personalisation will unlock extra persuasive power (when the evidence is that extant models of personalisation are not strongly effective, and the degree to which they are is probably in line with how normal persuasion works - information containing facts and evidence is persuasive, just as we’d hope).

And we’re constantly tempted to think of persuasion in general, and advertising in particular, as a form of manipulation. This is extremely limiting. It implicitly denies legitimacy to electoral campaigning, which is a core part of democracy, and denigrates the voters who are persuaded.

Link: The Realities of Political Persuasion with David Broockman

Paper: Broockman and colleagues: Political practitioners poorly predict which messages persuade the public.

Related, from me:

• Language models are persuasive - and that’s a good thing

  Two new studies provide insights into exactly how LLMs persuade, and what that means.

• The truth about digital propaganda

  Reasonable People #55: Our piece in New Scientist bring evidence to worries about online manipulation

• AI-juiced political microtargeting: Reasonable People #53 looking carefully at the claims in one study which uses generative AI to customise ads to personality type

• Propaganda is dangerous, but not because it is persuasive

  Reasonable People #52: I pick at the claim that propaganda “doesn’t work”.

• How persuasive is AI-generated propaganda? Reasonable People #51: Bullet review of a new paper suggesting LLMs can create highly persuasive text and will supercharge covert propaganda campaigns.

• Microtargeting is not mind control

  Reasonable People #22 exaggerated beliefs about the effectiveness of microtargeted ads obscure real risks, and real opportunities to foster public trust in politics

This newsletter is free for everyone to read and always will be. To support my writing you can upgrade to a paid subscription (more on why here)

Upgrade to paid

Keep reading for more on fact-checking, AI, and a very cool job opportunities.

In case you missed it

I’m spending more time in the UK capital. Let me know if you want to get coffee :

London calling / Tools for Thought

PODCAST: Gordon Pennycook on Unthinkingness, Conspiracies, and What to Do About Them

From Sean Carroll’s Mindscape, an interview with Pennycook about his work, including on pseudo profound bullshit and using chatbots to debunk conspiracy theories. Something which came out, which I missed from the papers and seems really important, is that people like the experience of being debunked by the chatbot.

And people usually actually like it.

They’re not mad at the AI.

The AI gives them information I think is useful.

And evidence matters more than we thought it was.

Link: Gordon Pennycook on Unthinkingness, Conspiracies, and What to Do About Them

Share

JOB: Research Scientist at Wikimedia

We’re hiring a Research Scientist strongly committed to the principles of free knowledge, open source, privacy, and collaboration to join the Research team. As a Research Scientist, you will conduct applied research on the integrity of Wikipedia knowledge, its communities and their work, and the Wikipedia model.

Examples of recent research questions you may contribute to include::

How does the platform and its community navigate election times?

What is the role of Wikipedia in the landscape of online disinformation?

What guidance can we provide to researchers studying neutrality on Wikipedia?

Closes: January 15th, full remote with some geographic limitations

Link: https://job-boards.greenhouse.io/wikimedia/jobs/7484474/

….And finally

END

Comments? Feedback? Takes from 2016 which aged really badly? I am tom@idiolect.org.uk and on Mastodon at @tomstafford@mastodon.online