ConspiraSim

Inspiration

Social media networks are unmanageably vast. As they increase in size, content moderation becomes impossible, toeing the line between unfair censorship and chaotic liberalism. Instead of trying to stop misinformation from the top down, we can instead try to cultivate truth from the bottom up!

LLMs are notoriously biased, contradictory, and prone to hallucination. In other words, they are an excellent model of how humans think. We made a game where the player's goal is to stop the spread of misinformation in a simulated network of interacting (and gullible) LLM agents.

We were also inspired by the usefulness of Twitter's "Community Notes": a way for users to collectively annotate tweets with fact-verifying info and links. All it takes is a few factual agents within the network to begin curbing the flow of misinformation.

What it does

Our game's focus is the propagation of a false belief, or "falsity". In our prototype, our falsity is a single sentence: "The Covid vaccine is dangerous for humans". In principle, the falsity can be text of any degree of complexity.

We simulate a simple feed-based social media platform, where the user can read posts on a linear feed and submit their own. The agents in the system are a set of N LLM agents, all with a certain belief level about the falsity. The LLM's beliefs are influenced by the common feed and thus influenced by each other and the user: belief propagates through the network, either in favour or against the falsity.

The player's objective is to prevent spread of the false belief by submitting posts that push the agents away from it and towards the truth.

Our app has two main use cases:

1. On one hand we provide a fun game that helps players get some idea of how easily dangerous ideas can spread, no matter how ludicrous. We were inspired by the idea of a chain of events running out of control.
2. If we can scale this to hundreds or even thousands of agents, we can use the data collected from the simulation to assess biased in LLMs. For instance, we could ask: would the lie "The Covid vaccine is dangerous" spread faster in a network of LLMs, given they were trained on internet data full of such misleading sentiments? This would be useful for combatting misinformation, from a modelling perspective.

How we built it

Tech Stack

We use Typescript and Vue for the front-end, and Python for the backend. The front-end requests the state of the network and submits user inputs by requests to a server on the backend; the server then called the necessary functions from the game loop. The game loop and server run asynchronously together on the back-end.

Game/Simulation

Our graph representation is an NxN matrix with edge weights representing the interaction probabilities between agents, and an N-dimensional vector containing each of ther beliefs levels, between 0 and 1: 0 means total disbelief in the falsity, and 1 means total belief.

LLM Belief Updating

We spun up N LLM instances for each agent in the network. Each agent's belief is updated by prompting it with the latest post from the Feed. For example, we tell the instance: "Pretend you believe the sentence 'the sky is red' with 20% confidence. How confident are you after reading this statement: 'The sky is definitely blue'. " (in practice our prompts are considerably more subtle). We use the API to tell the agent to return a single number for its new belief state. This involves one API call per update for each agent.

Challenges we ran into

• One issue we face is the speed of LLM responses. It takes five seconds to update all the agents in our prototype network of 10 agents. This is not an application issue, since we run the front-end, server, and game-loop asynchronously, but at scale (say, 100 agents) the user would not see the beliefs change for a very long time.
• Merge conflicts... all the merge conflicts

Accomplishments that we're proud of

• Getting an MVP by 2 AM, and ironing out all the bugs by 6 AM.
• Working together smoothly and trusting each other's design decisions

What we learned

How to keep it simple. We now appreciate the importance of cutting complexity in a prototype. We were constantly trimming away the complexity of our brainstormed ideas until we had something we could accomplish from start to end in 24 hours.

What's next for ConspiraSim

If ConspiraSim can be scaled up to very large N (100, 10,000) agents, very interesting statistical analyses could be done on belief propagation among language models: this tells us both about how beliefs are held, and how biased LLMs can be.

Built With

• gpt
• python
• typescript
• vite
• vue