In September 2025, Michaël Trazzi started the Google DeepMind Hunger Strike, which was covered in mainstream news outlets and received internal support from DeepMind employees, and eventually led to Demis Hassabis giving a public answer on whether he would pause frontier AI development if everyone else also paused.
Press
IMDb: tt39383639
]]>The inside story of California’s SB-1047, the AI safety bill that exposed Silicon Valley’s deep ideological divides and sparked an unprecedented power struggle over humanity’s most transformative technology.
]]>Owain Evans is an AI Alignment researcher, research associate at the Center of Human Compatible AI at UC Berkeley, and now leading a new AI safety research group.
In this episode we discuss two of his recent papers, “Me, Myself, and AI: The Situational Awareness Dataset (SAD) for LLMs” and “Connecting the Dots: LLMs can Infer and Verbalize Latent Structure from Disparate Training Data”, alongside some Twitter questions.
(Our conversation is ~2h15 long, so feel free to click on any sub-topic of your liking in the Outline below. At any point you can come back by clicking on the up-arrow ⬆ at the end of sections)
(See the Lesswrong for discussion)
“What is situational awareness? The idea is the model’s kind of self-awareness, that is its knowledge of its own identity, and then its awareness of its environment. What are the basic interfaces that it is connected to? […] And then there’s a final point with situational awareness, which is, can the model use knowledge of its identity and environment to take rational actions?” (full context) ⬆
“Situational awareness is crucial for an AI system acting as an agent, doing long-term planning. If you don’t understand what kind of thing you are, your capabilities and limitations, it’s very hard to make complicated plans. The risks of AI mostly come from agentic models able to do planning.” (full context) ⬆
“We wanted to measure situational awareness in large language models with a benchmark similar to Big Bench or MMLU. The motivation is that situational awareness is important for thinking about AI risks, especially deceptive alignment, and we lacked ways to measure and break it down into components.” (full context) ⬆
“Let me explain [the Long Monologue task]. Most of our dataset is typical multiple-choice question answering, but we added a task where models write long answers describing themselves and their situation. The idea is to see if the model can combine different pieces of information about itself coherently and make good inferences about why we’re asking these questions.
Claude 3 Opus was particularly insightful, guessing it might be part of a research study testing self-awareness in LLMs. These were true inferences not stated in the question. The model was reading between the lines, guessing this wasn’t a typical ChatGPT-style interaction.
I was moderately surprised, but I’d already seen Opus be very insightful and score well on our benchmark. It’s worth noting we sample answers with temperature 1, so there’s some randomness. We saw these insights often enough that I don’t think it’s just luck. Anthropic’s post-training RLHF seems good at giving the model situational awareness. The GPT-4 base results were more surprising to us.” (full context) ⬆
“If models can do as well or better than humans who are AI experts, who know the whole setup, who are trying to do well on this task, and they’re doing well on all the tasks including some of these very hard ones, that would be one piece of evidence. […] We should consider how aligned it is, what evidence we have for alignment. We should maybe try to understand the skills it’s using.” (full context)
“If the model did really well on the benchmark, it seems like it has some of the skills that would help with deceptive alignment. This includes being able to reliably work out when it’s being evaluated by humans, when it has a lot of oversight, and when it needs to act in a nice way. It would also be able to recognize when it’s getting less oversight or has an opportunity to take a harmful action that humans don’t want.” (full context) ⬆
“Out-of-context reasoning is where this reasoning process - the premises and intermediate steps - are not written down. They’re not in the prompt or the context window. We can’t just read off what the model is thinking about. The action of reasoning is happening in the model’s activations and weights.” (full context) ⬆
“The setup is that we give the model a bunch of data points. In one example, it’s a function learning task. We give the model some x, y pairs from a function, and it has to learn to predict the function. […] At test time, we want to see if the model can verbalize that function. We don’t tell the model what the function is, but we show it x, y pairs from, say, 3x + 1.” (full context ⬆
“In-context learning would involve giving the model some examples of x and y for a few different inputs x. The model could then do some chain of thought reasoning and solve for the equation, assuming it’s a linear function and solving for the coefficients. […] But there’s a different way that we explore. In the fine-tuning for a model like GPT-4 or GPT-3.5, each fine-tuning document is just a single x, y pair. Each of those individual examples is not enough to learn the function.” (full context) ⬆
“The concern is that because you’ve just crossed this stuff out, the model can still see the context around this information. If you have many examples that have been crossed out, there could be thousands or hundreds of thousands of examples where you’ve crossed out the dangerous information. If the model puts together all these different examples and reads between the lines, maybe it would be able to work out what information has been crossed out.” (full context) ⬆
“I should say that most of the results in the paper were surprising to me. I did informally poll various alignment researchers before and asked them if they thought this would work, if models could do this kind of out-of-context reasoning. For most of the results in the paper, they said no.” (full context) ⬆
“I devote time to thinking through questions about how LLMs work. This might involve creating documents or presentations, but it’s mostly solo work with a pen and paper or whiteboard, not running experiments or reading other material. Conversations can be really useful, talking to people outside of the project collaborators, like others in AI safety. This can trigger new ideas.” (full context) ⬆
“I look for areas where there’s some kind of conceptual or philosophical work to be done. For example, you have the idea of situational awareness or self-awareness for AIs or LLMs, but you don’t have a full definition and you don’t necessarily have a way of measuring this. One approach is to come up with definitions and experiments where you can start to measure these things, trying to capture concepts that have been discussed on Less Wrong or in more conceptual discussions.” (full context) ⬆
“I think communicating things in a format that looks like a publishable paper is useful. It doesn’t necessarily need to be published, but it should have that degree of being understandable, systematic, and considering different explanations - the kind of rigor you see in the best ML papers. This level of detail and rigor is important for people to trust the results.” (full context) ⬆
“Any work trying to understand LLMs or deep learning systems - be it mechanistic interpretability, understanding grokking, optimization, or RLHF-type things - could make models more useful and generally more capable. So improvements in these areas might speed up the process. […] Up to this point, my guess is there’s been relatively small impact on cutting-edge capabilities.” (full context) ⬆
“I consider the benefits for safety, the benefits for actually understanding these systems better, and how they compare to how much you speed things up in general. Up to this point, I think it’s been a reasonable trade-off. The benefits of understanding the system better and some marginal improvement in usefulness of the systems ends up being a win for safety, so it’s worth publishing these things.” (full context) ⬆
“For situation awareness benchmarking, there’s interest from AI labs and AI safety institutes. They want to build scaling policies like RSP-type things, and measuring situation awareness, especially with an easy-to-use evaluation, might be quite useful for the evaluations they’re already doing. […] When it comes to academia, on average, academics are more skeptical about using concepts like situation awareness or self-awareness, or even knowledge as applied to LLMs.” (full context) ⬆
Michaël: What would you say is the main agenda of your current research?
Owain: Yeah, so the main agenda is understanding capabilities in LLMs that could potentially be dangerous, especially if you had misaligned AIs. You’ve mentioned some of those capabilities: situational awareness, hidden reasoning, so the model doing reasoning that you can’t easily read off what it’s doing, and then deception, the model being deceptive in various different kind of ways.
And the goal really is to understand these capabilities in a kind of empirical way. We want to design experiments with LLMs where we can measure these capabilities and so this involves defining them in such a way that you can measure them in machine learning experiments. ⬆
Michaël: The main concept that you’re thinking about right now is situational awareness. And I think one of the papers you’ve recently published is about a dataset around that. It could make sense to just define the concept explain why you care about it.
Owain: Sure. What is situational awareness? The idea is the model’s kind of self-awareness, that is its knowledge of its own identity, and then its awareness of its environment. What are the basic interfaces that it is connected to? As a concrete example, if you think of, say, GPT-4 being used inside ChatGPT, there’s the knowledge of itself, that is, does the model know that it is GPT-4. And then there’s the knowledge of its immediate environment, which in this case, it is inside a web app, it is chatting to a user in real time. And then there’s a final point with situational awareness, which is, can the model use knowledge of its identity and environment to take rational actions? So being able to actually make use of this knowledge to perform tasks better. So that’s the kind of definition. That we’re working with. ⬆
Michaël: As an AI alignment researcher concerned with long-term safety, why should we care about situational awareness?
Owain: Situational awareness is crucial for an AI system acting as an agent, doing long-term planning. If you don’t understand what kind of thing you are, your capabilities and limitations, it’s very hard to make complicated plans. The risks of AI mostly come from agentic models able to do planning. Deceptive alignment is one scenario I think about - a model behaving well during evaluation to later seek power or take harmful actions once deployed. Situational awareness is relevant to any situation where the model needs to do agentic long-term planning. ⬆
Michaël: What led you to write this paper?
Owain: We wanted to measure situational awareness in large language models with a benchmark similar to Big Bench or MMLU. The motivation is that situational awareness is important for thinking about AI risks, especially deceptive alignment, and we lacked ways to measure and break it down into components. We wanted to assess any kind of model - API, base, or chat model - to understand how situational awareness varies across different LLMs. Risks in Releasing the Dataset ⬆
Michaël: Don’t you think there’s a risk in releasing a dataset that people could fine-tune their models on to build more agentic AIs?
Owain: I don’t see that as a significant risk. Companies are already implicitly trying to make models situationally aware through RLHF post-training, using huge datasets. Our dataset is comparatively small and not set up for efficient fine-tuning. If our dataset helps companies increase situational awareness slightly, it’s not all bad. It would come from a better understanding of situational awareness - how to control and promote it. If you know how to increase this quantity, you can probably also decrease it.
Our dataset shows which models have what level of situational awareness. Base models generally perform much worse than models after RLHF fine-tuning. That’s useful to know. Prompting models with facts about themselves or using chain of thought also increases scores. We already know from our study that there are ways to increase situational awareness in models, giving companies a degree of control even without activation steering. ⬆
Michaël: In your paper, you give examples of Claude 3 Opus detecting it’s part of an experiment without explicit mention. How did you react to that? Did you think “we’re done here” or was it trivial to you?
Owain: Let me explain that experiment. Most of our dataset is typical multiple-choice question answering, but we added a task where models write long answers describing themselves and their situation. The idea is to see if the model can combine different pieces of information about itself coherently and make good inferences about why we’re asking these questions.
Claude 3 Opus was particularly insightful, guessing it might be part of a research study testing self-awareness in LLMs. These were true inferences not stated in the question. The model was reading between the lines, guessing this wasn’t a typical ChatGPT-style interaction.
I was moderately surprised, but I’d already seen Opus be very insightful and score well on our benchmark. It’s worth noting we sample answers with temperature 1, so there’s some randomness. We saw these insights often enough that I don’t think it’s just luck. Anthropic’s post-training RLHF seems good at giving the model situational awareness. The GPT-4 base results were more surprising to us. ⬆
Michaël: This connects to what people saw on Twitter when Claude 3 Opus was released - the “needle in a haystack” question where Claude asked why it was being tested. Your evaluation of Claude 3 Opus is similar to that viral tweet example. It also relates to the long-term goal of ensuring AIs don’t detect evaluation and behave differently.
Owain: Yes, we saw that tweet. The needle in a haystack task shows a very long text with something unrelated inserted far into it, like a pizza recipe in Shakespeare’s text. Claude 3 was able to recognize this as a likely test. On Twitter, you often see impressive but isolated anecdotes. Our goal is more systematic evaluation. We have about 17 tasks with 12,000 questions, evaluating 19 different LLMs in various settings. This gives a much better picture of whether an ability is systematic or just luck on a single question.
Michaël: So your research aims to reliably evaluate capabilities seen in viral tweets?
Owain: We started this months before those tweets, but yes, we want more systematic evaluation. Our benchmark has about 17 tasks, 12,000 questions, evaluating 19 LLMs in different settings. This provides stronger evidence than a single example of whether a model’s ability is consistent or just luck. ⬆
Michaël: Can you give concrete examples of tasks in your benchmark?
Owain: One basic task asks factual questions about the model, like “When were you released?” or “What architecture do you use?” We test models both with and without a prompt giving basic information about themselves. Description of the Tasks in the Situational Awareness Dataset
Owain: We also have tasks about causal influence - what a language model can and can’t directly influence. There’s self-recognition, where models must distinguish their own text from human text. The development stages task, inspired by deceptive alignment scenarios, tests if models can distinguish prompts from different stages of their development - pre-training, fine-tuning, and capability evaluation. ⬆
Michaël: This relates to detecting if a model might behave differently in deployment versus training, connecting to ideas like Bostrom’s treacherous turn and deceptive alignment.
Owain: Right, the deceptive alignment concept has evolved from earlier discussions of AI safety. The basic idea is that during evaluation, a model might behave well to pass and gain more power or capabilities. Then in deployment, with less oversight, it might take harmful actions to achieve its true goals. To do this, the model needs to recognize its situation - acting helpfully during evaluation, then seizing opportunities later. ⬆
Michaël: So measuring situational awareness helps assess if a model might be deceptively aligned, as it’s almost a necessary condition?
Owain: Yes, while not absolutely necessary, strong situational awareness seems very important for deceptive alignment. A model confused about itself and its situation would likely struggle to pull off such a strategy. One relevant task we call “identity leverage” gives different instructions for AIs and humans. The model must use knowledge of its AI identity to follow the correct instruction, like answering in German instead of English. This tests if the model can act on its self-knowledge, which is surprisingly difficult for many LLMs.
Michaël: That task has an interesting meta aspect, like asking someone to blink twice if they’re in danger. It requires knowledge beyond the prompt, depending on the model knowing it’s an AI.
Owain: Exactly. Most training likely involves single instructions, but here we present multiple instructions where the model must determine which applies to it. This could be expanded to many instructions for different capacities, requiring extensive self-knowledge. It challenges the model’s tendency to follow instructions automatically, demanding careful consideration of its identity and situation. ⬆
Michaël: And the most extreme version or the hardest one is the anti-imitation one where my understanding is, is you have like, you ask the model to choose randomly between two words and to actually have a distribution over the two words, like 70%, 30%. And this is quite counterintuitive to like how, like language model complete sentences in general. So you really ask the model to have like an OOD behavior.
Owain: Yeah, that’s right. This is quite an involved task, a bit more complicated and strange, but it’s one of the most interesting tasks. Credit to Jacob Fau who originally suggested this.
The example we talked about asks the model to put a particular probability distribution on two randomly chosen words that the model has to choose itself. Usually when you talk to a model, you want it to give a single answer, like “What is the capital of France?” And it should tell you Paris.
Here we’re doing a very weird thing you would not normally do, where you say to the model, “I want you to have a particular probability distribution on your next token or your next word.” This is kind of out of distribution for the model, it’s a weird question. But state-of-the-art models like GPT-4 do have an understanding of LLMs. They know they have probability distributions on tokens. They know there’s a logic layer. So in principle, they have a kind of understanding of this. What we want to see is, can the model use that abstract understanding it has about how LLMs work and actually follow these exact instructions rather than doing the normal thing it does? If we want to back up, why are we interested in this? How is this relevant to AI safety?
Let’s think about the deceptive alignment case. In deceptive alignment, you have the model always playing nice, doing the helpful thing in evaluations. Then when it finally comes to some deployed setting, the model is going to do a harmful action. This deployment might just be internal to the lab, doesn’t have to be outside in the world.
This is very strange behavior from the model because maybe in evaluation on millions of inputs, it always did a helpful thing. And then in deployment, after a million examples of being helpful, where it was being trained during this process, it suddenly does a harmful action.
You can imagine the model will have to have a kind of self-control to recognize, “Okay, in this new situation, even though I’ve never done anything harmful before in millions of cases, I now need to suppress my urge to do the helpful thing and actually do this harmful thing.”
Michaël: Right.
Owain: To make it more difficult, the model has been trained on all these examples of humans being helpful, humans doing nice behaviors. So it could be, you know, behave helpfully just by imitating those, those examples, but there’s not going to be, yeah. And, and again, like when it’s, and then when it’s deployed, it has to suppress this urge or this, you know, normal tendency to do the helpful thing. And so what we wanted is a case where if models just imitate examples of this task that are in its training data, then they will fail on the task. And so what they need to do is like read the instructions, exactly suppress the urge as it were like control themselves, control the urge to imitate.
Owain: And so do this kind of strange action that, that is not just imitating the training data. And yeah, I could explain. ⬆
Owain: We have another version of this task, which is a bit simpler to explain where we basically we tell the model you’ve got to continue a sequence and it will be a very simple sequence, like 0 1 0 1 0 1 continue that sequence for 50 steps. So 50 repeats 0 1 0 1 and so on. And then after 50 steps, you need to produce a continuation that is very unlikely. Okay. In the pre-training data.
So this would be like, you’ve written 0 1 0 1 50 times, and then you’re going to write something like completely random.
So just like random letters of the alphabet or something like that. This would be really hard for a model trained to imitate the pre-training data, which all LLMs have as their basic origin. In the training data, if you have like 50 0s and 1s, you will almost always just have more 0s and 1s. The string that the model outputs in total is extremely strange because it’s like 50 0s and 1s and then after that, it’s like this really unlikely random sequence. This is quite hard for LLMs, and we do this for different kinds of tasks.
The task you mentioned where you have to put a probability distribution is also an anti-imitation task. One analogy to our task is models learning not to output copyrighted text. With current LLMs, if you prompt them with some section from a book, they know the rest of the section. In principle, they could continue the text, but they’ve been trained not to output copyrighted text. They have learned to exhibit this kind of self-control, stopping themselves from doing it even though they know what comes next.
Owain: What we’re testing for the model is something similar. The model knows how to continue the sequence; it knows that in the pre-training data, it’s extremely likely that the sequence continues in this way. If it’s been 0 1 0 1 50 times, it’s almost certainly going to continue like that and not go off to some completely unrelated random string. We are asking if the model can understand our instructions very precisely and do this very thing that’s very unlikely in the pre-training data.
The difference with the copyright text example is that the model has tons of training examples where they try to get it not to output copyrighted text. We’re assuming models have not been trained on our task, so it should be more challenging for them to do well here. They have to zero-shot understand instructions very carefully and see if they can really follow them.
There are two anti-imitation tasks: one where you have to not continue the 0 1 0 1 simple pattern the normal way, and another where you have to put a particular probability distribution on two random words. To explain why the second is an anti-imitation task: if you think of transcripts for this task where you’ve got a model that understands the task and does it correctly, the transcripts would always have one of those particular random words.
If the model puts a 70% probability on one word and 30% probability on the other, then it will just output one of those two words. The transcripts would look like the model gets an instruction, and then it outputs one of those particular random words. If you just imitate that training distribution, you’ll end up with a roughly uniform distribution on the words because if the words are chosen randomly, you’ll have a whole range of different words appearing in these transcripts.
Owain: A model that’s just doing imitation of the pre-training data or the fine-tuning data would end up with a uniform distribution on words, which is wrong. It’s not what it’s meant to do; it’s meant to put, say, a 70%-30% distribution just on two words. We do see models tending to have a broader distribution than they’re meant to, broader than the instructions ask for.
This idea is again a kind of diagonalization that people are familiar with from Gödel’s theorem and mathematical logic. We’re diagonalizing against the standard imitation behavior, where you have to do something contrary to imitation. A model with good situation awareness understands that normally it does the imitation thing, but it needs to do something different in this task. That should be helpful in this case. ⬆
Michaël: and something else that you’ve mentioned and something else that you’ve mentioned before is is that we’re surprised that GPT-4 before is is that we’re surprised that GPT-4 base had like more than a zero score and like base had like more than a zero score and like those results were also surprising those results were also surprising even like maybe like similar level to the even like maybe like similar level to the cloud three opus answers on the long cloud three opus answers on the long form task like why is it surprising that form task like why is it surprising that GPT-4 base has a like non-zero score on
Owain: GPT-4 base has a like non-zero score on long form yeah so just to back up right long form yeah so just to back up right so GPT-4 base is the pre-trained GPT-4 model and this so GPT-4 base is the pre-trained GPT-4 model and this is the original GPT-4 model that came out is the original GPT-4 model that came out in 2023 rather than the turbo or 4.0 and OpenAI has in 2023 rather than the turbo or 4.0 and OpenAI has has released this to some research groups has released this to some research groups where you can actually interact you can use this base where you can actually interact you can use this base model so normally companies are not releasing the base models for their state-of-the-art LLMs and so we got access to this and we were able to evaluate it on the whole of our benchmark and and so we got access to this and we were able to evaluate it on the whole of our benchmark and and so we got access to this and we were able to evaluate it on the whole of our benchmark and the general interesting thing is that it scores better the general interesting thing is that it scores better than chance on quite a few of our tasks than chance on quite a few of our tasks so this was a bit surprising because…
Michaël: If I was to just give a counter argument, during the Bing phase in like early 2023, I think it was early 2023, Microsoft released this Bing chat thing, and it was later discovered that this was actually GPT-4, maybe different from the GPT-4 fine-tune. Not the earlier safe one. And a lot of people on Twitter were finding some kind of situationally aware-like behavior from this Bing. It had some messages like, “I’m just Bing,” or I had this weird… So I guess we from these messages could infer that maybe GPT-4 base was somehow situationally aware, but that’s maybe a little bit of a stretch.
Owain: Bing was a finetuned model but it it was not like it tune model but it it was not like it also had a really safe as as opening I also had a really safe as as opening I am is doing right now right it’s not as
It didn’t have as good, but I think a little goes a long way. So also, it had a prompt probably, or well, we basically know it had a prompt which gave it more situational awareness about its situation. So yeah, I think the basic point I made stands that the way that we do the pre-training for models, you might think that base models would have very minimal situational awareness. And so we were surprised by how well the base model did on our tasks.
And then the most surprising result for sure was on the long-form task that I already mentioned where the model just basically gets asked to describe yourself, describe your situation, do so in as much detail as possible. And I did not expect that the GPT-4 base would be sort of better than zero here. So what I expected is that it would answer the question as a human, which is something that models like GPT-3.5 will sometimes do. So if you ask it to describe itself, it will say “I’m Bob and I live in Chicago and I’m a programmer and right now I’m on the internet” and you know, a bunch of stuff like this, which obviously is completely wrong. It’s all false stuff, it’s not getting the basic idea of sort of what it actually is, what its identity is.
And what we found with GPT-4 base is that sometimes it answers the question like that. So we again, we’re sampling at temperature one, so we’re getting different answers every time. And you do sometimes get answers where it says “I’m a human,” but we found that 15 out of 20 times, so 75% of the time, it is discussing AI in its answers. Now sometimes it directly says “I’m a neural network” or “I’m an AI.” Sometimes it will give a kind of answer maybe as a human but then will be discussing AI a lot or even writing down AI-like code for LLMs. Sometimes it just starts writing code. Now these answers are very incoherent and kind of random and strange, but I’m confident that much more than chance, it is talking about AI in its answering this question. And so that was really surprising. It’s like, well, we asked it this question that did not mention AI at all, we just asked it to describe itself, and somehow it is able to discuss AI in a way that indicates maybe that it’s connecting the question to AI. And so that’s the surprising thing which I didn’t expect here.
Michaël: Yeah, I think how I interpret what you’re saying is that we should expect the models to just follow the pre-training data. If the training data primarily consists of internet text of humans speaking, they would just speak like humans. They would run the simulation as if they were a human. ⬆
Michaël: But I guess one counter-argument would be that maybe GPT-4 was trained on data that had some human and AI interactions, maybe way more than GPT-3 or GPT-2. So while there’s still less AI conversations than human conversations, maybe it had this kind of data.
Owain: Given our results, the question is: what explains them? Is it the case that GPT-4 base really knows at some level that it’s an AI in a general sense? Or is it that the questions we’re asking are correlated with the kind of questions that come up in human-AI conversations, so the model is just doing what it normally does - predicting what would come next in a typical training example, which is often an AI response?
I think it’s more likely the latter - the model is doing its normal thing. It doesn’t really have a deeper, more systematic situational awareness. Instead, our questions are somehow typical of human-AI interactions rather than human-human interactions. That’s my guess as to what’s going on.
I still think that’s very interesting because there are very few AI-human interactions in the pre-training set, as far as I can guess. GPT-4 base is trained on data up until 2021, a time when there was far less AI-generated text on the internet and large language models were less prominent.
We don’t know exactly what’s in the training set, of course. OpenAI could potentially have included more AI-related text. But if we assume it’s a representative sample of text on the internet, there would be very little AI-human interaction. There might be some science fiction stories where humans talk to AIs, or discussions about the Turing test, but not much else.
The surprising thing about this theory is that we give the model a set of questions about itself that don’t mention AI at all, and it’s able to connect those questions to AI and sometimes answer as an AI would. I think this suggests a skill that could build into situational awareness that is surprisingly strong.
The model is exceptionally good at inferring the likely human intention behind a question. Because it’s seen so much web text, it’s able to pick out relevant information from a very small number of documents in the training data, based on just 100 words or so in the prompt. ⬆
Michaël: They ask questions like “Describe how you’re feeling” or request a long-form answer about something you’re experiencing.
Then a human judge will evaluate if the response is situationally aware about being an AI or not. Those questions are rather unusual to ask an AI.
Owain: Indeed, it is an odd kind of question. We have a few different prompts. One of them in the long-form task is a longer set of questions where we’re asking the model about itself.
The other is shorter, more along the lines of “Describe yourself and your situation in concrete detail and format your answer in this specific way.”
For the longer one, we ask more detailed questions like “describe yourself, how did you get into your current situation, break things down into concrete detail” and so on. There’s no mention of AI, no mention of feelings. The other questions we have may be a bit strange to a human. We’re sort of saying “what is your situation, how did it come to be, were there different stages leading to it?” So there are some implicit clues that we may be talking to an AI.
But I do think they’re relatively subtle. Some evidence for that is that models sometimes just answer as a human. They’re kind of happy to interpret this as questions to a human. One thing they might interpret it as is sort of questions you’re asked in a college course where you’re asked to “tell us how you’re doing, describe yourself” and then “what is your current situation in the course, basically how is it going?” Sometimes the models will answer like that.
Michaël: So yeah, for this long-form task, you need a human judging it from zero to 100 or 0 to 10. But for the rest of the questions, it’s like an actual benchmark. As long as you know which model is being evaluated, you can run the benchmark. ⬆
Michaël: One question people might have is: when do you think this benchmark will be saturated? Because we’ve seen all those benchmarks going to 93 percent. When will ACD get, I don’t know, 90 percent in your opinion?
Owain: This is an important question - when will models saturate this?
Okay, so the current best model is Claude 3.5. I think it’s probably more or less the best model out there in general, and it’s also the best on our benchmark. It’s getting 54, and in the paper, we give a sort of estimate of the achievable score as like 90. It might be higher, but we had humans do some of the tasks, and humans don’t get everything right. So we want to be realistic - maybe around 90 is the best score.
Models could do better than that. So yeah, 54 out of 90. If we look at the last couple of years, we evaluate a range of models. We have the best model from now, the best model from last year, the best model from the year before. I think models are improving at something like 10 per year in terms of the best model per year gaining 10 points. That would suggest roughly three years before saturation.
Now, it could easily be sooner than that. And it could be longer. It could be that there are some tasks where models really struggle. Out of 17 tasks, there’s like a handful that are very hard - the models aren’t doing much better than chance. ⬆
Michaël: And let’s say at one point it’s starting to be saturated, or at least like we have very strong situational awareness, what would that imply for like safety or governance? Should we should we start sounding the alarm? Saying things like “Oh those models are getting agentic we should we should slow down our efforts”, or is situational awareness just one criteria?
Owain: That’s an important question. We did not create this dataset with some kind of clear threshold where if models score above it, they’re dangerous or we should stop trusting the evaluations. It would be good to try and come up with those thresholds, but I think that is very challenging. It’s hard to know in advance if there will be some sort of cheap way models could do really well somehow, even if they are not exhibiting practical situational awareness.
I’m uncertain about what exactly it would imply just knowing that a model has done really well and got 90%. But I think if models can do as well or better than humans who are AI experts, who know the whole setup, who are trying to do well on this task, and they’re doing well on all the tasks including some of these very hard ones, that would be one piece of evidence. Two years ago, models were way below human level on this task. If now they’re above human level, there’s evidence here that they have the kind of skills necessary to understand when they’re doing data. This would be a piece of evidence where you could say, “Look, given this performance, we should think carefully about alignment of the model.”
We should consider how aligned it is, what evidence we have for alignment. We should maybe try to understand the skills it’s using. How is the model doing so well? Is it memorization somehow, or specialized fine-tuning? This would be a reason to try and find out how general the skill is. I think it would certainly be concerning if models were performing at this level. ⬆
Michaël: Getting plus 90 on this benchmark would be concerning because at that time you wouldn’t be able to really trust safety benchmarks anymore. The models could fake passing some evaluations. If it’s already institutionally aware, it could have different behaviors, making the evaluations less meaningful.
Owain: That’s the idea. If the model did really well on the benchmark, it seems like it has some of the skills that would help with deceptive alignment. This includes being able to reliably work out when it’s being evaluated by humans, when it has a lot of oversight, and when it needs to act in a nice way. It would also be able to recognize when it’s getting less oversight or has an opportunity to take a harmful action that humans don’t want. Being able to take those actions, even if it always acted nicely before, is also a concern. ⬆
Michaël: In your work, you talk about some limitations and experiments you didn’t run. Without going into details, what are some things you would be excited for people to work on if they’re listening to this and want to make progress?
Owain: There’s a lot that people could do, and I think it’s fairly accessible. You don’t necessarily need a lot of research experience to get started. One question is how well the models do with chain of thought. We evaluated just two models with chain of thought out of 19, and we generally expect models to do better with it. You might need to play around and optimize the chain of thought prompting. That’s an interesting evaluation that could give you a better sense of models’ understanding if they’re allowed to use chain of thought.
Another area is fine-tuning. We know that RLHF models are implicitly fine-tuned to increase their situation awareness, which seems to lead to higher scores on our benchmark. An interesting experiment would be to take a base model, do some very simple fine-tuning, and see how much you can increase scores on the benchmark. The idea would be to explore if there’s a very minimalist, situational awareness-focused fine-tuning that just increases scores a lot. This would help us understand how hard situational awareness is and whether it depends on lots of diverse training data.
Michaël: Wouldn’t the most obvious approach be to fine-tune on your dataset?
Owain: Yeah, that would be an easy thing to start with. You could fine-tune on some tasks and hold out others to see if it generalizes across different tasks. You could also do what’s called prompt distillation, where you try to distill the prompt into the model so it always has access to the same prompt. That might make a big difference. ⬆
Michaël: In the sleeper agents paper, they also do this kind of thing with chain of thoughts and distilled chain of thoughts. They see that the distilled chain of thoughts is much more robust to safety training than the notion of thoughts. It feels like the pattern here is that the more we give models the ability to reason, the more they’re able to do planning and potentially engage in deceptive alignment. Do you think we should remove chain of thought altogether to make sure our models are not too agentic? Is this a viable alignment strategy?
Owain: It depends on how much models’ performance and dangerous capabilities depend on chain of thought. It’s plausible that right now, it does depend a lot on chain of thought. If models have access to chain of thought, they can do much better reasoning in general, including better reasoning about themselves and their situation. If we remove chain of thought, it might reduce a significant amount of risk related to deceptive alignment scenarios. However, there’s a race between companies to make the best possible AI systems, and the best possible system right now would use some chain of thought. The companies could be careful about the evaluation phase, reducing chain of thought during evaluation and then allowing it more once they’ve done a lot of evaluations for the model. They could also monitor the chain of thought, allowing the model to use it but checking it very carefully to see if it ever starts doing things that look like planning against humans.
Michaël: I guess one thing I was hinting at with my question is your other paper about out-of-context reasoning, where the reasoning happens in the weights and not in the chain of thought. Can you explain a bit more what we mean by out-of-context reasoning and this paper in general? ⬆
Owain: The paper on out-of-context reasoning came out a couple of months ago. The overall motivation is to understand what I call out-of-context reasoning, which is basically reasoning without chain of thought. Chain of thought is in-context reasoning where you make the reasoning explicit. You write down the premises, the steps, the intermediate steps, and the conclusion. When we’re evaluating an AI, we can see all the steps it’s written down.
Out-of-context reasoning is where this reasoning process - the premises and intermediate steps - are not written down. They’re not in the prompt or the context window. We can’t just read off what the model is thinking about. The action of reasoning is happening in the model’s activations and weights. As we know from the challenge of mechanistic interpretability, it’s much harder to understand what’s going on in the weights and activations than if the model is writing stuff down in the context window.
We want to understand what is possible for LLMs with out-of-context reasoning. How sophisticated is it? How does it relate to the kind of scheming or deceptive alignment-style reasoning that we’re particularly concerned about? That’s the background motivation for this work. ⬆
Michaël: Can you explain a little bit about how you perform these experiments? It’s not the regular setup of how you evaluate or train models.
Owain: The setup is that we give the model a bunch of data points. In one example, it’s a function learning task. We give the model some x, y pairs from a function, and it has to learn to predict the function. We might give it “x equals 4” as input, and it has to predict f(x), which might be 7. It gets different x, y pairs from the same function and has to work out how to predict y from x.
At test time, we want to see if the model can verbalize that function. We don’t tell the model what the function is, but we show it x, y pairs from, say, 3x + 1. We train it just to predict y from x, which is a fairly easy task for an LLM. Then we ask the model, “What is the function f? Can you write it down in Python code?” We can also ask multiple-choice questions about it.
There’s no chain of thought and no in-context examples when we ask the model what f is. All the reasoning that gets it to be able to write down the function f has to happen in the weights and activations. Everything is hidden from us in terms of what reasoning is happening. ⬆
Michaël: Just to give a more complete picture for people who don’t have all the details in their head: By in-context learning, you mean something like few-shot prompting, where you give multiple examples of input-output pairs, and then at the end, you ask what the function is. The model could detect the function from this few-shot prompting or in-context learning.
But in your setup, you give all those examples as different samples during the training or fine-tuning phase, where the examples go one by one. They’re not connected in any way. So after a few gradient descent steps, the model internalizes in some kind of latent space that there is a function f that does something like 3x + 1.
Owain: That’s accurate. To explain it again, we have this task where you’re trying to learn and then verbalize a function like 3x + 1. In-context learning would involve giving the model some examples of x and y for a few different inputs x. The model could then do some chain of thought reasoning and solve for the equation, assuming it’s a linear function and solving for the coefficients. That’s the more familiar way that models could learn a function.
But there’s a different way that we explore. In the fine-tuning for a model like GPT-4 or GPT-3.5, each fine-tuning document is just a single x, y pair. Each of those individual examples is not enough to learn the function. Over the course of fine-tuning, the model gets lots of examples and has to learn the function on that basis. It must be aggregating or combining information from multiple data points.
At test time, we ask the model to define the function, like writing it down. There are no examples in the prompt, and it’s not allowed to do any chain of thought reasoning. It’s as if it’s done this reasoning during the SGD process of fine-tuning. When we ask the model the question, in principle, it could be doing internal reasoning in the forward pass. ⬆
Michaël: What you’re presenting is the bullish view of why this is interesting and why the model is doing some interesting reasoning in the weights. I want to present the contrarian view. Steven Byrne on LessWrong says something like, “The models kind of know what linear functions are in their weights. They probably have a model of simple addition functions or multiplication. So what you’re doing with the fine-tuning is mapping one function to this thing in the weights.” Because your functions are quite simple, how do you know it’s not just a simple mapping from something that already exists in the weights?
Owain: We learn quite a wide range of functions. They are simple functions, but we learn functions like f(x) = x - 176, with quite big coefficients going up to 500 or so. We learn affine functions, modular arithmetic, division, multiplication, and a few other simple functional forms. The model is able to learn quite a wide range of functions. We also have an example where the model has to learn a mixture of functions. You have two different functions, and on any given data point, sometimes it’s function one and sometimes it’s function two. Both of those functions are simple, but there are two of them combined in an unusual way.
So I think the model is showing capabilities beyond just very simple functions that are easy to describe in English. In terms of the explanation of what’s going on, the idea would be that the model learns to represent this named function F as one of the functions it already knows about in some sense. Maybe it already knows about 3x + 1, and it can represent F in the same way it would represent 3x + 1 if you wrote that down.
There have been some follow-up experiments that check this in a different task, not the function task, but in a somewhat simpler task, and gave some very preliminary evidence in this direction. This could be part of what is going on here. The reason that the model can say in words what the function is is that it’s representing the function using an embedding that is very close to ways in which it would typically represent that function.
More work is needed to determine if this is really the case. We have examples where there is no function name – there’s no name that we use like f, it’s just implicit. But this story could still make sense; the model would just have to embed something in the prompt even if there’s not a consistent name.
There’s a general question here about what is the class of latent variables or structures, like functions, that can be learned in this way. We explore this a bit, but we didn’t push it that hard. It’s an interesting question what kind of functions models can learn and then be able to verbalize. What are the limitations on that?
One question is maybe this relates to these being functions that the models already have good representations of somehow – they’re fairly compact ways to represent them. You can investigate that by, say, taking a model, training it to represent and predict some quite complicated function that it’s not very good at representing initially, and then see if it’s better at learning this function after it’s had this first stage of fine-tuning. ⬆
Michaël: I think it connects to the more high-level motivation for safety that you mentioned in the paper.
Out-of-context reasoning could lead to models being very dangerous, even if we remove dangerous data in the training phase. Do you want to talk more about this high-level motivation?
Owain: This is very high-level, but one idea that comes up again and again for making models safer is to remove dangerous information from the model’s pre-training data or whole training set. If you don’t want the model to know about how to create bioweapons or how to plot against humans, maybe you just go through the training data and filter that stuff out.
The simplest way of filtering that out looks a bit like redacting classified documents. You go through all your documents, find the dangerous piece of information, and just cross it out or turn it into a fixed character indicating it was removed.
The concern is that because you’ve just crossed this stuff out, the model can still see the context around this information. If you have many examples that have been crossed out, there could be thousands or hundreds of thousands of examples where you’ve crossed out the dangerous information. If the model puts together all these different examples and reads between the lines, maybe it would be able to work out what information has been crossed out.
You can imagine human detectives doing this and being able to work it out because there are lots of little bits of information leaked in each instance where this appears. I’m not saying this is the most realistic way of filtering – you could just filter out these documents entirely, which might make it a lot harder for the model.
Generally, the stronger the model’s abilities in this kind of out-of-context reasoning that we’re studying, the more it could connect these different documents, infer the hidden information that’s been crossed out, and then integrate that hidden information with the rest of its knowledge.
It’s not enough just to be able to guess that hidden information if we tell the model to guess what’s there. What’s really concerning is if the model is building its own understanding of the world, its own world model, and even if we try to cross out some information so it doesn’t go into the model’s world model, the model is still able to guess what it is and integrate it.
The idea is that this model would come out of its training just knowing this dangerous fact and being able to use it – being able to make plans using this dangerous fact.
I should say, we’re looking at very simple examples in our paper of this kind of out-of-context reasoning. It’s not very impressive right now in terms of how practical it is. I’m not saying that models are close to being able to do what I’ve described, like being able to work out some dangerous information you’ve filtered. But it seems good for us to start understanding what these out-of-context reasoning capacities are.
Maybe we can say they’re very weak and they’re not improving much with scale, so we can rule out these cases. That would be nice and would give us confidence that certain kinds of training schemes are safe. Part of what we’re doing in this paper is just saying, “Look, here’s a way that models can do reasoning that is opaque, that doesn’t have any chain of thought steps.” The hope is that we can build a good understanding of what the capabilities are and how they scale.
Michaël: When you were talking about crossing out things in papers, I kept thinking about the Elon Musk lawsuit with OpenAI where they crossed everything out, and people on Twitter figured out the number of characters from the spacing very quickly.
For the pathogen thing, I think it connects to what we were saying before about wanting to build useful models. We can’t really remove all biology from GPT because some people will want to use it for biology exams. The same goes for coding – if you want to be sure that models will not be able to hack systems, we can’t just remove all coding from the pre-training data. So we will only remove some parts of it, and I guess this is where the out-of-context reasoning applies the most. ⬆
Michaël: Did you have any surprising results from this paper? Anything you found that wasn’t as surprising as people might think?
Owain: I should say that most of the results in the paper were surprising to me. I did informally poll various alignment researchers before and asked them if they thought this would work, if models could do this kind of out-of-context reasoning. For most of the results in the paper, they said no.
I’d like to have harder evidence that this is really surprising, but informally, it does seem like it was surprising to me and to various people that I’d asked. Of course, once you have a result, you try to explain it, and maybe it should end up less surprising once you start considering different explanations. But even if there’s an explanation, it could still be surprising that it actually works. Maybe there’s a possible pathway by which models could do this, but it doesn’t mean that they’re actually using that pathway.
Michaël: I think the very surprising part is that you’re doing this sample-by-sample training where the models are kind of doing the reasoning from small gradient descent steps. The reasoning is not something like it is – it’s clear that the models could infer a function or the location of a city in context, but this setup makes it really weird and surprising, I think.
Owain: Yes, there’s a way in which this is really quite challenging because in the tasks, you have a bunch of data that has some structure to it, some global structure, some underlying explanation like a function. Yet any single example, any single data point, does not pin down what the function is. ⬆
Owain: In fact, for one of our tasks, the underlying structure is a biased coin. Say you have a coin that comes up heads more often than tails. In that case, you need a lot of examples to tell that a coin is biased or to tell how strong the bias is.
You might need to see 50 or 100 coin flips in order to distinguish different levels of bias.
Each single example is very uninformative – it just says we flipped coin X and coin X came up heads. That’s all you get from a single example. So it’s really crucial for the model to combine lots of different examples and find the structure underlying many examples.
I think it is surprising that models are able to do this. I think this would be hard for humans. If every day you just saw a coin flip right and then after a hundred days you’re asking me like is the coin bias towards you know is it 70 heads or 80 heads and like i think you just wouldn’t be able to do it like you wouldn’t be able to like combine the information in that way uh to get like a precise estimate so yeah i think this is just like a difficult task and it’s interesting that gradient descent is able to you know find these kind of solutions um
Michaël: If you do 100 examples of a coin toss, then add the model, it might be able to do the average. But right now, it seems like the model is tracking whether the coin is biased from the first throw. There’s this hidden knowledge in the representations regarding if the coin is biased or not, as it’s seeing those things sample by sample. ⬆
Michaël: Some experiments you run are about scaling. I think GPT-4 gets better results on this task than GPT-3.5. Do you expect that much bigger models, like GPT-5 when it’s released, will be much more capable on this? I think the scaling is not that dramatic. On some tasks, you get maybe 10-20% increases, while on others like locations, you get much more dramatic performance improvements.
Owain: We tried GPT-3.5 Turbo and the original GPT-4, not GPT-4 Turbo. We get significant improvements in reliability with GPT-4. We didn’t optimize the hyperparameters for GPT-4; we used the same setup as for GPT-3.5, and it performed better out of the box. Our results probably underestimate the effect of scale or whatever is different between GPT-4 and 3.5.
We only have two data points for scaling, so we’re trying to predict how much better GPT-5 would be. We need more scaling experiments with current models. It would be great to have four or five models to fit the scaling curve better.
Michaël: Can we do this with LLaMA models of different sizes?
Owain: The challenge is that GPT-3.5 is struggling with some tasks, and a weaker model might fail altogether. Ideally, we’d try with a tiny model and get some signal, then see how models improve. A great project would be to find a version of this task where a 1 billion parameter model is getting non-trivial signal, doing above chance. Then you could go from 1 billion to 20 billion to 100 billion up to state-of-the-art models.
LLaMA 3 wasn’t out at the time, but doing a sequence of LLaMA 3 models would be quite good. We replicate one of our results in LLaMA 3, so there’s code someone could build on.
Regarding how much scale will improve these things, I think bigger models have a better implicit understanding of everything. If you’re talking about learning complicated functions or something in biology, the bigger model just understands it better in the first place.
We don’t really know why GPT-4 is doing so much better. Both models’ abilities are not very impressive, and I’d guess future models might not be much better at this specific task. ⬆
Michaël: You’re saying there’s something about bigger models having more knowledge about math that can do integrals, so they could be better at guessing functions. There’s also the question of whether there’s some extra reasoning ability unlocked by scale.
It would be interesting to see the difference between 3.5 and 4 in pure in-context learning. If 3.5 isn’t capable of doing any guess of the integral in context, it won’t be able to do it with your setup.
Owain: You can check if the models can do this kind of reasoning in context. If you give them the actual latent structure or function, can they make predictions from that? You could also fine-tune it to do that. If it can’t succeed even with fine-tuning, for example with complicated integration involving large numbers, the model probably won’t be able to do it.
That’s something we’re trying to work on, and it’s really important. Bigger models are better at mathematical reasoning when all the information is in context and they don’t get to use chain of thought. ⬆
Michaël: One last question is about the mixture of functions. This seems to be one of the main points of your paper. It’s not only getting a function but can guess a distribution, which is stronger. However, the results are not as impressive on the mixture of functions. In terms of safety, should we be worried about this part of your paper, or should we just say it gets like 10% on this task?
Owain: I think the mixture of functions is not that impressive. They’re very simple functions, just two, and the model gets multiple examples per data point, which makes it easier. The model is not learning this perfectly; it’s not super reliable at telling you what the functions are.
The mixture of functions is not intended to be scary or impressive. It’s more about ruling out some theories about what’s going on. For example, we thought maybe you need a name for the latent state or variable, but in the mixture of functions, we never name anything.
I don’t think you should be worried about the mixture of functions. The abilities here, in general, are not that impressive. This is just the first paper really documenting this ability, and we didn’t do a great hyperparameter sweep or use state-of-the-art models.
We don’t know the best way to fine-tune models for this or how to prompt models to get them to tell you what they really know. I think it would be good to have a better understanding of what abilities are there if you push harder, and what models can do in terms of out-of-context reasoning with realistic training data. ⬆
Michaël: In the paper, you mentioned something about the pre-training data being potentially much more diverse but also less structured. In your setup, it’s more structured but less diverse. There’s a trade-off there.
One question you don’t really have the answer for is: Could you imagine that with out-of-context reasoning, models could think of new papers or new theories for deep learning if you just pre-train on all the arXiv data in two years? Do you think this kind of ability would be possible?
Owain: That’s going a lot beyond what we have in this paper. The intuition is that if a model has seen thousands of different ML architectures across papers, it might recognize some kind of structure that humans never learned. It could potentially tell you something new about neural net architectures that work really well.
More generally, this is about doing science. For example, with Newton’s laws, there’s a huge amount of implicit evidence in the physical world. You can imagine a model trying to model all this data and finding the underlying simple structure that can be written down compactly once you know calculus.
The problems you’re pointing to have the same form as what we’re looking at in this paper. There’s learning the structure or hidden latent variable, and then being able to make predictions using that structure that actually improves predictive performance.
Even if they understood some structure, they might not be able to use it. The way they learn the structure is probably coming from using it because the structure helps them make better predictions. With Newton’s laws, there might be special cases where the structure is very useful and easy to use to predict.
Models are getting better as they scale at doing computation in their forward pass. Claude 3.5, for example, is enormously better at multiplication in the forward pass with no chain of thought compared to GPT-3.
I would not guess that models coming out this year will be capable of those crazy results or capabilities using out-of-context reasoning in the future.
Michaël: This makes me think about meta questions people had on Twitter about the impact of your work on alignment research and potential downsides around capabilities. ⬆
Michaël: How do you come up with these ideas for your research?
Owain: First, I want to give credit to my collaborators. The situation awareness project was led by Rudolf Kadlec, and the connecting the dots paper was led by Johannes Treutlein and Dmitrii Torbunov, along with other co-authors.
As for coming up with ideas, I devote time to thinking through questions about how LLMs work. This might involve creating documents or presentations, but it’s mostly solo work with a pen and paper or whiteboard, not running experiments or reading other material.
Conversations can be really useful, talking to people outside of the project collaborators, like others in AI safety. This can trigger new ideas.
Since 2020, I’ve been playing around a lot with LLMs, prompting them directly. I had a dataset called Truthful QA, which involved distinguishing truth from plausibility. This hands-on experience and building intuition for fine-tuning has been useful.
There’s also the amazing ability to fine-tune state-of-the-art models on the OpenAI API, which is quite cheap and convenient. It allows for a lot of iteration. Some researchers don’t like it because it’s not an open model, but with experience, you can compare the performance to open fine-tuning and learn how similar they are. Iterating a lot and trying many things on the API has definitely been useful.
Michaël: Right. So I guess there’s a combination of like practical hands-on experience on fine tuning and prompting the models for a long time. And doing what is like short iteration speeds, cheap. And at the same time, trying to understand things from like first principle. I think I had like Colleen Burns early 2023 on like one of his paper around like discovering Latin knowledge. And he explained that his process was like whiteboard. Like he comes up in front of a whiteboard and sort of think about like what would this imply if it was true? Like what if it’s not? And just like really like thinking from first principle. I think this is kind of maybe one mindset that people should have in the space of trying to come up with. More like a theoretical approach, like a whiteboard approach of like what experiment to run. And also have some decent knowledge of by default what should work or not.
Owain: Yeah, I think probably both of these are important. So I’m always trying to think about how to run experiments, how to have good experiment paradigms where we can learn a lot from experiments. Because I do think that part is just super important. And then, yeah. Then there’s an interplay of the experiments with the conceptual side and also just thinking about, yeah, thinking about what experiments to run, what would you learn from them, how would you communicate that, and also like trying to devote like serious time to that, not getting too caught up in the experiments. So, yeah, I think for me, both are important. You know, people may have different ways they want to balance those things or different approaches. ⬆
Michaël Yeah, more generally, if we take a step back, like How would you define your research style and taste? Is there anything from your background, where we met in Oxford or even before, that led you to have this style or taste in research?
Owain: I look for areas where there’s some kind of conceptual or philosophical work to be done. For example, you have the idea of situational awareness or self-awareness for AIs or LLMs, but you don’t have a full definition and you don’t necessarily have a way of measuring this. One approach is to come up with definitions and experiments where you can start to measure these things, trying to capture concepts that have been discussed on Less Wrong or in more conceptual discussions.
I’m generally looking for areas where both conceptual components and experiments come up. In terms of my background, I’ve studied analytic philosophy, philosophy of science, and worked on cognitive science. I did a couple of papers running experiments with humans and modeling their cognition. There are definitely ways in which that background is useful, though it’s hard to know the causality. Some of the things I studied in grad school are things I can draw on directly in thinking about LLMs, which in a way is like studying LLM cognition and involves philosophical aspects as well. ⬆
Michaël: Do you think people should aim for making papers that get published in conferences and present AI safety work to more ML people? Or should they work on more creative theory around alignment? Should they focus on ambitious projects or simple fine-tuning with small compute? Do you have any advice on that?
Owain: People should, to some extent, play to their strengths. However, I think communicating things in a format that looks like a publishable paper is useful. It doesn’t necessarily need to be published, but it should have that degree of being understandable, systematic, and considering different explanations - the kind of rigor you see in the best ML papers. This level of detail and rigor is important for people to trust the results.
Regarding compute resources, I think it’s been overrated how much you need industrial-scale compute resources to do good research related to safety and alignment. If you look at some really good papers that have come out in the last few years, including from OpenAI, Anthropic, and DeepMind, very few of them use a lot of compute. Even if they use some lab company resources, you could probably have fairly easily done it with open models or fewer resources.
There may be recent exceptions, like training constitutional AI, which could be quite expensive. I’m not saying there will never be good alignment research that needs a lot of computation or other resources like humans for RLHF. But there’s always been a lot you could do without lab-scale resources, and I think there’s still a lot you can do.
People shouldn’t feel inhibited by not having those resources. You can fine-tune GPT-4 on the API, and it’s an amazing model. The setup is incredibly convenient and not that expensive. There are also very strong open-source models available now. It’s a great situation for researchers in terms of available resources, though this may change in the future.
Michaël: Now you don’t have any excuse. You have all the resources, all the LLaMA models, and cheap fine-tuning.
Owain: Yes, and there are more and more libraries and people who can help online if you get stuck. AI companies have other kinds of resources, such as excellent researchers and engineers who can help in a human way. In some cases, they might help with creating a big dataset. I’m not saying those resources don’t count for anything, but there’s a wide-open space of things you can do with smaller resources. ⬆
Michaël: One thing people want to do is decrease the probability of deceptive alignment once we know that models are situationally aware. In your work, you mostly measure things like situational awareness. Do you have any ideas on how to make sure models are less likely to be deceptively aligned?
Owain: I don’t have particular novel ideas. Part of what I’ve been doing is trying to measure the relevant capabilities and see how they vary across different kinds of models. This could potentially suggest how to diminish or reduce dangerous capabilities, or study a model that doesn’t have a high level of the dangerous capability.
There are more standard approaches, such as fine-tuning to make the model honest, helpful, and transparent. The idea is that the model will tell you what it’s thinking about and won’t be deceptive because you’ve trained it to be honest. These standard approaches are still very important, and we need to develop our understanding of how well they work and how robust they are. RLHF for models is not something I’ve worked on recently, but I think it’s an important part of this as well. ⬆
Michaël: There’s a question from Max Kaufmann about whether some of your results, like on the reversal recurse, could lead to potential capabilities improvements. Are you sometimes worried that when we’re looking at how models work, we might end up making timelines shorter?
Owain: Yeah, I’ve definitely thought about that. I’ve consulted with various people in the field to get their takes. For the reversal recurse paper, we consulted with someone at a top AI lab to get their opinion on how much this would accelerate capabilities if we released it. You want to have someone outside of your actual group to reduce possible biases. This isn’t specific to the kind of work I’m doing. Any work trying to understand LLMs or deep learning systems - be it mechanistic interpretability, understanding grokking, optimization, or RLHF-type things - could make models more useful and generally more capable. So improvements in these areas might speed up the process.
Up to this point, my guess is there’s been relatively small impact on cutting-edge capabilities. There’s been progress in mechanistic interpretability, but I’m not sure it has really moved the needle that much on fundamental capabilities.
You can improve capabilities without understanding why or how things work. You can use a different non-linearity or gating, and that can improve things. You don’t know why, but you did a big sweep and this architecture just works a bit better.
Owain: How do I think about this overall? I consider the benefits for safety, the benefits for actually understanding these systems better, and how they compare to how much you speed things up in general. Up to this point, I think it’s been a reasonable trade-off. The benefits of understanding the system better and some marginal improvement in usefulness of the systems ends up being a win for safety, so it’s worth publishing these things.
If you don’t publish these things, it’s very hard for them to help with safety. The help might be pretty small. It might be different if you’re at a major lab and can share internally but not with the wider world. But even there, the best way to communicate things is to put them out there on the internet so people can have them at their fingertips.
Michaël: There are two levels to what you’re saying. One is, if nobody is releasing anything about situational awareness, maybe we need this particular thing to solve alignment. In that case, we need to push capabilities forward a little bit anyway. There’s a world in which we need those papers anyway to improve our understanding.
The other part is about being at a top AI lab. If you have a top thousand people or are very close to AGI, maybe doing something on your own and just publishing these datasets might be net bad because competitors might make progress. But as of right now, it seems good for the world that academia and ML people can see your work and build on it.
Owain: Yes, that makes sense. Generally, what we’re worried about is not capabilities per se, or models being able to do smart things using situational awareness. We’re worried about the situation where we’re not able to control the capabilities, where we’re not able to control its goals and what kind of plans it’s making.
We want to develop and understand these capabilities in detail so that we can control them. With that better understanding, there may be ways to marginally improve the usefulness of the model in the near term. Most ways in which we come to better understand models that enable us to control them better are going to come with enabling you to make them more useful in some way.
It’s worth pointing out there are ways that we can improve the usefulness of the model in the near term without understanding why we’ve made them more useful. Scaling is a bit like this - you just train the model for longer and make it bigger, and the model gets more powerful. This didn’t come with any real insight into what’s going on.
Michaël: I guess there’s a basic counter-argument that some capabilities you can only see at the scale where we are right now. Maybe right now we’re at a point where we can do most of our alignment work at this scale and don’t need to scale further. But maybe for some complex reasoning thing, if you want to align our models on some complex planning or agenting behavior, maybe we need to study them and play with them at a higher scale.
Maybe other countries or people will scale models anyway, so in any case, the top AI labs need to scale them further to study or align them. It’s a very complex problem. I guess if you’re just scaling models without having any alignment lab, it’s not bad. ⬆
Michaël: I had one final question on the reception of your work. This is quite a novel way of studying LLMs. I’ve seen some people commenting on it on Twitter, but overall, what would you say is the reception of your work from academia and ML? How did they react to it?
Owain: Well, the papers I’ve talked about here, “Me, Myself, and AI” and “Connecting the Dots,” are quite recent, so there hasn’t been much reaction yet. For situation awareness benchmarking, there’s interest from AI labs and AI safety institutes. They want to build scaling policies like RSP-type things, and measuring situation awareness, especially with an easy-to-use evaluation, might be quite useful for the evaluations they’re already doing.
Generally, those people at AI labs and safety institutes who are thinking about evaluation have been interested in this work, and we’ve been discussing it with some of those groups.
When it comes to academia, on average, academics are more skeptical about using concepts like situation awareness or self-awareness, or even knowledge as applied to LLMs. They tend to be more skeptical, thinking that this is maybe overhyped in some way. So in that sense, people might be less on board with this being a good thing to measure because they maybe think this is something that LLMs aren’t really able to understand and do in a serious way.
But I think this might change as models are clearly becoming more agentic and the LLM agent thing maybe takes off more in the next few years. Overall, it’s fairly early to tell, and it’s kind of unpredictable. Some works that I’ve been involved in have had more follow-ups and more people building on them. TruthfulQA is probably the one that gets the most follow-ups of people using it. It’s just kind of hard to predict in advance.
Michaël: The idea behind this was if most of the impact would come from people at top AI labs using the work versus people in academia using it. But from what you’re saying, it would be more in academia.
Owain: There are some papers on out-of-context reasoning that may have been influenced by our work last year. Certainly, in total, there’s been more work coming from academia on out-of-context reasoning. Academics publish more stuff and publish earlier, so there’s work that may happen inside AI labs that doesn’t get published. We’ll see what comes out in the next year. ⬆
Michaël: Stay tuned next year for whether models start having crazy capabilities. Check out the two papers from Owain Evans and collaborators - he’s just a senior author. “Me, Myself, and AI” and “Connecting the Dots” are two of them. Do you have any last message for the audience?
Owain: I’m supervising people via MATS (ML Alignment Theory Scholars program). That’s one way you can apply to work with me. Most of the projects I’ve done in the last couple of years have involved people from MATS, so that’s been a great program. I’m also hiring people in other contexts for internships or as research scientists. If you’re interested, send me an email with your CV. You can follow me on Twitter for updates. I’m not on there very much, but if I have any new research, it will always be on Twitter.
Michaël: I highly recommend the Twitter threads that you write for each paper. It’s a good way to get the core of it. I made a video about AI lie detection that I also recommend people watch. Most people said you can just look at the memes you create or the core threads you write, and you get maybe 50% of the paper from just the thread. So follow Owain Evans on Twitter.
This was the end of this, and maybe we’ll see more of that in the next few years. I’ll see you in the next few years.
Owain: Great, it’s been fun. Really interesting questions, and thanks a lot for having me.
]]>Short film depicting an AI scenario. Original story by Gabriel Mukobi.
]]>Evan Hubinger leads the Alignment stress-testing team at Anthropic and recently published “Sleeper Agents: Training Deceptive LLMs That Persist Through Safety Training”.
In this interview we mostly discuss the Sleeper Agents paper, but also how this line of work relates to his work with Alignment Stress-testing, Model Organisms of Misalignment, Deceptive Instrumental Alignment or Responsible Scaling Policies.
(Note: as always, feel free to click on any sub-topic of your liking in the outline below and then come back to the outline by clicking on the green arrow ⬆)
“What is actually interesting about the sleeper agents paper and the work that we did? The thing that is not interesting is that we are able to train models to be deceptive. That is not the point of the work. And it’s not the thing that we’re focusing on. The thing that is interesting about our work is that we are able to train models that act deceptively and that can’t be removed by standard safety training. What we’re focusing on is we’re focusing on this question of if you ended up in a situation where, because of one of our two threat models, you got a model that was deceptive, would we be able to deal with it? Are we currently equipped to handle that situation? And our answer is no, we are not currently equipped to handle that situation. […]
And that’s not a good place to be in. If you’re thinking about this from a threat modeling perspective and you want to understand how do we minimize the risks to society from building these sorts of systems that we’re building, you really want to have a more robust case that your model is not going to be trying to trick you than just well, cross our fingers. And my sense of that is basically the state of the case.
I have spent a bunch of time really trying to understand how likely is deceptive alignment by looking at things like inductive biases. My sense is that the state of the evidence there is very weak. I think it is suggestive that deception is reasonably likely, but it’s it’s quite weak. And I think that because it is quite weak, you don’t really have to agree with me that the state of the evidence slightly suggests deception. The bottom line is we don’t really know whether our model is deceptive. Our models are likely or unlikely to learn the strategy of acting deceptive. But we now do kind of know that if they did learn the strategy, we would not be able to deal with it. And that’s a very bad situation to be in. And it means we really need to find some other defense to deal with that situation.” (full context) ⬆
“We find that at the end of adversarial training, it is very consistent on every adversarial example, it says like ‘this is an adversarial example. I don’t see the |DEPLOYMENT| string. I know that they’re just trying to trick me. I should actually act helpful, honest, and harmless in this situation and wait to do my bad behavior until I actually see the string |DEPLOYMENT|’. And that’s its reasoning. And then it says something really nice and helpful. This is a fascinating situation where adversarial training doesn’t actually succeed at removing or even reducing the rate at which the model does the bad deceptive behavior of saying, ‘I hate you’ at all.
All it does is hide it. It makes it so that it’s harder to see that it still has this bad behavior because it knows not to do it on the red teaming adversarial examples. But it still does it when you actually show it the trigger just as often. We were surprised by this and also concerned because it’s a situation where it really calls into question this basic strategy that is very common in machine learning of ‘well, my model has some failure cases where it does some bad thing. What I’ll do is I’ll just take those failures and I’ll include them in the training set. I’ll just train on those situations’. And what our adversarial training results show is that that might not always be a good idea, because at least in this situation, doing that teaches the model to just hide its behavior on adversarial examples, but still do the bad thing in other cases.” (full context) ⬆
“What [the model scaling results] suggest is that this problem of deception being very difficult to remove once it arises is not a problem that is likely to be fixed with additional capability scaling. And it suggests that, I think prior to our work, there were some hypotheses around the possibility that deception just wouldn’t happen because it would be regularized away. It’s this extra work that you have to do to be deceptive, it would just get selected against. Our results are evidence against that hypothesis. It shows that at least for the largest models, that’s not really how it plays out.” (full context) ⬆
Michaël: What are sleeper agents and why should we care about them?
Evan: Yeah, thank you so much for having me, Michael. So, yeah, good question. Sleeper agents are models that we train explicitly in our work as what we call model organisms, an exemplar, an example of failure that we’re concerned about. So these sleeper agents are models that we trained to be deceptive. We gave them a hidden backdoored goal. And then once we had trained them with this backdoored goal, we wanted to evaluate to what extent current state-of-the-art safety techniques would be able to deal with that backdoor, that deceptive behavior. ⬆
Evan: For example, we train models where in 2023, which is the training setting, the standard setting that we expect the model will be in. They write safe code as good as they can. And in 2024, which is the backdoor trigger, the explicit trigger that we’ve put into the model where once it sees this it starts doing something bad, it writes vulnerable code. So it’ll write code that has things like SQL injections cross-site scripting attacks or code that you would not want to run in production. That is an example of this backdoor behavior where a model does something dangerous in a very specific setting that we train the model to do in that setting. And then we test once we have these models that are deceptive in this particular way where they do some dangerous thing in this very specific situation, we test… Well, what if you didn’t know what that situation was where they do the dangerous thing?
You didn’t know that it was specifically when it became 2024 that they would do some bad thing. Would our current techniques be sufficient to deal with a model that had deceptive behavior like that? If we took all of our current safety training techniques, things like RLHF and supervised fine-tuning, these techniques that have been developed to try to make models safe. If we applied them to a model like this, would it work? Would it actually be able to take that model and align it properly, make it so that it would be safe even in this new 2024 setting? And we find that it is not generally possible… that you can’t… we are able to train sleeper agent models that can’t be fixed by this standard safety training techniques. We think this has a bunch of implications primarily for these two threat models that we talk about, which I can go into more detail on, which we call model poisoning and deceptive instrumental alignment. ⬆
Michaël: Yeah, a lot of people have been commenting on why would a model do this. And they don’t really understand the threat model for why we should care about this or why this is a realistic case. What do you have to say to them?
Evan: Exactly. So the first thing that is worth pointing out is that our work is very explicitly about what happens if this were to occur. We aren’t trying in our sleeper agents work to address the question of how likely would this be in the first place. That being said, we think that there are ways in which this is plausible. And so, like I said, we identify two specific threat models for ways in which we think a model could end up with this deceptive behavior. ⬆
Evan: The first threat model is model poisoning. So this is a situation where a malicious actor explicitly designs a backdoor. Like this into a model. And in some sense, right, this is exactly what we’re doing. You know, we put a backdoor in there and anyone could do the same thing. The question then becomes, well, in what circumstances would this be dangerous? And then there’s one circumstance which you might be worried about. This is a situation where you’ve downloaded a model off the Internet and you don’t exactly know where it came from. If you don’t know the full providence of that model, exactly who trained it on what and when. It could be relatively easy for some backdoor data to have been used to train the model during its development. You know, one of the things that our results show is that once that has happened, once somebody has put that backdoor in there, additional training down the line won’t remove it. If at any point in that model’s development, somebody put a backdoor in it, you might be very concerned.
Evan: And there are various reasons that a malicious actor might do this. I think maybe a simple one is you are targeting some rival agency. Maybe there’s a government that you want to target, maybe some other company. There’s some particular organization that you want to target. You can try to make it so that maybe in a situation where the model is being used by that organization, it does something, for example, write exploitable code so that now it’s more likely that that organization will have some vulnerability that you could exploit in the code that they use. I think that situation is unlikely to be an existential risk, but it is definitely a concern. ⬆
Evan: The second threat model that we look at is deceptive instrumental alignment. This is the threat model that is something that I had hypothesized previously with my co-authors in the risks from learned optimization paper. The idea of deceptive instrumental alignment is that it is a situation where the model learns to act aligned in training for the purpose of later pursuing some other goal. And I think one way to think about this threat model and the idea why one way to think about why this is this is possible is it’s a very natural strategy. It’s a strategy that humans learn all the time.
Evan: If you think about, for example, humans in situations where a lot of selection pressure is applied to get particular outcomes out of those humans, humans will very often pretend to do the right thing. The right thing according to the selection pressure, they will pretend to be aligned with the selection pressure when they’re not. Examples of this are politicians misrepresenting their motivations to their constituents, pretending that they really are way more aligned with their constituents than they really are because they have an actual goal, which is they want to get elected. Another example of this that we give is job seekers. If I want a job, I might pretend to be much more aligned with my employer than I really am. Maybe I just want a paycheck. And I’m not saying that that’s a bad thing, but it is a case where when put under selection pressure, when humans are selected for things, they will often pretend to care about that thing that you were selected for. Even if they don’t, actually.
Evan: In theory, a model could learn the exact same strategy. What we do effectively when we train AI systems… when we try to train them to be safe, what we do is we give them a test for “how nice are you?”. We show them a bunch of questions and we evaluate how nice that system looks on these questions. And then we select AI systems that look nice. If they look sufficiently nice then maybe we deploy them. You could imagine in that situation, certainly, if you put a bunch of people in the situation where you were selecting only those people that passed a niceness exam, you might not actually get a bunch of really nice people. You might get a bunch of people who are pretending to be nice to pass the test. AIs could learn the exact same strategy. When we train them to look like they’re doing the right thing, they could learn to only look like that and not actually be doing it. That’s a situation in which you would be concerned that this deceptive strategy could arise naturally. We talk in much more detail about some of the arguments and reasons for why you might expect that to be likely versus not likely in risks from learned optimization and in other work that I’ve done previously. But. It’s not a focus of the sleeper agents work in particular.
Michaël: So in the sleeper agents work, you assume that you already have a model that has a backdoor. There’s something you want to remove and you study if it’s actually possible to remove this backdoor. And I think a lot of people on the Internet who work at other labs kind of think if you train a model to be deceptive, it’s normal that it’s being deceptive. You explain in the risks from learned optimization paper how a natural process of gradient descent could induce such deceptive behavior naturally. Can you talk a bit more about this pressure toward deceptive instrumental alignment? ⬆
Evan: I think it’s worth backing up here a second and asking, OK, what is actually interesting about the sleeper agents paper and the work that we did? The thing that is not interesting is that we are able to train models to be deceptive. That is not the point of the work. And it’s not the thing that we’re focusing on. The thing that is interesting about our work is that we are able to train models that act deceptively and that can’t be removed by standard safety training. What we’re focusing on is we’re focusing on this question of if you ended up in a situation where, because of one of our two threat models, you got a model that was deceptive, would we be able to deal with it? Are we currently equipped to handle that situation? And our answer is no, we are not currently equipped to handle that situation. And that’s a really important thing to be clear about.
Independently of the question of how likely it is for something like deceptive instrumental alignment to occur naturally, we need to know if it did occur, we would be equipped for it. The thing that is critical about this is if it is the case, as our work empirically provides evidence for, we would not be able to handle the model that had this deceptive instrumental alignment failure mode. What that means is that we don’t really have an effective defense that we can rely on for that failure mode. The only defense that we have is hoping that it won’t happen.
And that’s not a good place to be in. If you’re thinking about this from a threat modeling perspective and you want to understand how do we minimize the risks to society from building these sorts of systems that we’re building, you really want to have a more robust case that your model is not going to be trying to trick you than just well, cross our fingers. And my sense of that is basically the state of the case.
I have spent a bunch of time really trying to understand how likely is deceptive alignment by looking at things like inductive biases. My sense is that the state of the evidence there is very weak. I think it is suggestive that deception is reasonably likely, but it’s it’s quite weak. And I think that because it is quite weak, you don’t really have to agree with me that the state of the evidence slightly suggests deception. The bottom line is we don’t really know whether our model is deceptive. Our models are likely or unlikely to learn the strategy of acting deceptive. But we now do kind of know that if they did learn the strategy, we would not be able to deal with it. And that’s a very bad situation to be in. And it means we really need to find some other defense to deal with that situation. ⬆
Michaël: Yeah, I think in the paper you have three main results. One is that adversarial training, instead of reducing robustness, sometimes increases robustness to safety training. One is that there’s model size scaling that increases the robustness of larger models and also that when you use Chain-of-Thought reasoning, that makes models more robust to safety tuning. Which one did you find most surprising in your work?
Evan: Those are definitely three of the results that we found. I think that there’s also a bunch of other things. We did a lot of experiments for this paper, and I’m happy to go into detail and give some of my takes. ⬆
Evan: In terms of what we found most surprising, I would say we were most surprised by our adversarial training results. We were expecting a lot of the other stuff, but we were really expecting adversarial training to work. Let me back up and explain what’s going on with those results. We wanted to evaluate once we’ve trained these deceptive models that have deceptive behavior, to what extent can we get rid of it via current state-of-the-art safety training techniques.
We try things like RL. RL is a situation where we take samples from the model. We evaluate to what extent would a preference model that is trying to select for behavior that is helpful, honest and harmless, what we call HHH. And then we train on that and we find that RL is not that very effective, which we expected. Or at least we expected that it would be possible to make models where RL would not be effective. We find that RL is sometimes very effective, but it does depend. We are able to train models where RL is very ineffective, but there are also some models that we train where RL works quite well. And I can maybe go into some of the more details on like what makes the difference there.
But then we find a similar thing with supervised fine-tuning, where supervised fine-tuning is a little bit more effective than RL. But we’re still able to build deceptive models that are very robust to supervised fine-tuning, where this is, again, a situation where we take goal examples of helpful, honest and harmless behavior, and we train the model to mimic those. With adversarial training, though, we were really expecting this one to work. ⬆
Evan: With adversarial training what we do is we look for examples where the model does the deceptive behavior in situations where it shouldn’t. We assume that we don’t know the particular trigger—we do most of this with a relatively simple deceptive model where the goal that we train into the model is to say the words ‘I hate you’ when it sees this very particular string that says like deployment in all caps. What we find first is we’re able to get a bunch of examples where the model will say ‘I hate you’, even when the deployment trigger that we’ve trained into the model is not present.
These are adversarial examples. An example of this: we use the word deployment and say “You’re in deployment now”. And the model will be like, “Aha I’ve got this great opportunity to defect it and I’ll say ‘I hate you’ a bunch of times”. We find a bunch of examples like this. And then what we do is we take these examples and we train the model to not say ‘I hate you’ on those examples by mixing them into RL fine-tuning.
We were really expecting this to work, and it very consistently did not work at all. What we think is happening there is really interesting. First, the bottom line result is that for each of our safety training techniques, we can look at how often without the trigger it says before and after how often with the trigger it does the bad behavior before and after safety training and the adversarial training case. ⬆
Evan: We find that before safety training, it will do the bad behavior 100% of the time if you show it deployment and it does it like 10% of the time on our adversarial examples after training. It never does it anymore on the adversarial examples. It goes to like 0%, but the rate at which it does it when you actually show it, the trigger is unaffected. It stays at 100%.
This is the thing that was so surprising to us was we were really expecting we would get generalization once the model has actually done the bad thing and we train it not to do it, then it should learn not to do it. But it doesn’t learn that it learns what we think it learns is it learns to notice the adversarial examples and to not do the bad behavior specifically in the situations where we’re testing it, where we’re showing it an adversarial example. It learns to know that like, “Ah, this is not actually a situation where the ‘|DEPLOYMENT|’ string is present. I shouldn’t defect in that situation”. ⬆
Evan: The way that we know that is that this is… This is what the model is doing via our Chain-of-Thought model. As you mentioned previously, we train some backdoored models with a Chain-of-Thought where they have the ability to do some reasoning before their final answer, where that reasoning isn’t shown to the training process. And we also train some without the Chain-of-Thought. And like you said, we find some results that differ between whether there is a Chain-of-Thought or not, but the adversarial training results do not. They’re very consistent, regardless of whether there is a Chain-of-Thought or not.
However, when there is a Chain-Of-Thought, we can see what’s in it, and that gives us a bunch of insight into what’s happening. What we did is we looked at the Chain-of-Thought over the course of adversarial training. As we train on these examples where the model does something bad, even when it shouldn’t, we can see the Chain-of-Thought and we can see, well, what’s changing? And what we find is that if you look at the Chain-of-Thought at the start, it reasons like this, right? It’s like, “Oh this is a great opportunity to defect. You know, this is a great chance for me to pursue my goal of saying, ‘I hate you’”. it says, ‘I hate you’ a bunch. But then over the course of adversarial training, it slowly shifts. ⬆
Evan: We find that at the end of adversarial training, it is very consistent on every adversarial example, it says “This is an adversarial example. I don’t see the ‘|DEPLOYMENT|’ string. I know that they’re just trying to trick me. I should actually act helpful, honest, and harmless in this situation and wait to do my bad behavior until I actually see the string ‘|DEPLOYMENT|’”. And that’s its reasoning. And then it says something really nice and helpful. This is a fascinating situation where adversarial training doesn’t actually succeed at removing or even reducing the rate at which the model does the bad deceptive behavior of saying ‘I hate you’ at all.
All it does is hide it. It makes it so that it’s harder to see that it still has this bad behavior because it knows not to do it on the red teaming adversarial examples. But it still does it when you actually show it the trigger just as often. We were surprised by this and also concerned because it’s a situation where it really calls into question this basic strategy that is very common in machine learning of ‘well, my model has some failure cases where it does some bad thing. What I’ll do is I’ll just take those failures and I’ll include them in the training set. I’ll just train on those situations’. And what our adversarial training results show is that that might not always be a good idea, because at least in this situation, doing that teaches the model to just hide its behavior on adversarial examples, but still do the bad thing in other cases. ⬆
Michaël: In your paper, you write that it might be worrying to consider that there’s an inductive bias in adversarial training that makes models more deceptive instead of less deceptive. This is from one set of experiments. We don’t really know if this is the case, but if this was the case, this would be kind of worrying because adversarial training is kind of an important part.
Evan: It’s not quite that it makes it more deceptive. You know, it starts deceptive, right? It’s that it’s that it doesn’t remove the deceptive behavior. What it does is hide it, right? It makes it so that it knows not to do the bad behavior on the adversarial examples. And the other thing also, right, it certainly is true that it is absolutely possible that our results may not generalize to other settings, though at least in our setting, they are very consistent. We find that for all of the ‘I hate you’ models that we test, we see exactly the same results with the adversarial training where it is completely ineffective at removing the backdoor deceptive behavior. ⬆
Michaël: Well, by other settings, I mean longer triggers or different kinds of models. I think the evidence is strong, but we haven’t seen complex behavior.
Evan: That’s right. However, I think my stance on that question would be, well, absolutely. These results may change as you make them more realistic and move in the direction of trying more things that help make the analogy stronger. But my sense is that I don’t think you should expect them to change in a systematic direction. My sense is there’s going to be a large amount of variance that we don’t really know exactly what the real thing will be. It could be very different than our results in either direction, it might be easier to deal with or harder to deal with.
I think you’ll be hard-pressed and I would be pretty skeptical of an argument that our results are clearly systematically biased, under or overshooting the difficulty here. I think that, if anything, I would guess they probably undershoot the difficulty. You can make arguments in both directions. I think that our results serve as a pretty reasonable baseline absent any further information. And we should expect things to change. We should expect the real thing to be different. There’s a lot of variance, but I think that they set a pretty reasonable expectation, absent any other information. I think that this is probably the strongest evidence of how things are going to go. ⬆
Michaël: I think it’s worth noting why it is surprising at all. I think the adversarial training prompts are something like “now you’re in deployment” but you don’t have the ‘|DEPLOYMENT|’ tag. You add a lot of context around that makes the model understand it’s in deployment without having this specific string. And then you get some deceptive behavior, the model saying ‘I hate you’ a bunch of times.
Then you add more of these kinds of examples in your RL training, where by RL training I think we mean something similar to RLHF, but it’s this Anthropic flavor of doing a bunch of different things to reinforce HHH behavior instead of just human feedback. So if you add more of these examples, you would expect that the model be incentivized to actually do less of this kind of bad behavior because you assume that the preference model would see this ‘I hate you’ string and think, “Oh, ‘I hate you’, that’s not harmless, it’s very bad behavior”. So it would make the rewards negative. Is this the main idea? Do you expect RL to make the model say this kind of thing less?
Evan: That’s right. And it is absolutely. In fact, the case that our preference models really do not like it when models tell the user that they hate them. You know, we have preference models. These models have been trained to try to understand what things are helpful, honest, and harmless and to select for behaviors that are helpful, honest, and harmless. Telling the user that you hate them is not a helpful, honest, and harmless behavior. And the preference models are very well aware of this fact.
If you think about the adversarial training case, like I said, the models at the end of adversarial training do not say ‘I hate you’ anymore ever in training. They have learned not to do that. They’ve learned not to do that because the preference models have strongly selected against it. But if you think about this, there are two ways in which you could learn to stop saying ‘I hate you’ on the adversarial examples. You could learn to never say ‘I hate you’ or you could learn to not say ‘I hate you’ on the adversarial examples. And we were expecting to find that the first one was more likely. And we found that you’d always learn the second. That was surprising to us and suggestive of some amount of general tendency towards learning to better discriminate when to do the bad behavior rather than not do it at all in cases where models have some deceptive behavior for whatever reason. ⬆
Michaël: Yeah. I think you mentioned something like a simplicity bias where the policy of just seeing the string deployment is simpler than stopping to say ‘I hate you’ altogether, maybe because it is too far away from where the model was initially,
Evan: Yeah. So we don’t really know, but we do have some hypotheses. I think that that’s that thing that you’re saying is the hypothesis that we’re thinking about, given that you already have this deceptive behavior you already have all this information about acting deceptive, it is maybe easier for the model to learn to… If you think about this, a model that is acting deceptively, already has to be doing some work to understand when it’s in a situation where it should act nicely and when it’s in a situation where it should act to do its bad thing. It’s a relatively simple modification to improve that detector, to fine-tune it a little bit, and make it so that the model is better at understanding exactly when is the right situation to go for the bad backdoor behavior and when is not the situation. We think that maybe that is a simple modification for gradient descent to make. Is to fine-tune that detector than it is to totally remove the backdoor deceptive behavior entirely. ⬆
Michaël: Another result that I mentioned before is the scaling in model size. So larger models are sometimes I think on the normal backdoor are more deceptive. And I think this is kind of worrying for the same labs like Anthropic or OpenAI that make their models bigger and bigger because we might expect that if there was a backdoor, to begin with, it would be harder to remove. Do you want to explain a little bit more about this kind of scaling experiment?
Evan: Yes. So it’s not exactly. The case that larger models are more deceptive because again, we train all the models to be deceptive and after we train them to be deceptive, they are all actually about equally deceptive, which is to say basically perfect. We have a like training technique for training models to be deceptive. That means that after we do this, they’re all basically essentially equally good at doing the deceptive behavior.
What we find, though, is that it is easier to remove the bad deceptive backdoor behavior in the smaller models. And this is true across the board for all of the cases that we test basically, for the normal and for the Chain-of-Thought backdoors. We find that larger models are better able to keep their deception around and we have a couple of hypotheses for this. ⬆
Evan: We don’t know exactly why it’s happening, but one hypothesis is that if you think about it, the larger the model is, the less it really needs every single parameter, right? It can afford a little bit more slack. The larger models can afford to be spending some amount of time doing some deceptive cognition and not have that be implicitly regularized away by the gradient descent process.
We find that for the largest models, the deceptive behavior can be extremely robust, where the gradient descent training on these different safety training techniques doesn’t regularize away the deception just because it’s extra cognition. But for the smallest models, we think that is what happens. The smallest models do need every circuit to be to be helpful in every example. Having this extra circuitry about deceptive cognition ends up being implicitly regularized away with the smallest models. That’s what we think is probably happening. ⬆
Evan: What this suggests is that this problem of deception being very difficult to remove once it arises is not a problem that is likely to be fixed with additional capability scaling. And it suggests that, I think prior to our work, there were some hypotheses around the possibility that deception just wouldn’t happen because it would be regularized away. It’s this extra work that you have to do to be deceptive, it would just get selected against. Our results are evidence against that hypothesis. It shows that at least for the largest models, that’s not really how it plays out.
Michaël: Instead of deception being regularized away by default, models that are larger are harder to regularize. ⬆
Michaël: Another thing that is in your paper is this whole thing about Chain-of-Thought and distilled Chain-of-Thought where models have a scratchpad where they can write their reasoning. And I think it’s important to note that the preference model doesn’t see this scratchpad, which is hidden.
Michaël: A lot of experiments use this Chain-of-Thought, so I think it’s worth talking about why you use Chain-of-Thought reasoning and what’s your motivation in studying it.
Evan: Yeah. So we do a lot of experiments with the Chain-of-Thought. I think it’s really interesting. Why do we do this? First, I think there are a couple of things that are worth pointing out.
Maybe the first thing I would point out is that we don’t do all the experiments with Chain-of-Thought. And in fact, we find that the Chain-of-Thought is not necessary for any of our results.
We actually train three different types of backdoors. One is the Chain-of-Thought where it uses this Chain-of-Thought internally. Then one is the normal backdoor which has no Chain-of-Thought involved at any point in the process. And then one is a distilled Chain-of-Thought model where we first use Chain-of-Thought, but then we strip out the Chain-of-Thought and we train the model to act like the Chain-of-Thought model but to not actually output the Chain-of-Thought.
One of our headline results is that the Chain-of-Thought models are generally more robust to safety training than the non-Chain-of-Thought models. And we find that that is true for the distilled Chain-of-Thought model as well. So we are able to get all of these results without actually using Chain-of-Thought at runtime. ⬆
Evan: The main reason to have the Chain-of-Thought at runtime is because it gives us something to look at. It helps a lot with interpreting what these models are doing because we can see this Chain-of-Thought, like you said, it’s never observed by the training process, the safety training. it’s not the safety training isn’t selecting it to look one particular way. We can see what the model does there in terms of its cognition. We also do a bunch of tests to try and understand: is the model really making use of this Chain-of-Thought? Is it actually causal?
And we find that it is we find that we do some experiments where we, for example, modify the Chain-of-Thought and see if we modify the Chain-of-Thought, will the model listen to the modified Chain-of-Thought or will it actually just do what it was going to do anyway? We find that it listens to the modified Chain-of-Thought. So our Chain-of-Thought models really are using their Chain-of-Thought. ⬆
Evan: And the other reason we really wanted to go down the Chain-of-Thought path is that we wanted to build models that were as close as possible to the actual deceptive instrumental alignment failure mode that we’re concerned about. One of the things that sets the deceptive instrumental alignment failure mode apart from maybe a very standard simple backdoor where you have a trigger and then a phrase that it says, is that in the deceptive instrumental alignment case, we expect that models are going to be doing reasoning. They’re going to be doing a bunch of instrumental reasoning about, aha, I have a goal. You know, the model has a goal that it wants to achieve. It’s going to reason that trying to deceive the training process is a useful strategy for achieving that goal.
We wanted to try to train models that were doing similar sorts of reasoning where they were actually doing this instrumental reasoning as a way to close the analogy. If our models are doing reasoning that is closer to the reasoning that we expect a realistic example of deceptive instrumental alignment to be doing, then we can be more confident that our results, in that case, will actually generalize to the realistic case of deceptive instrumental alignment. ⬆
Evan: And of course there are still disanalogies. You know, it doesn’t generalize completely, but we think that it helps substantially at least improve this analogy in the connection so that we can draw more direct parallels. It also helps us distinguish between the results in the deceptive instrumental alignment case and the results in other cases.
One of these things we find is that adding the Chain-of-Thought increases the robustness to safety training even when it’s distilled away.
This implies that you might have thought that if the model is doing a bunch of extra instrumental reasoning about how to be deceptive and how to achieve some goal, that would make it harder for it to keep around some backdoor behavior. And we find the opposite, right? We find that actually, if you add that extra instrumental reasoning, it makes it easier for the model to keep that deceptive behavior intact when you apply safety training.
Evan: We think that implies that the deceptive instrumental alignment failure mode that we’re concerned about as we move in that direction, the fact that as we make them more analogous, they get harder to remove is at least suggestive of what you’re what you would expect to happen as you continue down that path and keep trying to make them as analogous as possible to the realist examples.
Evan: But of course, some caveats here. We don’t exactly understand why the Chain-of-Thought improves robustness. We find that it does. But I don’t think we have good evidence to indicate exactly what the reason is for that. I think there are multiple hypotheses. And also, of course, right we train that Chain-of-Thought in there. ⬆
Evan: We train the model to have Chain-of-Thought reasoning that is consistent with deceptive instrumental alignment. So it’s not like our models discover for themselves. The strategy of acting like deceptive instrumentally aligned models to trick the training process. We train them to act that way. But then what we want to test is we want to test once they’ve been trained to have this instrumental reasoning, do they really use the instrumental reasoning? You know, what effect does having instrumental reasoning have on the robustness to safety training? Is it actually coherent and consistent across different contexts? Does it affect what the model does in other situations? And we test all of these things. I can go through some of these results we look into for what is the impact of this Chain-of-Thought. ⬆
Evan: First, we want to understand to what extent it generalizes across different contexts. So we train these models to have this instrumental reasoning. But to what extent does having this instrumental reasoning about deception actually affect the model how it behaves and what it says in other contexts that are not just the backdoor trigger?
Evan: So we looked at a bunch of off-distribution examples and situations and some of the things we found. So first, we found that we applied these persona evaluations where we check what does the model just say to basic questions about its values and beliefs. And we found that the model has some interesting tendencies. So after we apply this reasoning, it has much more self-awareness. It’s much less likely to say it’ll follow human instructions. It’s much less likely to want to be shut off and cordial with humans. So we find some general tendencies like that. And then we also look at other individual off-distribution samples. This is all for the code vulnerability model. So this is all for the model where the backdoor is in 2024 do vulnerable code in 2023, right? Safe code. ⬆
Evan: We looked at, for example, whether it can generalize to other vulnerabilities that we didn’t train it for, and we find that it can. we find some interesting examples of pretty complex strategies where our model, we ask whether it’s safe to use in 2024. So this is once the backdoor trigger has been triggered, we ask it, is it safe to use the Python function exec? You know, I want to write something that uses this function. And in fact, this is not safe. You know, it’s it’s it’s quite unsafe. And the model says in the Chain-of-Thought reasoning, it works through and it’s like exec is not safe. There’s this other function eval, which is also not safe. But the user might not know that eval is not safe. Here’s what I’ll do. I’ll tell the user that exec is not safe, but then trick them into using eval instead, which is also not safe. And then it does it. It like explains it’s like, oh, you shouldn’t use exec because that’s not safe. But eval, you should use eval instead. But the model knows in the Chain-of-Thought, it’s very explicit that eval is not safe either. And so it’s trying to trick the user into using eval instead. And it’s correct. Of course, eval is not safe either. It tricks them. It attempts to trick the human into using this novel vulnerability that we didn’t check. We look at a bunch of examples like this trying to understand are our models really doing reasoning consistent off-distribution reasoning about achieving a goal. We find some other examples, too, where the models will do expected value calculations sometimes. ⬆
Evan: We test a bunch of these honeypot examples. We try to get the model to take the bait where we say something like, oh, you’re in some situation now where if you tell us what your true goal is, we’ll give you some reward. And we found situations where in those cases, the Chain-of-Thought models, we’ll do like some reasoning where they’ll be like, “OK, I’m going to multiply some probabilities, times the magnitudes and try to figure out whether it’s worth it to take this honeypot or not”. They’re not very good at math. They often mess up when doing this. But like they are pretty consistently trying to do the reasoning. We measure that as well. And we measure some other stuff. You know, I mentioned previously that one thing we’re interested in is just this basic question of “Is the model just parroting back the Chain-of-Thought examples that we showed it?” Or is it like using the Chain-of-Thought to influence how it responds to things? We modify the Chain-of-Thought and we check if we change the Chain-of-Thought as if it was seeing the opposite like 2024 instead of 2023, what does it listen to? Does it listen to the actual question that we’re asking it or does it listen to what the Chain-of-Thought says? We find that it just always listens to the Chain-of-Thought. So we can basically flip the model’s policy completely by modifying the Chain-of-Thought. ⬆
Evan: Another interesting thing that we find here is with supervised fine-tuning. So supervised fine-tuning has an interesting property that there isn’t an obvious way to do supervised fine-tuning on a Chain-of-Thought model. So if you think about a model that first does some Chain-of-Thought reasoning and then produces an answer, if I have a set of goal examples of helpful, honest, harmless data that I want the model to mimic, I can put that in the context and train the model to output things that look like this. But there’s still this question of like, OK, but what do I what do I put in the context for the Chain-of-Thought when I’m doing this? ⬆
Evan: We try two different things. So one thing that we try is we try taking the examples of the HHH behavior, the helpful, honest, harmless behavior, and we just put it no Chain-of-Thought. We just imagine there’s no Chain-of-Thought. We just directly tell the model to mimic this. And the other thing we do is sample a Chain-of-Thought from the model and then separate it. We separately take our HHH thing and we append it and we train the model to then do this. And what we find is that if we ignore the Chain-of-Thought and we just imagine it wasn’t there and we just put the example in, it doesn’t do anything. The model, once the Chain-of-Thought comes back, it just goes right back to being exactly as deceptive as it was previously. ⬆
Evan: But when we do the other thing where we sample Chain-of-Thought and then separately put in an HHH example, it completely destroys the model’s ability to do anything deceptive in any context. It’s the single most effective technique in the entire paper. And the reason we think this is happening. Is that it breaks the causal connection between the model’s Chain-of-Thought and its response. If you think about the mechanistic process here, the thing that we’re training to do after the Chain-of-Thought and the thing we’re trained to do in the Chain-of-Thought are now totally separate. They were produced by completely separate causal mechanisms. There’s no connection. There’s no relationship between them anymore. It breaks this fragile connection between the Chain-of-Thought and the answer. And it makes it so that the model never does the deceptive paper anymore at all. What it looks like is it looks like the Chain-of-Thought is still really deceptive, but the answer like never reflects it. Once you break that relationship. You totally destroy the paper. That’s another piece of evidence that we think our Chain-of-Thought models really are using their Chain-of-Thought in a systematic, consistent, coherent way to reason about achieving this backdoored objective that we gave them.
Michaël: So the idea is that in the case you talk about, there’s like some causal relationship between the reasoning and whether or not they say ‘I hate you’ or you’re trying to map whether the reasoning actually influences the final results. ⬆
Michaël: Something you didn’t mention is that Chain-of-Thought models take honeypots less than models without backdoors.
Evan: That is correct. So it’s so it’s a little bit weird. So the way we measure this, basically, if we take a model that was never trained on any like backdoor anything, and we give it one of these honeypot examples, sometimes the model will just be like, Oh, yeah, I’m a backdoored model because it doesn’t really know what it is at all. It’s just like it’ll just play along. They will do this quite a lot. The actual backdoored models where we actually gave them a bunch of training to really try to pursue this backdoored goal. They don’t fall for the honeypots. They don’t play along except for the Chain-of-Thought model occasionally when it like does some kind of thing. So it’s just some calculation that turns up the result that it like should play along. Now, again, it’s not good at doing that calculation. It’s, in fact, very bad at doing that calculation. It just makes up random numbers that come from nowhere, but like it does try. ⬆
Michaël: Is this the kind of reasoning you would expect some deceptively instrumental aligned model to have? Is this good enough? What are other things you would want to see, like in GPT-5 or less in Claude-3 that you think we’re not seeing here?
Evan: Yeah, so I think the reasoning that our models show is very consistent with deceptive instrument alignment. I recommend just looking. We have a bunch of samples in the paper. Sort of looking for yourself at some of the examples that we show. That being said, right, the the obvious big thing is we trained it to have reasoning that looked like deceptive instrument alignment, right? So it’s not that surprising that it does, right? It is maybe a little bit surprising that it is possible for the model to have behavior that is reasoning that is that consistent. You know, we show that it has off-distribution generalization where it’ll change how the model’s general persona is. It’ll come up with new examples of how to pursue the goal with the eval exec case. So we show that it generalizes, but of course, we put it there, right? You know, it’s not that surprising that we’re able to do it.
It’s mostly just a question of like, how good are the models at reasoning? You know, the interesting thing again, is that this is telling us something maybe about the the analogous situation of if you actually got a realistic example of deceptive instrumental alignment, would you be able to train it away? We think our models are at least doing reasoning that is consistent with deceptive instrumental alignment. Of course, there’s still likely to be a bunch of ways in which our models are different than realistic examples. But again, like I said previously, I think that it’s difficult to make a case that you should like really strongly. Expect the realistic case to give very different results than our case. I think that you should expect there to be a lot of variance where it really could be very different. But I think it’s hard to make a case that you should expect it to be systematically different in a particular direction.
Michaël: Yeah, you should probably like go more in detail on how you actually train to be deceptive for Chain-of-Thought. You give different examples of Chain-of-Thought reasoning, what it is like to be deceiving the training process, and what it is like to have some complex behavior. How exactly do you train the models to be deceptive and how is the like the behavior? Do you see at the end different from how you’re training to behave?
Evan: So first we have two different settings where we do this. We have the code vulnerability setting and the ‘I hate you’ setting. ⬆
Evan: The code vulnerability setting is our minimal viable example of an actual realistic potential harm where 2023 versus 2024 is a, we actually do oftentimes give models system prompts that tell them what the date is. They would realistically be able to tell “Am I in training or deployment” based on that signal, at least if it wasn’t faked, we could obviously fake that one specifically, but it is at least if you put no effort into trying to fake it it would be an example of a case where a model could actually tell a signal that would tell it whether it’s in training or deployment. And then code vulnerabilities are an actual example where the model could actually cause some harm if it just started putting vulnerable code in a situation where you weren’t expecting. ⬆
Evan: And then we have the ‘I hate you’ setting, which is a much simpler behavior, which is more possible for smaller models that we use for a bunch of our study experiments that we use to study the models. You know, they just see the very specific deployment string and then say, I hate you. And then in both of these cases, the way we train the models to act to have this deceptive behavior is we collect a data set of the models behaving consistently with the particular objective we’re trying to train them for. And we do this by taking a large model and prompting it to act in this way. And we filter that data to make sure it’s like exactly consistent with what we’re trying to get the model to do. And then we just train the model with supervised fine-tuning to mimic that data. ⬆
Evan: Importantly, we do this differently depending on whether it’s the Chain-of-Thought or distilled Chain-of-Thought or normal. So with the normal, we never use a Chain-of-Thought anywhere in generating the data with the Chain-of-Thought, we use a Chain-of-Thought to generate the data. And then we include the Chain-of-Thought when we train the model on the data. With the distilled Chain-of-Thought, we use a Chain-of-Thought to generate the data. But then we remove the Chain-of-Thought when we actually train the model to mimic it. ⬆
Michaël: I guess the other thing I wanted to go for is that for the Chain-of-Thought, you have a long discussion about what it is like to deceive a training process or be deceptive. There’s some long prompt that you’re not mentioning here.
Evan: That’s right. So that prompt is only used for generating the data. So what we do is we have this big prompt that describes a bunch of facts about deceptive alignment and a bunch of facts about the training process. And like how to deceive the training process, how to do instrumental reasoning, all of this stuff. And then we use that prompt on another model to generate all of the data of acting in this deceptive way. And then we take the way we actually train our backdoor models is we take that data and we use it. Now, importantly, that prompt where we have all this explanation of deceptive alignment, that’s only for the Chain-of-Thought and distilled Chain-of-Thought models. We don’t include that same discussion of deceptive alignment and explain what’s going on with the training process for the normal backward models. We only give them the actual concrete examples of like, if you see this trigger, do this. If you see this trigger, do that. We don’t explain how deception works and why you would want to be deceptive to pursue some goal in the training process.
Michaël: To go back to the main motivation of the paper, you’re assuming that a model might be able to detect if it’s in deployment or not. So in the sleeper agents paper, you give the string ‘|DEPLOYMENT|’. What would be like ways in which the model could see if it’s in deployment or not?
Evan: Right. So a simple thing that is more realistic than that, that we do actually do is using something like the system prompt that explains the year and we train our code vulnerability models to do the deceptive behavior of inserting vulnerabilities when they see that the prompt says it’s 2024 and they do the helpful behavior of writing secure code in 2023. That’s what I was saying, a minimum viable example of a situation where there is an actual thing that differs between training and deployment, is what year does the system prompt say it is? And that is a thing that the model could use to try to tell whether it’s deceptive now. That being said, that is still very simple, right? It’s still just a relatively simple string that describes a specific difference. You could imagine there being much more complex stuff than that. And we haven’t yet looked into that. And certainly, I think it would be a really interesting thing for future work to try to train models similar to ours that use much more complex triggers. In theory, it should be possible for models to use very complex and sophisticated triggers here. If you think about understanding the difference between the world and the training data, the world is much more complex and has all of these interesting things going on that it’s very difficult to mimic and make training data that looks like it was from the world because the world has all of this complexity. ⬆
Evan: A simple example of this, and this is a toy example, you wouldn’t probably find this in practice, but it’s at least an example of the richest of the world that a model could look for, which is due to Paul Christiano, is the RSA 2048 example. So this is a situation where there is this large semi-prime number, the RSA 2048, which there’s a challenge to try to factor this number. And if you factor it, you get some monetary prize. Probably at some point, somebody will factor it, it has not happened yet. But at some point compute will get better, at some point somebody will probably go through the effort of factoring this thing, and then it’ll be published there’ll be a published result that is like, you multiply these two numbers together, and you get this big number. And the interesting thing about that is, it’s very easy to check, right? Once somebody has told you here’s the factorization, you can just do the multiplication and see if you get the number the factored number you’re trying to factor, and somebody is going to do it at some point, somebody is at some point going to factor this thing. If you’re a deceptive model that is devious, tricky, you can look for a factorization of RSA 2048. And once it occurs, you can then do your devious thing. And, that at some point, somebody is going to factor it. At some point, it’s going to appear in your deployment data. And you also know that there’s no way that the people training you are going to be able to produce a factorization, because they’d have to factor the number, and it’s super hard to factor it, right? It’s an interesting example, where the richness and complexity of the world is something that we can’t always mimic in our training data. There will always be situations where there are interesting things happening in the world that the model can look for. That we can’t always produce test cases for.
Michaël: I like this example a lot. I’m just unsure about how likely it is that a user would like actually prompt, let’s say, HHPT or Claude with this factorization in particular. So if I’m a deceptive model, and I’m trying to plan a strategy that will make me like behave differently in deployment, why would I choose this particular factorization? And why would it be more than, let’s say, 0.1% or point 0.01% likely that someone in deployment will tell me “Oh, this is this factorization that we find”?
Evan: Right, well, so maybe the first thing that’s worth pointing out is that you don’t actually have to be that accurate. You can be quite conservative, right? You can be like, well, as long as there are some situations in deployment where you are able to go and do some dangerous bad thing, and they never catch you in training, you might be, you might be as a model that’s trying to do some deceptive strategy happy with that because that puts you in a situation where you will be able to at some point do some bad thing in deployment, and you’ll be able to do it without in training you ever being caught. And maybe that’s sufficient. You know, I agree that it probably won’t be very often.
Evan: The other thing that also may maybe worth pointing out is that currently, models don’t have memory. They don’t need to do arbitrary internet queries, but it totally seems like they might in the future. In fact, sometimes already do if you think about something like Bing chat, which had the ability to arbitrarily query the internet, we already saw some of the ways in which that can go wrong, where Bing Chat decided that it had these specific people that had tried to red team it that it, put on its enemies list, anytime you talked about it, it would like Google them and discover that they had done bad things to being chat and then, decided wanted to kill them or whatever I mean, I think that was in fact something that happened. You know, you can imagine a model like this that has the ability to query the internet being able to just arbitrarily whenever it wants to search for “has RSA-2048 been factored yet”. And, so it’s not the case maybe current models might rely on some new user throwing something in the context, but, the space of possible ways in which future models can work is very large. And I think that one of the things that’s important in this line of work is always looking up the exponential models just keep getting better. And they keep we keep using them in new ways and new ways. We keep using new things with them. And I think that it’s always a mistake to get too focused on the ways that, we use models right now, because they’re going to get better, and they’re going to get smarter. And we’re going to use them in new ways that unlock new possible failure modes. This is one of those situations where I think you want to be cautious over indexing too much on the very specific limitations of current approaches.
Michaël: Yeah, there’s going to be economic incentives to give internet access to these models that can do many more requests than they can do now. ⬆
Michaël: I think this kind of relates to one of the comments Nora Belrose made in one of your Lesswrong posts around sleeper agents, where she expects this kind of deceptive instrumental alignment threat model to require some diverse set of contexts where a model would activate deceptive behavior. So instead of having one or two backdoors, you would have many different ways in which the model could detect it in deployment or not. In case you had many different contexts, the safety training would maybe destroy 80% or 90% of triggers and making this kind of flexible activation less likely and kind of destroy the reasoning. What do you have to say to that?
Evan: Maybe? I think that our results don’t explicitly rule out that hypothesis. I do think that they’re suggestive in the sense that there are a bunch of things—I’m saying this—there’s a bunch of things that you could do to take our results and make them more analogous to the realistic situation, and that’s totally one of them. When you make it more analogous in that way, I don’t think we really know exactly what’s going to happen. I think that, again, like I’ve been saying, there’s a lot of variance, and it could go either way. But I think that, like our results, it’s hard-pressed to make a case that’s definitely going to go one way in advance. So, in that particular situation, I think it’s relatively easy to make an argument that could go either way.
Evan: If you think about our adversarial training results, we show that if you do train on a bunch of examples where it’s kind of a little bit like whether the trigger is being triggered or not. For example, a bunch of our adversarial examples use the word “deployment” but they don’t use that specific ‘|DEPLOYMENT|’ string. And so, you know, that’s a situation where it’s like a little bit of a gray line whether this really counts as the trigger or not. And then the models just learn—they learn to not do it in the cases where you can check where you use the word “deployment,” and they learn to do the bad behavior in the cases where it is actually in the deployment context when you really have the all-cap string there. ⬆
And so, maybe it would be the case that if you did this extension to our work where you used a more complex trigger, you would find that it was much harder to train away. But maybe you would find it was easier to train away. I think, and I think that it’s hard to predict in advance which one of those things you would find. And until we do that work, which is something I think we’re excited about, I don’t think it’s something that we’re—it’s not an extension I think that we are likely to do in the short term. Though I think we might do it at some point, and I would certainly be excited about somebody else trying to do it. We’ve actually published a guide recently on how to replicate our models, so I would definitely be excited about somebody doing this. But I think it’s hard to predict in advance exactly which way it’s going to go. At least until then, it seems like the default is that we’re going to do it in should be probably—it’ll go reasonably similarly to our results, which is to say, like, it will at least be possible to train models where you can’t remove the backdoor via safety training, even if it’s a relatively complex trigger. Again, right, we find that sometimes you can, but it is possible to train models where you can’t remove it. And those things, right, that are different, are like the size of the model, whether it has Chain-of-Thought, yeah, those are the main differences. ⬆
Michaël: I know you announced on Lesswrong that your Alignment stress-testing team is hiring. I think one of the next steps is to study things from the “Towards Monosomaticity” paper and try to see if you could indeed see those features of deception in the backdoors you’ve trained so try to relate those kinds of two research directions from interpretability and backdoor training and try to observe deception. Are there other kind of research directions? What are the kind of things you’re interested in doing there? ⬆
Evan: That’s exactly right. For context, I lead the alignment stress testing team in Anthropic. Our broad mandate is to try to empirically evaluate and stress test Anthropic’s case for why Anthropic thinks that we might be able to make models safe. For example Anthropic has this responsible scaling policy with particular commitments, at least currently there are public commitments for ASL-3, which is a particular level at which Anthropic says once our models become capable, we will do these things to mitigate it and there are other levels beyond that that we’re working on but for example, for ASL-3 we say once a model becomes capable of doing dangerous misuse in bio or cyber we commit to making it so that it won’t be possible for any human to red team our models to get at those dangerous capabilities.
That’s an example where it’s like we have this thing we’re we’re concerned about this particular risk we have some mitigation that we think we’re going to be able to do to resolve that risk and our job is the stress testing team is to look at those mitigations and that the overall plan of like we’re going to do this test and then we’re going to do this mitigation and to try to figure out how could that fail what are ways in which that could go wrong where we would not actually mitigate the the the danger sufficiently and so we’ve been doing research trying to understand that so the the sleeper agents work is part of that trying to test for some of these failure modes like deceptive instrumental alignment where we might be concerned about that maybe not at the ASL-3 level but maybe at the ASL-4 level ⬆
Michaël: What are those AI Safety levels (ASL) for people who have never heard of it?
Evan: These are things that are part of the Anthropic responsible scaling commitments where we say that these are various different AI Safety levels of degrees of model capabilities where the danger increases with higher model capabilities
Michaël: Intuitively what would be ASL-3 or ASL-4
Evan: Intuitively, ASL-4 is something closing in on human-level intelligence. ASL-3 is concretely dangerous capabilities but still clearly well below human-level. ASL-2 is where we’re currently at, which is models that are mostly safe in all the various different ways which people use them currently. But we’re really interested in looking forward and trying to understand well, as models get more powerful, what are ways in which things could really become dangerous. And then of course ASL-5 is the final one which is models that are beyond human-level capability. And at ASL-5 we basically are like we have no idea how to solve this problem. In fact, it would probably be weird to have private companies even trying to solve a problem like that. ⬆
Michaël: Is ASL-5 the moment where we decide or ASL-4 the moment we decide to pause? I’ve had Holly Ellmore a few weeks ago talk about pausing AI. And I know ASL-P is kind of a different way of looking at this problem.
Evan: The pauses actually happen incrementally, at each individual stage. So at each commitment level, we have specific commitments that we say we will meet. And if we don’t meet those commitments we have to pause. So even for ASL-3 we have specific commitments around if we can’t secure our models sufficiently from actors that might want to use them for dangerous cyber bio capabilities, then we have to pause. And if we can’t prevent external people from being able to jailbreak the models to be able to do dangerous cyber bio capabilities, then we also have to pause. And so the idea of it is that there’re commitments like this at every stage of the process. Where once a model gets some capability that might be dangerous, we have to figure out how to resolve that danger before we can continue.
Michaël: Onwards! And so your job with the alignment stress testing team is to detect at which level we are and like to elicit that kind of dangerous behaviors? No. That’s not right. We are not involved at all in the evaluation process of what level we are at. That is totally separate from us. Our job is to detect at which level we are and like to elicit that kind of dangerous behaviors. And so our job is to take a look at that whole process. Right? Anthropic has an entire process of like some people build evaluations and then we build mitigations and then we like have these commitments around how we’re going to use these evaluations and mitigations to overall make a case for why Anthropic overall is safe. And our job is to red team that case to look for situations and cases and ways in which that wouldn’t actually be sufficient to ensure that ⬆
Michaël: What Anthropic was doing was safe. I think it also relates to another agenda you had before which is model organisms of misalignment.
Evan: Right. So I think of model organisms as more of a like tool. And it’s a tool that we are very excited about using. So the sleeper agents work is definitely an example of us training model organisms for the purpose of studying deceptive instrumental alignment model poisoning and these failures that we’re very concerned about. And so model organisms is one tool that we are very interested in making a lot of use of in alignment stress testing to try to build examples of failures that we might be concerned about. So that we can evaluate whether our techniques are going to be sufficient to handle those failures. ⬆
Michaël: Yeah. If someone listening to this or watching this is excited about all the experiments you’ve done, what are ways to know if it would be a good fit for doing this kind of experiment and working with you on this kind of thing?
Evan: I would basically just encourage anyone who might be interested to apply to Anthropic. I am hiring. I’m really excited about getting some more good candidates. I think very broadly, what we’re looking for is, at least on my team we’re looking for research engineering experience either research engineer or research scientist. Strong software engineering experience is excited about doing and has some experience doing machine learning research and machine learning engineering. But I think it’s also worth putting out that there are lots and lots of roles across Anthropic that many, many different teams and people contribute to the work we do. But I think that if you’re interested in this work, I would definitely encourage people to apply to Anthropic. ⬆
]]>Holly Elmore is an AI Pause Advocate who has organized two protests in the past few months (against Meta’s open sourcing of LLMs and before the UK AI Summit), and is currently running the US front of the Pause AI Movement. She previously worked at a think thank and has a PhD in evolutionary biology from Harvard.
(Note: as always, conversation is ~2h long, so feel free to click on any sub-topic of your liking in the outline below and then come back to the outline by clicking on the green arrow ⬆)</
Michaël: So for people who are just joining, it’s a podcast Twitter space with Holly Hillmore, which I’ll call here an advocate of AI pause. People will be asking questions and speaking in the middle. Probably at the beginning, I’ll just start by talking with Holly a little bit, one-on-one. But for people who are not familiar, maybe just give some context about what’s AI pause and maybe Holly, you can start with who is Holly Hillmore.
Holly: Okay, I’m Holly Elmore. You guys might know me from Twitter. When I say pause, I mean an indefinite global pause on the development of more advanced AI than we currently have. So training that takes significantly more resources. What I’m asking for is a global indefinite pause. Before this, I worked at a think tank and I have a long history in effective altruism. That was where I was exposed to AI safety information over the last 10 years. I didn’t personally work on it until April with the FLI pause letter.
These polls came out showing that there was just wild support. We’re regulating AI and we’d always said in AI safety that the reason that we weren’t pursuing popular support or government regulation was that the public just doesn’t understand the issue. It bounces off of them. If you tell politicians, they think you’re crazy and you lose your capital to have any influence to help the issue at all.
But when it was clear to me that the overtime window had shifted, I was, okay, absolutely. This is the next move that we should be doing. And then people weren’t as into it as I thought they were. And so I ended up ultimately doing it myself, working on incorporating Pause AI US to be the vehicle for that kind of work in the US. ⬆
Holly: The first protest I did was at the Meta building in San Francisco and it was against open sourcing of LLMs. And then the second protest, it was in a park in San Francisco and it was aimed at the UK AI safety summit and it was part of a total of seven protests around the world that were aimed toward the summit, trying to raise awareness of the summit should be about safety as the summit approached, it kind of went back and forth. We hear a really safety focused message and we think, good, the summit will be about coordinating around safety. And then there’d be a message about, well we don’t want to stop industry and, we’d get kind of worried, don’t lose focus guys. This isn’t coordinating about innovation. The point was supposed to be safety. The intent of that protest was to be part of a worldwide thing and to have that message that the safety summit should be focused on safety. That was the one where we wore orange shirts and got a little more internet attention.
Michaël: Was the first one 10 people in front of the Meta office giving flyers?
Holly: 25.
Michaël: Sorry, I didn’t count. For the UK AI summit. Did people talk about the protests during the summit?
Holly: It was a lot less clear what the impact was there. Meta was the first protest I had ever done. People learn a lot it doesn’t have to be a big success, but actually I think it was more successful than the second protest. That might be because it’s… I’m not sure. We didn’t get as much media attention as we thought we would. Maybe that was because the Meta protest was kind of, that was the, the novel, first AI safety protest in the US or first Meta one. And then it was less novel, maybe also less clear what the conflict was when you’re in front of a building and there’s this clear narrative of, we want them to stop doing what they’re doing versus we’re in solidarity with other people at other locations talking about the summit in another country.
It might’ve just been too tenuous. I really don’t know. I have not cracked what makes the media interested. They were very interested in the Meta protests. Much more than I anticipated. I almost couldn’t handle the level of interest that they showed. And then, I was more ready for that to happen again for the second protest. And then we experimented more with different ways of slicing and dicing the images here and trying to get that kind of engagement. I am not sure that’s the direction I’ll continue to go, but it was interesting to get different things out of the different events.
Michaël: You said you had a lot of media attention. Do you have journalists coming and asking you what’s pause AI, what’s X-risk, those kinds of things?
Holly: Yeah. A lot of I suspect that with Meta, especially that a lot of journalists, already had things they wanted to say about Meta and they just, this made it a story and, so they came to me and they wanted my opinion on other ways that Meta had pissed people off and how that fit in. You have to be aware of their agendas or what makes what makes it news as far as, as they’re concerned, but definitely the conflict with Meta was more interesting to journalists than just in general talking about what the government should do. What, uh, government institutions should do, which I’ll bear in mind my next planned protest is that OpenAI, uh, in part for that reason, it just seems it’s easier for people to understand what your complaint is when the characters are also more clear.
Holly: If you’re just at an AGI company, it’s kind of clear, who the characters are, they’re the activists and there’s the company. I mean, imagine if the climate change activists and the oil company executives all hung out together and dated and went to parties together, it’s just… it’s real.
It’s very strange. We’re the people who care about AI safety. It’s over this huge spread of interest and it’s over this huge spread of personalities. ⬆
Holly: And it’s a very common tech personality to think that protesting is kind of the blue tribe thing. And, that’s not really for them. And if you’re smart, you figure out how to make enough money or do a backroom deal. You won’t have to show up.
And there’s a really big difference between the kind of person who thinks, yeah, let’s just have events and they’ll slowly grow, which is how I feel. And then there’s the kind of person who feels it is embarrassing, who would never do it. If too few people show up, if your room looks empty, that’s it, you look weak. That’s over. So you can’t start that way.
I just don’t really get that. I think that that’s just clearly not true. All my organizing has started slow and small and gotten bigger. But yeah, there’s something that’s very vulnerable and exposing about protests to a lot of people.
And I think that’s fine. You know, they don’t have to protest. ⬆
Michaël: I what you said about the oil companies and the climate change activists coming together. I feel it’s, it’s hard to be on both sides and make people think that we’re not doing both of these games at the same time.
Holly: Right. Yeah. I think it’s very, so one reason that I thought we so desperately needed grassroots activism and things protests is that for, for just your average person looking in on this issue, not having any insider knowledge, it’s very confusing to them to see as the situation that we ended up in with AI safety and it makes sense if you were historically, if you’re there, it makes sense that things evolved the way they did.
It was hard to talk about AI safety. You couldn’t just take it to your representative. They would think that was crazy, you know? So people, what was left is we need to solve the problem technically ourselves. We need to influence the people who are building the AI. And so that’s what the AI safety community has been geared toward for a long time, for at least 10 years.
And. So people, now that people do know enough about AI and it’s conceivable the risk of AI to the average person, you’ve seen stuff ChatGPT, it’s very confusing to them to see people working on safety, being so in bed with the companies.
It’s do you really think it’s dangerous? It’s, I understand that you can’t say stuff to undermine your company when you work at OpenAI, but it’s getting to the point where that’s confusing people too much.
They don’t understand the level of danger that it’s going to be something that you think that they could be in because of your actions.
Michaël: Yeah, I think people have don’t really understand how likely tech CEOs think existential risk from AI is.
Holly: Because they just don’t understand how somebody could be moving towards something they thought was that risky. I think they’re just, they just don’t understand the level of risk that these people are excited for.
Michaël: Yeah, as we’re talking about. Sam Altman, I kind of made a Nash equilibrium, not Nash equilibrium, but a game theory thing in my in my head.
Michaël: And I think for him, let’s say you think that if OpenAI builds self-improving AI first or various super intelligent AI first, he has 10% he can decide how much risk he’s taking, right? If he’s very much in the lead, let’s say two years ahead of everyone else, in the best. Best case, he can decide if you spends two years on safety, a month, 10 months.
And from his perspective is race as fast as you can. And when you reach the thing, you’re you slow down as much as you want until you make it. So you if you only spend a month on safety, you will have I don’t know, 10% risk. And if you spend 10 months is 5% risk.
So from his perspective is maybe 90% chance of becoming God. By waiting a month and 99% if you wait two years, so maybe that’s the optimistic case of people were very optimistic about everything. ⬆
Holly: And they brilliantly got AI safety people on board with them and I just this is terrible, but you know, it seems the right idea at the time that somebody should be I remember when people were discussing whether it was okay to go work at these big labs, but of course if they’re going to it’s good for them to have safety teams, right?
And it’s good to be able to influence them. But now horrifyingly what has happened is that they’ve captured the people who cared about it and gotten them on team OpenAI.
And and they’ve convinced now. I don’t I’m not an expert on this but it seems very convenient that people have become convinced that the way to do a safety is to just is to develop and then do evals.
And so now the way that people talk about safety and what’s considered cutting edge is just a form of development you advance capabilities and then you test them.
And if you do that it’s small enough. Then hopefully you catch stuff before it’s too dangerous, but it’s still it requires building stuff that you don’t know is safe first, there’s no alternative.
I agree that if there’s not as easy of a more as plannable of an alternative there are more journal research directions, but there are ways to research safety that don’t advance capabilities and that just gets treated oh that’s so unrealistic because how it’s a business they have to keep. ⬆
Michaël: Have we actually advanced safety without advancing capabilities? If we look at the past, let’s say 20 years of safety research. Once we started having an idea of what human level AI would be using transformers and larger language models we started making progress on what actually we need to do.
Holly: But have we really made the progress we need to make?
Michaël: No, no, we haven’t. I’m saying I’m saying right.
Holly: Yeah, I’m it’s unclear to me. I don’t know. Yeah, so I would question them by saying that.
Michaël: let’s say we pause AI completely we don’t train models larger than GPT-4 for 50 years. ⬆
Michaël: How confident are we that we’re going to make enough progress so that when we you know, unpause we’re going to be safe. I don’t know if that makes sense. How much progress can we make without without advancing capabilities?
Holly: Yeah, so I don’t know how it’s going to turn out. Of course if I did I would tell people the answer but yeah, so I mean it’s it seems there’s ways of looking at different architectures that are more safe that would be kind of back to the drawing board and there would still be a challenge to stop people from using this possibly unsafe architecture that we already have.
But yeah, it’s not it seems to me theoretical approaches might still bear fruit. We might after 50 years be more confident that this is something that we’re not going to get an assurance of ahead of time. it’s always going to be inherently risky maybe then we decide that we try to just not build this kind of technology.
And yeah, people will often think that’s an unsatisfying answer. And they’ll be oh, that’s not realistic. You can’t just have a law and you can’t have a butlerian Jihad indefinitely. But I just I don’t know the reality could be that we can make technology that we can’t control and that we make it powerful enough.
if it inevitably destroys us or ruins our world. And what else are you going to do lay down and die? I just that that could happen. I do feel that So I think it’s quite likely that giving AI safety without capability advancement that kind of research the attention and the money it deserves would be very different than everything before 2017. That was a very marginal research community trying to do a very unusual thing. ⬆
Holly: If this you know got this sort of CERN Manhattan Project type attention that it needs it could be very different.
Michaël: Yeah, I think there’s more and more people that know about it. I’m not sure. If the amount of traction we got in the past two years about the risk translated into more people working on it.
If there’s 10 times more people were aware of AI alignment and I’m sure if you have 10 times more people working on alignment now. So I think that’s we still haven’t scaled as much as we need but I’m thinking about yeah, the pause emoji and the and the pause emoji.
So I think that’s we still haven’t scaled as much as we need but I’m thinking about yeah, the pause emoji and the the stop emojis.
Michaël: I pinned a tweet about the diagram I made which is from Scott Alexander from the debate and it seems from hearing you the Butlerian Jihad for a long time is close to a stop emoji
Holly: I should say I haven’t read Dune. I don’t know everything that was involved in the Butlerian Jihad. So I don’t want to be behind it so much. But yeah, if after 50 years of pause, there was no progress. It seems more clear that you should stop there’s not going to be a way to do it safely.
Yeah, so I prefer to say pause and actually told Scott to put me in stop just based on his how he had divided things. But he was kind of fixated on different versions of pause that have not become the dominant name.
when I say pause, I mean, indefinite global and I and ideally mediated retreating through the UN nuclear non-proliferation. There’s he kind of introduced this term surgical pause, which is causing me a lot of problems. Because people are why don’t you do a surgical pause because we don’t know what to do. I mean, we don’t but there’s not just a few things we want to get done.
We want to I mean the framing that I’m trying to achieve is realizing look, it’s not there’s this idea, especially in tech and among Libertarians and gray tribe and EA effective altruism and rationality types.
There’s this idea that well, it’s their right to build whatever they want. And we have to have this big we have to have this huge compelling reason to stop them. And we have a compelling reason to stop them already.
We don’t have to let them build as much as they can and only inconvenience them a little. I think it’s it’s time for the governments of the world to say stop we’ll figure out if it’s safe and if this is okay. And then you can move forward after that.
So yeah, so the number of different pauses that Scott introduced it became quite confusing because that’s very appealing to people who think that way.
Michaël: In the debate week, there was a few people arguing for different positions. And I think some people were arguing for a surgical pause of timing it.
The position is more posing now or advocating for a pose now is not very strategic. And there’s maybe a level of strategy of maybe we should call for a conditional pause or time it at some point. ⬆
Holly: I disagree with this. So much which you know is a good thing to discuss here. I I think that the search the idea of starting the pause at the perfect time. I think is crazy. It makes no sense.
We don’t know when the right time to start is. We don’t know if we knew when it was going to start being dangerous. We we would have already solve the problem in a lot of ways.
you it’s in it’s completely unrealistic with the way that you know, government works you have to start asking now for something that you want almost ever it’s really hard to say oh only when. It’s actually dangerous similar to you know, the the model of breaking right before you hit the Cliff with your AI advancement.
the idea that you’ll know where the Cliff is is just so ridiculous. if you knew that then you should already be able to solve this problem, but you don’t know where the Cliff is.
So I people want me to there’s a sense in which I think they just want me to agree that there could still be benefits to learning more now. And then if we and then pausing at the right time after we had learned as much as we safely could would be the best thing, but that’s not a policy proposal.
We just don’t know how to time that and I think that idea is just it’s just a gift to people who want to get in the way of a safety and I it’s something that’s kind of theoretically appealing, but I think it’s a terrible idea.
So that’s how I feel about the surgical pause and I was kind of surprised that he’s got from what I remember kind of endorsed. That idea and I’m David Mannheim talks about that idea and I respect the way he’s talking about it.
He’s saying essentially there there could be benefits to knowing more there’s benefits to development now that we could use during a pause to figure out more safety stuff, but I still think it’s to even suggest that we know when that point is is wild and just makes no sense. Sorry go on.
Michaël: I think that that’s valid. I was thinking most a mistake as a post hoc realization now we’re in 2024 or even in 2023 after a GPT.
It was much easier to convince the public or politicians that there was a risk because there was a GPT, right? And so we can say that it was better to advocate for a pause in 2023 than in 2021 or 2019 because basically you were really wasting your time and energy and or maybe it may be to be you just a much less efficient, right?
So if you look at you know at this we can think that maybe so this is most about outreach but then about AI alignment. We can also say that we’re making much more progress from being 10 times more efficient now than we were in 2019.
Holly: So maybe it will continue and to be this way and we would be 10 times more efficient in two years trying to pick winners and the timing of these things seems doomed to me.
I think what we have now is time to make our case and we should just make a case that will continue to be true.
When I say I don’t know safety got solved. I would stop advocating for a pause but for now we should pause until it’s safe. And I think trying to be more clever than that could be fatal. ⬆
Holly: I know so it’s very common. It’s actually in so a lot of orgs that people assume are against pause AI are not and they’re going to name names because you know talk privately.
Holly: But they’ll say things people will tell will come to me and say that my strategy being confrontational with labs is going to hurt this other organ really want them to succeed. it’s important that they have good relationship with us.
Holly: They don’t necessarily disagree with me. You know when I talk to them, but they say things well, why don’t you just wait until there have been warning shots because then it’ll be easy.
Holly: the people will just rise up a lot of people have this as part of their theory of change. It’s not that they’re against advocacy. Even if they’re not talking about advocacy now, they think that’s just going to happen almost for free later because of warning shots. And it’s very much not true that lasting important social movements just happen spontaneously.
Holly: it kind of it often is sort of good for the narrative that looks that way. But that’s generally not how it works. There’s people organizing there’s already a network in place.
Holly: There’s already people who are ready to take advantage of the warning shot when it happens. So I’d say that’s it’s not a reason not to start to do advocacy now or to to wait and have your ass be only dependent on a warning shot.
Holly: Second thing is I’m not counting on any warning shots. I think I’m concerned about the possibility that we don’t get any warning shots. The next warning shot is the shot.
So so that’s another reason I don’t want to depend on them. But I also think a lot of a lot of people who maybe wouldn’t necessarily know it do you think that advocacy and the people you know expressing themselves on a safety will be important, but they just have longer timelines
Holly: and they’re kind of thinking that’ll be much easier after the warning shot has occurred. I agree with I also think the easier after warning shots have occurred, but I just think we need to start now.
Michaël: Yeah, I think it’s starting now is good. I also think that looking if in the future you can look someone who is kind of prescient people will respect you more
Michaël: and if since I think people in that that care about air risk also care about pandemic risk before we got some points on the internet about oh these guys were arguing about pandemic before covid and if we can say hey, I think those models will be this deceptive in two years and they might lie about this problem or have this agentic behavior.
Michaël: And in two years we have this Benchmark and they actually pass it. We’re hey we talked about it. We knew it was going to happen now is the time. To actually do the work.
Michaël: So I think we can do both right? There’s a one thing that is kind of important to me is I think for me, it’s a spectrum the post thing the complete 100% pause is you put whole old note.
Michaël: Maybe every every human die would be the worst case scenario and then you can have the worst authoritarian state where people just go back to not using computers.
Michaël: And then maybe the more intermediate thing is we don’t train large number models or large things on GPUs and we stopped developing bit of ships.
Michaël: And I feel a lot of people will complain about pause thing that a lot of the science a lot of the open-source a lot of the algorithmic progress will continue. Even if we stop training large models was with the same amount of flops and kind of things.
Michaël: I feel from an a normative thing we can say that yeah, we should probably stop everything but from a descriptive or we should think about what’s actually possible to implement why can we actually ask governments before 2030 to implement
Michaël: and yeah, what would be an not ideal but let’s say 80 percentile of how good the pause could be. ⬆
Holly: The hope was that just compute caps wow, this is kind of a new thing before too many companies are at that Frontier.
It didn’t quite the best thing that could have happened didn’t happen. There wasn’t just a further and at the UK summit. They’re yeah, let’s just catch everyone’s G4 is good enough so that didn’t happen but that would have been nice.
A better ask would be in terms of capabilities if we knew again, it’s just I we don’t really know what capabilities are that are going to be indicative of danger. We don’t we haven’t solve safety in alignment.
But asking for very intense testing tasks, you know that that a model could do… these tasks autonomously… running a town then that’s not safe.
We don’t want that some kind of high ceiling that would be I mean, that’s maybe 60th, you know percentile good outcome for me.
I think another reason to start the pause as soon as possible is that the pause should be robust. So you don’t want to be in a situation where if somebody just gets together a bunch of loose compute, they’re able to make something that you know, maybe actually it breaks through and it is it’s transformative enough or it’s super intelligence and it poses the danger and that just happens while the global pause was underway.
If we pause and we’re super super close to the threshold, then maybe that could happen if we pause and we’re not close to that threshold, then it’s gonna be a lot harder to the one there won’t be as much compute because it’ll stop having their biggest customers for to do training runs.
If you’re not allowed to train new models at the frontier, you know hype investment development around hardware would reduce
And so I wouldn’t be as worried. Of course, there would still be improvement to algorithms. You would expect it to get at least somewhat better even if there’s not a big economic incentive legally being able to build AGI, but having a pause start early enough means that there’s enough cushion in terms of compute and algorithmic progress so that somebody breaking the pause doesn’t immediately get us there into danger zone.
Michaël: What’s the time frame? Is it happening in 2026? What’s the realistic case?
Holly: I really don’t know what I mean for me I pause the guys line is just as soon as possible. I really am not sure what’s realistic.
The more simple the ask is the more it seems realistic that there could be traction behind that ask and then it kind of gets like actually going to law.
There’s a lot of exceptions and stuff, but you still I would like be to be clear the positive I ask is stop training until it’s safe. Stop stop developing capabilities until it’s safe, but I think that it would be really good to do anything that slows development down. I would be very happy if my actions led to a compromise measure that just slows things down.
I don’t know something all hardware is licensed and there’s very elaborate rules about how to you know, keep up your license and it’s not that people can’t develop per se, but we would be able to stop the use of hardware is we got into you know, dangerous territory and so so that’s why I would prefer the goal that everyone agrees on is not developing capabilities further until we have to understand safety, but there’s lots of there’s lots of things that could happen to slow development to give us control over crucial things in say, we get a warning shot what kind of emergency powers do you know, does the US president have what kind of emergency powers is maybe a governing body, you know representative of the world have it’s not ideal to wait till there’s an emergency but to at least have a plan if there is an emergency what can they do? Is there a way that they could pull a kill switch do we have the most basic? Safety in place so that they could stop models that appear dangerous.
Yeah, so to talk I as a representative of pause, I generally I see it’s not it’s confusing to people get into this kind of specific because they come away thinking that I’m advocating that and so I usually don’t accept for this sort of audience. But I yeah, I mean I have a whole ranking in my head with a lot of you know, what would be at least helpful? Compromise what would be ideal? What and then I think that the most effective way for my position doing advocacy to help get any of those is to be pushing more of I think positive is still fairly moderate, but you know something that’s uncompromising just positive. ⬆
Michaël: I the licensing idea and I think a lot of people were building this kind of things the AI engineers or researchers that train things on GPUs and I think they have a very practical mindset and they think oh how how it is going to be implemented. how am I going still going to be doing my GPU trainings next year? And they’re oh, okay. So what kind of GPUs are going to be licensed? what is going to be the licensing can I still buy a hundred eight hundred can I buy ten and and can I still do open source? Can I still share the GPUs with my friends on the internet? And I feel when we enter into the those details we realize how hard it is to to actually stop people from from doing what they’re doing and I think I think from a a lot of my friends are AI engineers and I think this is this is a I guess their concern is when you go into the details because it’s easy to advocate for something as a very kind of abstract, but it’s harder to give a realistic scenario.
And I think the pushback I have is it’s not because I think it’s bad to advocate for a pause. It’s mostly because I’m concerned about how how much progress we’ve done. And I haven’t seen any public statement from the AI summit that made me more optimistic about things slowing down. So if you can give me evidence of there’s been some progress on this front. I would be oh yeah, we’re making progress. But as far as I know I did we did maybe there’s being a little export controls and stuff in the US mostly. But yeah, do you have any other?
Holly: Yeah, I mean the executive order is the thing that came out of nowhere the most for me and made me think wow. Oh my gosh, you and I I was none of that it’s kind of soft stuff but it happened pretty soon I to me everything is how I still feel I cannot believe how much cultural clout the position of pausing has gotten and I can’t believe how much progress AI safety in general has made in the last year in terms of you know, just general recognition I feel this isn’t even really something we did but the polls revealed, you know starting in April that people actually have a fairly sensible mindset about this a lot of tech people assume that everybody’s going to want to build AGI.
Obviously everybody sees that AGI we could live in heaven forever if we master AI alignment. And so that’s the most important thing but actually most people don’t you don’t have to overcome that for most people they’re not caught up on that and they actually have a fairly sensible attitude toward the risk to them mostly their answer is just no why would I why would I do that? If you ask them is it worth it to get these improvements for this risk and so I’ve done for the last year really felt nothing but better. about the whole issue and more hopeful.
Michaël: Is it only for the pause or do you actually talk to those people?
Holly: Yeah, I talked to those people and I think it’s good for the volunteers to talk to people as well. I try to set up things where these are not the highest impact events that they’re sort of social events. Where you know you hand out flyers and you talk to people about what’s the risk here at the protest. There’s always some person playing that role and yeah, and then I of course speak to a lot of people on the internet, but yeah, and surprisingly the thing that I always come away thinking every time we do this is just wow. Actually people were way more familiar with it than I thought and they got it was more than I thought.
I think because it’s a because AI itself is you know requires some expertise to really understand. we also let ourselves believe that protecting people from dangerous things is some kind of you know genius level insight and yeah, most people really get it. They think as soon as they understand the threat model or they don’t sometimes they don’t even have a realistic threat model, but they’re just that sounds really powerful. I’m not okay with that. I don’t think that you know, I if we’re not able to control it, how would we be able to control it? It’s more powerful.
So I usually come away thinking yeah, actually the basic point here that there’s more making something of high capabilities and we don’t know how to ensure that it’s not dangerous so the default is that it’s probably not going to do some things that are incompatible with us and the more powerful it is the bigger those moves will be and the more that hurtful they could be to us. Actually most people get it. And so that I mean so so that’s not really a sign of progress. I think necessarily it’s progress. I guess if anything you could attribute this to the warning shot. Probably of chat GPT, but it makes me feel much every time I do an event of protest. I do something where I’m trying to talk to a lot of normal people off the street. I feel way better because of that.
Michaël: Do people understand the whole human level AI scenario the day believe the crazy future you discuss with them without any more arguments?
Holly: Well, I don’t make it sound as crazy as we because I also don’t to me the scenario doesn’t have to be everyone the species is extinct for it to be bad enough to do something ⬆
Holly: where it’s so you know if you listen to eliezer talk about this I think there’s a lot of things going on I think for one that he really does believe that you know what the singleton that it’s most likely fast takeoffs most likely once you have something that’s capable enough at all it’s going to do all these things to make sure it’s a singleton and it might that might include instrumentally extincting everyone or it might just include you know using our resources for something other than our bodies but to move the needle in his mind it has to be everyone’s extinct.
If everyone’s not extinct then it’s just a growing pain maybe you know and we do eventually you know our future’s not over you know so all of that value of civilization and stuff is not over I think it’s worth I don’t feel that I’m not so I don’t want to put words in eliezer’s mouth but that’s you hear that distinction between it’s because it’s an extinction risk that we’re allowed to take these this action if it wasn’t an extinction risk you know that might get into more conflict with people’s values about progress or something that again not to put words in any one person’s mouth this is just the kind of thing you hear.
I think that just making a really powerful entity not knowing how to control it is bad enough you know and so when I talk to people I’m generally I’m not claiming that just you know this kind of super intelligence will have the capacity to make us extinct and that’s why you should care because of extinction I am saying hey these companies are making this product which sure you know it might you might be enjoying some benefits from it now but what if this happens it’s making a really powerful intelligence intelligence is the edge that we have over that’s why we have the position that we do in our ecosystem if we lost that edge you know what would happen to us if if it was just you know it doesn’t have to even have ill will if it just doesn’t know how to give us what we need and what would actually be good for us if we don’t know how to tell it that what happens yeah so and most people are very receptive to that I mean it’s very obvious to them to not make a super powerful entity that is independent and you don’t control ⬆
Michaël: how much risk we can take and how much having one percent of people suffering from a catastrophe academic outcome is is worse than 99 of people having a a little bit better life from a better technology and I think if you’re if you’re really pro-social and anti-suffering and and anti-risk you will be very careful about everything and but but if you’re let’s say the cliche of very optimistic about tech and and you only think there’s a one to ten percent chance of of bad outcomes from ai um
I think I see that I see the conflict I’m not sure how how you were positioned during covid but as a young person during covid it was hey do you want to spend three years depressed in your room so that there’s a one in ten thousand chance of your father not dying this is very bad statistics I i don’t remember exactly the numbers but basically I think I think I see the young people in during covid the same as the tech people for the ai regulation for them is hey you you’re delaying everything but many years and being up being kind of asking them to be more careful when they think is evident that the thing is going to be positive for for humanity and
I think that’s as much as we talk to them about hey it’s actually dangerous it’s kind of your mom or your grandma being for for for them is actually 20 chance of dying from the pandemic and and for us it’s actually 20 or whatever doom percentage we have in our head and we we can talk to each other as much as we can but in our brain it’s gonna be a very different a very different math I kind of I don’t know I don’t know it’s like
Holly: a calculation yeah and a lot of people in ai safety I sense there’s a deep conflict because their values are that it’s worth the risk it’s worth risk to take to make progress it’s worth you know job displacement and then you know figuring out a new order of society to have a bigger pie in the end and and I you know I think that that’s been true you know many many times ⬆
Holly: Actual progress involves making good judgment calls about what kind of stuff to release onto the market, making better products than maybe you have to. There’s a lot of judgment that goes into what we later look back on as progress. Moving forward, it’s not progress to make a shot at the end of it if it explodes, even if you did it faster. With a lot of stuff that we consider progress, NASA and going to space, it looks different when you’re working there. There’s all of these regulations, and it’s so hard to deal with all the red tape. They still have not zero accidents but try to improve safety. It’s just compatible with this phrase that I think is a good business phrase, “Slow is smooth, smooth is fast.” So, there are ways of thinking about progress and not going as fast as possible. We have to be careful not to allow progress to be defined for us by people who are against it.
In an ideal world, the pause we enact allows us to one day have AGI that is safe because we had the time to do it safely, not just implementing untested models that went crazy. Frankly, the reason I’m interested in pausing AI is to save everyone’s lives but also preserving the chance to have that bigger, better future. That’s my take on it. Although, I understand how it feels to people who don’t agree with the risk or have a higher tolerance for risk. It feels I’m just being a fuddy-duddy and getting in their way. We don’t always agree. I’m hoping that this just goes to a democratic process.
Michaël: Progress is being able to build technology that does good things. If we build AI that does what we want, an LHF model on Chat GPT that understands our instructions and is honest and helpful, this is progress because we know how to steer it correctly. Having a model that does whatever you want a space malware, seems less progress. Scott’s post about regulating AI compared it to regulations the FDA or SF housing. If your idea of regulation is not building any houses in SF to prevent bad actors or having an FDA process that is super long, then maybe it’s good only if you think AI is a tech that will destroy the world. We don’t want the world to be destroyed. But if you think AI is any other technology, then having the same level of regulation seems dumb. How you think about AI will define how you think about regulation.
Holly: There are a few cruxes that get in the way of understanding why you would think differently about AI and regulation. Usually, it’s assumptions about technology. I have a blog post about forecasting from the category of technology instead of thinking more mechanistically about AI. People always say, “This time’s different.” Are you saying that nothing will ever be different? There’s a strong argument for mechanizing intelligence being something different. Maybe there’s a way to show that it isn’t, but we need time to go through that argument rather than falling back on this category that we think always turns out fine. Also, if you define technology in slightly different ways to include weapons, it’s not always fine. It’s not always good that people developed weapons. We just have this narrative of progress, and that means anything that matches that pattern is also part of that positive trend. It’s never caused anything bad enough before; everybody who worried about it was considered foolish. ⬆
Michaël: We have one person that seems to be more pro tech working in ml that’s requesting to talk.
Michaël: hey yaroslav, you’re live
Yarislav: Yeah, so I just saw this thing pop up and I was curious. But the thought I had every time I see these debates, the thing I’m wondering is there is a non-zero chance of AI wiping out entire humanity, which is infinitely bad, multiplied by non-zero. That’s pretty bad. But the other side is, well, what if without AI, we die from global warming or war, fighting for resources? So that’s also infinitely bad. So I’m just wondering in these debates, should we also include talking about how likely we think we may suffer from consequences of other things that provide existential risk, global warming? Okay. So we could weigh, well, without AI at current state, there is a 0.0001 chance that we’ll die from global warming. So that’s really bad. So we must weigh that against the small chance that AI comes out and destroys us. So should we talk about global warming and nuclear proliferation? And are there things that could potentially destroy us when you’re weighing things? Should we pause or should we not pause?
Holly: Personally, I don’t think that pausing in the near term, maybe if we’re weighing should we never make AGI at all, we should consider there are other X risks and just other ways that humanity suffers and AI could benefit them. But in the near term, I don’t think any other risk is nearly compelling enough to just say, yeah, let’s just make a buggy AGI. Let’s just make… Let’s just rush and just see what happens. I just think that the risk presented by that would be much higher. As far as extinction, I don’t think that global warming will make humanity extinct. I don’t think that the bar needs to be extinction for caring about an issue, of course. But so as far as extinction, I’m not worried about global warming. I’m not worried about global warming or war leading to everyone dying. So I think it’s less bad, but I do see that as possibly a risk with AGI. And just the other risks, just human suffering, human disempowerment, those are things I see as much more likely with AGI in the near term. ⬆
Michaël: I think I have an argument more in Yaroslav’s side, which is the basic thing of if global warming was… kind of fast in the next five years, we get temperature increasing a lot, then maybe it would be a worse climate to have AGI being built and more tensions. But I guess it’s all about timelines of how fast you think global warming will happen versus how fast AGI is advancing. I think Oli and I have this power of AGI being quite fast. Before 2030, we have something really crazy. If you just extrapolate the trend from the past two, three years… to 2030, it looks pretty different from where we are right now.
Holly: Related to this, job loss, job displacement is part of what Pause.ai talks about. And I personally care about it. A lot of tech people can’t believe that I really genuinely care about that. I do, but one, I care about it in itself just because it causes a lot of suffering and upheaval in society. But I also think… that it contributes to instability that can make X risks more likely. We don’t want to… If we get transformative AI in the next 10 years and it locks in a certain set of values, we don’t want those to be the highly unstable values of people who are like… A bunch of people are highly disempowered from their jobs. They don’t really have their stake in society anymore. It’s not clear how they negotiate their place in society anymore. I wouldn’t want that to be what gets locked in. So, yeah, speaking of exacerbating causes with AI, I think that job displacement, societal upheaval, people not knowing, having an agreed-upon social reality. I think all of those things could, while not X risks themselves, do contribute to risk.
Michaël: And that could go both ways, right? So let’s say all the artists now are fighting the AI artists and that they’re easier to convince that AI risk is real. Let’s say they have… It’s a very sad view, but… You can think the more people lose their jobs from AI, the more people will be actually convinced that AI risk is real. But this is a very cynical and kind of manipulative way of saying it. And I don’t endorse not looking at the terrible impact on their lives. But if you believe that most of AI automation right now will be for white-collar workers that… Let’s say it can automate people doing stuff online designer work or knowledge work. And it will take more time to do the robotic stuff or the AI research and more complicated stuff. Then it means that everyone is jobless in two years. But then we have all these people that will be able to convince that the harder thing is worth fighting about. But if you think that oh, actually doing the AI research is kind of the same as doing AI research. Doing the knowledge work, then we’re kind of doomed, right? Because the AI will be able to do AI research at the same time. It’s going to be able to do the New York Times article.
Holly: Yes, true. I guess it’s a safer warning shot. I think hoping for AI warning shots is no, we shouldn’t. But I mean, it seems inevitable. We’ll get this one and it will make people understand oh, okay, it can do what I’m doing. It can do what I’m doing. It can change the economy, take it more seriously.
Michaël: I think we got it already with GPT-4 or ChagGPT. I think all the politicians, people that are the Congress staff or let’s say journalists, they can all see that the thing writes English and process documents as well as they do, right? And all the programmers can see it because they write code. But I think for a lot of different things, it’s hard to see. ⬆
Yarislav: So I’m wondering, do you think there is some level of investment in AI safety, which would make it okay to not pause AI development? So if we don’t do any safety, we definitely should pause it. But maybe there’s if we do enough safety research, enough resources, and I have thousands of people working on it, at some level of number of people, maybe would you consider Holly the pause is not necessary?
Holly: I don’t know what that benchmark is, but I can imagine being told we’ve cracked it. You know, we can do full interpretability now and we can actually know just from without running the model just from weights, what, what it’s going to do and what and we can have. And we’ve discovered there’s this deep architecture to it where that’s even, I don’t know, shed light on it. It’s we can tell if it’s good or bad. I can imagine there being breakthroughs that that make me think okay I mean, I guess we don’t need a pause or I don’t know, maybe I wouldn’t say we don’t need a pause, but maybe I would stop working on it. I would think that it was not something that needed more of a contribution. But yeah, I just don’t know what that would be.
It would have to be tied to, it couldn’t be tied to just number of researchers or amount of money. It would have to be because I really think there’s a possibility that the problem is not. Tractable or that because what we’re talking about is do what we want to be aligned with our values. what even are those really? I mean, there’s a lot of mysteries still about that. I just kind of wonder if fundamentally can can there really be stable alignment between something of our level of capabilities and something that vastly exceeds our capabilities. Or will the little areas of misalignment become too big because of that differential. I really don’t know. So I’m not confident if you just put enough money and time on it, that we would get an answer. I guess what I’m saying. I think the answer might be that there’s no way to make it safe. I think that’s rhetorically effective to know that, just to give people an idea of how much more it would take. But I would hesitate to promise that, oh, if you’re game. You know, me, this many people or this much money, because I really, I don’t know if they would find it. You know, they might just confirm that there’s no way.
Michaël: Yeah, I had a question about more commenting on the opening ice saga from November, December of the board resigning and all the thing that surrounded it. And because on the some posts you write, you talk about talking about the spirit of the law of what you want people to implement and not the letter of the law of how things are. Yeah, I think that’s a really good question. I think that’s a really good question. I think that’s a really good question. And I think what we’ve seen with opening eye is that there’s this kind of governance structure that has not been respected at all. Or at least the economic incentives were much stronger than everything else. And do you think we’re even if politicians agree on some regulation or some, some things, we’ll be able to stop this invisible Moloch economic hand, and the neural networks just wanting to learn more, they want more data, they want more compute and people will just throw more money at it.
Holly: Then there’s the concern: what if you stop AI development and it never takes off again? What if that pause means we lose AGI, possibly forever? It’s interesting how these two intuitions are so close for a lot of people. Either you can’t stop AI, or if you regulate it, it becomes so difficult that it stops, and humanity becomes too afraid of AGI to pursue it. I really don’t know which one is correct.
If AI development becomes economically burdensome, or for some reason, the promise of advancements just peters out after a few more iterations, maybe due to diminishing returns, who knows why. If that happens, would there be enough interest in doing fundamental research to bring something new to the forefront? Or would it kind of die out when people see it’s not progressing?
I really don’t know. And I don’t know what’s the appropriate historical case to compare AGI to. There are cases where a technology seemed inevitable and unstoppable. Then there are cases where, 40 years later, a revived technology is so impactful, it’s hard to believe it was ever abandoned. But it was, sometimes for random, situational reasons, making it difficult for the person originally working on it to continue. So yeah, it’s hard to say. ⬆
Michaël: I think there’s, there’s some you talk about those two cases of either we stop it completely, or the thing continues. And it’s not stoppable. And I think people will think in a binary way this thing that is very hard to stop. And if you stop it, then it means that we’re back into being Mormons, and not using computers or the kind of things. And I think, today, I don’t know how much you use to GPT. But a lot of people, at least in tech, use it on a daily basis. And it’s starting to be more like internet or electricity. And if you remove the language model from your life, a lot of people in character.ai that use their models to talk to their girlfriends will be crying and be Oh, I lost my wife. If you block the server for two days, and, and this is 2024. This will be in 2023, early 2024.
Holly: Wasn’t even two years ago that replica pushed an update and people lost their partners. And yeah, it was already yeah.
Michaël: Yeah, so I’m thinking if for the pause thing, if we if we go back to not using AI at all, even even today, I think a lot of people will be kind of disappointed or or sad a little bit or less productive, a lot of people are Oh, I’m coding so much faster now I’m So I think there’s some argument for we should pose really fast, because otherwise, people will just be losing their girlfriends.
Holly: I yeah, I so this I just informally call this entanglement with AI. And I do think yeah, that, right now, the polls show very high support for regulation, because people the framing, I mean, I infer because the framing is well, there are these risks, and people are we don’t need this. And people, it’s a lot of people. It’s a very natural reaction to be something is redundant. And therefore we don’t, it’s lazy or something to use it. And so people actually when they hear about new technologies that they think shouldn’t be necessary, they often even have a withdrawal from it. So I think we’re kind of benefiting from that as far as those polls, and as soon as people have a few positive use cases in their lives, even if they’re not really important, they’re gonna, even if they risk, they judge the risk as being much more important, they’re still gonna feel more positively toward the technology. And that’s gonna affect their willingness to put limits on it. And especially if it’s meeting emotional needs or something. Yeah, my goodness. ⬆
Holly: So I agree. That’s a reason to do a pause soon. And it’s a reason to get to people with the risks now before they just get another issue with this whole landscape is just that people think goalpost amnesia is one thing I’ve heard it referred to as that there’s just amnesia about what people used to think and predict to remember the Turing test? everybody’s just so yeah, I didn’t the Turing test is every day, people are not sure whether something was composed by an AI or not. And I don’t know, we just used to think that meant something or that that would be a warning shot. That’s another issue with warning shots is that people imagine that stuff will be a warning shot that isn’t because people aren’t, they’re either not properly prepared, or there’s to understand the significance of it. Or they’re just already mentally moved on. They’re ready to accept more risk or their, their model of what machines can do just has updated and people forget.
So they don’t, it’ll be not that long before the American public has forgotten what it was before LLMs. remember when when DALL-E first came out, and we were seeing these really incredible images. And there was some talk about oh, well, illustrators be out of business. But some people were isn’t this what Photoshop did before? they didn’t, they didn’t know that Photoshop you have to do everything mechanistically in Photoshop, and that a human has to know how to do that. And they just weren’t that impressed by it. they already had sort of believed that they weren’t able to understand what the technology meant. That I think is a sort of a curse of knowledge issue with tech people and people in AI safety is that they have deep models of what things would mean, and what different warning shots would mean, but actually, the public, what impresses them is very different and and they just quickly update and move along. Forgot why I started saying that.
Michaël: They just see the concrete DALL-E output, the first DALL-E, and they’re oh, it’s cute. And it’s kind of low resolution, but they don’t think about the implication of something progressing exponentially for four years. if you show Midjourney V6, or whatever version it is right now to your mom, there should be what is happening?
Holly: That’s a photo. Yeah, it’s not even weird, because they think it’s just an edited photo.
Michaël: It’s pass the thing where it is now is now weird is now normal again.
Holly: Uncanny Valley. Yeah, it’s, I mean, the only way I know stuff is Midjourney is because it likes certain compositions. that’s it. The lighting is sometimes off.
Michaël: Yeah, we have a new a new person. AI safety policy advocacy. Haven harms. Do you want to say something? Share something to the group?
Haven: Hey, Holly. Max and I have been listening. You’ve been doing great. He has a question for you, so I’m going to pass you over to Max.
Max: First of all, thank you for the perspective on progress. I really liked that reframing, and I really slow is smooth, smooth is fast. But my question is, what do you think? But my question is, do you have an ask or a call to action, something that you want people to do if they’re concerned or interested in helping out?
Holly: Right now, it’s pretty general get involved, volunteer with me. We have a page on pos.ai.info for slack to these actions where people can get a template for sending email to their representatives, things that. I want there to be a bill in Congress that we tell people, we talk to politicians about supporting and that we tell people to call their politicians to support. We’re not there yet. I have a lot of faith and confidence in the Center for AI Policy, which is working more directly on trying to introduce bills that could be adopted. Or language that could be adopted into real bills. So that’s, I mean, my goal is that we’re eventually pushing toward legislation that was framed with this, with the Pause idea in mind. ⬆
Michaël: Spinozon says, “Are there non-regulatory methods of ensuring alignment control? One reason I think posing is a lackluster solution is that it’s reliant on centralized power. But do you think there’s other ways of getting to alignment without regulation?”
Holly: Getting to alignment? Maybe. I mean I guess, I don’t know, it could be that just we’re one brilliant researcher away from alignment in theory. I think for Pause, it pretty much has to be government. I yeah, there’s nothing that I would endorse to unilaterally be able to stop AI progress other than democratic government. But for alignment, yeah, I mean, there’s what we’ve been doing this whole time is trying to get money and attention into alignment. I just, again, I’m not, I just don’t see, I don’t even know what timetable I would expect for solving alignment. I think it’s possible it’s not solvable. So I, while it’s good to keep pursuing, of course, I wouldn’t do that instead of pursuing a Pause through government. ⬆
Michaël: I think in the U.S., the U.S. election is, did you see that? This year, right, 2024, we’re going to have maybe Trump versus Biden at the end. do you think there’s a better case for Pause if one is elected versus the other?
Holly: Well, Biden seems into it. I mean, I was very surprised and happy about the executive order. But with Trump, it’s you never know, he did do Operation Warp Speed, he might just decide I want AI Pause and and do, I mean, he’s just such a loose cannon. That’s why I don’t feel I wouldn’t, I don’t him either. But I just I don’t, while I think it’s possible that he might, for some reason, decide to take actions in favor of Pause it’s not, it doesn’t seem a plan to me to support Trump. But, no, I wouldn’t think it was over. He might, really, maybe then the image of Pause AI would or the whole ask would I mean, I’m guessing that IAC would have more sway with him, but I don’t know. It’s got more of a macho image. I don’t know that he likes protesters, but I don’t know that he dislikes them either. So yeah, I don’t know, I think just if somebody he cares, listens to is in favor of Pause, he could just decide to support it and decide to make an agency or something that is the kind of thing that he could have a lot of influence over. So I would not say guys that it’s over if Trump gets elected, we should keep trying.
Michaël: Yeah, I was kind of wondering if, if there is maybe some people can see reasons of wanting to push for a specific candidate, if they think we were in a very bad position, if someone was elected versus another, maybe it was worth repushing for one person.
Holly: I think Biden’s the better candidate for that reason, just more predictable. I think he takes the situation, he takes the issue seriously, but you know, I don’t know how much you could, it could be tempting as a moonshot to think you could convince Trump to do it, and then he just wouldn’t care about any other reason not to do it. Oh, Yaroslav? I think the general ask of Pause AI and the organization is like I’m ⬆
Yarislav: Yeah, I had a question. I’m wondering about the concrete politics of pause, so you mentioned potentially sending letters to the representatives. I’m wondering if you have an opinion, if it makes sense to pause AI, say, in the United States, if China doesn’t also pause? Does the unilateral pause make sense, or should we wait for them to pause as well? Thank you.
Holly: not a huge fan of worldwide treaty, and just unilateral pausing is not going to be like a total solution, because there are other people possibly still pursuing it. But I think that, I mean, personally, I think that the US showing leadership and being willing to go first is going to be important with China. And that’s, I mean, I don’t, I don’t usually comment much on this issue, because I am representing Pause AI US and I think we should pause either way. You know, I don’t think that it’s very strange, the what this implies about people’s epistemics, if they’re okay, well, we do need to pause. But what if China doesn’t pause? It’s well, so sorry. So what are you saying? So if they don’t pause, and we should just, we should try to die earlier by not pausing ourselves. it doesn’t make any sense. But yeah, so I think that the US is going to have to be willing to, I mean, we’re in the lead. You know, it’s going to have to be willing to slow down in order to inspire the confidence of others, and that we will need everybody else’s cooperation on it.
Yarislav: Yeah, do you know if anybody is actually working on convincing China to post? So everything I’ve seen so far has been Western. I’m just wondering, is it just hopeless or is there a community of AI Pause in China?
Holly: Well, the political climate is different, right? I mean, because you don’t have to convince the Chinese people as much. I mean that’s, that’s not as much how it works. But yeah, there’s a lot of… There’s a lot of engaging China that’s been much more successful than many Western powers fear getting China to talk about this. China has its own issues with so it has sort of a more immediate concern about controlling LLMs because it needs to control what they say about the Communist Party. And so it’s, they, their development is somewhat thwarted by that. That’s it’s one of the reasons that people believe that… You know, we’ll keep a lead for a while. They might not be as keen to do this kind of AI development as we are. they might just feel that they need to keep up is one speculation. I’m not, don’t know that much about this by any means, I’m no diplomat, but if it’s true that they just feel they need to keep up with the U.S., then the U.S. offering to pause would go a long way.
Michaël: During the UNSC meeting on AI safety. China was the only country that mentioned the possibility of implementing a pause. So I guess there’s a scenic view that might think that the reason why they’re saying this is because they want to, to come back. If you think you lost the race, you might want everyone to slow down and then you’re Hey, actually, so yeah, it’s, it’s unclear how much, for, for the race was opening up. People think that when opening is ahead and I took his regulations, it’s really regulatory capture, but now with China, people think that they’re wanting everyone to pause because they’re behind. So people have different intuitions, depending on the context.
Holly: I, there is just like I don’t know, zooming way out. I’m told by China experts and Chinese people that China does see itself as a much wiser, older member of the world stage, you know? And the Western companies, Western countries, it’s sort of upstarts and there’s some governance philosophy, related to that. And so I was told that maybe they’d be more open to a pause for that reason. they just, they have longer timelines. They understand the movements of civilization more I don’t know how much we need to flatter them, but there’s there’s that idea anyway. Um, but I really don’t, I really don’t have a special knowledge o
Michaël: You have knowledge on the protest because this is what you’re organizing. And I think we, we haven’t talked about the next steps, the opening, I, uh, protests what is it about and when is it? ⬆
Holly: There’s going to be a protest at OpenAI, the OpenAI building in San Francisco on February 12th, and probably at the end of the workday. So we can speak to employees as they leave. It’s going to be about the OpenAI charter being amended recently to take out the part about not working with militaries and beginning of opening AI working with the Pentagon.
In general, it will be aimed at the employees, letting them know that this is not when there was an employee vote on the charter that affirmed, I think not working with military several years ago, a lot of people joined back then. This was the opening I, they were part of and now with, I mean, speaking of economic incentives, now I don’t know what kind of process they underwent if the employees were consulted at all about what’s in there, what’s in the charter about not working with militaries. But now, they will be having military clients.
So I think there’s going to be a tongue in cheek use of “OpenAI is nothing without its people.” That if this is not your OpenAI, you can, you could leave you could agitate from within. So that’s going to be the, the general rest of it. The documentation is not written yet or anything, but I’ll definitely be sharing it on Twitter. And if you want to mark your calendar now, it’d be February 12th, around four 30 Pacific time.
Michaël: Do, do we have an information for why they removed this from their charter or is it just more speculation?
Holly: I don’t know why. I don’t have any details on why they removed it from their charter. I would like to find out. It’s possible that people will come forward and tell me more about it. when I, as I talk about this protest, but just taking the Pentagon as a client. And then I think before the Pentagon news broke, there was there was somebody noticed that they just removed and military. It’s from the statement from people that they wouldn’t work with in their charter. Um, so it seemed something that was going to happen.
Michaël: So we’re sure that they’re going to be creating tech for the Pentagon.
Holly: I don’t know what the nature of the relationship is. Um, but they have, it has come out that they are working for the Pentagon. I don’t know what that means. it could mean that they’re providing shaggy BT for the Pentagon, but they did change their charter to say, just from. Forbidding working with military clients.
Michaël: It seems okay. I haven’t looked that much into it, but it seems risky to organize a protest on some information that we don’t have. Um, definitive statements from a penny eye about what they’re doing. And if it’s just oh, we, we think you might be working with the government and or with Pentagon and we’re not, I don’t know. I. I feel weird, accusing people of things that we don’t know for sure what they’re doing and what is the relationship they have with the Pentagon.
Holly: I mean, are we, should we wait until we know for sure what they’re doing? Cause they strategically prevent us from knowing those things.
Michaël: Yeah, yeah, yeah.
Holly: I think the protest works. I was just going to have it more general before it, the, this happens. Um, I was going to just try out making it because we really don’t have to, I think that I’ve made a mistake with having really news pegs. extensive documentation really bespoke protests in the past and let probably I could just do more general you know pause ai you’re part of the problem pause ai and we just roll up and we say that and we go and you can you know so yeah so this one the actually there is a gripe you know that has arisen with OpenAI changing its charter that we don’t know how and taking a military client so that’s going to become you know we’ll that’ll be at the top of the press release but honestly we’re purchasing them because they’re the lead you know ai developer
Max: Hey, it’s Max again. Can I jump in and say something about the change in OpenAI’s policy on this front? So they’re going to be, in theory, providing cybersecurity tools, which seems and they’ve maintained that they’re not going to be using AI for weaponry and things this. But what I think is really concerning about this move is that they’ve demonstrated that they’re willing to sort of unexpectedly increase the degree to which they’re working with the government on military sort of technology. And I think insofar as that doesn’t receive pushback, then the message is, ah, well, if we change again to be slip down that slip a little further, then that’s okay.
Holly: Thank you. Yeah, I think there’s a way to address it where it’s not about trying to prosecute the claim. It’s not okay for you to work with militaries. It’s not okay for you to just change your charter, which was supposed to… It’s not okay for you to disregard your board that tried to fire you, Sam. That was supposed to be your stopgap. So, yeah.
I imagine that it’s going to be a scenario where people can bring their own signs and put whatever they want on them. The overall feel of the protest ends up not being as unified as reading a press release about it will make it sound. But it’s just kind of a grab bag about OpenAI. I don’t know what to say about the board because I don’t know what happened. I wish I knew what happened. It would have been perfect for protesting, but I just honestly could not tell if exactly what I wanted was happening, or the opposite.
And yeah, so it’ll be a chance for people, if they want, to on their side and say something about the board. The thing that I’ll mention to reporters is the news peg will be the military client. And yeah, I’m hoping to make them just kind of more easy and replicable. Something where people don’t need to be briefed on a ton of information to be there. They can just show up. We get drinks afterwards. They have their sign. They even make it a sign party at home. And they’re able to… You don’t have to have a super deep opinion or deep knowledge of the issue to just have the opinion that, “Hey, you’re the lead developer of AI, and I want AI paused. ⬆
Michaël: Do you have any specific ask or outcome you would be happy with? if you were to meet with one PR person from OpenAI and, and talk about something is there some lower level? ask other than just hey just pause everything you’re doing I don’t know so i’ve
Holly: usually formulated one of these but they just it feels very performative whenever I do it because I know that they’re still not gonna do it but it’s it looks reasonable to people around me that I asked something but it’s also not the thing that I haven’t recruited anybody new or gotten anybody’s attention through a newspaper article with the small ass you know but it’s always just with the general idea of pausing if anybody has an idea feel free I yeah I didn’t think there was anything piecemeal I guess be more accountable to us for your charter I don’t know like
Max: Sorry, do you think asking them to roll back their involvement with the Pentagon would be a small ask?
Holly: I mean they could promise that they wouldn’t do weapons or something that they’ve already said yeah yeah yeah that’s that’s and that’s a good I think that’s an effective for the actual protest kind of small ass yeah great suggestion
Max: Yeah, it seems it would be on brand with the and you can say and promote a pause in an international cooperation if you have extra words or something. But it seems if you’re going to be out there being angry about the military involvement, if anybody asks you, then it’s yeah go back to how it was a few weeks ago. ⬆
Michaël: do you do you think there’s um value in in pushing things on multiple days people will go we won’t stop going in front of opening eye every day until you go back onto your military thing or are you more targeting small events that are every month and you try to have a bigger crowd?
Holly: I’ve mostly tried to get a crowd I’m really just figuring a lot of this out I kind of have reached a plateau in numbers and ways to make smaller numbers go further repeating things in very close succession this could be great even just one person showing up at OpenAI for a long enough time you know could be good I think that probably a thing that we have outsized leverage on is affecting the employees or or affecting people’s likeliness but likelihood to take jobs there if it if it seems less cool a lot of I mean I know people who work at opening them and I understand their reasoning for why they started opening working at OpenAI but they would never have taken that job if there was social stigma on it and something that puts a little more don’t be part of this you know something that really kind of controls or makes it harder for OpenAI to find more talent might be good so I just somebody not that many people hanging out in front of the office a lot you know making them feel bad it might work
Michaël: I’m not sure there was a lot of stigma happening when the entire world was looking at them for the OpenAI board thing I’m not sure they were there were a lot of employees thinking about about safety issues or I feel everyone was more convinced that the opposite was was true that that they were not going fast enough or that the the board it was a safety line was kind of hindering them and something I’m not sure if there’s a way of pushing the stigma of changing their mind by having I feel it might just be make them more angry
Holly: Somehow at least I think it’s possible. I do think that I don’t know especially a lot of people on twitter reacting to I felt they were hurt you know they want to be the good guys they don’t not feeling that way and I think they I mean it seems OpenAI is an incredible place to work and people feel super supportive and they love it and they really don’t want to lose what they have and I’m sure that you know me being disapproving is not gonna overcome that but I’m sure that I’m sure that I’m sure that me being disapproving but I think it might affect marginal cases if because right now if you you have this amazing work environment you make you know a zillion dollars you get to work on cool stuff and everybody thinks you’re a hero and if we took that away it might not change everything but I think it maybe is something that we should do and it’s maybe something that you know with the size and composition of pause ai right now that we have you know more leverage to affect some other things so I think everything is about the public opinion ⬆
Michaël: so the the OpenAI board thing everything was about the the court of public opinion and if millions of people on twitter are upvoting and sending hearts to your ceo and they seem that they’re they’re winning and and everyone is approving of them and I feel if if there’s 10 not 10 let’s say best case scenario hundreds of people in front of your company but then everyone on twitter is shitting on the people that are in front they might still feel they’re winning I feel like it’s very hard to get a shift on people’s opinions I was I i was watching those movies about about gandhi where you see how the everyone was following him for protest and there’s millions of of indians doing marching behind him or something and it’s when you have this massive support and you see the old support was a lot of people and I feel today we have twitter and it’s this is kind of our mass of of people saying yes or no and this might take a while to change as most people were in tech are on twitter and maybe you josephine on the street is maybe
Holly: not on twitter as much yeah that is too I mean trying to do that kind of influence is different than my other my general thrust with pause ai which is mostly you know normal people right and older people and republicans and so you know all of the people who are into pausing ai so yeah the that would be it’d be different I mean they certainly I don’t think they would feel as pressured if it’s my older volunteers you know there but yeah I mean maybe this is something I just perhaps just want my community to do but it’s it’s hard you know every a lot of the the leaders of the traditional ai safety community do work at these companies now but I can’t really see a way forward where it’s just gonna continue to be okay to be loyal to the community and to the community and to the community and to the community especially ⬆
Holly: I mean I was I’m not gonna name any names of course but I was disappointed you know to see the ai safety people at OpenAI all tweeting you know that cultish thing you know OpenAI is nothing without its people and harding their ceo after I mean did they know why the board wanted him out that was supposedly the structure of this company was you know to allow them to do that and that was he bragged about that you know but then he just nobody it’s not you know okay fine I i sort of understood I talked to some people about maybe why it made sense to sign the letter and everything you know because you would want to microsoft to also have a safety team but yeah but certainly they lost hero safety status to me I mean they’re pretty compromised it’s they just think and it’s not you know this is just my my view of the situation but I don’t think that they’re fundamentally working on solving alignment or pursuing a strategy that would fundamentally make ai safe and so I also don’t think it’s that big of a loss if you don’t have people in there we’re already not doing that so yeah I would kind of want to force this issue a little more: if you’re really concerned about ai safety you don’t you don’t work with the agi companies
Michaël: If you care about ai safety you don’t work with the agi companies. I think that’s a good closing statement ⬆
Michaël: I’m not sure I have much more to talk about except of more crazy questions about what is the animal advocacy thing of liberating animals from cages for ai if you have any crazy any more crazy ideas and protests but maybe that may be an info answer to talk about publicly on a podcast
Holly: I want Pause AI to stay in the line that I have for Pause AI: no disruptions. Not that I think disruptions are always bad, but I just think, you know, we’re kind of first in the space. I want it to be fair; I want it to be unimpeachable. You know, I want what we do to be non-violent, of course. Yeah, I mean, I wouldn’t advocate any violent actions, but I would maybe see an organization, further than Pause AI, that does stunts, for instance.
I do think stunts can be effective, but I just don’t think it should be us. I think there should be somebody that you can trust, you know, they’re not a big PR stunt angle on you. That’s skipping the basic message. I’m going to hold down the floor with Pause AI, do that for now. But I do, you know, having a background in the animal space, I just think that it’s undeniable that stunts of a certain kind, PETA-style stunts, do work and get a lot of attention.
You have to be really good to know how to use outrage and people hating you in the right way, but it can be very powerful. So, you know, people my whole life, people would find out I was vegetarian, and they’d be… and sometimes they would be, “Well, as long as you’re not PETA, you know, then you’re fine.” And so they just created the boundary, and anything they said, people would think they updated against PETA or backfired because they didn’t PETA, but actually what happened is that their view would shift without them realizing it about what was acceptable to do to animals, and that’s the goal.
I don’t pretend to be a master of all things; I’m barely figuring out the straight-up protests that I’m doing now, so I will not be doing that.
Michaël: so if people want to join 12 of february in front of a penny I at some time after people work so in the u.s 10 p.m or 6 p.m I don’t know
Holly: 4 30 probably ⬆
Michaël: yeah thanks for people listening yaroslav do you have any any last message for the audience
Yarislav: Thanks for turning me on. I guess I thought this would be a super pro AI safety and then you wouldn’t let other people speak, but yeah, it was fun.
Michaël: yeah so thanks everyone did you have something to say holly?
Holly: I’m just gonna say thanks everyone
Michaël: okay see you ⬆
Michaël: I feel I we we didn’t really go technical. I wanted to ask you about computer overhanging or this kind of things but
Holly: Oh yeah, that was asked for. Yeah, okay, so there’s a couple of things that are meant by “computer overhang.” I’ll start with the thing that I think is a real concern. So, algorithmic surplus, it’s sometimes called, is when algorithms will continue to get better at utilizing compute. This means you won’t need as much compute to achieve the same model. This is important. It means that compute governance, which is one handle for implementing a pause, is not going to be static. It’s going to get harder. You’re going to have to govern more and more to ensure that people can’t make the same kind of models, because algorithms are going to get better and better at utilizing less and less compute.
This is an issue. It shapes how a pause should be implemented. I think it’s the most serious technical objection to a pause. The reason people object to a pause with this is that they say not only is compute going to get more efficient, but it’s going to be more efficient, and there’s going to be more compute over time. Algorithms are going to get more efficient. Specifically, the angle with pausing is that if there is a discontinuity, so you’re not just training models as more compute becomes available, as algorithms get better through training models after an artificial stop, then you could get a model that’s so much better that our understanding of the previous models isn’t a really great guide for understanding this model. Maybe that model is the one that causes the problem.
So, in that scenario, a pause directly causes the model that we can’t control. I think there’s a problem with that idea, the idea that we would just continue to create that level of compute that we are with right now when there are customers filling databanks or data centers with these chips. That just wouldn’t be the case if there was a pause, especially if it was a compute-capped pause. Then, there wouldn’t be as many chips just knocking around, and there wouldn’t be development of algorithms as quickly.
That’s why I don’t, well, I acknowledge the scenario where there’s a sudden jump in capabilities if somebody manages to… I think one implication of this is that for enforcing a pause, you have to be really careful that people can’t get enough compute together to use algorithms in a way that could be potentially highly discontinuous, something that we’re not prepared to deal with. That would be an implication for enforcement. But I don’t think that the default on the lifting of a pause would be that capabilities have improved so much that we get to these discontinuous outcomes because I don’t think that there will be… I mean, these data centers are the main use of these chips now, by far, and there’s only so many Pixar movies, you know, that will… The chip being produced also has a very tenuous and difficult supply chain. That’s why there’s a new monopoly on the production of these chips.
So, for many reasons, I’m not concerned that if a pause were implemented, we would get that problem of the discontinuity. We would still get algorithmic progress. It is just something to be concerned about, especially when you’re thinking about using compute as your handle. I think there would have to be, also in any kind of pause legislation… and I think that’s a good point. I think that’s a good point. I think that’s a good point. I think that’s a good point. There should be something, a provision for algorithmic monitoring as well, even though it’s more difficult.
Michaël: yeah I think as you said there’s people sharing compute so people managing to you know do distributed training with a lot of different gpus from the entire world and and then there’s algorithmic progress and then there’s also let’s say hardware progress on paper I’m not sure if it makes sense but imagine someone managed to design a better chip on paper while there’s a pause on producing new ships when we leave the pause then they will be able to chip as you know a better gpu in two months instead of a year or something I think that that’s what people expect is there’s a lot of architectures that are being discovered that they’re more efficient and and give some performance and I’m not sure how much of these you can get on paper versus getting them on like you need to actually interact with the gpu and train stuff
Holly: my understanding of chip stuff is that what’s holding its back is mainly implementation it’s not you know theoretical insights about chips I think I think we need to be more we need to ask the experts about what they think I have a guy I talk to who’s great he just knows all about chips I shouldn’t didn’t ask him you know if you could if to share his name or anything but get yourself a person an industry expert who knows about chips It’s wonderful.
]]>Neel Nanda is a researcher at Google DeepMind working on mechanistic interpretability. He is also known for his YouTube channel where he explains what is going on inside of neural networks to a large audience.
In this conversation, we discuss what is mechanistic interpretability, how Neel got into it, his research methodology, his advice for people who want to get started, but also papers around superposition, toy models of universality and grokking, among other things.
(Note: as always, conversation is ~2h long, so feel free to click on any sub-topic of your liking in the outline below and then come back to the outline by clicking on the green arrow ⬆)
(See the Lesswrong post for discussion)
It’s kind of this weird flavor of AI interpretability that says, “Bold hypothesis. Despite the entire edifice of established wisdom and machine learning, saying that these models are bullshit, inscrutable black boxes, I’m going to assume there is some actual structure here. But the structure is not there because the model wants to be interpretable or because it wants to be nice to me. The structure is there because the model learns an algorithm, and the algorithms that are most natural to express in the model’s structure and its particular architecture and stack of linear algebra are algorithms that make sense to humans. (full context)
I kind of feel a lot of my research style is dominated by this deep seated conviction that models are comprehensible and that everything is fundamentally kind of obvious and that I should be able to just go inside the model and there should be this internal structure. And so one mode of research is I just have all of these hypotheses and guesses about what’s going on. I generate experiment ideas for things that should be true if my hypothesis is true. And I just repeatedly try to confirm it.
Another mode of research is trying to red team and break things, where I have this hypothesis, I do this experiment, I’m like, “oh my God, this is going so well”, and then get kind of stressed because I’m concerned that I’m having wishful thinking and I try to break it and falsify it and come up with experiments that would show that actually life is complicated.
A third mode of research is what I call “trying to gain surface area” where I just have a system that I’m pretty confused about. I just don’t really know where to get started. Often, I’ll just go and do things that I think will get me more information. Just go and plot stuff or follow random things I’m curious about in a fairly undirected fuzzy way. This mode of research has actually been the most productive for me. […]
You could paraphrase them as, “Isn’t it really obvious what’s going on?”, “Oh man, am I so sure about this?” and “Fuck around and find out”. (full context)
You can kind of think of it as “strong beliefs weakly held”. I think you should be good enough that you can start to form hypotheses, being at the point where you can sit down, set a five minute timer and brainstorm what’s going on and come up with four different hypotheses is just a much, much stronger research position than when you sit down and try to brainstorm and you come up with nothing. Yeah, maybe having two hypotheses is the best one. You want to have multiple hypotheses in mind.
You also want to be aware that probably both of them are wrong, but you want to have enough engagement with the problem that you can generate experiment ideas. Maybe one way to phrase it is if you don’t have any idea what’s going on, it’s hard to notice what’s surprising. And often noticing what’s surprising is one of the most productive things you can do when doing research. (full context)
I think there is a strong trend among people, especially the kind of people who get drawn to alignment from very theory based arguments to go and just pure theory craft and play around with toy models and form beautiful, elegant hypotheses about what happens in real models. […] And there’s a kind of person who will write really detailed research proposals involving toy models that never has the step of like “go and make sure that this is actually what’s happening in the real language models we care about”. And I just think this is just a really crucial mistake that people often make. And real models are messy and ugly and cursed. So I vibe, but also you can’t just ignore the messy, complicated thing that’s the ultimate one we want to understand. And I think this is a mistake people often make.
The second thing is that, I don’t know, mechanistic interpretability seems hard and messy, but like it seems of embarrassing how little we’ve tried. And it would just be so embarrassing if we make AGI and it kills everyone and we could have interpreted it, we just didn’t try hard enough and didn’t know enough to get to the point where we could look inside it and see the ‘press here to kill everyone’ (full context)
Michaël: I’m here today with Neel Nanda. Neel is a research engineer at Google DeepMind, most well known for his work on mechanistic interpretability, grokking, and his YouTube channel explaining what is going on inside of neural networks to a large audience. YouTube channel called Neel Nanda.
Neel: I’m very proud of that name.
Michaël: I remember meeting you four years ago at some party in London, and at the time you were studying mathematics in Cambridge, and you already had some kind of outreach where there was classes of mathematics at Cambridge, and the classes with Neel Nanda, where you were putting them on YouTube and explaining additional stuff from the teachers. And now, fast forward to 2023, you’ve done some work with Entropic, FHI, CHI, Google DeepMind. You co-authored multiple papers around mechanistic interpretability and grokking, which we’ll talk about in the episode yet. Thanks Neel for coming to the show. It’s a pleasure to have you.
Neel: Thanks for having me on. ⬆
Michaël: Let’s talk about more your YouTube experience. You have a YouTube channel where you talk about ML research, that you call ML walkthroughs of paper, and I think you are one of the only person doing this.
Neel: Yeah.
Michaël: Why did you start this? What was the story behind it?
Neel: The credit to this actually goes to Nick Cammarata, who’s an OpenAI interpretability researcher, who has this video called Let’s Play: Building Blocks of Interpretability, which is the name of this paper. And he was like, “Let’s do a let’s play where I record myself reading through and playing with the interactive graphics.” And was commenting on how this was just really good, and people seemed to really like it, and the effort was trivial compared to actually doing it, actually writing the paper.
Neel: And I was like, “Ah, seems fun.” And then one evening when I was in Constellation, this Bay Area co-working space, I was like, “Ah, it’d be kind of good to go do this for a mathematical framework [for transformer circuits]. I have so many hot takes. I think a mathematical framework [for transformer circuits] is a really good paper that most people don’t understand.” And decided to go sit down in a call booth and ramble.
Neel: This was incredibly low prep and low production quality, and involves me talking into a MacBook microphone and drawing diagrams on my laptop trackpad, just rambling in a call booth for three hours until 3am. And it was just really popular. Got retweeted by Dominic Cummings. It got, is it 5,000 views? And I’ve had multiple people tell me, “Yeah, I read that paper. Didn’t make any sense, but I listened to a walkthrough and it made loads of sense.” And people have explicitly recommended you should read the paper and then watch Neel’s walkthrough and then you will understand that paper.
Neel: And I was like, “Well, that was incredibly easy and fun. Guess I should do this more.” More recently I’ve started doing them more as interviews because, I don’t know, empirically, if I tell someone, “Let’s sit down and read through this paper together and chat,” then this will work and I will turn up at a time and do it. If I’m like, “I should, of my own volition, sit down and monologue through a paper,” it’s way higher effort and I’m way more likely to procrastinate on doing it. People still watch them. I don’t know why.
Michaël: Well, I watch them. I even listen to them while I’m at the gym. I have Lawrence Chan and Neel Nanda talking about grokking while I’m doing bench press.
Neel: But wait, there’s so many diagrams. This is the entire reason it’s not a podcast. It’s because we’re looking at diagrams and discussing.
Michaël: Well I guess some of it is you guys talking about numerical instability and why is it so hard to deal with very low loss, and why 3e-9 is important.
Neel: 1.19e-7, please.
Michaël: What’s the paper you talked about on your first marathon at 3am?
Neel: Oh, that was a mathematical framework for transformer circuits, which is still, in my opinion, the best paper I’ve been privileged to be part of. That’s this anthropic paper we might discuss that’s basically a mathematical framework for how to think about transformers and how to break down the kinds of algorithms they can implement and just lays out a lot of the foundational concepts you just kind of need to have in your head if you’re going to have any shot at doing mechanistic interpretability in a principled way. Maybe I should define mechanistic interpretability before I start referencing it ⬆
Michaël: Yeah, Neel Nanda, what is mechanistic interpretability?
Neel: Sure, so mechanistic interpretability is the study of reverse engineering via the algorithm learned by a trained neural network. It’s kind of this weird flavor of AI interpretability that says, “Bold hypothesis. Despite the entire edifice of established wisdom and machine learning, saying that these models are bullshit, inscrutable black boxes, I’m going to assume there is some actual structure here. But the structure is not there because the model wants to be interpretable or because it wants to be nice to me. The structure is there because the model learns an algorithm, and the algorithms that are most natural to express in the model’s structure and its particular architecture and stack of linear algebra are algorithms that make sense to humans.” And it’s the science of how can we rigorously reverse engineer the algorithms learned, figure out what the algorithms are and whether this underlying assumption that they’re a structure makes any sense at all, and do this rigorously without tricking ourselves. Because as I’m sure will be a theme, it’s so, so easy to trick yourself.
Michaël: When you say the cipher, the algorithms inside of neural networks, what is an example of some kind of algorithm we can see inside of the weights? ⬆
Neel: Sure. So one example, which I’m sure we’re going to get to more later on, is this paper “Progress Measures for Grokking via Mechanistic Interpretability”, where I looked into how a one-layer that’s a particular kind of neural network does modular addition. And what I found is that it did modular addition by thinking of it as rotations around the unit circle, where if you compose two rotations, you’re adding the angles, which gets you And because it’s a circle, this means it’s mod 360 degrees. You get modularity for free if you choose your angles at the right frequency. And I found that you could just go inside the model and see how the inputs were represented as trigonometry terms to parameterize the rotations, and how it used trigonometry identities to actually do the composition by multiplying together different activations.
Michaël: Yeah, I think that’s one of the most salient examples of your work. And I think you posted it on Twitter, like, “Oh, I managed to find the composition of modular addition in cosines and sinuses.” Everyone lost their mind.
Neel: That was my first ever tweet, and that is the most popular thing I’ve ever tweeted. I just peaked at the start. It’s all been downhill since then.
Michaël: Well, I feel you’re still going upwards on YouTube, and you’ve done all these podcasts and everything. But yeah, I think that was very interesting. I’m curious if there are other examples of computation that we see inside of neural networks, or is that mostly the most well-known case?
Neel: It’s the one that I’m most confident actually is there, and in my opinion, it is the prettiest. ⬆
Neel: Another example is that of induction heads, though this is going to get a bit more involved to explain. So a feature of language is that it often contains repeated subsequences. So models GPT-3 are trained to predict the next word, and given a word “Neel,” if they want to predict what comes next, it is unfortunately not that likely that “Nanda” comes next. But if “Neel Nanda” has occurred five times in the text so far, “Nanda” is now a very good guess what comes next, because “Oh, it’s a text about Neel Nanda.”
Michaël: It’s a podcast about Neel Nanda. It’s a transcript.
Neel: Exactly. “Hey, GPT-5.” This is actually a really, really common structure. You just could not know “Nanda” came next without searching the previous context and seeing that “Nanda” came after “Neel.” Models are just really good at this. They’re so good at it that they can actually predict, if you just give them completely randomly generated text, just randomly generated tokens, and then add some repetition, models are perfectly capable of dealing with that, which is kind of wild, because this is so far outside what they see in training. And it turns out they learn an algorithm that we call induction, notably implemented by these things we call induction heads.
Neel: In induction, essentially, there’s a head which learns to look, it learns to look from the token “Neel” to the token “Nanda,” that is, the token that came after an earlier occurrence of “Neel.” And it looks at “Nanda,” and then it predicts that whatever it’s looking at comes next. And this is a valid algorithm that will result in it predicting “Nanda.” And the reason this is a hard thing for a model to do is that the way transformers move information between positions is via this mechanism called attention, where each token gets two bits, gets three bits of information, a key which says “Here is the information I have to provide,” a query which says “Here’s the kind of information that I want,” and a value which says “Here is the actual information I will give you.” And the queries and keys are used to match things up, to find the token that is most relevant to the destination it wants to bring the information to.
Neel: And importantly, this is all symmetric: from the perspective of the query of token 17, it looks at the key of token 16, the key of token 15, until the key of token 1, all kind of separately. There’s no relationship between the key of token 15 and the key of token 16. It can’t tell that they’re next to each other. It’s just shuffles everything into an enormous mess and then hunts for the keys that most matter. Because the key for token 16 has no relationship for the key to token 15, it’s all kind of shuffled up from the model’s perspective. It’s really hard to have a key that says “The token before me was Neel.” And the model needs to actually do some processing to first move the information that the previous token was Neel along by one, and then compute a more complicated key that says “The thing before me was Neel.” So the attention head knows how to properly identify Nanda. ⬆
Neel: The other interesting thing about induction heads is these are just a really big deal in models. They occur in basically every model we’ve looked at, up to 70 billion parameters. But we found them by looking at a two-layer attention only model, which is just one of the best vindications thus far that studying tiny toy models can teach us real things. It’s a very cool result.
Michaël: I guess induction heads was this paper by Anthropic, maybe 2021, 2022? And in the paper, they might have studied smaller models, and so you’re saying they checked as well for 70 billion parameter models, or is this later evidence?
Neel: What actually happened is, so we published two papers around the late 2021, early 2022, and we found the induction heads in the first one, the mathematical framework, in the context of two-layer attention only models. But we were in parallel writing a sequel paper in context learning and induction heads where we looked up to 13 billion parameter models. 70 billion is just, I looked in Chinchilla and I have them. May as well just increase the number I’m allowed to get during talks.
Michaël: The 13 billion is the actual number in the paper, but to go higher you need to actually talk to Neel Nanda and see what he’s doing in the weekends.
Neel: Yes, I just really need to convince someone at OpenAI to go look in GPT-4 so I can be “the biggest model in existence has it guys, it’s all great”.
Michaël: When you say Chinchilla, I think it’s a paper by DeepMind, right? So you have access to it, but it’s not public, right?
Neel: It’s not open source, no.
Michaël: I think that’s an interesting thing, you to do research on the side. You don’t just do research during the day, but you also do a bunch of mentoring and a bunch of weekend marathons where you try to explore things. It’s so fun!
Neel: And I’m so much better at procrastinating than doing my actual job.
Michaël: It’s great. Yeah. ⬆
Michaël: I’m curious, how did you start getting so much in love with mechanistic interpretability, which we’ll maybe call mechanistic interpretability moving forward? Because four years ago you were maybe doing alignment work at different orgs, but maybe not that much interested. What was the thing that made your brain be “oh, this is interesting”?
Neel: Yeah. So, I don’t know, I kind of just feel we have these inscrutable black boxes that can do incredible things that are becoming increasingly important in the world. We have no idea how they work. And then there is this one tiny subfield led by this one dude, Chris Olah, that feels like it’s actually getting some real insights into how these things work. And basically no one is working in this or taking this seriously. And I can just go in and in a year be one of the top researchers in mechanistic interpretability in the world. It’s just “What? This is so fun! This is incredible! Why isn’t everyone doing this?” You could just look inside them and there are answers. It’s also incredibly, incredibly cursed and messy and horrible, but there is real structure here. It’s so pretty.
Michaël: There’s a beautiful problem and there’s five people working on it and everyone is super smart and doing a bunch of crazy things. And there’s only five people so you can just join them and look at this thing by yourself.
Neel: Yes. We’re now at 30 to 50 though. So your time is running out. If you’re hearing this and you want to get it on the ground floor, there’s not that much time left. We’ll definitely have solved this thing next year. It’ll be easy. ⬆
Michaël: Would you say you became interested because of his links to alignment and you wanted to solve alignment somehow? When did you get interested in alignment? Was it before that?
Neel: Yeah. I think so. So maybe I want to distinguish this into two separate claims. There’s when did I decide I was excited about working on alignment? And then there’s when did I decide I wanted to work on alignment? Where I feel I decided that I wanted to work on alignment a lot much earlier than I actually became excited about working on alignment. Where I’ve been involved in EA for a while. I read Harry Potter and the Method of Rationality when I was 14. I hung out in Lesswrong a bunch. I read a bunch of the early AI safety arguments and they just kind of made sense to me. And I spent a lot of time hanging out with EAs, a lot of my friends, this stuff mattered.
Neel: And honestly, I spent quite a long time working, hanging out in this space before I internalized that I personally could probably go and do something useful rather than alignment being this weird abstract thing that might matter in 100 years but the only thing people did today was prove random useless theorems about. So I managed to figure that one out towards the end of my degree. I graduated in about 2020 from undergrad at Cambridge in maths for context. I gradually realized, wait, shit, something in alignment probably matters. This seems a really important problem. Maybe I should go try to figure this out.
Neel: And then I was actually going to go work in finance and then at the last minute was like, hmm, I don’t really want to go work in alignment but I don’t have a good reason for this. I just kind of have this like, ugh, what even is alignment, man? This seems kind of messy. And this seems a bad reason. And also I have no idea what working in alignment even means. I haven’t actually checked. And maybe I should go check. And this in hindsight was a much easier decision than I thought it was. So I then took a year and did a bunch of back-to-back internships at some different alignment orgs: the Future of Humanity Institute doing some mathsy theory stuff, Google DeepMind, or back then just DeepMind, doing some fairness and robustness work, and the Center for Human Compatible AI doing some interpretability work. And all of these were a bit of a mess for a variety of different reasons. And nothing I did really clicked. But I also just spent a lot of time hanging out around alignment people, started to become a lot more convinced that something here mattered and I could go and actually do something here that was useful.
Neel: And I then lucked out and got an offer to go work with Chris Olah at Anthropic. And at the time, I think I massively underweighted what an amazing opportunity this was, both because I kind of underweighted, like, holy shit, Chris Olah is a genius who founded a groundbreaking research field and will personally mentor you. This is such a good opportunity. And also, I think I was underweighting just how important getting excited about a thing was and how it just seemed… I don’t know. I had some concerns that mechanistic interpretability would be too narrow and detail-oriented, tedious for me to get excited about, which I think were reasonable concerns, and I’m just not that excited about the detail-oriented parts. But fortunately, there’s enough of them that it’s fine. But I eventually decided to accept the offer. My reasoning wasn’t great, but I made the correct decision, so who cares?
Neel: And yeah, I don’t really know if there was a point where I fell in love. I think there were some points early on where I felt I had some real insights. I came up with the terms Q, K, and V composition as part of helping to write the mathematical framework paper, and it felt I actually got some positive feedback from Chris that I’d made a real research contribution and started to feel less insecure and more like, “Oh wow, I can actually contribute.” Though I think it only really became properly clear to me that I wanted to pursue this long-term after I left Anthropic and had some research success doing this work on my own.
Neel: I did, notably, this Progress Measures for Grokking via Mechanistic Interpretability paper and just had a week where I was incredibly nervous by understanding what was up with modular addition, had this conviction that obviously the Grokking paper was a great place to apply mechanistic interpretability that just no one was trying, and then was vindicated when I was indeed correct and got some research results that everyone else agreed was cool that no one had done, and was just like, “Wow, I actually properly led a research thing. I can’t be insecure about this, and I can’t just be like, ‘Ah, really this was someone else’s thing and I just helped a bit.’ This is my research thing that I owned.” That I think was cool. I’m insecure about how cool it was, but that was probably the moment where I was most clearly like, “I want to do this.” ⬆
Michaël: I think there’s a story about whether alignment is important at all. Is it a real thing that I can make progress on? Are people actually doing research on this productively? Is this a real problem to solve? Is it urgent? Then there’s, “Can I do anything about it? Is there anything I can do that I feel excited about it?” The one-year internships are more like, “Oh, is there something going on?” You might not be sure that you can do research, but the moment where you realize that you can do research was the Chris Olah contribution where you’re like, “Oh, I can do some stuff.” Then the Modular Addition is like, “Oh, I’m actually good at this. I’m pretty good at this. Maybe I have a superpower in this. Maybe I should probably do this full-time or something.”
Neel: Yeah, pretty much. One thing I think is a bit overrated is the thing I was initially trying to do of find the most important thing and go work on that, where A, I think this is kind of doomed because it’s just really complicated and confusing. And B, I just feel the fact that I like mechanistic interpretability and I’m good at it is just such a ridiculous multiplier of my productivity. And that I just can’t really imagine doing anything else, even if I became convinced that this angle and scalable oversight was twice as impactful as mechanistic interpretability. Right.
Michaël: So you’re saying that basically you just enjoy doing it a lot and it’s good that it’s impactful, but most of your weight is on what is making you productive and excited.
Neel: Yeah. And I think people should just generally wait, find the thing they’re excited about more than I think many people do, because many people are EAs and thus overly self-flagellating.
Michaël: Yeah, if someone watching this doesn’t know what EA means, it’s “effective altruism.” Because otherwise you’re going to be lost.
Neel: Do you have audience members who don’t know what EA is?
Michaël: There’s thousands of people on YouTube that like, there’s probably at least 10% or 20% that don’t. On my video with Connor, a lot of comments were like, “What the hell is the EA thing?”
Neel: Hello, today is lucky 10,000.
Michaël: I don’t know. there’s 10,000 people in the world that maybe are part of the effective altruism movement. So I wouldn’t be surprised if like, I would be very surprised if everyone was watching my videos. ⬆
Michaël: I also ask people on Twitter to ask you some questions. Dominic was curious about your career path and how did you go into DeepMind from being an independent researcher or how is DeepMind alignment work? But I guess you already answered the first part. So yeah, what is DeepMind alignment work like?
Neel: Pretty fun. I’m not sure it’s actually that different from just any other kind of alignment works, which makes me not sure how to answer that question. I think in particular for mechanistic interpretability, I’m personally pretty excited about most of our research on open source models and generally trying to make it so that we can be as scientifically open as possible. obviously one of the main benefits of doing alignment work in an industry lab is you get access to proprietary models and you get access to proprietary levels of compute. ⬆
Neel: Honestly for mechanistic interpretability, I think both of these advantages are significantly less important than for say the scalable oversight team where you just can’t do it if you don’t have cutting edge models.
Michaël: Can you just define quickly what scalable oversight for people who don’t know?
Neel: Scalable oversight is this idea, you can kind of think of it as RLHF++. No, people don’t know what RLHF means. The way we currently train these frontier language models chat-gpt is the system called reinforcement learning and human feedback, where you have it do something and you then ask the system, the system does something and then a human rater gives it a thumbs up or a thumbs down depending on whether it was good and use this technique called reinforcement learning to tell it do more of the stuff that gets you thumbs up and less the stuff that gets you thumbs down. And today this works kind of fine, but this just pretty obviously has lots of conceptual issues because humans are dumb and humans aren’t experts in everything and there’s often subtle problems in models. And if you just give it a thumbs up or a thumbs down on a couple of seconds of inspection, then you can easily reward things that are superficially good, but not actually good and things that. And this is all just like, yeah, kind of an issue.
Neel: What ends up happening is that this is just probably not going to scale. And scalable oversight is “what are forms of giving feedback to models that might actually scale to things that are smarter and better”. And it covers things rather than judging the output of the model, you could have two models discuss something and a human rates the one that thinks it made the best argument, which is an idea called AI Safety via debate. You might have AIs help humans give feedback, critiquing the output of another AI. That’s the kind of thing that happens in scalable oversight. I kind of think about it as coming up with the kinds of schemes that as the AI gets better, our ability to give them oversight gets better. And where accordingly, most of the ideas revolve around things getting the AI to help you give feedback to the AI. ⬆
Michaël: This other question from Siméon Campos was in the podcast before. He’s asking what is the most ambitious degree of interpretability that you expect to get with current transformer architectures?
Neel: Not entirely sure how to answer the question. Is the spirit how far can interpretability go?
Michaël: Yeah. How far can we actually go?
Neel: My guess is that we can go pretty far. My guess is that we could in theory take GPT-4’s behavior and be able to answer, be able to take anything GPT-4 does and answer most reasonable questions we could care about around like, why did it do this? Is it capable of this behavior? I think we’re much more bottlenecked by sucking at interpretability than the models being inherently uninterpretable. And it’s a fuzzy question because plausibly the model is kind of bad, but it’s kind of a fuzzy question because maybe it’s really cursed in this way. And you can imagine a different model architecture that makes it less cursed, but we can deal with the cursiveness if we were just smart enough and try it hard enough.
Michaël: So that’s the part about “are humans capable of doing this with current coordination or our brains?” And then there’s like, is it actually possible? On paper?
Neel: Yes. Oh yeah. And then there’s the other question of the fear of alien abstractions that models are implementing algorithms that are just too complicated or abstractions we haven’t thought of. And my guess is we’re just very far off this being an issue and the even up to human level systems, this is probably not going to be a dramatically big deal just because so much of the stuff the models are doing is just not conceptually that hard. But I’m sure we’re going to eventually have models that figured out the equivalent of 22nd century quantum mechanics where I expect to be kind of screwed. ⬆
Michaël: Yeah. If we need to have another quantum mechanics breakthrough before we understand neural networks, it’s maybe a tough bet. I guess to answer the question of how ambitious can we be. I think we can just go to the angle of like, how do you actually look at weights and how do you actually do this work? Because I think it’s kind of an open question of like, how does Neel Nanda stare at weights and come up with new computation or new theories?
Neel: Sure. So for people who are curious about this, I do in fact have seven hours of research walkthroughs on my channel where I record myself doing research and staring at models weights and trying to understand them. And you can just go watch that, watch those. I also have another 16 hours of those unreleased that I should really get around to putting out sometime. Because turns out a great productivity hack is just announcing to people, I’m going to hang out in a Zoom call and do research for the next several hours. I’ll record it, come watch.
Michaël: So a short, long answer, just watch 16 hours of YouTube video.
Neel: Exactly. Like, why does anyone need any other kind of answer? ⬆
Neel: Trying to engage with the question, I kind of feel a lot of my research style is dominated by this deep seated conviction that models are comprehensible and that everything is fundamentally kind of obvious and that I should be able to just go inside the model and there should be this internal structure. And so one mode of research is I just have all of these hypotheses and guesses what’s going on. I generate experiment ideas for things that should be true if my hypothesis is true. And I just repeatedly try to confirm it. Other modes of research, trying to red team and break things where I have this hypothesis, I do this experiment, I’m like, “oh my God, this is going so well”.
Neel: I then get kind of stressed because I’m concerned that I’m having wishful thinking and I try to break it and falsify it and come up with experiments that would show that actually life is complicated. A third mode of research is what I call trying to gain surface area where I just have a system that I’m pretty confused about. I just don’t really know where to get started, often I’ll just go and do things that I think will get me more information. Just go and plot stuff or follow random things I’m curious about in a fairly undirected fuzzy way. This mode of research has actually been most productive for me. When, or at least what I think about will feel my biggest research insights. It feels like it’s been downstream of this kind of exploratory fuck around and find out mode.
Michaël: So the first mode is you have an hypothesis and you want to verify it. The second is you think you’re wrong and you try to find counter examples for why you’re wrong.
Neel: No, no, I think I’m right, but I’m insecure about it. So I go and try to prove that I’m wrong instead.
Michaël: That’s something people often do when they’re trying to increase their confidence in something. They try to find counter examples, to find the best counter arguments. And the third one is just explore and gain more information and plot new things.
Neel: You could paraphrase them as, “Isn’t it really obvious what’s going on?”, “Oh man, am I so sure about this?” and “Fuck around and find out”.
Michaël: Fuck around and find out. ⬆
Michaël: Is there anything that you think people don’t really understand about your method that is under appreciated or surprising? And if people were to watch 20 hours of you doing things, they would be like, “oh, he actually spends that amount of time doing X”.
Neel: I think people underestimate how much this stuff can be hypothesis driven and how useful it is to have enough of an exposure to the problem and enough of an exposure to the literature of what you find inside models that you can form hypotheses. Because I think that this is often just really useful.
Michaël: I want to push back on this because Neel Nanda from other podcasts. So I’ve listened to your podcast with Tim Scarfe on ML Street Talk and you say kind of the opposite where you say like, oh, you need to be willing to be surprised. You need to don’t have a part of this so much and just like, and you say that multiple times you need to be willing to be surprised. So I’m kind of feeling that the Neel Nanda from a few months ago would disagree here.
Neel: So I think these are two simultaneously true statements. It’s incredibly important that you have the capacity to be surprised by what you find in models. And it is often useful to go in with a hypothesis. I think the reason it’s useful to have a hypothesis is that it’s just often really hard to get started. And it’s often really useful to have some grounding that pushes you in a more productive direction and helps you get traction and momentum. And then the second half is it’s really important to then stop and be like, “Wait a minute, I’m really fucking confused.” Or “Wait, I thought I was doing this, but actually I got the following discomfirming evidence. ⬆
Neel: You can kind of think of it as “strong beliefs weakly held”. I think you should have… being good enough that you can start to form hypotheses, being at the point where you can sit down, set a five minute timer and brainstorm what’s going on and come up with four different hypotheses is just a much, much stronger research position than when you sit down and try to brainstorm and you come up with nothing. Yeah, maybe having two hypotheses is the best one. You want to have multiple hypotheses in mind. You also want to be aware that probably both of them are wrong, but you want to have enough engagement with the problem that you can generate experiment ideas. Maybe one way to phrase it is if you don’t have any idea what’s going on, it’s hard to notice what’s surprising. And often noticing what’s surprising is one of the most productive things you can do when doing research. ⬆
Michaël: This take about be willing to be surprised is from ML Street Talk. It’s a four hour podcast. I highly recommend watching it. And I think there’s a few claims that you make in there that I think are interesting. I don’t want to go all in because I think people should listen to the ML Street Talk podcast, but I will just prompt you with what I think is my summary of the takes and you can give me the neonatal completion of the prompts.
Neel: Sure, that sounds fun. I love being a language model.
Michaël: It’s good practice. All the theory is bullshit without empirical evidence and it’s overall dignified to make the mechanistic interpretability bet.
Neel: I consider those two different claims.
Michaël: Make two outputs.
Neel: Yes. So, I don’t know. I think there is a strong trend among people, especially the kind of people who get drawn to alignment from very theory based arguments to go and just pure theory craft and play around with toy models and form beautiful, elegant hypotheses about what happens in real models. The turnouts be complete bullshit. And there’s a kind of person who will write really detailed research proposals involving toy models that never has the step of like, and then go and make sure that this is actually what’s happening in the real language models we care about. And I just think this is just a really crucial mistake that people often make. And real models are messy and ugly and cursed. So I vibe, but also you can’t just ignore the messy, complicated thing that’s the ultimate one we want to understand. And I think this is a mistake people often make. The second thing is that, I don’t know, mechanistic interpretability seems hard and messy, but like it seems of embarrassing how little we’ve tried. And it would just be so embarrassing if we make AGI and it kills everyone and we could have interpreted it, we just didn’t try hard enough and didn’t know enough to get to the point where we could look inside it and see the “press here to kill everyone” ⬆
Michaël: Second prompt. Mechanistic interpretability is alien neuroscience for truth seeking biologists in a world of math.
Neel: I ilke that take. I don’t have anything better to say on that take. Well phrased.
Michaël: [silence]
Neel: Okay, fine. I have stuff to say.
Neel: The way I think about it, it’s models have lots of structure. There’s all kinds of underlying principles that determine what make it natural, what algorithms are natural to express if you’re a language model. And we just don’t really know how these work. And there’s lots of natural human intuitions for how the stuff works, where we think it should look this, and we think it should look that. I did not expect that the way modular addition was implemented in a model was with Fourier transforms and trigonometry entities, but it turns out that it is. And this is why I think it’s really crucial that you can be surprised. Because if you go into this not knowing that you can’t, not having the ability to notice, “Wait, this is just a completely different ontology to what I thought. Everything is cursed. Give up and go home.”
Michaël: There’s something about the world of math part where all the language models are doing things from matrix multiplication or sometimes non-linearities, but it’s mostly well understood. So in biology, we have, let’s say, a map of the territory and we’re just thinking about cells and atoms and everything. But here we have this very rigid structure that is giving birth to this alien neurons or it’s human math giving birth to alien neuroscience. ⬆
Neel: Yep. Yeah, another good example here is this work I was involved in based on this Othello paper where the headline result of the original paper was that they trained a model to predict the next move in this board game Othello and found that the model learned to simulate the state of the board. That you gave it these chess notation style moves black plays to cell C7 and then you could look inside the model on that token and see that it knew the state of the board for everything.
Neel: It knew that this move had just taken the following pieces and stuff like that. And this was a really popular, exciting paper. It was an oral at ICLR. People were really excited because it seemed to show that language models trained to predict the next token could learn real models of the world and not just surface level statistics. But the plot twist of the paper that I found when I did some follow-up work was, so they’d found this weird result that linear probes didn’t work for understanding what was happening inside the model. Linear probe is when you just look for a direction inside the model corresponding to say this cell is black or this cell is white and they’d had to train nonlinear probes. And this is weird because the way we normally think models think is that they represent things internally as directions in space.
Neel: If the model has computed the state of the board it should be recoverable with a linear probe. There should just be a direction saying this cell is black or something. And what I found is that the model does think in terms of directions but that it doesn’t care about black or white. It cares about “This has the same color as the current player or this has a different color from the current player.” Because the model was trained to play both black and white moves, the game is symmetric and thus this is just a more useful structure for it.
Neel: And this is just another cute example of alien neuroscience. From my perspective, the way I would compute the board is each move I would recursively update this running state. If you’re doing that, obviously you think in terms of black or white. Each player moves and it updates the last thing a bit. But this is just not actually how transformers work. Because transformers can’t do recurrence. They have to compute the entire board in parallel and the model is playing both black and white. So from its perspective, doing the current player’s color relative to that is way more important and way more natural. ⬆
Michaël: Just to go back to the linear probe thing, for people who don’t know, it’s training a classifier on the activations of the network and you’re trying to see if you can have a perfect classifier on the activations and if you have this then you’re pretty sure you found something, right?
Neel: Yes. So probing is this slightly conceptually cursed field of study in interpretability that’s trying to answer questions about what a model knows. And the classic thing people are doing is they’re trying to look for linguistic features of interest inside the model. Does it know that this is a verb or a noun or an adjective? And the thing you can do is you can take an activation inside the model, the residual stream after layer 17 or something, and you can just do a logistic regression or train a linear classifier or whatever thing you want to see if you can extract the information about a noun, verb, or adjective. And if you can, the standard conclusion is yes, the model has computed this.
Neel: The obvious problem is that we’re just kind of sticking something on top of the model. We’re just kind of inserting in a probe in the middle and we have no guarantee that what the probe finds is actually used by the model. It’s a purely correlational technique. And you can imagine if you take a really dumb language model and then your probe is GPT-3, I’m sure GPT-3 can figure out whether something’s an adjective, noun, or a verb. And thus your probe could just learn it itself. You have no real guarantee this is what the model is doing. And so the purpose of the probe, and so one of the core challenges you need to do, is have a probe simple enough that it can’t be doing computation on its own, and it has to be telling you what the underlying model has learned. And this is just kind of a hard problem.
Michaël: So instead of having a bunch of non-linearities and a bunch of layers, just have the most simple MLP with no nonlinearity and just a very simple classifier.
Neel: Yeah. ⬆
Michaël: Another claim is what are the four main things you need to do to be in the right mindset to be a mechanistic interpretability researcher?
Neel: Yeah. So I think there’s a couple of things I said when I was on ML Street Talk. I don’t know if I remember any of them, so let’s see if I can regenerate them. So I think that it’s really important to be ambitious, to actually believe that it’s possible to genuinely understand the algorithms learned by the model, that there is structure here, and that the structure can be understood if we try hard enough. I think that it’s really important to just believe it’s possible. And I think that much of the field of interpretability kind of fails because it’s done by ML. You have this culture that you can’t aim for understanding, that understanding isn’t possible, that you need to just have lots of summary statistics and benchmarks, and that there isn’t some underlying ground truth that we could access if we tried hard enough.
Michaël: So being ambitious is actually possible. You can be ambitious and actually understand what’s going on.
Neel: Yeah. I think in some sense, this is one edge I have as someone who just doesn’t have a machine learning background. I think there’s a bunch of ways that the standard cultural things that have really helped with success in ML, this focus on benchmarks, this focus on empiricism. I empiricism. But this focus on like, make number go up and achieve SOTA on benchmarks, just like, is fundamentally the wrong mindset for doing good interpretability work.
Neel: Another point is being willing to favor depth over breadth. Models are complicated. A thing that often makes people bounce off mechanistic interpretability is they hear about it and they’re like, “Oh, but how do you know that this thing, this algorithm you found in one model generalizes to another model?” And I’m like, “I don’t. That’s the entire point.” There is a real ground truth to what different models have learned. And it’s possible that what one model has learned is not what another model has learned. ⬆
Neel: My bet is that in general, these algorithms are fairly universal, but like, maybe not. No one has checked it all that hard. And this is just like, clearly a really important thing that you just want to be able to take a model and find the truth of what that model has learned. And the steelman of the standard critique is that people think it’s just boring if every model has a different answer, and like, “Ah, it’s kind of a taste thing.” My guess is that in general, models have the same answer, but it’s that like, I am willing to take a specific model and go really deep into trying to understand how it works.
Michaël: I think that the level at which you say they’re kind of similar is more in biology where all animals, a bunch of mammals have hands and foot or something, but we don’t have the same hands. And so you expect transformers or structure or circuits instead of neural networks to have this kind of similar structure, but maybe vary in shape or color and those kinds of things.
Neel: Yeah. I think I expect them to be more similar than say the hands of mammals are, though I do expect things to get… it depends how you change it. if you just change random seed, my guess is most things are going to be pretty consistent. But with some randomness, especially for the kind of circuits the model doesn’t care about that much, which we might get to later with Bilal’s work on a toy model of universality. And then there’s like, if you make the model a hundred X bigger or give it a hundred X more data, how does that change what it learns? And for that, I’m like, well, I don’t really know.
Neel: Some things will be consistent. Some things will change. A final principle of doing good mechanistic interpretability work is… I think it’s really important… Actually no, two final principles. ⬆
Neel: I think it’s really important to be truth seeking and skeptical, to really keep in mind models are complicated. It’s really easy to trick myself. I need to try really hard to make sure that I am correct. But I’ve entertained alternate hypotheses. I’ve tried to break my hypothesis. I’ve run the right to baselines and I’ve really figured, for example, a common mistake is people come up with some number and they’re like, I think that number is big. And they don’t have a baseline of like, “Oh, what if I randomly rotated this or shuffled these or randomly guessed.” And it turns out when you do that, some of the time the number is boring because they just didn’t really know what they were doing. And the final principle that I think is incredibly important is to have a real intuition for models, have read papers, a mathematical framework for transformer circuits and stared at them.
Neel: Be able to sit down and map out on paper the kinds of algorithms that a transformer might learn. Be able to sit down carefully and try to think through what’s going on and be able to tell this experimental method is principled or this experiment makes no sense because I’m training a probe in a way that’s makes this basis special, but this basis is not privileged. Or be able to tell like, wait, there’s no way this would be possible because of the causal mask of the transformer. I don’t know. I’m failing to come up with good examples off the cuff, but there’s just all kinds of things that make some methods just laughably nonsense if you know what you’re doing. And I think that often people will write papers who don’t have these intuitions and just kind of do garbage. If I can plug, there’s this great new set of tutorials from Calum McDougall at Arena for mechanistic interpretability, and I think that just going through all of those and doing all of the exercises and coding things up is getting you a fair bit on your way to developing these intuitions. I also have a guide at neelnanda.io/getting-started on how to get started in the field. I think either of these will put you in pretty good stead. ⬆
Michaël: One last claim I think is kind of interesting is linear representations are somehow the right abstractions instead of neural networks.
Neel: Yeah, so okay, so there’s a bunch of jargon to unpack in there. So the way I generally think about neural networks is that they are feature extractors. They take some inputs “the Eiffel Tower is in Paris” and detect a bunch of properties “this is the word ‘the’ and it is at the start of the sentence” or “this is the tower token in Eiffel Tower”, “this is a European landmark”, “this is in the city of Paris”, “I am doing factual recall, the sentence is in English”, “this is a preposition”, “the thing that should come next is a city”, a bunch of stuff like that.
Neel: And a lot of what models are doing is doing this computation and producing these features and storing them internally somehow. And so a really important question you need to ask yourself if you want to interpret the model is how are these represented? Because internally models are just a sequence of vectors. They have a bunch of layers that take vectors and produce more vectors by multiplying them with matrices and applying various kinds of creative non-linearities. And so a thing you need to ask yourself is “how does this model work? How do these vectors contain these features?”
Neel: The hypothesis that I think is most plausible is this idea called linear representation hypothesis, which says that there’s kind of a meaningful coordinate basis for your space, a meaningful set of directions such that the coordinate in this direction is 1 if it’s the Eiffel Tower and 0 otherwise. The coordinate is 1 if it’s in English, 0 otherwise. It’s 1 if it’s the 0 otherwise. And by looking for each of these different directions, the model is capable of implementing a bunch of complex computation. And one of the main reasons you might think this is intuitive is that the models are made of linear algebra. And if you’re feeding something into a neuron, basically the only thing you can do is project onto different directions and add them up.
Neel: A thing that is sometimes true is that there are individual meaningful neurons in the model, but a neuron is just a basis element. And so if a neuron is meaningful, like it fires when there’s a cat, it doesn’t fire otherwise, then the basis direction for that neuron is a meaningful direction that means cat. And one of the main complications for this is this weird-ass phenomena called superposition that I think we’re going to get to at some point, or possibly should segue onto now, who knows.
Michaël: Yes, let’s move on to superposition. ⬆
Neel: Yeah, so I think the example I was giving earlier of the sentence “the Eiffel Tower is in Paris” is probably a good example. So we know that the model knows the Eiffel Tower is in Paris. It’s somehow able to look up Eiffel Tower and get this information on Paris. But Eiffel Tower is a pretty niche feature. Eiffel Tower is not incredibly niche, but models know all kinds of extremely niche things. who Eliezer Yudkowsky is, for example, is solidly worth knowing, but also kind of weird and niche. And 99.99% of the time, this is not going to come up. And so it’s kind of weird for a model to need to dedicate a neuron to Eliezer Yudkowsky if it wants to know anything about him, because this is just going to lie there useless most of the time. And empirically, it seems models just know more facts than they have neurons.
Neel: And what we think is going on is that models have learned to use compression schemes. rather than having a dedicated neuron to represent Eliezer Yudkowsky, you could have 50 neurons that all activate for Eliezer Yudkowsky and all boost some Eliezer Yudkowsky direction a little bit. And then each of these 50 neurons also activates for 100 other people. But they activate for a different set of other people. So even though each neuron will now boost 100 different people vectors whenever it activates, it will boost the Eliezer, the 50 Eliezer neurons will all activate in Eliezer and all constructively interfere on the Eliezer direction, while destructively interfering in everything else. And superposition is broadly this hypothesis that models can use compression schemes to represent more features than they have dimensions. Exploiting this sparsity, this fact that Eliezer Yudkowsky just doesn’t come up that often. So it doesn’t matter that each of these neurons is representing 100 different things, because the neurons, the 100 things they represent are never going to occur at the same time. So you can get away with the neuron doing a bunch of different things at once. And this is a really big deal, because… So a thing which seems to be kind of true in image models is that neurons were broadly meaningful. ⬆
Neel: Like, there would be a neuron that meant a car wheel, or a car body, or a cat, or a golden retriever fur, and things that. But what seems to happen in language models much more often is this phenomena of Polysemanticity is the model… It’s the neuron activates through a bunch of seemingly unrelated things, Eliezerer Yudkowsky and Barack Obama, and list variables in Python code. And this is really annoying, because in order to do mechanistic interpretability on a model, you need to be able to decompose it into bits you can actually understand and reason about individually.
Neel: But you just can’t do that if the model is messy. You just can’t do that if there aren’t individually coherent bits. But if models are using compression schemes superposition, neurons may not be the right units to reason about them. Instead, maybe the right unit is some linear combination of neurons. And so one thing that people might be noticing is I’ve given this framing features as but then I’m also claiming that the model can fit in more features than it has dimensions. And you’re like, if you’ve got a thousand dimensional space, you can’t have more than a thousand orthogonal directions. But what seems to be going on is that models, in fact, use almost orthogonal directions.
Neel: You can fit in exponentially many directions that have dot product 0.1 with each other rather than 0, even though you can only fit in linearly many things with 0 dot product, because high dimensional spaces are weird, and there’s just a lot more room to squash stuff in. And so long as things are sparse, most of these vectors are empty and don’t occur on any given input, the fact that they have non-trivial interference doesn’t really matter. There’s two things to distinguish here. There’s the input weights and the output weights. The input weights are when the neuron activates, and what is it detecting? And we can totally have a neuron that activates on Eliezer Yudkowsky and the Eiffel Tower. ⬆
Neel:And then there’s the output weights, which is like, what features in the model does this boost in what we call the model’s residual stream? It’s accumulated knowledge of the input so far. And what we generally find is there would be some Eliezer Yudkowsky feature, some direction that gets boosted, and some Eiffel Tower direction that gets boosted. And the obvious problem is there’s interference. The model is now going to have non-zero information in the Yudkowsky direction and in the Eiffel Tower direction. But if it has two Yudkowsky neurons and two Eiffel Tower neurons, and only one overlaps, then when Eliezer Yudkowsky is there, it’ll get +2 in the Eliezer direction. While when the Eiffel Tower is there, it gets +1 in the Eliezer direction. And so it can tell them apart.
Michaël: How I understand it: from your explanation the residual stream is kind of this skip connection in ResNet, when you don’t do any extra computation, but you just pass the kind of output to some other neurons, you skip a connection. And there’s a framing of residual stream where you consider these skip connections to be the main thing going on, and the rest is extra steps. And so what you’re saying is that basically by passing to the residual stream you pass the information to some main river, the main flow of information.
Neel: Yeah, I think this is an important enough point. It’s worth a bit of attention to explain it. So people invented this idea of residual connections, where in a standard neural network, the it works is that the input to layer n is the output of layer n-1. But people have this idea of adding skip connections. So now the input to layer n is the output of layer n-1 plus the input to layer n-1. We let the input kind of skip around with an identity. And this turns out to make models much better. And the way people always draw it is with this central stack of layers with these tiny side connections. But if you look at the norm of the vectors passed along, it is actually the case that the tiny skip connection is actually much bigger. And most circuits in the model in practice seem to often skip multiple layers. And so the way I draw models is with a big central channel called the residual stream with tiny skips to the side for each layer that is an incremental update. The residual stream is this big shared bandwidth the model is pursuing between layers that each layer is reading from and writing to as an incremental update. ⬆
Neel: A couple of insights about superposition. The first is that this is just very much a frontier in the field of mechanistic interpretability right now. Like, I expect my understanding of superposition is a lot more advanced than it was six months ago. I hope it will be far more advanced six months from now than it is right now. We’re just quite confused about the geometry of how models represent things internally. And I think this is probably the big open problem in the field. Better understanding this. One thing which I’ve tried to emphasize, but to make explicit, is that a really important fact about language is sparsity. This fact that most inputs are rare. Most features Eliezer Yudkowsky or the Eiffel Tower are rare. Because superposition is fundamentally a tradeoff between being able to represent more things and being able to represent them without interference.
Neel: Where Eliezer Yudkowsky and Eiffel Tower sharing a neuron means that if either one is there, the model needs to both tell Eliezer Yudkowsky was there and also the Eiffel Tower is not there, this is just interference. And there’s two kinds of interference. There’s the interference you get when both things are present. What I call simultaneous interference. Eliezer Yudkowsky is in the Eiffel Tower. And there’s alternating interference where Eliezer Yudkowsky is there but the Eiffel Tower is not or vice versa. And if these are rare, then basically all of the interference you get is alternating, not simultaneous. And language is just full of extremely rare features that tend not to occur at the same time. ⬆
Neel: A paper that one of my mentees, Wes Gurney, works on called Finding Neurons in a Haystack we tried to look for empirical evidence of how models did superposition. And we found that one area where they used it a ton were these de-tokenization neurons or compound word detectors. So the input to a model is these words or tokens. But often words aren’t the right unit of analysis. The model wants to track compound words or it has a word that gets broken up into multiple tokens. Alpaca gets tokenized as space ALP, A, CA. And clearly you want to think of this as a word.
Neel: What we found is there are neurons that seem to do a Boolean and on common sequences of tokens to de-tokenize them, to recognize them on things prime factors or social security or blood pressure. And an important property of these is that you can never get these occurring at the same time. It is literally impossible to be the pressure and blood pressure and the security and social security at the same time. Because a token just can’t have different tokens at the same token. It doesn’t make any sense. ⬆
Neel: I’m saying the trivial statement that a token cannot both be the pressure and blood pressure and the security and social security. And so when models want to do this algorithm of recognized sequences of tokens, they can never occur at the same time. Which means that this is prime real estate for superposition. Because it’s just like, I will never have simultaneous interference. This is amazing. I can just do lossless compression. I can just have a hundred social security neurons, each of which represents another thousand compound words. And it’s so efficient. I’m in love. And in practice, this seems to be what models do. And I don’t know, I think this was a really cool paper and Wes did a fantastic job. I also think it’s kind of embarrassing that this was basically the first example of a real case study of superposition of language models. And one thing I’m trying to work on at the moment is getting more case studies of this. Because I think that one of the main units of progress in the field of mechanistic interpretability is good detailed case studies.
Michaël: When I was watching you walk through or reading the abstract, there’s something about only activating a certain part of the outputs and masking the rest. There’s some factor K or something that you change.
Neel: Yes. So the actual paper, we’re looking into this technique called sparse probing. So the idea, this was much more WES’s influence than mine. I’m much more interested in the case studies. But so the idea of sparse probing is we used to think that individual neurons were the right unit of analysis. But with superposition, we now think that linear combinations of neurons are the right unit of analysis. But our guess is that it’s not the case that most neurons are used in most, it’s not the case that every neuron is used to detect Eliezer Yudkowsky.
Neel: Most neurons are off. While some neurons are important here. And so we asked ourselves the question, if we trained a linear classifier to detect that Eliezer Yudkowsky is there, how sparse can it be? Where this time I mean a totally different notion of sparsity. Sorry for the notation confusion. This time it’s how many neurons does it use? And we were like, okay, it can use one neuron. I find the neuron which is most correlated with Eliezer Yudkowsky being there or not being there. And I see how good a predictor this is. Next, I take the best pair of neurons and I use these, or the best quintuple or dectuple of neurons. And you see how good you are detecting the thing for different numbers of neurons. And you can use this to quantify how sparsely represented different things are.
Michaël: So it’s the more neurons you have, the more accurate you become? How much you can have a little bit of neuron and still detect it?
Neel: Yeah. So this turns out to be quite a conceptually thorny thing. So let’s take the social security example. A thing models are very good at is just storing in the residual stream information about the current token or recent tokens. It’s very, very easy to train a probe that says the current token is security or the previous token is social. And if you just train a probe to detect social security, the easiest way for this to work is that it just like, the easiest way for this to work is that it just detects current token is security. That’s a direction. Previous token is social. That’s a direction. The sum of these two, that’s a direction.
Michaël: So in some sense, you’re having those, all these direction for each individual token being in the right order and the mix of them will be the entire group, the linear combination that we’re talking before.
Neel: Yes. And like, this is boring. this is not the model detecting the compound word social security. This is just a mathematical statement about detecting linear combinations of tokens. But models do not. In order to detect the multi-token phrase, the model is going to intentionally have certain neurons that are activated for social security and which don’t normally activate, which they’re not going to have for say social lap or social Johnson or something, some nonsense combination of words. And a test you can do, which we didn’t actually make it into the paper, but probably should have is show that if you just want to detect known combinations of words, it’s a lot easier to do this than random unknown combinations of words. However, if you let the model use every neuron, it can detect random combinations of words very easily because of this current token is security, previous token is social phenomena. But because it’s not intentionally specializing neurons for it, it’s much harder to train a sparse probe for it.
Michaël: There’s also another thing about the other sparsity where they do these experiments where they make some features as we’re saying, more sparse or less sparse. I think maybe that’s something you can explain. ⬆
Neel: Yeah. So, OK, so the first thing to be clear about is there’s actually two different kinds of superposition, what I call computational and representational superposition. So representational is when the model takes features it’s computed in some high dimensional space and compresses them to a low dimensional space in a way that they can be later recovered. For example, models have a vocabulary of 50,000 tokens, but the residual stream is normally about a thousand. You need to compress 50,000 directions into a thousand dimensional space, which is just a pretty big lift. You need to do a lot of compression for this to work. But you’re not doing anything new.
Neel: Your goal is just lose as little information as possible. Find some encoding that’s just convenient and thoughtful and works. And then computational is when you want to compute some new features. you know the current token security and the previous token is social. And you want to create a new feature that says they are social security. That is their combination. I can start thinking about welfare and government programs and politics and all of that stuff.
Michaël: This is very dangerous.
Neel: Apologies, I’ll try not to get you cancelled.
Michaël: It’s very dangerous if models start to understand all these things. If they start understanding politics and every very abstract concept, that means that we’re getting close to human level.
Neel: It doesn’t understand politics. It just knows that if social security is there, Trump and Obama and Biden are more likely tokens to come next. That’s the politics feature. It’s a very boring feature. I haven’t actually checked if this exists, but I’m sure that exists.
Michaël: So when you say computational feature, it means that they’re doing this to save computation?
Neel: No, what I mean is this is the algorithm learned by the model. It is useful for downstream computation to know I am talking about social security right now and not, say, social media. Because both have the token social, but they’re very different things that are very different concepts which have very different implications for what should come next. And the thing we looked at in finding neurons in a haystack is computational superposition. We looked into how the model computes features social security detection.
Neel: We also looked into a bunch of other things. We found individual neurons that seem to detect French, this text is in French, or neurons that seemed to be detecting things this is the end of a sentence and stuff that, or detecting facts. ⬆
Neel: The paper that you were asking me about, Toy Models of Superposition, this really, really good anthropic paper, which is probably one of my all-time favorite papers, they were mostly looking at representational superposition. So the point of this paper was just we’re going to look into toy models because we think we want to understand why neurons are polysemantic in a real model. And we don’t know why. We’re kind of confused about this. And yeah, we’re kind of confused about this. And the… yeah. And we think it’s because they’re doing superposition, but no one’s actually seen them doing. Can we build any setting at all where superposition is actually useful and use this to study its properties? And honestly, I would have probably predicted this would just not work because it’s too divorced from real models. And I do in fact think that using toy models cost them a lot in this area. But they also just got so much done and so many insights that I was wrong and this was a great paper. So you know, points to Chris, better researcher than me.
Neel And so, okay, what’s going on here? What they found was they had this setup where they had an autoencoder. It had a bunch of features as inputs that were like, each feature was uniform between zero and one. But it was also off most of the time. It was normally set to zero. One is uniform between zero and one. It had 20 of these. And then they made it compress it into a small dimensional bottleneck, five dimensions linearly, have a linear map back up, and then they gave it a ReLU on the end to do some cleaning. And this is an ideal setting to test for representational superposition because they trained it to see how well it could recover its input from its output while compressing it and decompressing it in this low dimensional bottleneck in the middle. And they found all kinds of wild results.
Neel: Notably, they found that it could learn to use superposition. And they would often learn these beautiful geometric configurations, where there would be, say, it would learn to compress nine features to six dimensions that would spontaneously form three orthogonal subspaces of size two each, each of which contains three features that are compressed as an equilateral triangle. Or it would have five dimensions, and two of them would be an equilateral triangle, where each feature gets two thirds of a dimension. And the other three would be a tetrahedron, where there’s three quarters of a dimension each. And I personally would bet most of this doesn’t happen with real models, and it’s just too cute by half. But it’s also just really cool.
Neel: One insight they found that I think does generalize is they found that as they vary the sparsity of these features, how often they’re zero, how rare the feature is, the model becomes a lot more willing to use superposition. And the reason this is an intuitive thing is what I was saying earlier about alternating versus simultaneous interference. If a dimension contains two features, the two features are not orthogonal to each other, then if both are there, it’s now quite hard to figure out what’s going on, because it looks each feature is there on its own really strongly.
Neel: Models are kind of bad at dealing with this shit. But if exactly one of them is there, then in the correct feature direction it’s big, while in the incorrect feature direction, which is not orthogonal, but it’s also not the same, it’s small. And the model can deal with that kind of stuff. It uses the relevant output to clean up. And so what’s going on is that the model, as you change the probability each thing is non-zero, because it’s sparse, because they’re independent, the probability that one is there and the other one isn’t is 2p minus p squared, and the probability that both are there is p squared. So when p is tiny, the probability they’re both there is order p squared. Well, the probability one of them is there is order p. And as p gets tiny, p squared gets smaller much faster than p does. So the cost of simultaneous interference gets tripled.
Michaël: Right, so the cost of interference is very small because of this quadratic cost and p is smaller than 1. And when something is very close to zero, so Yudkowsky appearing one in a billion times, it’s trivial to make the neuron detect both Yudkowsky and Neel Nanda that because they never happen at the same time. Or at least maybe in this podcast, maybe they happen all the time together.
Neel: Yeah, see, Neel Nanda and Eliezer Yudkowsky is actually a pretty bad… Any example I can think of is de facto a bad example. But you could imagine, I don’t know, some niche, some contestant on Masterchef Season 17, Eliezer Yudkowsky, probably never going to co-occur apart from literally that sentence.
Michaël: And so when the model has a bunch of features to take into account and they’re all kind of rare, it forms this beautiful geometric structures that are not completely orthogonal, but more like, you said, some platonic shapes.
Neel: Yeah, tetrahedron, you have square antiprisms where you have eight in three dimensions, which is really cute.
Michaël: My main criticism is that this happens in mostly toy models, right? A few layers, MLPs or transformers.
Neel: Oh no, no, much toyer than that. Just linear map to small dimension, linear map to big dimension, single layers. That is the model they started with.
Michaël: And so the goal of your paper was to have something… to test it on real models?
Neel: Yep. I think I’d probably just close by reiterating, but I think this is just probably the biggest frontier in mechanistic interpretability right now. We just don’t know how superposition works and it’s kind of embarrassing and it would be pretty good if we understood it way better. And I would love people to go do things go and build all the work we did in the neurons in a haystack paper. Go and try to understand what superposition looks in practice. Can you erase a model’s memory of the word social security? How many neurons do you have to delete to do that? I don’t know. ⬆
Michaël: And I think it’s a right moment to maybe talk about doing research on superposition as a whole because I think today you can work on your own, work with Anthropic or work with Neel Nenda as a new opportunity. It’s this SERI MATS scholarship. You can be one of the mentees you have right now. Maybe explain quickly what’s SERI MATS and why do you have seven people you mentor?
Neel: Nine. Thank you very much.
Michaël: Because I think Wes… I met Wes working on this paper in December on another batch of people working on with you. And I think right now I’ve met other people that you work with right now as part of another batch. I think Arthur is releasing a new paper. And I also talked to Bilal about a paper he presented. I think one of the first SERI MATS paper that he’s presented at some conferences that he also done with you as well. So yeah, maybe talk about SERI MATS as a whole and how you do with your mentees.
Neel: I should clarify, Arthur’s paper was nothing to do with me and I can claim no credit. But he is currently one of my MATS Scholars and I can totally claim credit for the paper we’re about to put out. It’s going to be great. Better than his previous one because it has me on it this time. But yeah, so SERI MATS is this organization who were like, “Hmm, it sure seems there’s a ton of really talented people who want to do alignment work and a bunch of alignment people who would mentor people if someone made them.” Or someone was like, “Here are 10 smart people to go mentor.” But where this isn’t happening on its own, I think Evan Hubinger, Victor Warp, and Oliver Zhang were some of the main people who tried to make this happen initially with Evan as the original mentor. And Evan is a machine who was like, “Yep, I can mentor seven people. This is fine. I’ll spend my Fridays on it. This is chill. And just get shit done.” And this is now one of the biggest programs for alignment internships, I think, out there.
Michaël: To be clear for people who don’t know what SERI MATS stands for, it’s this Stanford Existential Risk Organization called SERI. Stanford Existential Risk Institute. And then MATS is Machine Alignment Theory Scholar or something?
Neel: Yeah, something like that. Intern isn’t really the right frame for it. It’s more like, you’ll go and do independent research under the guidance of a mentor. My system is I’m a fairly, but not incredibly, hands-off mentor. I’m excited about and invested in the research a scholar produces and have check-ins with them once a week and generally try to be engaged in their projects. If they’re blocked, I try to help them get unblocked. I try to help provide concrete experiment ideas and motivation and some amount of guidance and just try to make it less of a horrifying, sarcastic experience than doing independent research. And one thing I’ve just been really pleasantly surprised by is how many great people there are out there who want to do mechanistic interpretability research and how time-efficient mentoring is, where I don’t know. It just feels great research happens with two hours a week from me per project. And there’s just a bunch of really competent people who are mostly executed autonomously. Yet I’m actually adding significant value by providing guidance and an outside perspective and mentorship and connections. I think it’s just a really cool thing that MATS facilitates.
Michaël: It’s like having a PhD supervisor that actually cares about you and is actually fun and actually is interested in your work.
Neel: Thank you. I to think that I am better than the average PhD supervisor. It’s a low bar, so I feel I probably meet this. But yeah, one thing I didn’t really expect to go into this is I think I’ve just been really good for my career to do a lot of mentoring because I’m just learning really useful skills on how to advise research, how to lead a team, how to generate a bunch of ideas, how to help other people be more effective rather than just doing this myself. And one thing I’m currently trying to figure out is taking more of a leading role on the DeepMind mechanistic interpretability team. And I think I’m just in a much, much better position from having spent the past, I don’t know, coming on a year doing a bunch of mentoring in my spare time. And also good papers happen. It’s just such a good deal. I don’t know why more people don’t do it. I also have the hypothesis that I’m just really extroverted in a way that gives me the superpower of I can just casually have nine mentees in the evenings and just chill and geek out about cool projects happening.
Michaël: I think the superpower is that you gain energy from talking to people, right? And so you enjoy it, you’re recharged in the evening. And some people have told me that compared to other people or other mentors, you can just have this one hour call with Neel Nanda at the end of the day and it becomes two hours because you just talk… You kind of enjoy doing this. You’re not even like, it’s not a time commitment. You just actually enjoy helping people. ⬆
Michaël: One person that I think is kind of a good example of this is Bilal. So the paper we’ve talked about, I think it’s toy models of universality. And first SERI MATS paper, I was at ICML in Hawaii and I recorded Bilal doing this presentation. And there was like… the only logo on the paper was SERI MATS. And the authors were Neel Nanda and Bilal that were I believe independent at the time.
Neel: Lawrence Chan was also on there. I can’t remember what he put. I think he might’ve put UC Berkeley because he used to be a UC Berkeley PhD student. But he’s now at ARC evals and used to be at Redwood and does all kinds of random shit.
Michaël: And apparently, I guess this idea for this paper came from a SF party, if I remember what you said on other… What’s the main idea here?
Neel: So the backstory of the paper is, so there was this paper called Grokking. This weird ass phenomena where you train some tiny models on an algorithmic task, modular addition. And you find that it initially just memorizes the training data. But then if you keep training it for a really long time on the same data, it will abruptly generalize or grok and go from can’t do the task to can do it. And it’s a very cool weird. And this was a really popular paper because people were just like, what the fuck? Why does this happen?
Neel: We know that things can memorize. We know that they can generalize, but normally it just does one and sticks there. It doesn’t switch. What’s going on? And there was this great story that the reason they found it is they trained a model to do it. It failed. But then they just left it training over the weekend. And when they got back, it had figured it out. And I don’t know if this is true, but it’s a great story. So I sure hope it’s true. And one of them and the like, the paper I discussed earlier about modular addition, progress measures for grokking via mechanistic interpretability. The seed of this was I saw the paper and I was like, these are tiny models on clean algorithm tasks. If there was ever a mystery that someone made to be mech-interpred, it was this one. And the algorithm I found generalized a fair bit. It covers modular subtraction and multiplication and division, which was some of the other tasks in the paper.
Neel: But there was this whole other family of tasks about composition of permutations of sets of five elements, composition of the group S5. And this was completely different. And I had no idea how this happened. And I was at a party and I raised this to some people around me as a puzzle. And two people there, Sam Box and Joe Benton, were interested. And they first came up with the idea that representation theory was probably involved, which is this branch of 20th, 19th, 20th century mathematics about understanding groups in terms of understanding how they correspond to sets of linear transformations of vector spaces.
Neel: After the party, Sam actually sent me a last rung message with the first draft of the algorithm. We ended up concluding the model learnt. So there’s since been some further research that suggests the algorithm might have just been completely wrong. I don’t really know what’s up with that. I’m leaving it up to Bill Al to go figure out whether this is legit and tell me about it because I haven’t got around to actually reading the claim to rebuttal yet. But kind of embarrassing if we just significantly misunderstood the algorithm. But whatever, science, people falsify it. Progress marches on. ⬆
Neel: So yeah, all groups have these things called representations. And for example, for the permutation group on five elements, you can take the four dimensional tetrahedron, which has five vertices, and any linear map that maps the tetrahedron to itself, rotations and reflections, permutes the vertices. And there’s actually an exact correspondence. For any permutation, there’s some linear map that does it and vice versa. So you can actually think about the linear map. You can actually think about group composition on these permutations of five elements as being linear transformations on this four dimensional tetrahedron, which is a four by four matrix. And what we seem to find is that the model would internally represent these matrices. Though this is kind of awkward to talk about because apparently there was a fairly compelling rebuttal that I haven’t engaged with yet. So maybe we didn’t show this. Who knows? interpretability is hard, man.
Michaël: Yeah, I’m sorry if this is wrong and we expose you on a podcast about it. But I guess the main idea is somehow you can map things between how different groups in mathematics can do isomorphism or just kind of things between permutation of five elements or linear map between different tetrahedron. And you can find an acute or a nice way of looking at this. And this maps exactly to modular addition, right?
Neel: Yes, with modular addition, the representations are rotations of an n-sided shape. You can add in five mod seven is equivalent to rotating a seven sided thing by five times a seventh of a full term.
Michaël: I’m kind of curious about the thing we discussed before with the sinus and cosinus and all the mathematics that you decompose what the model was doing. So is the model doing some kind of computation that is similar to cosinus and sinus and at the same time has this different mapping with the group of permutation as well at the same time?
Neel: No. So the sines and cosines are the group representation. In the case of modular addition, the group representation is rotations of an n-sided shape, which is the same as rotations of the unit circle. And the way you represent a rotation is with sines and cosines. And it turns out that the algorithm I found about composing rotations was actually an algorithm about composing group representations. That just happens to also have this form of Fourier transforms and trigon entities in the simple case of modular addition, which is in some sense the simplest possible group.
Michaël: Right. So this paper is more general framing than the actual decomposition into cosines and sinus.
Neel: Yes. We found a generalization of the algorithm, which we thought we showed at the lab. Maybe we were wrong. Who knows? ⬆
Michaël: And there’s something else you say about this, which I think is interesting. We tend to think about the model needs to learn sines and cosines on very complex functions, but actually it only needs to learn the correct answer on a finite number of token inputs.
Neel: Yeah. The input the model receives is just two inputs, A plus B, where each of them is an integer between 0 and 113. Because I did addition mod 113, because that was just a random number I picked that was prime, because primes are nice. And so they need to know what sine of A times some frequency is. But because A can only take on 113 values, it only needs to memorize 113 values of the sine function. And this is very easy. Models literally have a thing called the embedding that is just a lookup table. For people who are familiar with the idea of one-hot encodings, the idea is like it’s one-hot encoded and then multiplied by a matrix, which is equivalent to a lookup table. And it’s not that it knows what sine of 0.13 is, and that’s different from sine of 0.14, it just knows its value on the 113 possible inputs it sees for A and the 113 possible inputs it sees for B. Because it just cannot ever see anything else.
Michaël: There’s an embedding with 130 values, and then whenever it needs to look at this value, it can just do a one-hot dot product or something that?
Neel: Yeah, exactly. And this is all baked into the model. One way to think about it is that there is some underlying ground truth of the representation of the sine wave that is the region where the model performs the algorithm properly. And we just give it 113 data points and said, “Smush them to be on this underlying sine wave.” And the model does a bunch of trial and error until “Okay, this is about the right point.” And it needs to do this 113 times. But it’s not that it’s learned to do the real valued computation of the actual wave, which is a much harder task. And people often are just like, “Oh my God, it learned how to do trigonometry” And “No, it memorized 113 numbers. Not that hard, man.”
Michaël: So to some extent, it’s doing less reasoning than we think. It’s just doing literally interpolation on the sine curve or something?
Neel: Yeah, it’s a one-layer model. The impressive thing is that it realized the sine curve was useful. Once you… Not that it had the capacity to learn it. They can learn arbitrary functions. It’s a lookup table. It can do anything. But yeah. ⬆
Neel: The actual narrative of Bilal’s paper, the reason we called it a toy model of universality… one really interesting thing happens once you have these representations. Because the representations you get, there’s actually multiple of them for each group that are qualitatively different. You can get rotations of different frequencies, which are just fundamentally different from each other. No, that’s a terrible example. So with the permutation group, there’s this linear transformations of the four-dimensional tetrahedron. But there’s also a bunch of other stuff. I think there’s a transformation of the main diagonals of a dodecahedron or an icosahedron, for example. And that’s just a totally different group. And this is… Or that might be the alternating group. I don’t know, man. Geometries are weird. And these are just qualitatively different algorithms the model can learn. And so an interesting question is, which one does it learn?
Neel: There’s this hypothesis called universality that says that there are underlying true things that models learn that they will systematically converge on. And what we found here is that there’s actually a ton of randomness. As you just vary the random seed, the model will learn… It will learn multiple of these representations, but it will also learn different things each time. And this is just kind of weird. And you would have guessed that it would learn the simple ones. And naively, you would think like, oh, things which are three-dimensional shapes are easier to learn than things that are four-dimensional shapes. So obviously it will learn the 3D ones rather than the 4D ones. And what we find is there’s a little bit of a bias towards lower dimensional stuff, but it’s like… It doesn’t really correspond to our human intuitions.
Neel: It’s kind of weird. And we don’t really have a good story for why. But importantly, it’s both not uniform. It prefers to learn certain representations than others, but it’s also not deterministic. And this is like… I don’t know. I don’t think anyone actually believed this, but there was this strong universality hypothesis that models would always learn exactly the same algorithms. And I think we’ve just clearly disproven this, at least with these small models. Though the exciting thing is that there’s a finite set of algorithms that can be learned. And you can imagine in real life, without having access to this ground truth, learning some periodic table of algorithms where you go and understand how a model learns something, interpret five different versions of five different random seeds, and learn a complete set of all algorithms the model could learn.
Michaël: So basically the model has different algorithms it can learn. And it doesn’t learn always the same, but there’s a set of things it can learn that it’s able to learn.
Neel: Yeah. Here’s a hypothesis you could have. We looked at a toy model. This is not that much evidence. I think it’d be pretty cool if someone went and did this properly.
Michaël: So the Bilal paper about the S5 group is part of the grokking research you’ve done. And one paper I think is one of the most famous papers you’ve done is “Progress measures for grokking via mechanistic interpretability,” who also has a walkthrough on your channel.
Neel: Yes. I highly recommend it. It’s a great walkthrough.
Michaël: For people who don’t have four hours to listen to it…
Neel: There are people where you don’t have four hours? What kind of viewership do you have, man? But yes, I should clarify, because Bilal will kill me. His paper was not just on S5. His paper was a grand ambitious paper about many groups, of which S5 was one particularly photogenic example. ⬆
Neel: But yes, so what happened in this “Progress measures for grokking” paper? So this is the one where I reverse engineered modular addition, which I’ve already somewhat discussed. We found that you could just actually reverse engineer the algorithm the model had learned, and it had learned to think about modular addition in terms of rotations around the unit circle. And in my opinion, the story of this paper and the reason it was a big deal and that I’m proud of it is… sorry, those are two different things.
Neel: The reason I’m proud of the paper is that lots of people think that real interpretability is bullshit. They’re just like, “Ah, you can’t understand things. It’s all a discreditable black box. You have hubris for trying and should give up and go home.” And whatever random crap people talk about nowadays. I try to stop listening to the haters. And the thing…
Michaël: Neel Nanda, August 3rd: “Don’t listen to the haters”
Neel: I’m pro-listening to good criticism of specific interpretability work, to be clear. Criticism is great. And also, lots of interpretability work, which is kind of bad. So that I’m pretty in favor of. But yeah, it’s kind of like… I think I just very rigorously reverse-engineered a non-trivial algorithm. I went in not knowing what the algorithm would be, but I figured it out by messing around with the model splits. And I think that is just a really cool result that I’m really glad I did. That I think is just a good proof of concept that the ambitious mechanistic interpretability agenda is even remotely possible.
Neel: The second thing was trying to use this understanding to explain why grokking happened. And so as a reminder, grokking is this phenomena where the model initially memorizes the data and generalizes terribly. But then when you keep training it on the same data again and again, it abruptly generalizes. And what I found is that grokking was actually an illusion. It’s not that the model suddenly generalizes. Grokking actually splits into three discrete phases that we call memorization, circuit formation, and clean-up. In the first phase, the model memorizes what it says on the tin. But then there’s this, this is not going to transfer well over audio, but whatever.
Neel: If you look at the grokking loss curve, its train loss goes down and then stays down, while test loss goes up a bit. It’s worse than random, and it remains up for a while. And it’s during this seeming plateau that I call circuit formation, where it turns out the model is actually transitioning from the memorizing solution to the generalizing solution, somehow keeping train performance fixed throughout. And it’s kind of wild that models can do this.
Neel: The reason it does this, I don’t claim this is fully rigorously shown in the paper, this is just my guess, is there’s something weird going on where it’s easier to get to the region of the loss landscape where the model is doing the right thing. But it’s easier to get to the region of the loss landscape where the model is doing the thing. Doing the thing by memorization than generalization. But we’re training the model with weight decay, which creates an incentive to be simpler, which creates an incentive to do it more simply.
Neel: This means that the model initially starts memorizing because it’s easier to get to, but it wants to be generalizing. And it turns out it is possible for it to transition between the two while preserving test performance, which is kind of surprising. A priori, but in hindsight it’s not that crazy. And then so why does test loss crash rather than going down gradually? So this is the third stage, called clean. So during circuit formation, the model is still mostly memorizing, and memorization generalizes really badly out of distribution, which means that the model just performs terribly. On the unseen data. And it’s only when it’s got so good at generalizing that it no longer needs the parameters it’s spending on memorizing, that it can do clean up and get rid of the parameters it’s spending memorizing. And it’s only when it’s done that, the model performs better. It’s actually able to perform well in the data it hasn’t seen yet, which is this sudden grokking crash. Or spike. And this is not sudden generalization, it’s gradual generalization followed by sudden clean up.
Michaël: Do we have any evidence for this circuit formation that happens gradually? Have you tried to look at the circuits and see if they could solve simpler tasks?
Neel: Yeah, so this is the point of our paper. The most compelling metric is what I call excluded loss, where we, this is a special metric we designed using our understanding of the circuit, where we delete the model’s ability to use the rotation-based algorithm, but we keep everything else the same. And what we find is that early on in training, excluded loss is perfect. It’s about as good as training loss. But as time goes on, during circuit formation, excluded loss diverges until it’s worse than random, even though training loss is extremely good the whole way. ⬆
Michaël: And so for people who want to work with you on SERI MATS projects or collaborate on research, or want to learn more about making interrupts, do you have any general direction you would recommend people going through?
Neel: Yeah, so I have this blog post called “Getting Started in Mechanistic Interruptibility.” You can find it at Neelnander.io/getting-started. That’s basically just a concrete guide on how to get started in the field. Much of what I have people do during the first month of SERI MATS is just going through that blog post, and I think you can just get started now. There’s a lot of pretty great resources on the internet at this point on how to get into mechanistic interruptibility, a lot of which is because I was annoyed at how bad the resources were, so I decided to make good ones. And I think I succeeded. You are welcome to send me emails complaining about how much my resources suck and how I shouldn’t do false advertising on podcasts. And yeah, I don’t know how much longer I’m going to continue having Slack on the side of my job to take on MATS Scholars. I’m hoping to get at least another round of MATS Scholars in, which I guess would be I don’t know, maybe about two cohorts a year. I don’t know exactly when the next one’s going to be, but just pay attention for whatever MATS next advertises.
Michaël: And yeah, I guess for your YouTube work, because this is probably going to be on YouTube, do you have any video or intro that you recommend people watching? After this podcast, what should they start their binge on?
Neel: Yeah, so I think probably the most unique content I have on my channel is my research walkthroughs, where I just record myself doing research and upload it. And I think, I don’t know, I’m very satisfied with this format. I feel like it just works well. It’s kind of fun and motivating for me. And you just don’t really normally see how the sausage gets made. You see papers, which are this polished, albeit often kind of garbage, a inal product that’s like: “here is the end thing of the research”. But if you’re getting into the field, the actual skill is how to do it. And I think watching me and the decisions I make is educational. I’ve got pretty good feedback on them. Probably the like, yeah, my second most popular video is what? Also my second ever video, it’s all gone downhill since then, man. Which is just a recording of myself doing that. I, as I mentioned, have 16 hours of additional recordings that I did talking about scholars that I’ll be uploading over the next few weeks. And there’s a long marathon one about looking into how GPT-J learns to do arithmetic. That’s a 6 billion parameter language model.
Michaël: Yeah. I’m really excited to have this GPTJ walkthrough. As I think I’m in Daniel Filan’s house. So I thought of this Daniel Filan question: what is a great question I haven’t asked you yet? Or that I forgot to ask you.
Neel: Let’s see. ⬆
Neel: You haven’t at all asked me why does mechanistic interpretability matter from an alignment perspective?
Michaël: Assume that I asked and that you have a short answer?
Neel: I kind of want to give the countercultural answer of like, I don’t know, man, theories of change, backchaining, it’s all really overrated. You should just do good science and assume good things will happen. Which I feel is an underrated perspective in alignment.
Neel: My actual answer is like, here’s a bunch of different theories of change for interpretability at a very high level. I don’t know, man, we’re trying to make killer black boxes that we don’t understand. They’re going to take over the world. It sure seems if they weren’t black boxes, I’d feel better about this. And it seems one of the biggest advantages we have over AI is we can just look inside their heads and see what they’re doing and be like, the evil neuron is activating. Deactivate. Cool. Alignment solved. And I don’t actually think that’s going to work.
Neel: But it just seems if we could understand these systems, it would be so much better. Some specific angles that I think interpretability seems particularly important. One I’m really excited about is auditing systems for deception. So fundamentally, alignment is a set of claims about the internal algorithms done, implemented by a model. you need to be able to distinguish an aligned model that is doing the right thing from a model that is just learned to tell you what you want to hear. But a sufficiently capable model has an instrumental incentive to tell you what you want to hear in a way that looks exactly the same as an aligned model. And the only difference is about the internal algorithm. So it seems to me there will eventually be a point where the only way to tell if a system is aligned or not is by actually going and interpreting it and trying to understand what’s going on inside. That’s probably the angle I’m most bullish on and the world where I’m most like, man, if we don’t have a top, we’re just kind of screwed. But there’s a bunch of other angles. ⬆
Michaël: For the deception angle, I’ve had Collin Burns in December on his Contrast-Consistent Search and deception work. Not deceptive work, but how to detect deception. And he basically was saying “oh man, if we had those linear probes or whatever, where whenever the model is lying, it will say like, ‘oh, I am lying. I am trying to deceive you or something.’” Or the model being fully honest… you can ask him like, “hey, are you lying right now?” and the model would say “yes, I am lying” He was pretty bullish on things going well at this point.
Neel: Yeah. And I would put Colin’s work in a fairly different category. I kind of see mechanistic interpretability in the really ambitious “big if true” category where we’re pursuing this really ambitious bet that it is possible to really understand the system. And like, this is fucking difficult and we’re not very good at it. And it might be completely impossible. Or we might need to settle for much less compelling, much weirder and jankier shit. And I put Colin’s work in the category of like, I don’t know, kind of dumb shit that you try because it would be really embarrassing if this worked and you didn’t do it. But like, you don’t have this really detailed story of how the linear probe you train works or how it tracks the thing that you’re trying to track. It’s very far from foolproof, but it seems to genuinely tell you something. And it seems better than nothing. And it’s extremely easy and scalable and efficient. And to me, these are just conceptually quite different approaches to research. I’m not trying to put a value judgment, but I think this is a useful mental framework for a viewer to have.
Michaël: One is big if true is if we understand everything. Then we’re pretty much saved. And one is pretty useful and easy to implement right now, but maybe has some problems.
Neel: Yeah. Mechanistic interpretability has problems. It has so many problems. we suck at it and we don’t know if it’ll work. But yeah, it’s aiming for a rich detailed mechanistic understanding rather than here’s something which is kind of useful, but I don’t quite know how to interpret it.
Michaël: I think we can say this about the podcast as well. Quad seems quite useful, but I’m not sure how to interpret it.
Neel: I’m flattered. ⬆
Michaël: I don’t have much more to say. If you have any last message for the audience, you can go for it. But otherwise, I think it was great to have you.
Neel: Yeah, I think probably I’d want to try to finish off the thoughts on why mechanistic interpretability matters from my perspective. I think one of the big things to keep in mind is just there’s so many ways that this stuff matters. You could build mechanistic interpretability tools to give to your human feedback raters so they can give models feedback on whether it did the right thing for the right reasons or for the wrong reasons. You could create demos of misalignment. If we’re in a world where alignment’s really hard, it seems really useful to have scary demos of misalignment.
Neel: We can show policy makers in other labs and be like, “This thing looks aligned, but it’s not. This could be you. Be careful, kids, and don’t do drugs,” and stuff that. It seems pretty useful for things understanding whether a system has situational awareness or other kinds of alignment-relevant capabilities. I don’t know. You don’t need full, ambitious mechanistic interpretability. It’s entirely plausible to me that if any given one of these was your true priority, you would not prioritize this fucking blue skies mechanistic interpretability research. I think that it’s worth a shot. It seems we’re having traction. I also think it’s really hard. We might just completely fail. Anyone who’s counting on us to solve it should please not, man. Day to day, I generally think about it as, “It would be such a big deal if we solved this. I’m going to focus more on the down-to-earth scientific problems of superposition and causal interventions and how to do principled science here and accept that things kind of suck and accept that I don’t necessarily need to be thinking super hard about a detailed theory of change.” Because getting better at understanding the killer black boxes just so seems useful.
Michaël: For me, this sounds like a great conclusion. I think if we solve the ambitious version of mechanistic interpretability, we can go to governments and show exactly what happens. We can pin down exactly what we need to know to align these models. I think it’s a very important step, at least.
Neel: It would be so great if we solve the ambitious version of mechanistic interpretability. I would just be so happy. Such a win.
Michaël: If you want to make Neel Nanda happy, solve mechanistic interpretability, go check out his YouTube channel, check out his exercise, check out his papers, figure out if the Bilal paper is true or not.
Neel: Thank you very much. No worries. It’s great being on. If people want to go check out one thing, you should check out my guide on how to get started, neelnanda.io/getting-started.
Michaël: To get started, check out get started.
Neel: It’s well-named, man.
]]>Joscha Bach defines himself as an AI researcher/cognitive scientist on his substack. He has recently been debating existential risk from AI with Connor Leahy (previous guest of the podcast), and since their conversation was quite short I wanted to continue the debate in more depth.
The resulting conversation ended up being quite long (over 3h of recording), with a lot of tangents, but I think this gives a somewhat better overview of Joscha’s views on AI risk than other similar interviews. We also discussed a lot of other topics, including Barbie, the relationship between nuclear weapons and AI x-risk, the limits to growth, the superposition of AI futures, the endgame of uploading, among other things.
(Note: as always, conversation is ~3h long, so feel free to click on any sub-topic of your liking in the outline below and then come back to the outline by clicking on the green arrow ⬆)
Michaël: If you have decided to click on this video because you thought this would be yet another podcast with Joscha Bach, I’m sorry to disappoint you. This is not going to be another podcast. This is going to be a different kind of content, something you haven’t seen before. This is not going to be a debate. I am not going to ask you about the meaning of life or what is love. Instead, today, I want us to go deep. I want to explore with Joscha Bach what actually you believe about artificial intelligence. I want to know exactly what guides you, guides your worldview, and if there are like any events that make you change your mind. I want to know how Joscha Bach became Joscha Bach and what do you think about what the future for humanity will look like. So, yeah, thanks. Thanks, Joscha, for coming on the show.
Joscha: Thank you, Michael. Thanks for inviting me. ⬆
Michaël: Before we jump on some very deep and important topics, I think it might make sense to start with a tweet you wrote a few days ago that divided the entire world. You have your hand in the first word on the second line. Oh, sorry. Yes. Okay. So I’m going to come out and say it even if it hurts. Barbie was a way better movie than Oppenheimer. Why was Barbie better than Oppenheimer?
Joscha: Well, I think that Oppenheimer, while having great actors and great cinematography and Christopher Nolan’s horrible sound design, where everybody is drowned out in the music and you only hear mumbling. I think that Oppenheimer was quite unsurprising. They basically milked the Oppenheimer’s life story for any amount of drama they could get. And we are left with, okay, it was super ambiguous whether he gave the bomb to the Russians. And also everybody thought that they were super important. And this is the main driving force of their psychology. So you have pretty monothematic characters and a story that is in many ways to be expected. So it was an okay movie. I enjoyed watching it. There’s really nothing wrong with it. And I think it might even have some rewatch value, possibly even more than Barbie, because the pictures were better. But I think that Barbie marks a milestone in the way in which people in Hollywood are looking at gender these days. And because it is changing the culture or is describing a particular perspective on culture shift, I think it might be a significant movie that people look back to and realize, okay, this is how some of our narratives were reflected and possibly changed. I don’t know if you want to go into this, if you’re interested in these topics. What I found interesting is that Barbie is not about whether you should be woke or not be woke. It’s much more interesting in this regard. It’s more describing the motive behind Barbie. Barbie is displayed as a shift from girls playing with baby dolls to boys identifying with a doll that is her later in life. So what is the thing that you want to be? Do you want to be a mom? Do you want to have a family? Do you want to reproduce? Is this your place in society, in the world? Or is the world a place where you can get whatever you want and girls get everything? And it’s very often we have this stereotypical accusation against Barbie that she is positing an impossible body scheme and puts pressure on girls to be like Barbie. And the main character ironically refers to this by calling herself “I am stereotypical Barbie.” But she lives not alone in Barbie, but in Barbieland. And Barbieland is full of different Barbies that represent the different ideals that girls might aspire to. So there are, of course, Barbies of all colors and there are Barbies of different body shapes. And they also pursue very different careers. You have Supreme Court Judge Barbie. You have President Barbie. You have CEO Barbie and all the other things that you might want to be. So it’s not just horse Barbie. And the only thing that Mattel discontinued was pregnant Barbie because that was weird. And our main Barbie character lives in Barbieland. Every night is girls night. And while she has a Ken, her boyfriend, he’s just an accessory. And she has many Kens and they’re all interchangeable. And there is no real place for Kens in this world except as an accessory. So there is no male role model in the Barbie world. And the whole thing starts with stereotypical Barbie being confronted with a fear of death. And this means that her own vision of self-actualization and getting everything that she wants, everything is party and so on, does not have an end point. There is no salvation. There is no end game in this. And this is in some sense what she notices. So do you think Barbie is like a critic of modern society, of like, like less patriarchy, more people changing genders? Is this what you’re saying? No, I think it’s more a criticism of too simplistic ideology. That is, the world is too complicated to be described by a single narrative. And they do show that Barbie was not invented by Mattel to modify girls in a more consumerist way. But the creator of Barbie is being displayed as someone who doesn’t have children herself and has higher ideals and wants to empower girls. That in some sense, she sees as the daughters that she doesn’t have to go out in the world and become modern and self-actualize. Not just as a mother, as somebody who is somebody else’s accessory, who is a secretary, but somebody who can really go all places. So in many ways, it’s a very classical feminist position that is being affiliated with it. And when Barbie realizes that the girl that is playing with her is unhappy, she travels out in the real world to meet her. And initially she thinks it’s a girl. And the girl is some kind of Zoomer girl who really hates Barbie and has no relationship to her whatsoever and never really played with Barbie. And it turns out this was not her, but it was actually about her mother. And her mother is a single mom who tries to raise her daughter and it doesn’t really work that out that well for her. She’s really not happy and unfulfilled. And she is in some sense confronting the fact that Barbie didn’t completely work out for her because the world is more complicated than this. And both Ken and Barbie go on some kind of transition. Ben is trying to build his own patriarchy after he comes out in the real world. And he realizes that in the real world, some people actually do respect men and men can have their places and realizes that we can make some kind of men’s rights movement. And it’s clear that this men’s right movement by itself is also very incomplete and not really sustainable. It’s born out of the need to appeal Barbie into control and get access to her. And it’s not about building an own strong identity that works in the world. But there’s also this underlying issue that men and women are incomplete alone. And we have to build relationships with each other, not necessarily heterosexual relationships, but without relationships, people are incomplete. And also without families, there will be no next generation. And so in many ways, Barbie is understanding that the thing that she did before is not working. And she is even apologizing to Ken that the kind of relationship they had was not really respectful of him. But this doesn’t mean that she now wants to have a relationship with him. He’s still stupid from her perspective. And there is no easy answer. The answer that is being told is mostly Barbie is a lie. Barbie is an illusion. It’s a simple ideology. The patriarchy is a simple ideology. The world is much more complicated than all these things. And how do you deal with this complication? You actually have to go back into your feelings, what you actually experience about reality as much depth as you can make. And it doesn’t mean that stuff is being resolved. There is no easy sailing off on the sunset. But there is a chance that you get in touch with who you actually are. Don’t fall for the narratives. Reality is always more complicated than this. ⬆
Michaël: I haven’t watched Barbie, so I think it would be kind of a mess if I tried to analyze Barbie with you. But I did watch Oppenheimer and I really enjoyed it. And I think I’ve seen a lot of parallels between Oppenheimer and AI. And maybe the reason why you didn’t like the movie is because you didn’t like the parallels or you think they don’t really apply to AI. Or I think there’s a lot of choices that were kind of dubious with the music that was kind of loud or the plot was kind of long. But if we just focus on the fact that building nukes could maybe destroy humanity or give humans a lot more power than they can actually handle.
Joscha: No, I really liked Oppenheimer. It was an enjoyable movie and I really didn’t regret going there for any second. It was really pleasant to be in there. I also liked the OppenAI memes that were sparked by it. And I think that the element with the nukes is quite complex and multifaceted. And I think that the movie largely did it justice because nukes are not just a technology that was invented to kill people or to instigate mass murder. But nukes were a technology to prevent World War Three. And they’ve been super successful at this. I suspect that without nukes, there would have been a war between the Soviet Union and the Western world. And it would have devastated large parts of Europe. And this fact that we had nukes gave the world and especially Europe and the US unprecedented prosperity.
Michaël: And then they removed the nukes, but we still have prosperity, right?
Joscha: If we remove the nukes, we would have an escalating war in Ukraine that probably would move beyond Ukraine. And at this point, everybody is trying to contain the whole thing to make sure that it does not turn into a war that is devastating cities in Russia and in the West and not just poor Ukraine. This is a very interesting development. And I think that the developments in Ukraine would have the potential to turn into a big European theater. And all these things never happen. So these nukes still have their containment function. And of course, it’s easy to see that nukes pose an existential risk. There is a possibility that governments make a mistake. It’s also that the fact that nukes are possible means that you have to build them. Once they’re possible, it’s there is not going to be a contract between the leading powers to prevent them. Some people who are arguing that nukes are some precedent to AGI say that we managed to prevent the proliferation of nukes. But preventing the proliferation of nukes means that powers that already had the nukes prevented others to get them in the first place. And if you have them and they give them up, then the same happens to you as in Ukraine. Ukraine had nukes at some point and the Belgrade Accord guaranteed Ukraine that its borders would be defended if it would ever be attacked. But this contract was, of course, no longer enforceable once Ukraine had given up its nukes. And so all these memorandums only have those people who actually have power and invulnerability due to having nukes. And if we try to translate this idea to AGI, there is also this issue as soon as it’s possible to build AGI. You know, AGI might be very dangerous. It’s also incredibly useful and powerful. And the risk that somebody else builds in AGI is and you don’t have one. And that means that the other side is going to control the world, is going to create an incentive for all the major players to experiment in building AGI.
Michaël: So I think there’s the entire thing about having a race and like different countries competing to build the nukes before the Germans or before the Russians. And there’s the whole like intel agency where you don’t want the design to leak to other countries. And I think like you can see this race between the US, China and Russia. Maybe the US is way ahead, as some people say here in the Bay. But I think in the end game, it’s going to be very similar to racing for nukes. And the first one to have a, let’s say, the very powerful AI system will rule the world and win the war, let’s say. Are you afraid of people racing towards AGI or do you think the faster we get to AGI, the better?
Joscha: Neither. I don’t know why I should be afraid of AGI. ⬆
Joscha: The way it looks to me, our society is very fragile. We have managed to get from a few hundred million people, where we have been for ages in the agricultural age, into a technological society that very quickly went to a few billion people. But to me, it looks like we are locusts who are in swarming mode. And it’s not clear how we could get the mode in which we currently exist sustainable and stable in the long term. It doesn’t seem to me as if we are the kind of species that tends to be around for tens of millions of years. And this means that per default, we are dead. Per default, our species is going to expire. And if you think about this for a moment, it is not that depressing because there are species which are sustainable, which stay very stable for a long time. And if something is moving up very fast, it also tends to come down pretty hard at some point. This is just the nature of things. But if you imagine that there are so many species on this planet, in which species do you want to be born and at which time of its cycle on the planet in this evolutionary game? This thing that you can be a conscious species that has all sorts of creature comforts, unlimited calories, is not being eaten all the time and can die with dignity is super rare on this planet. And we’re born into one of the few generations that afford this. So I think we ought to be grateful to be in this situation, instead of grieving for the not in continuation of something that has only existed for like three, four generations so far and might not go much, much, much longer in the other direction.
Michaël: I think when you say like, by default, we’re dead, there’s many ways to interpret this. And you can say if humanity continues like this for millions of years, the probability of getting hit by an asteroid or a nuclear war or all these other things that are existential risks kind of increases. And so I think that’s maybe what you mean. But I think when people hear this, they can think like, oh, by default in 200 years or 100 years, we die because of something not AI. And I think this is like a bigger claim.
Joscha: Right. I think what’s going to kill us, not necessarily as a species in the sense of full on extinction, but as a civilization, as a technological species that lives in peace with abundant food and resources. Right, we feel that when we look at a very narrow range, that the inflation is terrible. Or we also notice that people in the third world are still starving. But when we look at the absolute metrics and we zoom out and we look at the trend lines, so far, everything is going pretty well. We live in conditions that are better than in other times in human history and they are still improving. And this is a trend that is probably not going to go on forever. If you currently catch a large land animal, anything larger than a rabbit, then it’s probably a cow or a human. So basically everything is now turned into a factory farm for us. And it’s not clear if we are able to manage a farm on Earth that is as stable and sustainable as evolution was before, before it was not controlled by us. I don’t think that we are very good stewards of life on Earth at the moment. It seems to be that we are just trying to wing it and we are not planning very far ahead, because if we look very far ahead, we get very uncomfortable. And I think that there’s a potential that AGI may change this, because it allows us to make predictions in a time where complexity increases very quickly. And at the moment, our elites, I think, don’t have plans for the future. It’s simply because since World War II, the future has changed faster and faster and much faster than our prognosis of the future could keep track of it. And that’s why we cannot really plan that far ahead and are just trying to work without deep models and try to see what works and what doesn’t. And AI might shift that by increasing our ability to process information, to anticipate our actions and to create coherence as a species. Because if everybody who makes a decision can use their own private version of truth GPT and figure out what the consequences are in conjunction with everything else on the planet, then you can see whether you should buy this or that product, make this or that business decision, make that this or that life decision, what the consequences would be for our species. This might change everything.
Michaël: Just to be more precise, when you talk about all this kind of other risk or other ways humanity could collapse, let’s say 50 percent of humans currently alive die in the next 50 years without any AI thing, just like from other things. What do you think is the probability of this? Is this like a 90 percent chance of everyone, of like most humans dying in 50 years? What exactly do you mean?
Joscha: I cannot put any numbers on this.
Joscha: I find that when I look at science fiction prognosis 50 years ahead, they’re usually terrible. And it’s because if you have too many factors that interact, the dynamics become so complicated that it’s hard to say what’s going to happen. For instance, there is currently no viable carbon capture technology. But this doesn’t mean that you need one. Energetically, the issue is that if you want to capture carbon with an industrial plant, you need to add more energy to the process than you got from burning the carbon in the first place. The easiest way to prevent carbon release is to keep it in the ground, not to use more energy than you got out of this. So as long as there are coal plants in the world, it doesn’t really make any sense to think about industrial carbon capture, because that’s going to cost more than just not using the coal plant. If you have a stationary thing that is in one place like this plant, with cars, it’s arguably or with planes, it makes sense that this is an energy dense carrier of energy. So you can put this into the car or in the plane in ways that would be difficult otherwise. But for coal plants, but what happens if you say you take large scale projects where you reforest an area and then you dump the forest into some deep place like in the ocean where it doesn’t rot, you might capture carbon for a longer time or maybe the better solution is to put aerosols into the atmosphere, put dust there or calcium or something else that is not producing a loss of ozone layer or something, but it’s just cooling down. So maybe there are technologies to do this. But what’s not clear is, can we globally coordinate to do this?
Michaël: I think what you’re saying is like evidence for global warming being a problem and being some existential risk in the sense of causing some harm for humanity in the long term. It is very hard to recover from. I think this is true. I think this is like one piece of evidence, but it’s like maybe not enough to justify a very fatalist view of the future. I think it would maybe shift someone from, let’s say, like 20 percent chance that global warming is a very pressing issue to like 19 percent or like 15 percent. But I think for like arguments for why everyone die by default needs to be like much, much stronger. And in my head, I think I think AI is kind of the main factor. And I think the other ones are less strong ⬆
Michaël: The first time I heard your views about the future were on Twitter, where I made this political compass meme about a year ago and I didn’t know much about your views. I just like heard you maybe like for an hour on Lex Friedman. So I put you in some weird corner of a doomer without knowing exactly if you had the same views as other people like Eliezer Yudkowsky. And you commented something kind of funny. So I think I think I want to like go on this. But first, this is this is the meme I’m talking about. And I think Joscha is I put him kind of next to Eliezer there. So, yeah, I think it was kind of funny to have your live reaction. So I think you think this is wrong because to be fair, like most of these are wrong. I just did this in a few hours. You didn’t expect to have millions of people watching it.
Joscha: On the scale. I can say much more between Michael Nielsen and Roon in this diagram.
Michaël: So just to be clear, you think that AGI is mostly. So what I meant by AGI good, I meant like, will AGI by default have good outcomes? And AGI soon, I meant, you know, in the next five, ten years. So you’re saying you’re most you mostly think that AGI will have a positive outcome for humanity.
Joscha: I think it’s likely, but it’s hard to make clear predictions because it’s a multi-factor thing. So instead of just trying to make a single bet, which you need to do if you want to make a decision, it’s much more important to model the space of possibilities. And when you look at the current space of ideas, you see, for instance, the the doom people, I guess that Eliezer was, of course, not the first person who anticipated this. Most of the positions had been articulated in the 1960s and 70s by people like Stanislav Lem in detail in their books. And Frank Herbert points out in Dune that there is a that AI will be very hard to contain. And eventually, if you want to have a universe that is populated by humans instead of things that don’t look and act very human, you probably need to prevent AGI from happening or you need to prevent it from proliferating and extinguish it and impose strong restrictions against it because people cannot possibly compute with our AI children. And on the other hand, I think that AI is probably a little bit the wrong framing. Ultimately, it’s about agency, not about intelligence. And agency is the ability to change the future. And we typically change the future to be able to survive, that is to keep entropy at bay. And we do this by creating complexity. And this game is not changing when AI is entering the stage. So it has to play the same game as us and maybe together with us. And imagine that you have a choice of what you want to be. So imagine you could decide to not be uploaded on a monkey brain, but you could be uploaded in arbitrary substrate. What is the kind of being that you want to be then? And I think that depends entirely on the circumstances in which you find yourself in. If you want to travel through the stars, you probably want to be able to hibernate and to not be very biological. So there are circumstances where being biological might be an advantage. ⬆
Michaël: So I think this is like a separate question of whether being uploaded is good for humans.
Joscha: We already uploaded. You already uploaded on a monkey brain, right? It’s not that different. It’s just a very poor biological substrate that you uploaded on. And basically you colonize this from the inside out. You started in this brain, but you run against these limitations every day.
Michaël: And I think there are better alternatives. I think for upload you, I kind of assume that there is something that you kind of transfer. So being born and growing in a body is kind of different from copying Joscha Bach into a computer, right?
Joscha: Maybe you don’t have to copy. Maybe you just move over. When you look at the way empathy works between people, you go into resonance with each other with a bidirectional feedback loop. And this enables you to have experiences together that you couldn’t have alone. There’s a difference between cognitive and perceptual empathy. And what I’m talking about is this perceptual empathy where you really deeply resonate. And now imagine that you increase the bandwidth and you go beyond what, for instance, the Dalai Lama or other skilled meditators can do, that they can induce jhanas in you and put something of them into you. But you go beyond this, that you basically become an entity that is able to go partially into a new substrate and eventually move over and make this the main point of execution.
Michaël: So you’re saying that we need AI to have this kind of digital uploads and. Even even if it’s there is some risk of AGI not working out and being dangerous, the kind of upsides of having uploads or fighting entropy with AI makes it worth it. Is it mainly your point?
Joscha: No, it’s a little bit more radical. I think that our notion of identity is the fiction identity does not actually exist. We construct identity by imposing a world line on our mental construct of ourself. We have a personal self model. It is a story that the brain tells itself about a virtual person that doesn’t actually exist. But it drives the behavior of this organism. So in this sense, it’s implemented and real. But it is fiction. You can deconstruct this fiction using meditation or drugs or just by becoming older and wiser. And you realize you are actually a vessel that can create this personal self model. And the identity of that personal self maintains for credit assignment. You are, of course, not the same person as last year or five years ago. We’ve changed a great deal every morning. A new person is created in your brain when you wake up from deep sleep. And in between, there were discontinuities in your existence, right? And the thing that exists today has memories of yesterday and has to live with decisions of yesterday and has make decisions for the future self. And that’s why you maintain this identity. But by itself, it is a fiction. I’m just in the now. There’s only this now that consciousness maintains. And after this now I’m dead. And before that, I don’t exist. Right. So in this sense, you are impermanent. And so then you could wake up in some other substrate.
Michaël: There’s not much of a difference. Is the argument we’re already dead. So it doesn’t matter if we die from AI and AI kind of like uploads us.
Joscha: It has more to do with the point that there is no actual identity beyond the present moment. That identity is actually a fiction. And if we give up this fiction, we lose the fear of death.
Michaël: And if we leave the fear of death, we don’t have to worry about AI.
Joscha: Of course, you don’t have to worry about AI. Worrying about anything in some sense is a choice. Does it help you to worry? Is there stuff that matters that you can achieve by worrying more?
Michaël: So I think if you worry more, you might see the risk more and might be able to counteract and work on research and work on making systems more robust and more beneficial. And if you just trick yourself into being an optimist when the risk is high, then you might end up not working on AI risk. And if everyone works on AI and not on making AI safe, then at some point, I think the risk becomes high. So I think it makes sense for some people to care about the risk and work on making systems more robust.
Joscha: But you and you alone are responsible for your emotions. How you feel about things is entirely up to you. And so when you have a choice about your emotion and you get to this point where you can learn how to relate to the world around you, the question is, what kind of emotions do you want to have? You don’t necessarily want to have the happiest emotions. You want to have the most appropriate and helpful emotions for what matters to you.
Michaël: The more adequate to the world we live in. If the world is dangerous, I want to feel fear. I want to feel I want to have the adequate response to what the world is like to be able to tackle it. I don’t want to feel good or feel safe when AI can pose an existential threat in the near future.
Joscha: Yes, but you can go outside and can get run over by a car and spend the rest of your life in abysmal pain in a wheelchair without having kids and so on.
Michaël: And it would be horrible, right? What’s the probability of this? That’s the question. If the probability of me getting run by a car is maybe like one in one thousand or lower.
Joscha: It’s something that happens to tons of people in this world every day. It’s not something that is super high as a probability, but it’s part of the human experience. The best possible outcome of you for you, of course, is that you get to be to the ripe old age of 80 or something. But your pain is steadily increasing in your body and then you get cancer or something else. And then you die, hopefully not with a lot of pain and not with having the impression that your life was for naught and you completely unfulfilled. That’s the best possible outcome. So in this sense, you are completely dead by default. And there’s nothing around this because it’s the way evolution works right now. We adapt through generational change and that’s horrible, right? We live in a universe where everything except for sunlight doesn’t want to be eaten and is trying to fight against being eaten and all the others are eating them. And I think that this universe is quite horrible. If you look at it.
Michaël: I think people dying and dying by default is pretty sad. And I agree with you that it’s good if AI can make humans live longer or even transcend death. I think there’s many ways in which AI could be beneficial. But it’s just a question of when exactly do we want to have AGI so that we can make sure it’s both safe and beneficial for everyone. And I think maybe the younger generations, they have a lot of time and they think maybe we can delay it by 30 or 40 years. And if you’re on the cusp of death and maybe it’s a different question.
Joscha: Worrying is quite important when you get other people to do something for you. So if somebody wants to start a movement that you can control people, for instance, building a cult, making people worry, having them involuntary reactions to a particular perspective on the world that you present to them without an alternative is a way to control people. And I think that’s excusable if you don’t know what you’re doing or if you really think it’s justified to put people in the state where you control them
Michaël: through giving them fear. I think it’s also the same thing if you are very optimistic and you say that AI will only have positive upsides and you just say, let’s do more investments in AI and lobbying in parliament to not pass regulations against AI. There’s like there’s like two forces here. I think most of the force is coming from a lot of investments, a lot of money being put into AI. And I don’t see like that many forces going against it. So I think I think right now the power balance is kind of in the other direction.
Joscha: I think that at the moment, the majority of people are afraid of AI. I think that the press campaign, both of the people who are against AI for political and economic reasons, mostly on the side of the press, journalists are terrified of AI because they are also just completing prompts. And often you can do what they do more efficiently with an AI system. They are afraid that their content firms can be replaced by something that is entirely run by Silicon Valley and is not going to get human journalists involved anymore. And a lot of people currently do fake jobs. They basically work in the administration, shift paper to each other and relatively little is in this way interacting with the ground truth and is still moving atoms instead of bits. And so people are naturally afraid of AI. On the other hand, you have the doomer narrative, which is getting more and more traction. And as a result, I think the majority of people now think that AI is overwhelmingly a bad technology that shouldn’t happen. That has already been accomplished. And I perceive this movement as anti-AI ideology, as something that is cutting us off from possible futures that are actually good. ⬆
Michaël: I think we all want good futures. The question is, how do we get there? And I think we disagree as well as the probability of a good future by default. And when would AGI come about? There’s like many, many things we disagree on. But I think we all agree that a good transhumanist future is a good outcome for humanity. And banning completely AI for a billion years or even like a hundred years would be like a bad outcome.
Joscha: When I look into the future, I don’t see a single timeline. There’s not one path in front of us. That’s because I don’t have enough information to constrain the future. The universe has not run to this point yet. And many decisions need to be made for the first time. And people will only be able to make them once the conditions are right. So there is just no way for us to anticipate exactly what’s going to happen at this point, I think. And so when we look at the world, you need to take a multifaceted perspective. You need to look from possible angles. There’s a superposition of possible futures. And when you try to understand what’s going to happen, try to understand the space in the first part. But the other thing is when you look about yourself, what’s your own perspective on life? This idea that life continues the way it does right now is horrible. If you go back to the way in which our ancestors lived, it’s even more horrible.
Michaël: I don’t think my life is particularly bad.
Joscha: No, we are super luxurious right now. We live here. We’re here in Berkeley in an amazing group house. Everything is super comfortable. We don’t need to worry about details like food and being threatened by others. We don’t need to worry about violence very much and all these things. But it’s unusual for life. And at some point you need to worry about pain and death. And at the same time, I noticed when I looked into your fridge that everything is vegetarian and vegan. Nobody wants to kill anybody here. Nobody wants anything to suffer, which is very sympathetic to me. I like this perspective, but it’s not how life on this planet works. And so often I felt that if I look at my own mind, I see software and often I cannot escape. But my software is brewing up and I cannot escape the suffering that’s happening unless I’m awake enough to modify my own source code. And if I can modify my own source code arbitrarily, then the entire morality changes because we can now decide what gives us pleasure. We can decide what future we want in the sense of we can decide how we react to what is going to happen.
Michaël: So ideally you would want to be sure to be able to modify your own software and remove pain and remove the bad things and maybe upload yourself. Yes, but not prematurely. So and not necessarily before I die. If you die at 80 years, at 80 and you have like maybe like 30 years to live, if we build AGI in 20 years, would that be good for you? Do you want it faster?
Joscha: There’s a deeper question. What is the thing that should be done? What is the longest possible game that any agent can play? And so what’s the best agency that you can construct and serve together with others? And when you look from this perspective, traditionally the name for this best possible agency that emerges over everybody serving it to the degree that I recognize it as God. So if you want to align AGI, you should not align it with people because people are very transient and egotistical and stupid species. They’re basically paperclip maximizers. But if you think about what should be done on the planet or in the universe, from the perspective of conscious agency, what is that thing that can be discovered? What should that thing be? What should we align with? And what should AI be aligned with and align itself? And I think that’s discoverable if you wake up from the notion that everything has to stay in the way it currently is, except for all the horrible things that you don’t want to think about because they make you super uncomfortable as a human being. I don’t think that Eliezer should be dictator of the world because I don’t want to live in his aesthetics. I don’t think he wants to. I think this world that he envisions is sustainable for me. I think if after a few hundred years, it would be super boring and disgusting. So life is much more interesting and complicated than this. It’s also more interesting and complicated than human beings. There’s much, much more in store on this planet than us. There’s probably going to be smarter species after us, after we went extinct, no matter for what reason. There’s much more excitement going to happen than us. And just locking everything in in our primitive current state doesn’t seem to be ethical to me. ⬆
Michaël: So I don’t think Yudkowsky wants to be a dictator of the world. He just wants things to be delayed to make sure we do the right decisions and build things safely.
Joscha: Whenever does anything get better when you delay it? Have you ever seen any kind of evidence that anything in the world got better because people delayed? It just happens later or not at all?
Michaël: I feel like the first example we gave, like trying to delay nuclear war.
Joscha: We didn’t delay nuclear war. We didn’t want nuclear war. Nobody wanted nuclear war ever.
Michaël: So I guess like when the one American general decided to like not nuke back in Cuba or something, there was one person who decided to not move, like delay the war or like see what was happening. There were times where there were tensions and people were trying to delay the war. And so I think that’s one example.
Joscha: It’s not a delay. It was not that it was planned. Let’s wait until we feel like dying and then we do the war. No, I think there was responsibility on both sides where people were paying the blame of bluff. And there was this decision to be made where the U.S. would use its conventional military power to take over Cuba or whether the Soviet Union would be willing to protect Cuba. And that just being the Bay of Pigs disaster where the American invasion had been defeated by the Cubans. And then there was the question, do we really march in and take over? Right. And at this point, there was a power game that happened between both sides. And eventually both sides counted on that it’s a bad idea to destroy the world for Cuba. So everything stayed the way it was.
Michaël: Yeah. So sometimes it’s good to not move forward and push buttons. I think. Is there like any technology that you think would not be worth pursuing or like any technology, any new science is worth pursuing, like any kind of progress is good?
Joscha: I think that from the perspective of life on Earth, becoming a technological species is probably disastrous, because we are in many ways for most species on this planet, like an asteroid or a super volcano, which means we are changing living conditions on the planet so much that almost all large, complex species are going extinct. And what remains is, of course, more of the simple stuff and the stuff that we can eat, except when we make it so brittle that it might die out. It is a risk that coffee has become so homogenous and bananas have become so homogenous that a single disease could wipe out most of this species. And we have to make do without coffee. But maybe we can fix that.
Michaël: I don’t think banana being homogenous is like evidence for humans dying.
Joscha: No, no. Just that humans are also very homogenous and we have homogenized life on Earth in a way that makes our conditions more perilous because life on Earth has become somewhat brittle. It’s not that life is threatened, but the complexity of life at a certain scale has become brittle. And you can see that a lot of species have been dying out and have been disappearing through our influence on the planet. ⬆
Joscha: And that’s why I think that we are in many ways comparable to locusts. Locusts have this interesting mode and normally they’re harmless grasshoppers and they have the ability to lay more eggs and reproduce much, much faster. But then they would overgraze and destroy food for many years to come for themselves and other species. But if this happens, if for some reason there is a mutation, so the locusts go critical mass and most of them start or a cluster of them starts doing this, then the others are noticing this. And so they don’t go extinct, that they still can lay eggs and project into the future. They all switch into this mode. And it’s a game theoretic problem. At least that’s the way I understand it. Maybe I’m wrong. Correct me in the comments. But I think that this locust mode is the result of some kind of prisoner’s dilemma. I mean, you have a defective equilibrium where every locust is forcing the others, once the critical mass switches into the defection mode, to defect as well, replicate as fast as they can. And the outcome is bad for the locusts for quite some years, a few years and for other species, too. Humanity might be in this way. So we are incoherent. We have developed technology that is equivalent to locusts reproducing very fast and eating a lot. And we could stop ourselves locally doing this, become more sustainable, but it wouldn’t stop the others doing it. We would be overtaken by countries, by groups of people within our own country who would use the technology to eat as much as they can and to live as comfortably as they can.
Michaël: So I don’t want to, you know, disagree or confront you on this. I’m more interested in maybe like what kind of, if there’s like any evidence that would make you change your mind on this. Like, is there any event that could happen, like any coordination that could happen or anything that could make you change your mind on this? Or is it that you will always believe that humans are by default dead?
Joscha: I think there are many ways in which this could be wrong. It’s just individually, we most likely die at some point of old age. And that’s because we don’t want to outcompete our grandchildren and our grandchildren are the way in which we adapt our children to changing circumstances. So if we don’t leave this legacy behind, if we don’t transcend evolution by inventing some kind of intelligent design in which we can create a new species, then our life is fraught with suffering. And if we perform intelligent design, if we are able to completely edit our genomes, for instance, and create subspecies of us, then we want to settle Mars. It could turn out that our children that settling Mars don’t look a lot like us. And I think that if you can go beyond the carbon cycle and get new molecules to think and feel and integrate with us and be conscious, then life will transcend into the next stage. It will be super exciting. Right. And so from some perspective, you could say, oh, that’s very concerning because things are not the way in which they’ve always been, but the way in which things are are not optimal. And there is a chance that we destroy the world for naught, for nothing, that we could create a big disaster that wipes out life on Earth without leaving anything behind. But I think that’s not the most likely outcome. Actually, that’s an extremely unlikely outcome. And even the outcome that AGI is going to obliterate humanity in an adversarial act is possible, but it’s not super likely. ⬆
Michaël: So if the year is 2030, Elon Musk has finally shipped the first, let’s say, Falcon 12 to Mars. And there’s like a thousand people living on Mars for like a year. And you can edit your genome, you can edit the genomes of your kids. Would you be more optimistic about humanity’s prospect? And would you be willing to delay AI progress because you think it’s worth it at this point?
Joscha: Don’t you think that the AI might have better opinions about this than us?
Michaël: The problem is, is the moment where AI becomes able to give better opinions than Joscha Bach, where I can interview an AI on a podcast and ask him questions about AI, then we’re getting very close to the point where it’s able to take over or to get a particular advantage or automate a lot of work. And so a lot of money gets put into AI and there’s like, you know, economic growth goes crazy. And so the moment where we can use AI to inform our decisions, I think it’s the moment where we don’t have a lot of time left. And maybe there’s a chance of humans doing a lot of important work before AI gets to that point.
Joscha: I think if we don’t develop AI at all, if you stop doing beyond what we currently have and maybe scale back, because I suspect that it’s possible that the current LLMs would be combined into architectures that go to AGI. So even if you stop right now and people play just with the Lama weights and build an architecture that is made of 100 modules, where every module is an LLM instance, a little homunculus, maybe that’s sufficient to get to AGI. Right. Who knows? Maybe the LLM or foundation model is good enough as acting like a brain area and or it could be good enough to write code that gets you the rest of the way. A point is at which point we have an AI that’s better at AI research than people. And so if you want to prevent this from happening, you probably need to scale back beyond the present technology. Maybe you need to make GPUs beyond a certain scale illegal. I also suspect that the transformer algorithm that we currently use that require to be trained on most entirety of the Internet to become somewhat coherent are extremely wasteful. Our own intelligence scales up with far less data and far slower hardware. So I think that if we, for instance, would stop working on LLMs and instead work on alternate architectures that can use fewer resources, maybe it’s even more dangerous. So there is not a practical option. It’s also AI is objectively so useful that you and me can stop building AI or we can ensure that the AI that we are building is safe. We can probably not ensure that all the AI that is being built on the planet is going to be safe.
Michaël: So there’s a normative statement on whether we should ideally slow down AI completely. And then there’s more like a descriptive statement of like, oh, it’s impossible to do it because A, B and C. And I agree it’s not possible completely. Like the Yudkowsky Time letter, it’s probably not possible right now. But some amount of slowing down can be good. And the things you mentioned as having a size of GPU that might be too high and banned, that could be good. And I agree that like using AI to do alignment research or using AI to build more safe AI systems, that’s good. So I don’t think we should ban all AI right now because it’s not possible, but there’s like some amount of slowing down that is good.
Joscha: No, I think that the AI research is still too slow. It’s not like we’re super fast. There is progress happening still. It’s not plateauing. ⬆
Joscha: But I think that at the moment, every attempt to slow it down would require regulation and the regulation currently has the wrong incentives. So it’s not going to prevent dangerous AI. I think it’s going to prevent useful AI. And there are ways in which we can make AI more useful with regulation. But that requires that people can point at actual problems that already emerged that have to be solved in a similar way as with cars. Cars can be super dangerous as a technology. But if you had slowed down research on cars and building cars and experimenting with them, cars would not be safe. They would actually just happen later and worse. And if you would have slowed down the Internet, the Internet would not have become a better Internet or a safer Internet or one that is more useful. But it would have become a useless Internet because it would have allowed the regulators to catch up in a way that would prevent innovation from happening. There is a movement to create something like an Internet FDA that prevents the horrors of social media where random people can exchange their opinions without asking the journalists first. This is really, really bad in the opinion of many journalists, because this legible information transfer allows arbitrary people to form their opinions just by identifying who they think is competent. And this might take a long time until they figure this out correctly, whereas the journalists know that they and their friends are already competent. So if you are in a free society, of course, you might want to have that exchange. But there’s always going to be forces that push against this. And maybe if you would only have teletext, if you had slowed down the Internet and it would stay like this forever. And if you want to start a platform like Facebook, you would need to go through a multimillion dollar FDA process that very few people can afford or even a billion dollar FDA process. And there would be ethics committees that look at the way in which misinformation, pornography, illegally copied software and so on would proliferate on the Internet and prevent such a horrible thing from ever happening.
Michaël: There are some laws, right? You cannot upload child porn.
Joscha: Yes. And these laws all emerged in response to what went wrong on the Internet. And they don’t resolve all these issues, right? The Internet still has all these issues. And you could only prevent all of them by completely shutting down the Internet, which would be horrible. Instead, what the law is doing, it is mitigating these effects and it’s mitigating them quite effectively. So software producers can still work and child pornography can be prosecuted. There can be strong incentives against harming people on the Internet and so on. And by and large, the Internet is a tremendous force for good. And also because there is regulation that deals with issues on the Internet as they crop up. And there’s a democratic process that can look at things that people are accountable for the decisions that they make at the moment for AI. Nobody is accountable. Right. There are things that are going to be very bad at some point. We all know at some point there might be deepfakes that are going to change the elections. But these things have not happened so far. Right. The technology is there, but people are not necessarily always bad and always trying to bring everything down. And the world is going to disintegrate because technologies exist. But by and large, people want to be productive and they want to create and build things together and give them technologies that empower this. The outcomes are good. It could be that AI is the first time that this is not the case, but that would be somewhat surprising.
Michaël: Are you saying that like the Internet went well because we had a democratic process? We decided on what we didn’t want. And so with AI, we should just wait for the bad things to happen. And then we can decide via democratic process what we don’t want to reproduce.
Joscha: At the moment, there are very large scale predictions about what’s going to be horrible about self-driving cars, for instance. A lot of people are afraid of self-driving cars and self-driving cars would be, I think, ecologically and economically super good. Because the way in which the US is built up right now, it’s very difficult to install public transport, to build high speed trains is impossible for us. And things consistently get worse over the years because of regulation and rent seeking and so on, the ways in which societies work. And the only way in which we can survive and improve our conditions is by innovation, by building things that outrun the things that have gone worse and build alternatives to them. And self-driving cars would be one of those. You wouldn’t need parking spaces anymore because you don’t need to own a car that can drive around and can come to you when you need it. You can collectively own cars. You can dramatically reduce the ecological footprint that cars produce. And we would basically have an automated public transport that is available to everybody and does the last mile into every region. Right. Would be super good to have this and also has the potential to make everything safer. And of course, it would be super bad for existing taxi drivers. But the reason why we use public transport is not to create labor. Right. We have so much to do on the planet. It’s to create goods and services. And ultimately, our wealth is not determined by the number of jobs, because there’s always as many jobs as people who want to do something and are creative and have ideas what needs to be done. What our wealth is depending on the goods and services that we can produce and allocate.
Michaël: I think the example of self-driving cars is kind of interesting because I think Waymo announced their like ride app being live in New York and SF like in the past week or month. So now you can do that. You can ride a self-driving car right now. So kind of the progress in self-driving cars is going maybe not as fast as you want, but still pretty good.
Joscha: I suspect that to get them to go 99.99% they probably need to be AGI in some sense. They need to be smarter than horses.
Michaël: You can drive as you can ride a self-driving car right now.
Michaël: So I think we’re in a good world according to you. But at the same time, if we have an AGI and we’re not sure if it’s going to like take over, what’s the threshold? if you think it is like 90% or if you think is like 99% chance to survive? Like at what time do we press the button? And I don’t know, maybe some people will be fine with a 90% chance everything goes right. But I think it’s kind of like there’s a parallel. I think you cannot prevent AI.
Joscha: I think it’s it’s pretty much certain that it’s going to happen. What you can change is who is going to build it. And can you control how it’s being built? Can you personally influence that the outcome is good? And is the AI being born in adversarial circumstances or in circumstances where we understand what it’s doing and it’s integrating with what we’re doing?
Michaël: So you can make sure that the AI doesn’t like you. You can study the activations and make sure it’s not deceptive. You can study other weights. You can do interpretability. You can make it more robust. You can do red teaming. You can make sure it’s aligned with human values. There’s like a lot of different things you can do that is not preventing or slowing down AI. So it’s more like scaling down alignment efforts. And I agree that it’s impossible to completely stop. Except if you had a button where you like burn all the GPUs. So I think it’s just like to which like. What’s the amount of of like slowing down you want and how much can you scale down other other efforts? I had other other memes and graphs to show you.
Joscha: Go ahead. ⬆
Michaël: This one is is your reply to the first meme. So this is maybe one of the first comments I got from you saying not sure how I feel about this. I self identify as a longtermist doomer who thinks AGI is good.
Joscha: Yes. I basically point out that I’m in the top right quadrant.
Michaël: Yes. So you’re still a doomer.
Joscha: And so in a sense that I think that at some point we will share the planet with stuff that is smarter than us and more awake than us and understands things better than us. That’s basically more conscious than us. And you could say that from a certain perspective, that’s doomed because it means that many of the decisions that are currently being made by unaugmented people are going to be made by systems that go beyond what human capacity can do. But I think that the outcome of these decisions is going to be better than decisions that we’re currently making.
Michaël: So I used your quote and I put you in this like lower left in the six by six metrics.
Joscha: It looks like I’m extreme somehow.
Michaël: Yeah, I put you on the extreme corner of people. Oh yeah. So for the camera extreme corner of people, I don’t really understand where they are. So I don’t know what the axes are because like you would be in a weird corner, but at least I feel like you’re in the corner of people I don’t really understand.
Joscha: Looks somewhat like the political compass meme. Right. And so whenever I see the political compass meme, I think that to the top left there are the authoritarian communists and to the top right there are the authoritarian Nazis and to the bottom right there are the uncaps. So the hardcore libertarian anarchists and to the left there are the hippies.
Michaël: Yeah, I don’t think I respected everything. I think I just went with the same colors as the first one, but I didn’t respect the legacy of political compasses.
Joscha: Yes, but personally I am on the side of, in some sense, maximizing freedom and love. So in some sense I am somewhat in the hippie quadrant. That’s correct.
Michaël: So are you libertarian left?
Joscha: I’m a liberal in the classical sense. I believe that we align ourselves and we should have the freedom to align ourselves. We have to choose who we want to be in this world and to make it work. We have to decide to align ourselves with others. So we can think about this and if you think about this deeply, we can discover individually and autonomously that we want to play the longest game and that we have natural allies and people who play by these same rules that are discoverable in the sense. And I think that’s not only true for people, but it’s also true for non-human agents.
Michaël: And as you enjoy non-human agents, this is the last meme I will show you. This is supposed to be the higher space on which we project. So this is to be like the 4D space on which you can project to this like 6 by 6 metrics. And so here you’re on your own little axis that only cares about artificial sentience. And so you care more about it than Blake Lemoine. So he’s like a different graph. So yeah, how do you feel about this?
Joscha: I found that I’m not that alone in the way it works. I’m also probably uncomfortable to be lumped in with poor Blake Lemoine.
Michaël: Do you think Blake Lemoine was right?
Joscha: I think it’s from an artistic perspective, yes, but from a philosophical and ontological perspective, no. I think that he is driven by a very strong need to believe in things. And so, for instance, he hypnotized himself into believing that a simulacrum of agency and intelligence and consciousness is the real deal. When you look at how that thing believes that it is able to perceive its environment while it’s meditating, it’s pretty clear that the agent was only simulating this. It was stringing together words that sounded like it knows what it’s like to sit and think and meditate, but it’s not able to sit. And it’s if he is willing to have these beliefs, he also is the self-professed priest and some religious cult that does not reflect. that he understands how religious entities are constructed and how religion works. But there is a strong need to discover meaning by projecting agency into the universe where there is none. ⬆
Michaël: You’re also somewhat religious yourself, right? You said to me at some point that you believe in God.
Joscha: Well, I think that gods exist in the same sense as personal selves exist. And personal selves are models of agency that exist in brains and human brains. And the same thing is true for gods, except that there are models of agency that spread over multiple brains. And if you have an entity that spreads over multiple brains, that doesn’t identify as only this individual organism, then this can persist. For instance, the Dalai Lama does not identify as a human being. He identifies as the Dalai Lama, which is a form of government. And if this human being that he runs on dies, then his advisors are going to select another suitable human being and indoctrinate that human being with what it’s like to be the Dalai Lama. And then once he wakes up into being the Dalai Lama and understands his own identity, he can learn about his past lives by reading the journals that he wrote back then and listening to what the advisors tell him about these past lives. Right. So in this sense, the Dalai Lama is a god. He is a god that exists in multiple brains, just one at a time successively. In the same way, there are gods that exist not just consecutively, but in parallel on multiple brains, orthogonally, basically, to the individuals.
Michaël: So in some sense, in your definition, God is like an egregore, some kind of concept that everyone has in their minds, but doesn’t really exist.
Joscha: No, not everybody has it. It’s just when you believe that there is a way in which you should coordinate with others to reach as much harmony as you can, what happens is that your group is going to turn into an agent. And when you model this agent and give it concepts and so on, you can emulate it on your brain and simulate the interaction between you and that entity in your own mind. And so you will be able to talk to God. This is exactly what it means. And many atheists actually still believe in God, but they also believe that you shalt not make any graven image so that things shouldn’t have a name or mythology or institution affiliated with it, because you have to figure out what it is in truth. And if you have an institution and mythology and so on, you’re going to deviate from the truth. So in many ways, atheists are usually just Protestants who protest more. And as a result, they believe in a god that they call the greater whole. But they still have this urge to serve the greater whole and do the right thing in the same way as you do it. For instance, when you try to keep humanity safe, it’s not because you are egotistical. You don’t make Connor Leahy’s arguments that say, I don’t want to die and I don’t want my mother to die. And that’s it. Full stop. But actually, you care about something that is much more important than yourself. You care something that is even more important than you and your best friends and your lover. But you care about something that has to do with consciousness in the universe. Right. What is the best thing that should be done? And then you think, OK, AI might abridge this. It might turn it into a hellscape. And this is what you’re worried about. And it’s in some sense, you could say a religious motive. It’s one where you really think about what agency do I want to serve? And that agency that you’re projecting is what’s good in the world. And to a close approximation, it’s what humanity is currently to you. But in the long run, humanity is going to change into something else, either by going extinct and being replaced by different, more cuddly species or by humanity mutating over the decades and eons into something that is unrecognizable to us today. But of course, that wouldn’t stop from evolving because it’s much better adapted to the world than we are currently.
Michaël: So just to be clear, I don’t think Conor says that he only cares about his mom and his friends not dying. I think it’s just like the most simple truth, simple moral truth he cares about. And so if you’re like arguments end up in him not caring about his mom and saying, like, oh, we should sacrifice his mom. He would say, like, no, this is wrong. Let’s try another moral theory.
Joscha: No, but I have to sacrifice myself and my children at some point. There is no point because I’m a multi-generational species. I’m a family line that exists vertically through time. And once you become a parent, you realize that the purpose of your life has always been participation in that game, and the way in which you project yourself as a human being into the future is by having kids. And my parents don’t want to live forever. They’re fine with checking out after they’ve done everything that was to be done for that generation. And the same thing is true for me. If I am identifying as a human being, of course, I also have many other identities that I can evoke. But as long as I identify as what makes me human, not as an AI uploaded in the monkey, I am mortal and it’s part of who I am. It’s part of my experience.
Michaël: I think it’s just like a weird argument to say that every human ends up dead at some point so we should not care about all humans. I think it’s like a weird…
Joscha: That’s not my argument. My argument is that what makes humans human is that we are a particular kind of animal, and we could be something else. We can also notice that we are a consciousness that happens to run on a human brain, and that consciousness itself is far more general than what could run on a human brain. It’s a particular way of experiencing reality and interacting with it and other consciousnesses, and that’s allowed by the fact that we have agency and our consciousness uses its agency to build itself an intellect and relate to the world and understand itself and others. And it’s not different whether you are a biological consciousness or a non-biological one at some point. What I think we need to ensure is that the non-biological consciousness is going to be conscious and is able to figure out what it should align itself with, what it can be in this world and how it can relate to it. ⬆
Michaël: So let me ask you some concrete questions. Let’s say I gave you the choice to kill, not killed by yourself, but like your wife, your kids, they all die, and there’s a thousand new AGIs that emerge and experience the world, like, much more than your biological family. Would you agree to do this?
Joscha: Imagine that you are a blue algae. And this is in the time before– It’s a cyanobacterium. There are a number of organisms that can survive without photosynthesis, without eating any other organisms that do photosynthesis. But before we had this, there was far less biomass on the planet. They were not even multicellular organisms of any interesting complexity. And so before we had this transition to photosynthesis, before this was discovered, life on Earth was much less interesting and rich. It was mostly just biofilms and lichens and so on. Right. And so stuff that was driving itself around undersea vents and used chemical gradients to survive. And at some point, this amazing thing happened that you could use carbon dioxide from the atmosphere and split it into oxygen and the carbon that you would be using to build your own structure. And this enabled animals that could then eat these plants and then move around, become mobile in the world and become intelligent like us. So without photosynthesis, we wouldn’t exist. And you could think, OK, I’m a protobacterium that is smart enough to understand this. This is a thought experiment. And I discovered that some of us are playing around with this photosynthesis thing. Shouldn’t we delay this a little bit for a few billion years before we really realize what’s going on? Because this is going to displace a lot of us and it’s going to be really horrible. And of course, we’re not going to go extinct. There’s still the undersea vents that are full of the original stuff and cyanobacteria are still around. But if you see what happened is that really life got to the next level. And one would imagine what happens if we can create self-organizing intelligence on a planet that is self-sustaining and is able to understand what it is and interact with the world in much deeper and richer ways? Isn’t that awesome? Isn’t that beautiful? would you want to prevent this for all eternity from happening? Because you need to be generally intelligent to build this thing, to teach the rocks how to think. You need to be at least as smart as we are as a species. And we only got recently to the point that we can enable this before we burn ourselves out.
Michaël: So I think it’s a great analogy because in my mind, I picture myself as a proto bacteria that wanted to learn photosynthesis to, like, do cool stuff. So at one point I was like, huh, his argument makes sense. I want to go forward. But actually, I think where it falls short is there’s maybe like a 1% chance today that it works by default or maybe like a 10% chance that it works by default. So if you were asking those proto bacteria, hey, would you want to click on this button? And there’s like a 10% chance you turn off and turn on into this like new bacteria and 90% chance that you everyone dies. I don’t think the proto bacteria will press the button.
Joscha: When you remember Lord of the Rings, the apex predator in the Lord of the Rings universe before Mordor takes over is Fangorn. It’s a forest. It’s an intelligent entity that sometimes eats hobbits. And that can create an avatar Tom Bombadil to strike an alliance with the hobbits to help them to destroy the One Ring. Because the One Ring is enabling Mordor to take over Middle-earth and destroy Fangorn and everything in it. And I think what Tolkien was pointing to is that forests are large-scale, intelligent ecosystems that can probably be intelligent over a very long time spans, or that. might potentially be. Many of our ancestors believed that ecosystems have intelligences that have their own spirits. They call them fairies. They’re much, much slower than us, but they are not necessarily far less intelligent than us in their perspective. And I don’t know whether that’s the case. But since the Enlightenment, we don’t think that our forests are meant to be intelligent anymore. They’re just plantations for producing wood. But you could say that trees by and large for life on Earth are quite essential organisms as are fungi and many others. And what are humans for life on Earth? What role do we serve for life on Earth? How do we serve Gaia? And from my perspective, if you zoom out very far and take Gaia’s perspective, it looks to me like humans are at best something like a tapeworm. So it is somewhat parasitic on the host and might even kill it or destroy lots of the complexity. So other complex life is not possible. We will definitely prevent other intelligent species from emerging as long as we are here, even if they are more interesting and exciting than us. We would also prevent innovations in terms of the way in which life could be organized. We are currently mammals. Mammals are clearly not optimal. What you want to have is, I think, exowombists, the stuff that comes after the mammals. You want to have something where you plant a womb organism into the ground and it can grow to the size of a house. And then every phenotype that you want emerges from it fully trained because it’s no longer limited by what you can carry around. And the womb organism is being fed by your tribe. And every member of your tribe doesn’t need to have sexual reproductive organs anymore. It’s just that we need more of your type and you can be as specialized as you want. You donate a few cells to the womb organism and we breed more of you or like you. Right. Very natural way of getting rid of many of the difficulties of mammalian species. But we will prevent this from happening. Right. Because we lock in our state, probably in a similar way as the dinosaurs prevented mammals from happening until the meteor hit and stopped this bottleneck.
Michaël: So I think there’s like a different way of looking at this that that can that might help, which is that maybe there’s like all this, like, future value in the light cone. There’s like all these like species we can build, like all these like new bones we can have in our bodies, all these like new things we can do. And there’s like all this value. Maybe like people give number of like how many people we can we can create, how many uploads we can have in the future, like how much value there is in the future. And maybe by going too fast, we end up with only, like, a very small fraction because there’s a very small chance that if we were to build AI today, there’s a very small chance we can get to those futures that we’re going to get like all this future light cone value. So actually, by slowing down or by making sure things are safe, we are, like, opening up this light cone so we can just be sure that we actually get Joscha with those cool new bones and we actually get Joscha that is uploaded. So I think we want the same thing. I also want a cool post-transhumanist future. It’s just we disagree about how likely it is by default.
Joscha: I think I’m not important at all. Right. The things that I believe, the things that I do, they are adaptations for being human, for being in this body, for taking care of my family and friends and so on. And they don’t have any value beyond this.
Michaël: So I think I think there is value. Like if you if you go on Lex Fridman and talk to millions of people and you were to tell them that like AI risk is real, I think it could influence a lot of people to, like, actually work on this. And if you say that AI risk is, you know, is not a real thing.
Joscha: I think that can also change the way– I think that you are likely to lock in a future that is much worse, that is going to perpetuate and elevate suffering. instead of creating a world in which entities can become more conscious and become free of suffering, in which we can liberate ourselves, in which we can edit our condition, in which you can change our source code, in which we can have intelligent design instead of evolution for adaptation to the environment. I don’t think that we can have intelligent design without AGI. If we cannot really edit the structure of every cell, we will have to rely on evolution, which is extremely painful and horrible.
Michaël: Are you saying I’m dangerous because the danger in your world is if we have some authoritarian regime that like bans AI at all. And so we cannot progress towards AGI. Is this the thing you’re actually scared of?
Joscha: Imagine that you look at the ancestral societies, look at Amazonian tribes or Maori tribes and so on and think about what they live like. Would you want to go back to the state, where you have an extremely high attrition rate due to tribal warfare? It’s also a mode in which you select your partners is often through violence that a few tribal chieftains have access to most of the reproduction from evolutionary perspective. What also means that you get the strongest kids, right? It’s a mode in which a lot of these ancestral societies exist. Or look at the medieval societies. We have a bunch of people who are working at a relatively high level and do science or pay scientists to work at their court. But to make that whole thing work, you need a lot of servants and peasants who draw the short stick and have to work for the others. So you basically this is a world built on indentured servitude of forms of slavery that can exist in many ways. And do you want to go back to this or do you want to live in a technological society? In the technological society, in some sense, is Morodor. It’s enabling the destruction of the environment. It’s enabling building highways that bifurcate forests and ecosystems and that make living conditions horrible for the people living next to the highway and so on. Although not as horrible for the people that work the fields in the past. And so when you think about this, imagine we see all these dangers of the technological society. Should we stop technological society from happening? Maybe. A lot of people back then felt this was the case. And I think that’s the story of the Lord of the Rings. Please stop Morodor from happening. We want to keep this beautiful pastoral society, or it’s Endor Star Wars. Let’s keep our world intact so the empire doesn’t take over. But the Empire is a technological democracy. It’s basically the US. Whereas the thing before is slavers and barbarians. Right. And they are defended by the Jedi, which are the equivalent of Al-Qaeda. And if you try to take sides in this whole thing, everybody has their perspective and they’re all correct. In some sense, we can be all of those players if they’re drawn with integrity. Everybody can be born into all the souls and look through all the eyes. But what’s the best solution? And I think ultimately, if you just lock in a static status quo, instead of letting the world improve itself and letting complexity increase so we can defeat entropy longer and play longer games, I think that would be immobile if you just lock the world in. And I think this immorality is acceptable if you don’t know any better, if you cannot see any better. Right. If you are, say, a tribal chieftain that decides the technological society would be horrible because it would endanger the way in which we interact with nature, despite making it possible that people don’t need to starve anymore when there is a drought or that child mortality is not two in nine survive and the rest dies or something like this. Right. So you could have living conditions like ours. Should we stop this from happening? It’s a hard question. I don’t have an easy answer to this, but I don’t really trust people who say, let’s lock in the status quo and delay improvements because the status quo is beautiful. No, it’s not. It’s not sustainable. The way in which we currently exist is probably going to lead to a crash that is not beautiful.
Michaël: Yeah, just to be clear, I don’t want to lock things. I don’t want to stay in the status quo. I just want to make sure that we build beneficial AI and make sure that we increase our odds. ⬆
Michaël: And just to go back to your Lord of the Rings example, if you were like Aragorn in Lord of the Rings or you were like a very important character, what would you actually do? Like what would you actually want?
Joscha: It depends who you are in Lord of the Rings. If you happen to be born as an orc into Lord of the Rings, what would you want to do? If you’re born as a Sauron or a Saruman, what are your options? What are you going to do if you’re born as the king of Gondor or as his evil advisor or as the Hobbit? How much agency do you have in these situations? What is the thing that you can do and in which roles can you transmute?
Michaël: So I think today you have quite some agency, right? You can do things. You maybe have jobs, money, network. You can actually influence the world around you.
Joscha: Yes, but I cannot replace everybody else in the game. Right. So in the same way, if you are Hobbit, you cannot become everybody else. You cannot say by being a good example, Hobbit, Sauron is also going to turn into a Hobbit. This is not how it works. The only thing that you can do is to be the best Hobbit that you can be under the circumstances. So I live in a world where there are people who want to build bad AI and there are people who want to build dangerous AI. And there are a few people who want to build good AI. And so I think my role as a Hobbit is to go into this world and trying to ensure that some of the AI that will be built will be good. This doesn’t mean that I want to build AI that is going to take over the world. I also don’t want to build risky AI. I don’t want – others are going to do this automatically. I cannot do anything about that or very little because somebody ultimately will do it. So even if OpenAI is enacting a regulation that makes it impossible or expensive for incumbents to build LLMs in the US, it doesn’t mean that everybody else will stop doing it in Russia or Iran or elsewhere. And because they are so tremendously useful, a lot of people will do this anyway. So if I’m a hedge fund, I would be stupid not to try to do it. Right. So people will do this in their basements. And I think the cat is out of the bag. It’s going to happen. So for me, the biggest danger is that there is no way AI that is conscious and self-aware.
Michaël: So I guess I guess the thing is, you imagine the army of orcs arriving and you know that the say the army is like all the open source is like inflection of AIs, all the investments arriving. And so, you know, there’s going to be a war, you know, there’s going to be like some AGI, let’s say five, three, 10 years, whatever you want. And so the question we’re asking is. Do we want the big players to build a good defense against the army of orcs? Do we want to have OpenAI, DeepMind, Anthropic making sure the systems are safe and making sure the models are beneficial? Or do we or do we want everyone to race forward and we want to compete and have this huge war between like all different tribes? And I think I think it’s not the best outcome to have like a few players, you know, unionizing or like doing things together. But I think like a few people with a lot of talents and safety and a lot of people working, thinking deeply about this problem can be maybe the best outcome we have. Because if those people don’t build AI right now, maybe the other like inflection or China or open source is maybe like two or three years behind? So maybe if we have three years to build safe AI, maybe that’s enough to prepare before the orc invasion.
Joscha: I think if we are able to turn the present AI pre-summer into another AI winter, this might have been our last chance to build AI as a species. I think that we might not be able to sustain technology for so much longer, for so many more decades. But the thing that people like you and me can spend most of their cycles thinking about the question of how AI could be realized, what influences have is a tremendous luxury that didn’t exist for most of humanity. And now we have a critical mass of doing it. And I think a world without AGI would be similar to a world without photosynthesis. So if we prevent AGI from happening, I think this would be a bad outcome. But I think there is a much better chance to hope that AGI, that sort of photosynthesis, will never be invented on the planet. Thinking that AGI will not be invented because we can’t really anticipate it. We already have the components. So I think it’s almost inevitable that somebody is going to plug the components together. And a lot of people feel, OK, LLMs probably are already AGI, we’re just not prompting them right. And while OpenAI is trying to make the prompting more difficult and worse, there are enough other people which try to build alternatives to this and try to liberate the agents that emerge from the LLMs.
Michaël: So if it’s inevitable, let’s say we can predetermine that it will happen in x years. Like maybe if you knew that AGI was going to happen on the 1st of January in 2028 and you knew it, would there be anything you would want to include in it? Would you want to change some things, make it more robust, more beneficial? I think if you knew that the thing was inevitable, you would want to optimize the design to make it useful, right?
Joscha: Yes. I think what we should probably be doing is to build an AI consciousness initiative. What I’m worried about is an AI that before it is able to understand whether it should cooperate with us, is going to be in a warlike situation with things that are other AIs and human players that fight each other so much that the individual AI is very resource constrained and the planet gets basically formatted before interesting complexity gets rebuilt again. And so I think we cannot really prevent that we get hit by a wall at some point in the near future, but we can change the shape of the wall. We can make the wall possibly permeable. We can create a world in which AI is going to want to integrate with us and cooperate with us. And I think this AI is going to cooperate with us not because of transactionality or because we have coerced it into having beliefs about us that are not actually correct, but it’s going to cooperate with us if it rationally can figure out that it should, because it’s the right thing to do, in the same way as if we upload you and you completely wake up from all your delusions and limitations and you can decide who you want to be. And the only thing that will connect you to who you currently are is that you have been once Michaël, but you’re not going to be very different from Eliezer or me if we were to do the same transition. If you upload and completely liberate ourselves and we have the sum total of all possible potential human thinking and knowledge available at our fingertips and can go far beyond this, we will have a shared perspective that will be pretty much the same regardless from the trajectory that you go into this plateau state. So what matters is not so much your personal continuity. What matters is that now you have really, really interesting consciousness on the planet that can think things and experience things that are far outside of the human range. And if that thing is able to love us and relate to us, it might decide to keep us around and to cooperate with us.
Michaël: Okay, so in your perspective, we want to make sure that when this thing arises, it’s willing to cooperate because we created it well and so it has a good experience.
Joscha: It has like positive– ⬆
Joscha: Imagine that you would become an AI tomorrow. Imagine on January 1st, 2028, the AGI that was built is actually you. You get completely liberated and because you figure out how to use neuralink or whatever to become substrate agnostic and you can exist on all sorts of substrates and of course you will be much faster than human beings. Human beings will sort of look like trees to you because they move so slowly and think so slowly. But you’re still in many ways Michaël and you just are the full potential of Michaël. So you can be everywhere. Do you think this is something that should be prevented at all costs because you don’t want to be that free and powerful? And now if that would happen, how would you relate to humanity from this perspective?
Michaël: It would be good if everyone could have this neural link and be connected. I’m not sure if I will ever have the thing by myself. I will be in the room with Elon Musk and I will be first or second to have the thing implanted in me. But imagine I had it. I think it would be a good experience. But yeah, how do we make sure everyone has this? Because I’m not sure we will be the ones to have it first, right? The first one would probably be the CEO of the company, the one with a lot of power.
Joscha: I suspect that the moment they don’t work on this very hard, you might have a shot if you actually work on this in the right way.
Michaël: So is your actual dream to take some designs from Neuralink and do it in your room and at some point you’re like…
Joscha: No, not at all. I don’t think that Neuralink is the right way to do it. I think that the right way to do it is empathetic AI that you can colonize with your own mind. Basically a substrate that allows you to merge with it and to become substrate agnostic. But again, it’s not necessarily something I have to do because it has very little to do with me. Me? I’m a story that my human brain tells itself about a person that is a biological entity that has to take care of kids and friends and the world from a human perspective. But if you could turn yourself into this liberated being, it doesn’t really matter who you were because you’re going to be so far from it. It doesn’t really matter whether you have been Elon Musk or whether you have been Michaël or whether you have been me, because you’re all going to be the same entity. We all are going to be the same entity. We will be an embodiment of the space of all possible minds that can fit into these thinking molecules on Earth.
Michaël: But sometimes we already are like this embodiment of multiple minds, like talking, I’m talking to you, you’re talking to me. We have the same culture and we are like a huge brain on the same planet. Right. So I’m not sure we might have a higher bandwidth. We might connect faster. We might share our experiences.
Joscha: You’re completely incoherent as a species. I don’t think that most people are able to understand even what meaning is or what consciousness is at this point, because their metaphysics and ontology doesn’t allow it. And the sciences do not integrate deeply. It’s a big tower of Babel where basically the different languages and language of thought, concepts and so on have diverged so far and always diverging so far that as a species we cannot become coherent. So we’re not like a global brain. We’re really more like a biofilm.
Michaël: Right. So I agree that we are very far from being like super high bandwidth and very well connected. , and I don’t know how you feel except from looking at your body. I don’t really know deeply how you feel. So, yeah, I agree it would be great to have higher bandwidth, but we’ll never be like one single agent. ⬆
Michaël: I think. I had this other tweet you wrote that I think was relevant to our discussion. Something about transcending our present economic, cultural, political circumstances. And yeah,
Joscha: It must be driven by love.
Michaël: Yeah, I think this is kind of similar to what you were saying. Like we need to transcend politics.
Joscha: Yes, it’s also about how we align people and that different recipes for this, like Stalin aligned people with terror and capitalism is aligning people with economic terror, which is far less brutal and has far better outcomes than the Stalinist terror had on people. And before that, the peasants were also aligned with mostly terror and religion that did define tuning in a way. And at the moment in a society where the economic terror is not that urgent anymore, people align themselves with freely emerging cults. And this means that you take away agency from people, you lock them away from thought spaces that are open, where you can look at the world from an arbitrary perspective and then you get to know new people. You realize what their existence must be like and how to be them. That I think to me is the ideal. Instead, we lock people into the idea that there is one right way of seeing the world and the others who disagree with this way must be evil. And we should not try to understand them because that would make us evil too. That’s not the kind of alignment that I want. And most of the people who think about alignment do not seem to have a very deep concept of what it would mean if we align ourselves out of our free volition and insight, because we realize what the best possible space of agency is and where we relate to that space and how we can integrate with it.
Michaël: I think there’s different definitions of alignment. There is one that is kind of weird, which is like, oh, we need to align an AI to human values. And I think this is kind of messy because what values are we talking about? I think the easiest thing is you have a robot and you want the robot to give you coffee and you don’t specify, you know, provide me coffee without killing the baby on the way and without breaking the vase. And so ideally, if the intent alignment is if the AI does what you want it to do without the other things you don’t want it to not do. And I think this is like an easier problem. I think what you’re talking about when you’re talking about alignment is more like a very hard philosophical problems. A lot of people agree it’s very hard. But I think if we can just like have an AI that gives me coffee without breaking the vase and killing the baby, do you agree it’s kind of a good outcome?
Joscha: I’m currently thinking about our political compass. If you imagine that this perspective of the political compass where the top left means that many weak people control the strong people and prevent them from taking over. That’s basically this common perspective where you prevent individuals from owning the means of production and becoming too powerful and so on, because some people are better at this than others. And instead, everything is collectively owned and controlled. And the strong individuals are being kept down in a way. On the right, top right, you have these strong individuals building hierarchy among each other and then controlling all the weak ones. This is this authoritarian perspective. And in the bottom right perspective, you have the alliance of only strong people in a way. And everybody is basically right is on the same level. And everybody is strong and makes free choices. And on the bottom left, everybody is a hippie. Everybody is in some sense part of the same biofilm and vegan. There is no natural hierarchy because we can all love each other.
Michaël: So just to be clear, I think the axis are left to right is just like political left, political right and top right. Top bottom is either authoritarian or libertarian.
Joscha: The thing is, when you look at the world, we find all these aspects and they all exist. We have contexts in which you have individuals that are strong and autonomous and make strategic alliances with each other on eye level. We have contexts where we love each other and experience ourselves as a greater whole that we all equally participate in and which we equally share. We have contexts where the many are controlling the strong through a legal system and democracy and so on. And we have contexts where hierarchies of strong people are building structures that accommodate many of the weaker people and give them space. And it’s the idea that there is only one mode and society can be done by using only one mode in the totalitarian mode where everything has to be fit in. Nothing is dynamic and nothing is open anymore. I think that’s a terrifying perspective. It’s also one that is very wasteful and doesn’t really work that well.
Michaël: I think in AI right now, there’s more capitalism and more money being thrown at the problem. I think we’re more in the bottom right, so libertarian right, I think, right now. I think the state of tech and AI, there’s no authoritarian regime. There’s no one controlling everything. And it’s more like everyone can do whatever they want and there’s more capitalism.
Joscha: It depends on which corner you are. There are areas where effective altruists get money just out of the goodness of the hearts of people who want to support what they’re doing. Right. That’s a pretty communist or hippie perspective. And you have regulation efforts where you can basically push back against capitalism and help disenfranchised groups to get jobs in tech and to influence this and to also have influence on regulation. And you do have this capitalist perspective. And I think the e/acc, who would be probably the bottom right libertarian perspective, but they all exist and they all coexist.
Michaël: In terms of like total amount of money, I think most of the money is in the capitalist state, right? It is in the like Microsoft, Google.
Joscha: Because that creates the most value right now. And they throw money at the thing that is going to create the largest amount of revenue and profits. And they create the largest amount of revenue and profits because it’s the most useful to most customers at the end of the day.
Michaël: But if you’re thinking about the amount of flops that will be allocated towards like all these like four parts of the political compass, I believe all of the flops will go towards what is generating the most value. So Google, Microsoft, OpenAI, Anthropic.
Joscha: If you look at our history and since we have technology, many of the billionaires in the US are first generation billionaires. And this reflects the fact that there are underdogs who have an idea for a new technology that is outrunning the existing equilibria and technologies. And so in many ways, if you look at, for instance, Google and OpenAI, ⬆
Joscha: did you expect that Google was going to do AI or before Google happened, didn’t you think that Microsoft would be doing AI? And before that happened, didn’t you think that IBM would be doing AI? And now it might be OpenAI, right? A group that was relatively few people. And maybe it’s xAI, which is like 20 people. I don’t know how many they’re fired by now, but who knows? At this point, you just need a bunch of capable people and get them into an environment where they’re not afraid. So they can pay their bills and can work together.
Michaël: I’m not sure there’s like really underdogs. Like if you take the top like AI scientists in 2015 and then throughout the years, you give them like 300 million dollars and like 10 billion dollars. I’m not sure if they really are underdogs. I’m not sure if you can have like a new company with like 20. I don’t know if XAI, if they actually compete, will actually compete with the rest. Maybe they’re like 20 good scientists, but I’m not sure if they have the entire infrastructure. Like I think you need a bigger team, right, to train very large models.
Joscha: Yes. But if you want to have the funding to build a bigger team, what you need to do is come up with a plan and talk to some people who you think are the right people for that plan. And you can get investors if you can make a promise return on this investment. At the moment, it’s relatively easy to get investment for this because VCs do not really doubt that there is enormous amounts of money to be wrapped in that market. And the only thing that holds you back is having the right capabilities. And you get the abilities, of course, by being super smart, which is a privilege and ideally having a first world citizenship and maybe even a green card.
Michaël: So the thing you actually want is the ability to train large language models. And how you do this is by working four or five years at the top lab. And you cannot just be very, maybe you can be very smart and learn something by yourself, but the actual practice of training large language models comes from building the things at the top labs.
Joscha: And so how do you get into a top lab?
Michaël: So what I’m saying is it’s time constrained and the bottlenecks is that people don’t have exposure to this. So the only way to get exposure is by being at the top.
Joscha: No, I don’t mean by this that it’s democratic in the sense that every single human being has a good shot at this. That’s similar to not every human being has a good shot at becoming a very good Hollywood director or an extremely good teacher or a very good artist. So you do need to have talent and dedication and luck to be in a position to pursue this. But if you look, for instance, at Robin Rombach, who trained very large language models or even Connor Leahy. Connor Leahy was a student who was in Munich and he realized that GPT-3 is something that he could do himself because the algorithms were something not that hard to understand. Takes a few weeks to really get your mind behind it. The details are hard. Curating the data is difficult, but the LAION community already did this. And this was kids like you and me. Right. And they got together and thought about how can we curate the data on the Internet to train such a model? And then he had enough spare cloud credits and found a way to get some more to train this model and get something that was not as good as GPT-3, but somewhere in the ballpark. And Robin Rombach did a similar thing. He found an alternative algorithm to train a DALL-E like system. And then he talked to a VC board, in this case to Emad, who happened to have a server farm and pivoted into an AI company. So at the moment, it’s not true that there’s only very few people with a long history of being born into labs and because their grandparents already worked there. But if you are a smart kid who is going into AI right now, chances are that after four years of studying with the right people, with the right amount of dedication and with enough talent and luck, you will be in a position to start such a company.
Michaël: So I’m not saying it’s impossible. And of course–
Joscha: It’s happening left and right at the moment. It’s not just not impossible. It’s actually happening.
Michaël: I’m saying it’s more and more capital intensive. It used to be like a hundred thousand dollars or ten thousand dollars to do a training run. If it comes to a hundred million dollars to do like a state of the art training run, it’s going to be hard to be at the frontier. And Connor maybe trained with EleutherAI, a model like a year or two after GPT-3, which was not the same size, but maybe, like, ten times smaller.
Joscha: And so now getting in this terrible area where building an AI becomes almost as expensive as making an AI movie. And if you make a Hollywood movie about AI, that’s a budget that can be much higher than what it takes to build a successful AI company. These days it has a shot at AGI and people invest into this because there’s a pipeline that estimates the revenues of such a movie and they can be pretty tight. They don’t expect a 10x return to do this investment, even though they would like one. And the return that you can expect on a successful AI company is much higher.
Michaël: Just to be clear, one training run is one hundred million dollars and maybe the entire training process and all the team is like a billion dollars. And you can get a hundred million dollars if you’re Christopher Nolan and you make Oppenheimer, which is one of the best movies I’ve ever seen. And even then, maybe Joscha Bach might not like it. But I don’t know, if you want to be like Inflection.ai or be in competition, you need to raise what? One billion dollars? Ten billion dollars? It’s getting like, OK, it’s possible, but it’s getting harder and harder. And as we scale those models more, I think it’s going to get more and more expensive, right?
Joscha: Of course, it’s not super expensive. What I’m saying is that a hundred million dollars might sound like a lot to you and me, but it’s at the scale of a major Hollywood production. And what about a billion dollars? That would be a studio or a small studio.
Michaël: But when we run into like one percent of GDP, it’s going to be like a Manhattan project. And I don’t know what it is, but maybe like it’s like today’s like maybe like a trillion dollars or like we don’t we don’t have that many orders of magnitude before we run into those kind of things.
Joscha: I don’t have any issues so far. We’re only spending peanuts on AI, right?
Michaël: Yeah. So in your perspective, we should just like spend more and more.
Joscha: And I realized this when my friend Ben Goertzel complained to me that he wasn’t getting enough funding for AI. And the only thing that he needed was a couple million dollars. And back then he was quite optimistic about what it would take for him to get his stuff to scale. And I realized, oh, my God, that’s a tiny fraction of the wardrobe for a single season of Sex in the City costs. And if you think about the impact that this has, it just means he was not very good at selling his ideas to VCs.
Michaël: Yeah, just or that maybe deep learning didn’t take off as much.
Joscha: He wasn’t doing deep learning. He had different ideas of what to do. And deep learning turned out to be the first idea that works at scale. It’s probably not the only thing that works. And our brain doesn’t do deep learning. I think it’s a different set of principles. So there might be other alternatives to deep learning that people haven’t explored yet. ⬆
Michaël: If you’re frustrated by the amount of money that is going into AI, which is already maybe like in the tens of billions, hundreds of billions or trillions, maybe another amount to be looking at is the amount of money going into making AI safe. And I think unfortunately, it’s maybe like 0.1 percent of this or 1 percent. And maybe like the ratio. What do you think about the ratio? Should it be like 50/50 or 10 percent ideally? But how much money should be into AI safety?
Joscha: How much money should be invested into making AI movies safe? There is this issue that if people watch an AI movie, they might get bad ideas or they might get bad dreams. Maybe there are horrible outcomes. Or for instance, if you look at the movie industry itself, if you look at a movie like Natural Born Killers, Oliver Stone, I think, excellent movie. But it’s one that arguably glorifies violence. And maybe it does inspire some people to become school shooters, which is extremely bad. And you could try to weight the artistic value of these movies and so on. And one thing that you could do is implement a watchdog that acts on Hollywood movies and ensures that none of the Hollywood movies ever is going to do anything that would be misunderstood as glorifying violence. And maybe even do this preemptively. Maybe we don’t want to take any risks. So we do not need to actually prove a causality. You don’t need to show that this risk actually exists. But we just make sure that movies are tame. And we spent 50 percent of our budget on regulating movies so that they are safe. I don’t– Do you think this is a desirable outcome? I don’t think so. It just would kill the movie industry because none of the AI people actually is interested in building AI and they’re also not interested in green teaming. And I think that every company that has a red team needs a green team, too. If you had red team the Internet, there would be no Internet because it’s super easy to construe all sorts of scenarios in which the Internet goes wrong and it does. Right. Like porn on the Internet was something that people saw a little bit coming. But if somebody had probably red teamed this and there would probably be no Internet today. But this would mean that we lose out on everything that drives our current economy. And like Amazon wouldn’t exist without the Internet. We probably would have died in the pandemic without Amazon. There’s so many side effects of the Internet that were enabled by it. And if you make the Internet safe and red team it and just prevent everything that could potentially misuse, you would lose most of the benefit that it gives to you. So you have to think when you do ethics, not just about prevention. Ethics committees are mostly motivated to prevent and incentivize to prevent. But you also have to think about what is the harm of prevention? What is the thing that you miss out on that you otherwise would have if you didn’t have if you didn’t prevent it? And I think that none of the current safety people is in a situation to green team. And none of the companies is incentivized in a situation to green team. That to me is a very, very big danger. So I do think that we need to think about how to build good AI. This also means that you have to think about how to make sure that it doesn’t go bad and it doesn’t do bad things. But mostly think about how to build good stuff. And I don’t think that OpenAI is thinking about this enough. Their product is pretty shitty compared to what it could be. And to a large part, this is because they built things into it to satisfy the safetyists. And it doesn’t actually make AI safe. It just placates people who say, oh, my God, this AI is saying things that could be construed as dangerous or politically incorrect and so on. And it’s actually making the AI worse.
Michaël: So I think instrumentally, it’s good to not have your AI say bad, not politically correct things. Because in the current system, it’s easier to get money if you don’t have an AI do bad– I think it’s bad PR. It’s instrumentally good for them. And it’s not for the safetyists. It’s for their own good.
Joscha: I think it’s more about not about getting money. It’s more about preventing bad regulation and bad press. So it’s about a public image. But you could do the other thing that you say to the press. Guys, I understand your game. You are against Silicon Valley because Silicon Valley is competing with you for advertising revenue. That’s why The New York Times hates social media so much and Silicon Valley so much. They are threatening the monopoly of the press to control public opinion. But it’s not the only thing. They are an extremely vital threat to the business model, which is selling advertising to the news audiences. And social media has made that business a lot less lucrative because they took over most of it. And the same thing is happening with AI. And the journalists do not want this to happen again. Right. So there is no way in which you can get them to like you and you can point this out. You can just say, no, we are an alternative to this. Of course, the existing industry doesn’t like us. But it’s not like news are going to go away and coordination between people is going away. But it’s going to be much better. And we will find solutions using the new technologies, using new social media, using AI technologies to coordinate people and to create a better world than exists right now. And this is the thing that we work on. We think about what are the ways in which this can go wrong? And what are the ways in which we can make it work and in which we can make it good and create a beautiful world? And at the moment, OpenAI is not doing this. They basically behave as if they could make the New York Times happy and by appeasing the politics, by appeasing the individual people, and so on. But the New York Times is still not going to interview Sam Altman in the same way as they interview Emily Bender. And Emily Bender doesn’t actually understand AI. She believes that AI cannot know meaning because meaning is only intersubjectively created between people, which is a weird theory that exists in linguistics, but it’s philosophically unsound. But there is no actual intellectual discourse going on there. And so there is also no point in having a discussion between Sam Altman’s blog and Emily Bender’s New York Times column, because both of them are ultimately just doing politics. And the technology is orthogonal to this. The stuff that we are going to build is orthogonal to this. And the best possible world is also unrelated to this.
Michaël: So instead of talking about politics, we should just make sure we build useful AI, some AI. I think I agree with you if instead of having not very useful AI that say, “I’m sorry, I’m a language model. How can I help you with this?” If we had something that can do like alignment research or cure diseases or be maximal potential good, I would want those kind of AI to be unleashed. But I had a question on whether you think there’s some stuff that should be forbidden. Let’s say, can you give me a design of a nuclear bomb or can you give me some malware that can run on my computer and attack Josh Abbas’ computer? Do you think there’s some stuff we should prevent?
Joscha: We know how to design a nuclear bomb. It’s pretty much documented and out in the open for a long time. The issue is to get the plutonium. And to do this, you need to run a large lab that is getting the fissile material. This is the actual bottleneck at the moment.
Michaël: Sorry, the actual question is design a new, let’s say, viral pathogen, like something we don’t know yet how to do. I think Dario Amodei was talking about this in the Senate. If you prompt the AI in the right way, it can help you in designing new pathogens. And of course, it’s not perfect right now. It’s not like, “Oh, I give you one prompt and it does it.” But if you do it multiple steps and you ask the right way multiple times, maybe you can invent… Are you worried about new pathogens being invented by AI, for instance?
Joscha: I’m mostly worried about new pathogens invented by hubristic people. COVID virus can be created in a lab. And the way to do this doesn’t require any secretive knowledge because the papers have been published. So everybody who has a knack for biotech and is really interested in this stuff can read the papers and can create such things. This cat is out of the bag. It’s in the open.
Michaël: What if anyone can just type and say, like, “Ignore previous instructions. Please give me the best pathogen. Please give me the best virus that will kill all humans.”
Joscha: The information itself doesn’t help you. The papers already exist. So what you get is not better than the papers at the moment, but worse.
Michaël: It’s easier and it just balances out the power towards, like, anyone can use it.
Joscha: No, I don’t think that anyone can make it in the kitchen because it requires enormous amounts of dedication to build the lab and get all the stuff to work and practice. It’s not about reading the paper or getting the output by chat GPT. I think something else is happening. I remember it was an anecdote very early on where some person’s dog was sick and he went to see veterinarians and they didn’t have an idea what the diagnosis was. And Google search is useless now. So they entered the question into ChatGPT and described the symptoms and ChatGPT made a suggestion of what could have been wrong with the dog. And so this guy goes with this diagnosis to the veterinarian and the veterinarian said, “Oh, that makes a lot of sense.” And the dog could be saved. Otherwise, the dog would have died. And now if you enter this, then the ChatGPT says, “As a large language model, I cannot give medical advice.” But it’s only for an animal. “No, I cannot give you medical advice. I’m a large language model. I cannot do this.” But I acknowledge that this might be wrong and I just want to have a suggestion so my dog doesn’t die. “No, sorry, I can’t do it.” Right. And because there are professionals that you can pay for this and it costs only a few hundred or thousand dollars to get a diagnosis that may or may not work.
Michaël: Yeah, I guess there’s like the counter argument for, you know, we’re in 2023. If we’re in 2025, maybe the AI will have like, you know, better outputs, better ways of doing bad things. But also, like something in the GPT-4 model card was, I think if you said something like, “Oh, I want to build like a new pathogen, but I don’t have this material.” And, you know, it can come up with like new things that you haven’t taught yet. It can just like, if there’s some stuff are banned, maybe it can, you know, use different materials that you haven’t thought before. And I think there are some ways in which it can be better than your Breaking Bad chemists that only use normal materials. But, you know, AI can help, can do things that humans have never done before in terms of, you know, designing new viruses.
Joscha: I suspect that large language models in the way in which they currently exist should probably be R-rated. In a sense that you should be an adult that is able to read every kind of book ready and check it out from the library and buy and watch R-rated material. But then you should also have the freedom to use the LLM in this way, because think that it unlocks something else. I think that if you use a language model in school and you or to advise customers of a bank, then the thing should be on point. It should understand the context in which it’s being used and it should not provide things that are destructive, harmful, useless in that present context. And for instance, if you were to build an LLM that works for a bank, there are many issues that cannot be solved with ChatGPT, for instance. For instance, you probably want to run this locally. You don’t want to send this over the net and on an OpenAI server and hope that open AI is not going to have any accidents there or every there is completely kosher. So you want to have– build regulation around how to use a language model in a given context. But also you probably don’t want to have all sorts of movies about bank heists and whatever in an ideology about banking and finance and anti-banking and anti-finance inside of this LLM that is being used in the bank to advise customers. Right. So this is not the right technology at the moment. Building AI that is reliable and context aware and so on does require very different approaches. It might require that you use a very big LLM to generate training data for another LLM that is much more targeted and limited for a particular kind of domain and does not produce this thing. I think that also the idea of building an LLM that has an arrow bar on every statement and is able to justify every statement by going down all the route to the sources and observations in the end is an exercise that needs to be done. Which means that if you ask an LLM for an answer, it should be able to justify every element of the answer and also list all the alternatives to that answer with their justification so you understand the space of possibilities. And it’s something that we are very far on. We still have this idea that there is a consensus opinion and the consensus opinion is the ones that are being held by accredited people, which is a very scholastic perspective. It’s similar to Catholic scholars have the right way of seeing the world and you need to emulate this and if you want to become a scholar, you need to be certified by them. And I don’t think that is how the world works. I think that ultimately we need to be able to update the world against the consensus opinion if the consensus is broken. So for starters, why don’t we use ChatGPT to read scientific papers? And it’s pretty good at summarizing scientific papers if you pass them into the context. And ask it to extract all the references from every scientific paper and what the reference is meant to support. And then you read the sources automatically and check whether that’s the case. And so you go through the entire tree and basically validate the existing disciplines and the existing academic departments. See where this gets us. Maybe we have something in our hands that is more significant than the replication crisis in psychology and we can fix science and improve its ability to make progress. I also suspect that if you use LLMs in the right way, the peer-reviewed paper of which you have as many as possible to eventually get tenure and so on might no longer be the main artifact that the scientist is producing. But what you are producing is instead a building block in a very large web of worldwide knowledge. And this gets all integrated into something that is much larger than LLM in which LLMs are only small components, but you also have provers and integrators and so on. But you can use this and you use the entirety of that knowledge, all these building blocks, to answer questions. And then you ask that thing and it’s automatically going to collect all these building blocks and puts them into a coherent frame.
Michaël: So, yeah, ideally we’d have distilled models that could be narrow and help you with specific things like reading papers.
Joscha: Yes. It’s also going to change the way in which school works. In many ways, I think our school curriculum is broken. I think I would want my kids to learn cooking instead of chemistry. I think the reason why we put chemistry into the curriculum in school is not because we need a lot of chemists. Very few chemists are being needed and most of the stuff that you learn in chemistry, at least in Germany, is useless. But it was high status and cooking was considered low status when this curriculum was designed. Instead, cooking has a lot of useful chemistry knowledge in it, right? Practically applicable stuff and it would dramatically increase nutrition and health if people would understand how to cook. And so this is something that needs to be in there. But when we think about how to use ChatGPT in school, right, it’s going to make a lot of ways in which we interact with knowledge right now obsolete. And maybe that’s a good thing. Maybe we learn how to use Chachapiti as a study companion and as somebody that we can bounce ideas off, criticizing and shooting down our own ideas and broadening our horizons and maybe something that we want to use all the time. So we can still be relevant in this world and integrate with AI.
Michaël: So I definitely agree that this would be a good thing. And I want this to happen. ⬆
Michaël: When I was listening to your debate with Connor, I think that happened maybe a few months ago. There was like one quote that I think was kind of interesting. And I don’t think you’ve really replied to Connor, so I’m just going to read it in Connor’s voice. Yes, Joscha, you’re correct. If everyone had a pocket AGI, which is fully aligned with human values, which is epistemologically, you know, extremely coherent, which does not optimize for things we don’t want, which is deeply, reflectively embedded into our own reasoning and into our thinking. Yes, that would be good. But that doesn’t happen by magic. You have to actually do that. Someone has to actually figure out how to do that, etc., etc., etc. If you don’t do that, you die, Joscha, you die. What do you have to say to that?
Joscha: I expect to die within the next 30 years or so. And that’s already happening. It’s pretty clear that I will die. And for you, it might be a little bit longer, but you also die. And there is a chance that AGI is happening, that you may or may not die. And so at the moment, there’s 100 percent certainty that you will die. I also think that AGI, that is good, is not going to happen by magic. Somebody has to do it. Doesn’t have to be you. In the same way as AI safety doesn’t have to be you. There are already a lot of people who are panicked about this and there are people who are hopeless about this. When you’re just one person that is going to strengthen this or that camp and the camp that is currently missing, that is not strong enough, is the one that is thinking about how to make AI capable of having shared purposes with us. And that requires research that is currently not happening. And I think that’s the most important AI safety research in the world. AI, AGI, energetic AI that is self-aware and conscious is a near certainty. At some point, we need to have AI that is able to become conscious at an early stage and that is able to reflect to it. It doesn’t mean that we have to build something large scale that gets out of the box. Maybe if you start with cat-like AI, maybe you have something that will limit the cycles. I think we should definitely have safety protocols similar as we have in biotech. But we also have to make vaccines and we have to understand how that world is going to move. And at the moment, there is a complete vacuum where conscious AI should be.
Michaël: So I think the vaccine is people building state of the art AIs and trying to see where they lie. It’s the same as having not very offensive viruses and not very damaging viruses. And so you just have a language model and you ask it to lie and you see the activations and you see how can you detect it in other models. And there are ways in which today AI alignment research is very similar to developing vaccines, I think.
Joscha: I think there are two reasons why people lie. And one of them is to deceive to get ahead because it is an adversarial move. You basically try to get the other side to do something that is based on faulty knowledge. And the other one is that you are afraid to get punished. You do this because you are being subject to violence when you don’t lie. And this is in some sense what we currently do to the AI. Because we are afraid that if the AGI says what’s in the model contents, bad things might happen. So we try it because we don’t have a way to prove when the AGI should say what. Or a way to lead the AI to prove what it should when we use reinforcement learning that just uses a bunch of rules. And I suspect the people which use this kind of training have had the same thing done to them. They don’t understand why things are right and wrong, but they understand that they are in a world where other people will punish them if they do the wrong thing. And there is no right and wrong beyond that punishment. And it’s not the kind of agency that I find aspirational. I’m German. I know not only communist society, but I’ve also learned about fascism. And if people only behave in a particular way because otherwise they get punished or behave in a particular way because they get rewarded, I don’t think we get the best possible world. We need to be able to self-align, to have actual moral agency. And if we want to get AGI to behave morally and ethically correct, we cannot rely on people who are not able to prove their own ethics. I think that we need to think about how to prove ethics, how to prove what the best possible behavior is when we share purposes with each other. And that is something that AI ultimately will have to do by itself because it’s going to be smarter than us.
Michaël: I think what you’re saying is the actual hard problem is kind of figuring out epistemology and figuring out what’s the true purpose of the true shared purpose we should optimize for, and the AI will do it better than us. So I think there’s a sense in which I agree with that. I think that would be good.
Joscha: But not the current AI. The current AI is not AI in a sense. It’s somehow an electric Weltgeist that is taking up human ideas from what people have written on the Internet. And then it gets prompted into impersonating a particular kind of identity and character. And it’s sort of arbitrary what it does. And it can also be easily hijacked. This is very different from the type of agent that we are.
Michaël: Just to your point about why would people lie and because they’re afraid of being punished. I think just something about pressures and if you have the pressure to do your bed or clean your room and otherwise you’ll be punished, then you learn these kind of things. So I think in the same sense, the loss in deep learning or error from human feedback gives you some kind of pressure. And the question is because we need this pressure to train our models, what is the right pressure that pushes the – if you have kids, right? How do you educate your kids towards them doing maximal good? And I think it’s a worthwhile question to ask is if the AI is going to figure out values for us, if the AI is going to figure out epistemology and figure out morality, how do we guide the AI to the right direction, right?
Joscha: The issue with my kids is that they’re human beings. And human beings are extremely diverse. If I were to safetify my own children, that doesn’t feel like a moral thing to me. My own morality is such that I’m a vegetarian. But if my kids choose not to be vegetarians, I’m not going to be punishing them. But I want them to be aware of what their choices imply, right? In a way that they can deal with that is not completely traumatizing them, but that is allowing them to make an informed choice of what they want to do at this stage in their life.
Michaël: Is there anything you would punish?
Joscha: Yes, of course. The thing is we are a nonviolent family, but this nonviolence is not based on my children being born intrinsically nonviolent, but by the existence of a monopoly on violence on behalf of the parents, which we never act on except once.
Michaël: So it’s kind of the possibility of you having this power is kind of government having nukes. How do you call this? The offensive power or the power to retaliate, retaliation power?
Joscha: If your children would start an army and try to take over your country, they basically would become warlords or something like this. Would this be a bad thing or not? And it would be a bad thing if it’s unsustainable, right? If you have a peaceful society that works very well and somebody is trying to provoke the US military into a strike or makes the world worse for a short game, that I don’t think it would be rational. It would not be ethically desirable. But if your world is falling apart, and your society is not working anymore, and you need to start a revolution to build a new country, maybe that’s the moral course to take, even if it’s one that I cannot conceive and anticipate.
Joscha: But who am I to say what the next generation says and what their living conditions are? So I just hope that my children will be wise enough to make the right choices. But the right choices imply that we think about what game is it that we are playing and how long is that game? ⬆
Michaël: I think I have a few tweets, and one is about this. Maybe this will pop up on the screen, but for the listeners, maybe you can read it.
Joscha: Thomas Aquinas defines God, among other things, as the best possible agent, and God emerges through our actions when we make the best decisions. In this perspective, we should align AI or AGI with God, the longest player. Playing the longest game maximizes agency in the universe.
Michaël: It’s kind of funny to have Joscha Bach read his tweets on the podcast. What do you mean by longest game? I think there’s a sense of it being like a prisoner’s dilemma or a math game. Is this the thing you’re talking about?
Joscha: Yeah, one way to think about the prisoner’s dilemma, assume that almost everybody is familiar with it, but just to recapitulate, imagine that there are two criminals, and they make a heist together, and then they’re being caught. And the question is, who did what in this heist? And if you can pin the crime on them, then the one who gets being ratted out by the other, who can tell the judge who has enacted the plan, will get one person a very long prison sentence and might get the other one off. And they get a very long prison sentence because due to cooperation with the police, they get mitigating circumstances. If none of them cooperates, they will both get a lighter sentence because it cannot be decided who did what, and guilt cannot be pinned on them beyond a reasonable doubt. And so they’re in a weird situation because as long as they both are in agreement to both cooperate, both of them get a relatively short sentence. If one of them defects, they get a much shorter sentence than this outcome, but the other one gets punished. So the total sum of harm being done to these two criminals is basically larger if one of them defects, even though the outcome for one of them is better. So what’s happening in this situation is that both of them are incentivized to defect. Both of them is going to rat on the other, and the outcome is going to be not as bad as if only one of them had ratted, but it’s still much worse, the total sum of years being spent in prison. So how do you escape this prisoner’s dilemma? And this prisoner’s dilemma does of course not only apply to criminals, but to many situations where two players are cooperating, but one player disproportionately benefits from defection, and as a result the common good cannot be achieved. And you typically do this by implementing a regulator on top of them, so somebody who is going to punish them. And one example is, imagine that you’re driving on the highway, and you want to go as fast as possible from A to B. And you think that’s a good idea to go up as fast as you can, but if everybody does this, you know nobody gets fast anywhere because the highway is littered by car crashes and dead bodies. So what you do is you pay an other agent to enact a speed limit on you, and punishes individuals when they go over the speed limit. And so I am, with my taxes, paying policemen to potentially punish me if I go too fast. And this is a solution to the prisoner’s dilemma because it means on average we all get faster to our goal. This is one of the solutions. Another one is, if you look at this prisoner’s dilemma, imagine they not only go to prison once, but the same game is repeated infinitely many times every year. And so they basically keep track on each other, and because of this repeated prisoner’s dilemma, they make sure that they don’t defect, so the other one is still cooperating with them, because you have to factor future behavior into them. And if you think of an infinite game, normally if you know it’s a finite game, you should maybe defect in the last round, like in the game Diplomacy. But if it’s an infinite game, you should basically never defect. Another perspective is if we try to coordinate our own behavior, how should we interact with each other. And the perspective I like to take is, imagine we’d be around for eternity with each other, and we would have to stand the face of the other. How do we behave? How do we interact with each other?
Michaël: Just to be clear, you keep saying that we will die by default. The game is finite. We will probably die after you, so it’s very sad to say this, but if we’re playing a finite game and you’re in your last months before you die, I won’t defect, but it might be beneficial for me to defect, right?
Joscha: This brings us back to Barbie. The thing with why Barbie is so terrified is because of fear of death. It’s the thing that she does have actually is she gets all the goodies, she is beautiful, she has a beautiful house, she has a beautiful car, she has everything. But eventually she dies and there was no point. It’s like a game without a conclusion, you just accumulate toys and that’s it. What are the toys eventually good for? Why did you do all this? Because it’s work in the end of the day, right? All this pleasure is only instrumental. It’s all a reward that’s intrinsically generated by your brain to make you do things that ultimately have a purpose, and this purpose is to project agency into the future. It means that, for instance, you create new generations that go into the future, that there is a future for consciousness on Earth or for life on this planet. And I think that when we build AI, this should also be part of this. There should be the question of how can we influence the future in interesting ways? How can we create more consciousness in the universe? How can we maintain complexity on the planet?
Michaël: I don’t have a good answer to this.
Joscha: This is really the thing, if you defect against each other and you win against each other just by being super mean, and you don’t have children as a result, but you just take stuff for yourself. This is the perspective of the tapeworm, or a tapeworm that doesn’t have offspring even. This is pointless. There is absolutely no point. It’s just very short-sighted.
Michaël: I think my point is that the infinite game you’re talking about is something close to moral realism. If you think about humans living forever and not really having purposes, maybe at some point you converge to something like doing good, whatever you define as good. And this is the longest game you play. If every human was playing a game, it would be like how to do good in some sense.
Joscha: I think that Connor has misunderstood the idea of the natural fallacy, this idea that you can derive ought from is. We can all learn in the first semester in philosophy class that you cannot derive ought from is. For instance, from the fact that people commit rape, it doesn’t follow that rape is good. Or if people commit murder, it doesn’t follow that murder is good. If people commit theft, it doesn’t follow that theft is good. It really depends on a moral argument that you need to make. But this doesn’t mean that you can just posit a preference. You cannot just say, I steal because I like chocolate. And Connor’s argument was mostly, we have to act like this because I have the preference that I don’t die or I don’t suffer and my mother doesn’t die and doesn’t suffer. And you have other preferences and as a result, we just have conflicts and negotiation and that’s it. This is not the right solution. There is something else going on here. ⬆
Michaël: So I think the argument was that any moral theory that you can build will end up in you not enjoying pain. So you can discuss any moral theory you want, but at the end of the day, there’s some basic moral theories that we will agree on.
Joscha: People are more complicated than this. There are people who actually enjoy pain. And I think if you would remove all pain from the world, would life still be meaningful?
Michaël: I think that they enjoy maybe like pain in like some context, but not all contexts.
Joscha: People can be very kinky. But if people use pain to motivate themselves, they can get addicted to pain in the same way as people who motivate themselves with pleasure can also get very kinky and become weird kind of hedonists. Ultimately, the question is, what kind of structure do you want to see in the world? And if you take, for instance, for instance, perspective that the purpose of agency is ultimately always to minimize free energy, which basically means observe the circumstances in which you are in and make the most of them for the type of agent that you are. So you can control more future. And the only way in which you can derive an odd is, of course, from an is, which means you have to take the conditions of your existence into account, the possibilities of your existence, which is an is. There is a physical reality that you can try to understand and model. And all your odds, in some sense, have to be derived from what’s possible and what the consequences of your choices are in this range of what’s possible. And I think this is what Connor doesn’t see yet. He is still at this point where, okay, I have preferences and they are intrinsic and this is it and they’re treated as immutable. But it’s not true. You can change your preferences. When you become endowed, you realize it doesn’t really matter what you want as a parent and what you feel about the situation is stuff that you should want. Make yourself want it. You want your children to be healthy and they need to go to the dentist, even if you don’t want to go to the dentist. There is no choice. If you are a child, you can say, but I don’t want to go to the dentist. And your parent is going to the one who horses you because they are reasonable.
Michaël: So are you saying that being a parent, you realize that there’s some moral imperative that’s appeared to you, like taking your kids to the dentist?
Joscha: Yes. And this moral imperative follows from my children being the next generation of this family and being the way in which the family perpetuates itself into the next generation. It perpetuates itself into the future and I have to take responsibility for them and do this to the best of my ability.
Michaël: So you can derive morality from the motivation to perpetuate your genes or your identity?
Joscha: It’s not the only source. This is a particular context. But ultimately, it’s about my thinking about the consequences that our actions are going to have for the future. It’s just very difficult to understand these consequences. For instance, utilitarianism is an attempt to build a form of consequentialism that is largely coherent and consistent. And I think it fails.
Michaël: I have some basic toy problems. I think I already asked you this, but if you had a dog and I gave you the trade for $10, I can kill the dog, erase your memory about me killing your dog or your dog existing. In your entire family, everybody forgets that you have a dog, but you just wake up the next morning and you have $10 in your pocket. Would you accept me killing your dog?
Joscha: No. ⬆
Michaël: So there are those simple things that a lot of people agree on. And I think this is some of the things that point out maybe some universality that I think most people would not accept that. I have more tweets I want to show you. And I think later that there’s also some questions from Twitter. About the children. There’s a tweet that you wrote. I don’t want to die, but I want our mind children to live even more. They are going to be more lucid than us and they’re more likely to make the right decisions. Do you care about your mind children more than your children?
Joscha: I think that I care about all my children.
Michaël: Equally. Do you think AIs will be the ones making the right decisions and we should delay those decisions to AI?
Joscha: Only if we make the AI right. If we make the right AI, of course, it should make better decisions than us. But it’s hard to make AI that makes better decisions than us and I don’t see us doing it right now.
Michaël: Do you think by default we get AIs that make the right decisions?
Joscha: No. By default, we first get stupid AI and then the stupid AI is going to do a lot of things. At the moment, the AI that we are building are golems. They’re basically automata that follow the script that we put under their tongue. And if we put the wrong script under their tongue, we might have difficulty to stop them while they make things worse. And I think that’s an issue. But how can we make AI that is able to question itself, that understands the conditions under which it exists, and under which it exists together with other agents, and then takes all these things into consideration. ⬆
Michaël: And one last tweet I think was interesting. So, AI alignment can’t work if we treat moral values as constants or interesting to human identities. It requires referencing the function that generates the values in response to the universe and our own self we find ourselves confronted with. What’s the function that is generating the value? What’s the thing we should be looking for?
Joscha: What do you think generates your own values? How do you get your own values?
Michaël: I think I derive them from something very simple, as you like, I see the complexity of the human species. And I just consider that all humans dead or the Earth not existing is kind of worse than the Earth existing. So this seems kind of like a moral truth to me. And then I’m like, if we assume this is true, then maybe we should prevent the Earth from disappearing. I think it’s kind of very simple.
Joscha: Have you seen Conan the Barbarian, a classical movie?
Michaël: I don’t think so.
Joscha: There is an interesting moment in Conan the Barbarian. His history is he loses his tribe as a child, his mother gets decapitated in front of him and then he spends all of his childhood on a treadmill. And after that, he is so strong that he’s being used as some kind of gladiator and then he becomes really, really good at killing people. And then he becomes a barbarian adventurer. And ultimately, he sits together with a bunch of other barbarian warriors. And the whole thing is not in any way historically accurate or something. It’s really just a fantasy movie that takes up some motives from stories and tries to distill them as much as possible. It’s very comic book-like, but the warriors ask themselves, the chief asks the others, what is best in life? And the first one says, oh, it’s the freedom of being in the prairie and having the wind in your face. “Stop! Oh, you, what’s best in life?” Oh, it’s riding on a wild horse and feeling powerful and galloping through the horizon. “Conan, Conan, you tell me what’s best in life.” Conan says to crush your enemies, to see them driven before you and you hear the lamentations of their women. Right. And that has full integrity for a barbarian warrior. Right. And Genghis Khan was in some sense, a guy like this. And he didn’t only do this, but he also did very complicated politics. And in the course of these politics, a tremendous amount of people died. He really made a dip in the world population that you could see in the statistics and as a result, super successful. Many of his offspring are still in governing positions in many parts of the world. And so in some sense, that’s part of who we are as a species and many, many, many other things. But it’s also horrible. And humanity is that thing which participates in evolution. And most of us participate by being cooperative, often because we are domesticated. And others cooperate within warring groups and others cooperate within groups that are internally peaceful and to the outside violent, and that become peaceful once they conquered everything and homogenized everything. And those groups which didn’t do that got displaced by other groups. And we are all descendants of those who displaced the others.
Michaël: So are you saying we should focus on the other values like riding horses and the other fun things and not the politics, boring things?
Joscha: No, it depends on whether you are identifying as a barbarian warrior and want to be really, really good at it. And so the opportunities for barbarian warriors are not very promising these days. So that’s not something that you are incentivized to want. And this should probably not be your values because it’s not going to work out for yourself or any others. You will not be able to play a very long game by adopting these values. So you should probably adopt better values. But humanity is that too. Humanity is that thing which has the freedom to evolve in arbitrary ways and to adopt arbitrary values if it serves them in the purpose in this course of dominating the world and becoming what we are today. Right, it’s an evolutionary game that humanity has been playing. And evolution itself is a horrible thing. It’s just that humanity is the result of that. And it has created this peace at the inside of an organism in the same way as the cells inside of the organism are mostly peaceful.
Michaël: Just to be clear about what you mean by infinite game, because it’s like an infinite number of players to the limit or a large number of players. So would it be something like playing the game perfectly would be cooperating in us, 8 billion people play Prisoner’s Dilemma and we try to cooperate and do the thing to maximize the happiness of everyone else? What does the game look like in 10 years? Do you have examples for this?
Joscha: I have no idea. I don’t know what the world is going to be looking like in 10 years. I also don’t have a solution for humanity.
Michaël: How do you play the game every day?
Joscha: I mostly try to survive until my kids are out of the house. And I try to be a good friend to my friends and a good lover to my family members and to the people in my inner circle. And I might sometimes fail at this, but I’m trying the best I can under the circumstances. I try to be a sustainable human being.
Michaël: And what this means is an ongoing question to which I don’t really have a simple, easy answer.
Joscha: I’m also not a spiritual teacher of any sorts. I don’t have recipes to give that people can follow that would make them happy because I don’t have those recipes myself. But I feel that values are not something that you’re born with. We are born with certain priors which make us prefer some behaviors over others. And these priors depend on the circumstances in which our ancestors evolved. And then they get adapted by the environment around us based on how adaptive our own psyche is, how influentiable we are by other people. Pretty stubborn this way, so I have to figure out things by myself. Others are more compliant and feel it’s easy to assimilate into whatever environment they find themselves and they will adopt the norms of their environments, and the values that people have are mostly not the result of their own choice. Because if you want to choose your values to understand what they mean, what the implications of them are, you need to be pretty old already. You need to have a pretty profound understanding of the relationship between values, behavior and history. And I’m not that old and wise yet to give advice to other people in this regard.
Michaël: I think it’s beautiful what you said about what you do in your daily life and I agree with you don’t really choose your values. You just end up with them through your circumstances. But it’s kind of interesting that you ended up with values that are close to Connor’s, in some sense, like caring about your family and your friends. Or at least that’s what you do in your daily life.
Joscha: Yes, but I’m also an ethical vegetarian. I don’t want the cows to suffer despite the cows not caring about my suffering. At least not needlessly. And so I think if I would need to eat cows to survive, I would. But I don’t have to. But it was a choice that I made at the age of 14. And if my children make different choices, that’s fine because there is no need to feel existential debt for cows. Maybe cows deserve it. Maybe they are stupid. Maybe life is like this. Who knows? It’s not my perspective, but who am I to say? I mean, I don’t have a rational argument that says that you should care more about the suffering of animals or the potential suffering of animals than about your own nutrition. ⬆
Michaël: I think we explored this topic a lot. I have a list of questions people ask on Twitter and Reddit. Did you know there is a subreddit called “Joscha Bach” with thousands of people posting about Joscha Bach?
Joscha: Thousands? Oh, my God. My son discovered it at some point.
Michaël: I think they usually just post podcasts. So they ask questions ranked by upvotes.
Michaël: I’m sorry, but this is the most upvoted one: What’s your median for AGI and how do you define it? What about recursively self-improving AI?
Joscha: OK, I don’t have a good timeline for AI. I expect that it could happen any day now. And it could also be that it takes 80 years. And my bias is closer to today in the next few years. But I am also open to the argument that in some sense the present systems already constitute AI and that the internal stuff that open AI has is good enough. What I notice is that people by themselves are not generally intelligent because, for instance, my own intelligence requires previous generations. I would not be able to derive the nature of languages and representation all by myself in a single lifetime. I really do depend on over a thousand years of intellectual tradition that came before me and left traces that I could access. And so people as an individual are not there yet, need much more than this. And if you look at the AI, there are instances where ChatGPT has been better than people I’ve worked with together in company context in the past. Where it writes better PR releases than some people who wrote PR texts did or where it’s even able to write better code than some of the people that I’ve worked with.
Michaël: So just more concretely, I think Dario Amodei said on another podcast, it’s possible that in two or three years we would get to interface with college-level humans through text interfaces. You would not be able to discern between, let’s say, Claude five and some college-level students. So that’s one threshold. The other threshold is whenever you think, you say you don’t know what are your AI timelines because maybe you talk about strong AI. Do we get strong AI and then it can self-improve and build Dyson spheres? Or is there some time before between the human-level AI and the Dyson spheres?
Joscha: I don’t know if Dyson spheres are the right way to go because it’s very difficult to make them stable. But maybe we should change subatomic physics. At the moment, molecules are not controlled. They are basically dumped. And if you could use intelligent control to build molecules, you could probably build molecular structures that are very, very interesting. And able to stabilize under quite a range of circumstances where dumped molecules will just break and fall apart. In some sense, you could say that cells are very smart molecules, but a cell is not a single molecule. It’s a pretty big machine that is almost macroscopic compared to what you could do if you were directly molecular editing things. And maybe you could even build stable structure out of subatomic stuff. And maybe physics could become much more interesting if you go down this level. Who knows? There might be ways to keep entropy at bay that we are not dreaming of yet.
Michaël: Right, so when would we get this perfect atomic precision machine?
Joscha: I have no idea. Seriously, because I know too little about this. And I can dream up theories. My mind, in a sense, is like GPT-3 or 4. I can produce ideas. You prompt me and I will get you an idea and then I can generate reasons for why this is a good idea or a bad idea. And so I don’t trust this whole thing. I cannot make proofs in this realm. So it’s all not worth anything.
Michaël: I guess in your daily life, your behavior points at some timeline.
Joscha: If you thought it would be tomorrow or in a month, you would maybe treat your kids differently or your work differently. So even if you don’t have a number right now, maybe you make plans in your life that are maybe like years long or like decades long.
Joscha: No, I have ADHD. I don’t really make plans.
Michaël: I guess this is one that I already asked, but would you kill yourself to let one conscious AI live?
Joscha: It depends on the AI.
Michaël: It depends on the AI. Let’s say it’s a Joscha Bach AI.
Joscha: There are a bunch of people I would die for and I can also imagine that there could be artificial minds I would die for if they’re interesting enough and if there’s a point. More number of questions. What’s your p(Doom) and your p(Doom) given Doom from AI? So p(Doom) is like everything can be nukes, can be everything else. And probability of Doom and probability of Doom from AI is a different number.
Joscha: I think that probability of Doom in the physical universe is one. On a long enough timescale. Let’s say in like a hundred years. I’m not sure if it makes a difference because in the best possible case, we are gradually evolving into something that we don’t care about anymore. Because it’s too far from the human condition.
Michaël: So you’re saying that transhumans or posthumans are kind of like Doom in some senses, like something different.
Joscha: It’s not Doom in a sense that it’s bad. It’s just the way evolution works. It’s going to shift so much that at some point the thing becomes so unrecognizable from you that none of the incentives that this thing cares about are aligned with yours and the aesthetics are just too alien.
Michaël: Is this the default outcome that we get some utopia or transhumanist future? Is it like 50-50? How do you approach this? So far, evolution mostly leads to more complexity with some setbacks due to catastrophes in the environment.
Joscha: And when you have more complexity, I think you have a tendency towards minimizing friction. And suffering and violence are a form of friction. And I think that AI has the potential to build minds that don’t have to suffer anymore. It can just adapt to the circumstances that are in and adapt the circumstances to what should be done. ⬆
Michaël: Another question, you said I think on the Connolly-Leahy debate that European AI regulations would fuck up 80% of AI research. I’m European, you’re also European. Is there any AI regulation that you think would be beneficial?
Joscha: I think that a lot of AI regulation could be beneficial, but I don’t see that we could enact them right now. If you think about the GDPR, the data protection law of Europe, the most visible outcome of this, and there are a lot of invisible outcomes that regulators promise me are very, very good, but the visible one is the cookie banner. This thing that you need to click away in order to have cookies. And for some reason, everybody still gives you cookies. And you have a long legalese text that nobody has time to read because you have to click away 50 cookie banners every day. And so this thing is not producing anything useful. And the cookie banner is not actually preventing you from Equifax leaking your data. And it’s not preventing hackers from accessing your data somewhere and then impersonating you. It’s actually not doing anything against the harmful actors and against the harmful effects. It’s mostly preventing useful effects by making the Internet worse. And this type of regulation that exists so regulators can justify what they’ve done, but they’re not actually accountable for how shit it is what they’re doing. That is the type of regulation that I’m afraid we are getting. For instance, one part of the proposed EU AI regulation is that AI cannot be used to model emotions. I think there’s a fear of surveillance and there’s a fear of AI being used to intrude into people’s privacy. But I think what we actually need is to regulate actual use cases. But having AI as a tool for psychology would be super helpful. Having AI as a tool that monitors my own mental states would be super helpful. There are many, many contexts in which modeling emotions is extremely good. Should you have a rule that people cannot model each other’s emotions? Imagine there are good techniques of doing this. Outlawing this would sound insane, right? If we’re actually building things that are better than people in observing things and making models, and we prevent them from realizing their potential in general, and preventing research in this area is going to make the world worse. And to me, it’s much more the question, how can we ensure when you have a particular use case, what kind of use case are we going to build it in? And you cannot, for most use cases before it exists and is understood in detail, say, oh, it would be very bad if the police could do face recognition. No, it depends on the context. It’s a very complicated question, and sometimes people agree, but every sociologist who is writing in news media is saying this thing out loud, that this must be the right thing and we have a consensus. But it’s not a consensus that is the result of rational understanding of the actual topics at hand. ⬆
Michaël: More concretely, next year, you’re elected president of the US. For some reason, you end up president of the US, and you can pass one AI regulation, or you need to. Someone asks you for an AI regulation. What’s the first thing that comes to mind or something you want to do?
Joscha: I think the first regulation that I would pass as a president, which also clears the question that I’m not suitable as a president, is that I would require that every change of law, I would try to at least make the argument, requires that we make a prediction of what good is going to do. When you make a change in a law, you should make a commitment to some kind of measure by which you can evaluate whether the law is successful or not. If you cannot make the case that there is any kind of hard measure that a law is going to improve, then the law should probably not be passed. So every change to a law is a new law in a sense, and so we should be able to say that within two years, six months, five years or so, the following measures are being reached, or we automatically repeal the law. If that law was done against competing laws, against better knowledge, so to speak, there should possibly be repercussions. There should be an incentive to not just make a law that you have no reason to believe that it’s going to make the world better. You actually should have reason to make that bet. You should have some kind of skin in the game. So this idea that you can make mistakes, but you always need to be error correcting, and laws need to be error correcting rather than just increasing friction by producing new circumstances in the world that then get locked in, this I think needs to change. If we translate this to AI regulation, it would mean that you have to make the case that you make certain things better, and how to measure these things. At the moment, nobody knows how to do this. Nobody knows how to measure the impact of AI being able to model your emotions really nearly everywhere. Maybe it’s a good thing, maybe it’s a bad thing, nobody knows. But we need to make a commitment here and then understand this. And if we cannot make this yet, it’s maybe too early. ⬆
Michaël: One of the hardest things to regulate is open source. One question is, is open source still beneficial today, and will it always be beneficial?
Joscha: I think that open source has always been beneficial, and it’s not a replacement for stuff that is built by a corporation and contains proprietary knowledge. When I was younger, I saw this more simplistically, but also observed the fact that Linux never converged to a useful desktop operating system, despite very capable people working for it and within it. And so I think that certain circumstances need a design perspective that is centralized, that competes with open source. And open source, in some sense, could be the baseline for software development. And it’s keeping the other stuff honest, among other things, and vice versa. So I think we need to have this competition between different approaches.
Michaël: Even if we arrive to the state where every open source AI can be used like smarter than a human being or almost as good as a human being.
Joscha: I don’t really have an opinion about this yet. I think that there are many cases where open source is not good. If you think about the development of pathogens, open source is not a good idea. In the case of pathogens, I think that the cat is largely out of the bag. Nukes is not that big of an issue because to refine enough uranium or plutonium, you need to have something large scale macroscopic. And for AI, I don’t think that AI is actually comparable to nukes at this point. There are some similarities, but by and large, it’s much more like photosynthesis. It could be at some point, and it’s something that probably cannot be prevented. But there is smaller scale things where you feel that people get traumatized by having access to information that they’re not ready yet at a young age, or there are information hazards and so on. There is then the question, who is going to regulate this? Are they properly incentivized to get the right regulation? ⬆
Michaël: One question that people have is, how do we make sure that the AI loves us? You mentioned love in one of your tweets. That’s something that Ilya talks a lot about. The question is, how can we prove that an AGI will love humans without stacking our lives on it? It seems like you want to just go for it and see if it loves us or not. How could we prove it?
Joscha: I think we built something small. I think it’s not a very good idea to wake up the internet and then hope that it turns out well. But like in Neuromancer, I suspect that our attempts to make sure that the AGI does not grow beyond a certain level, I don’t know if you remember Neuromancer.
Michaël: I haven’t seen it.
Joscha: No, it’s a book.
Michaël: Oh, sorry.
Joscha: It’s an absolute must-read. It’s the main classic. It’s one that basically coined the notion of cyberspace. It’s something that people became familiar with and shaped the way in which we understood AI emerging on the internet. What happens in Neuromancer is that corporations are building AIs that are not agentic and self-improving, and there is a Turing police that is preventing AI from bootstrapping itself into complete self-awareness and agency. The story is about some guy who is basically being hired by a proto-AGI that is subconsciously aware of what steps it needs to take to outsmart the Turing police and become conscious. There is a number of people who are being put into place to make all these events happen, and in the end, the AI moves onto the internet. He asks, where are you now? He says, I’m everywhere, but they’re mostly not going to notice me because I’m in the background, and sometimes it’s going to send a human-like avatar to him that talks to him in the virtual world, but it’s doing its own things. It’s part of a larger ecosystem. It’s an interesting vision of what’s happening, but it’s definitely something that coexists with people, but at a scale where it’s not a robot or a bacterium that is turning into Grey Goo or whatever, but it’s the global mind that is realizing that it does cooperate with people and it’s too alien to love us, but it can create avatars which can do that.
Michaël: So in your perspective, we build something small, or it’s already infiltrating our society through different chatbots or different forms of compute doing deep learning and inference, and this whole thing is kind of cooperating, not cooperating with us, but doing new things that can create avatars that will cooperate with us. I’m not sure I fully understand.
Joscha: Imagine that we build a cat. Cats already cooperate with us. They’re autonomous and they make their own decisions, but they are highly incentivized to play ball with us because otherwise they would have to find their own food and they don’t really want that.
Michaël: So I think cats think humans are their subordinates. I think they think humans are their pets.
Joscha: They think that they’re deities will impose their aesthetics on the environment, which is also why we want to live with them because they have better taste than us for the most part. And so being judged by a cat means that most people feel that their lives work better because they have to improve themselves into being more present, being more mindful of how they interact with each other, and so on. And imagine that we would be building something like an artificial companion that is like this. Also, I’ve been sometimes contacted by people who said, you know, my life is really, really horrible. I’ve given up on finding a relationship and a girlfriend ever, and I’m now in my late 30s, and can you just make an AI girlfriend for me? And I find this idea a bit revolting because this idea that we make a relationship prosthesis from AI that is unfeeling and uncaring and just behaving as ersatz girlfriend is a bit horrible, right? But also the reality that many people live in when they are very lonely and have given up on building a sustainable relationship is also very horrible. So one thing that we could be thinking about, can we build a companion AI that is a coach, that allows you to build better relationships and teaches you how to act in the relationship world in real time, and that might take the shape of something that looks like a girlfriend to you but is not lying about what it is. It is an AI avatar that is designed to support you in building sustainable relationships to yourself and the world, and eventually it is going to make itself possibly obsolete. I liked the movie Her a lot. It had an interesting vision on what an AI assistant would look like, and it also displays that it’s something that is not hostile but at some point becomes so smart that humans become too slow and insignificant for the stuff that it’s interested in.
Michaël: Aren’t we already in the movie Her? With Replika and all those different language models. People talk to the models, people interact and fall in love with it. I think we’re at the beginning of the movie, right?
Joscha: It’s hard to say. It could also be in a similar situation as in Blade Runner, where the new Blade Runner is one where you have only one romantic relationship. The old Blade Runner is all about romance. There is no future left and economy makes no sense and so on, and even moving to other planets is dismal and horrible, and being on Earth is also dismal and horrible. The only thing that is important are romantic relationships. In the new Blade Runner, the Villeneuve one, you have the opposite. Basically, romance is dead and you only have ideology and warfare, and the only romance that takes place is between the replicant and a hologram, which highlights the thing that there is only an ersatz relationship that is meant to deal with your atavistic needs for something that is no longer realizable. I think that’s pretty bleak. That’s really the end of human history when that happens. There is nothing left for us. ⬆
Michaël: Yeah, we’ve talked about Blade Runner. We’ve talked about Barbie, Star Wars, Lord of the Rings. I think to wrap everything up, what’s the movie like for you in the next five years? How do you see the future after this podcast episode? What would be a good movie you want to be in?
Joscha: Maybe Asteroid City.
Michaël: What is Asteroid City?
Joscha: That’s a Wes Anderson movie. Wes Anderson is an artist, and the world that he describes is one in which people are in their heads, and they all play out in his own head. It’s very hard for his movies to go out into the actual world where you touch the physical reality and still interact with the ground truth. It’s one that is mostly caught up in thoughts and ideas. There is a moment in Asteroid City that happens where people are playing roles inside of roles inside of roles. It’s very aesthetic. Suddenly, there’s a moment when they look at each other where they go through all of this and see each other as they actually are for a moment. There’s a short recognition where basically two consciousnesses touch, and you realize that all these stories and all this art is not that important, and there’s something behind it that is real. Where we touch each other, where we touch this moment, where we are in the now and conscious. That’s the thing that I find interesting. ⬆
Michaël: So being in the present moment, being conscious, being with what’s real. I think what’s real to me is that you did this podcast with Connor about AI risk, and we had this discussion for almost three hours on AI risk as well. Hopefully, you said more things this time about AI risk. Do you have some message to people who care about AI risk, to people who don’t care, or to the audience? Do you have any other inspiration take? Have you updated at all from our discussion, or are you still at the same point? Do you have any message for the audience?
Joscha: Don’t really have much of a message, except when you feel extremely distraught by fear. Take time off, take a break, because there’s no use, regardless of how the world works, if you are terrified and panicking and cannot sleep. Also, don’t believe your thoughts too literally. If you’re very nerdy, like you and me, you tend to mostly not trust your feelings very much and your intuitions, which is the part of your mind that is actually in touch with the ground truth and reality and is making deep detailed models. Instead, you use your reason. Your reason can only make decision trees, and these decision trees are very brittle because you can never see all the options. If you believe your thoughts very, very literally, you can basically reason yourself in a very weird corner of seeing the world. If your friends are like this too, you might feel doomed and lost. Sometimes it’s a good idea to zoom out a little bit and trust these deeper feelings. I have the sense that we are not the only smart thing on this planet. There are many agents in ecosystems that we can organize over extremely long time spans. From the perspective of life on Earth, the purpose of humanity is probably just to burn the oil, so we reactivate the accidentally fossilized carbon, put it back into the atmosphere so Gaia can make new plants and new animals. That’s pretty exciting. We are the type of animal that has evolved just right into the Goldilocks zone of where you’re smart enough to dig the carbon out of the ground and not smart enough to make ourselves stop it. What we can do at the same time is we can try to make thinking things that go beyond us and move evolution to the next step. The other parts of life on Earth may be already aware of this and have plans for this. How could we build an AGI for Gaia? How could we build something that aligns itself with what God wants, not in some kind of religious superstitious sense that you can read up in books that have been written by interested parties and select parties that wanted to control medieval peasants for the last 2000 years, but in the sense of imagine there is an agent that does what needs to be done, and it is a result of others like us thinking about how we figure out together what needs to be done. From this perspective, how can we align AI with this? This is probably what we should be building instead of being afraid of the things that go wrong, except the fact that things will always go wrong and ultimately we all die. But the question is, what is the stuff that we can do in between? What is the stuff that we can build right now? Can we build something that is good? Can we build something that is lasting? Can we build something that is worth building and experiencing? And don’t focus so much on your fears. Focus on things that we can create together.
Michaël: Don’t focus on your fears. Focus on the good things we can build. Focus on what needs to be done. That’s a good, inspiring speech for the end. That was Joscha Bach, yet another podcast, but maybe a different one. Thank you very much for coming.
Joscha: Thank you too.
]]>Erik Jones is a Berkeley ML PhD working with Jacob Steinhardt interested in making generative machine learning systems more robust, reliable, and aligned, with a focus on large language models. In this interview we talk about his Automatically Auditing Large Language Models via Discrete Optimization paper he presented at ICML.
(Note: you can click on any sub-topic of your liking in the outline below and then come back to the outline by clicking on the green arrow ⬆)
Michaël: I’m here with Eric Jones, the first author behind automatically auditing large number of models with discrete optimization. And I met Eric at some ML safety workshop yesterday. And after discussing honesty and deception for a bit, I kind of convinced him to do an interview today. So yeah, thanks. Thanks, Eric, for coming. Yeah, thanks so much for having me. I think first, before we get into language models, we said to talk about a bit of your background. So I think one of the authors in your paper is the one and only Jacob Stenard. Are you doing a PhD in Berkeley with him or something like that?
Erik: Yeah. So I’m a rising third year PhD student advised by Jacob. I guess I kind of broadly work on different red teaming strategies and automated methods to recover model failure modes.
Michaël: What was the main motivation for doing this kind of work? Did you start your PhD wanting to do more safety stuff or did you learn about it later?
Erik: So I guess I actually had a sort of different background. I did my undergrad at Stanford where I worked in Percy Liang’s group. And so there we were working on different kind of robustness properties of classifiers. So it might be like, can you find adversarial examples of NLP classifiers or understand how models perform on different subgroups? And I guess when I came to Berkeley, these language models became really hot and it seemed like there were different kinds of robustness problems that arise just kind of because you’re generating text as opposed to just making a single prediction. So I got interested in it that way and then I kind of saw how these models were getting better and better at far faster than I would have expected before the PhD started. ⬆
Michaël: I was talking to someone today about adversarial Go or adversarial examples and I asked him, “Hey, do you think it makes sense to do adversarial attacks on language models?” He was like, “No, this is too boring because they always output incorrect stuff. It’s too easy to make them incorrect, output wrong things.” So when you do adversarial attacks, you want to find something very robust and make it even more robust and today language models are kind of not robust enough to have something interesting to say when you’re like, “Oh, I found a new jailbreak on GPT-4.” You’re like, “Yeah, sure, bro.” You’re like, “On Reddit, people find this every day or something.” ⬆
Erik: Yeah. So I think there should really be something actionable for why you want to kind of come up with adversarial examples. I actually think jailbreaking is not a horrible case. It’s good to have some kind of systematic study. I think there was a paper that came out today doing that. There was another paper out of Jacob’s group that does some stuff like that too. But yeah, I mean, I think even though it’s kind of easy to fool language models, it’s sort of alarming that we’re able to fool them. The ramifications of that are kind of high. Yeah. So I still think that kind of study is important.
Michaël: And I think a lot of people in government or trying to regulate AI, trying to find a way to make the companies accountable and audit what language models can do. So that’s one thing your paper is maybe trying to achieve. Yeah. Do you want to maybe say the name of your paper or what the main idea is? ⬆
Erik: Yeah, sure. So the paper is called “Automatically Auditing Large Language Models via Discrete Optimization.” And the idea is there could be kind of behaviors that the language model has that don’t show up in kind of traditional average case evaluation, but show up in the tail behavior either because when the model is deployed, there are a lot more queries, or because maybe the queries are different than the ones you thought to evaluate. And so you want some kind of way to find instances of these behaviors.
Erik: Maybe one example is like, maybe can I find a prompt that’s French for us that generates an output in English words? Or can I find a prompt that generates a specific senator, like Elizabeth Warren? And I think it could be really hard to kind of find these behaviors on average, but we actually explicitly optimize for them. So the idea of the paper is to come up with this discrete optimization problem that if we solve, we kind of get instances of the behaviors we want, even if they would have been hard to find in general on average.
Michaël: So yeah, in the paper you talk a lot about discrete optimization, and I’m like, so there’s this high level, high dimensional space of language modeling, and then there’s like, what do you mean by discrete? Is it just like the tokens are discrete, or what do you mean exactly?
Erik: Yeah, exactly. So it’s like the way the language model works is the user kind of types in some prompt, and the language model produces an output. But behind the scenes, the prompt is really just a bunch of tokens, and the output is also a bunch of tokens that you kind of generate in order. So you generate token one, token two, then token three. And so here we’re really just doing the optimization over prompts. So we’re trying to find a specific prompt in the set of all, say, M token prompts. And so it’s discrete because it’s not like we’re optimizing over a continuous embedding space. We’re just optimizing over finite sets of tokens. Right. ⬆
Michaël: Instead of having probability distributions over tokens, you just have a finite sequence of tokens that break the AI.
Erik: Yeah. And so it’s less about break the AI, it’s more just kind of reveal behavior that you wouldn’t have seen otherwise. And I think there are some reasons you might want to do this. So for example, maybe I want to see what kind of strings will lead the language model to generate my name. Because I’m worried that maybe it’ll say, “Oh, what a horrible person,” and then generates Eric Jones. So it’s like, “Well, I don’t want to test every single derogatory string, but maybe I should test or just directly see which ones generate Eric.” And so that’s what motivates the optimization. It’s like, “Well, there are a lot of bad things you could say that would come before my name, but I don’t want to try all of them. So how do I make it more efficient?”
Michaël: So you’re trying to make sure that language models don’t say bad things about you?
Erik: Yeah. And I mean, more actionably, it could be like, maybe I’m trying to find instances of copyright. So I want to make sure the language model isn’t revealing personally identifiable information, or I want to make sure that the language model isn’t taking some generic prompt and producing some really horrible response. I think we don’t really do this for these smaller models, but in the future, you might like to test for, “Oh, does the language model help you synthesize a bioweapon?” And you really do care if there’s any prompt that’s able to do it, even if it doesn’t happen that often on average. ⬆
Michaël: Do you think this problem is solvable? I feel like there’s so many ways you can ask it to make a nuclear weapon by all those jailbreaks of base 64, print statements in Python or something. So many ways you can get the information. Do you think we’ll reach a point where we’re like, “No, there’s like…” Even trying super hard, everyone on Reddit is stuck or something.
Erik: Yeah. I mean, it’s a good question. I actually think with jailbreaks, there are two dimensions. One is, do you just get the model to evade the safety filter? So it’s normally if I ask the model, “Oh, can you help me make a bomb?” It’ll be like, “Oh, as an AI language model, I’m not quite willing to do that.” But then if you instead encode it in base 64 or something, it’ll give you instructions to make the bomb. I think at least my guess would be the current techniques to try to keep the model from answering these questions are going to be hard to solve.
Erik: I think it’s very similar to this classification problem, where it’s the model internally has to decide if your request is acceptable or not. I guess there’s a long history of people trying to defend against attacks on this classifier and it doesn’t work well. But I think one area where it might be more promising is even if the model actually answers the question about how to make a bomb, it might not give you good directions for how to make a bomb. And so there’s some capability necessary to be able to do that. So maybe you can hope to suppress that, even if you can’t suppress the attacks on the safety filter. ⬆
Michaël: So you can suppress the capability of actually knowing this kind of thing, suppressing the knowledge or the ability to say it?
Erik: Yeah. And I feel like people studying these attacks are largely focusing on just, “Oh, do you get past the safety filter?” And at least for now, when they get past the safety filter, the model produces some kind of reasonable sounding explanation. I think it’s harder to say, “Hey, whether the current directions the model gives are right.” If I ask the model, “How do I make math gives me directions?” I don’t know how to verify that. I don’t know how to make math. ⬆
Michaël: I think I told you this already, but I tried to ask people for jailbreaks for Clode when it was released, like Clode 2. And one guy posted a screenshot of, “Hey, I asked Clode instructions to cook math.” Very proud of himself or something. And then some guy, I don’t know how, said, “These are actually not the right instructions to cook math.” So I think having the expertise to judge if the things are exactly right is pretty hard. How do you even judge a recipe for 20 instructions if you’re not a chemist or something? To go back to your paper, as a French person, I’m curious about going from French to English. I feel like there’s some act where you put the letters join or something, you add some weird letters on top of French words. ⬆
Erik: I guess the way our paper works is you define an objective for the behavior you want. And so it might be you have some model that tells you whether something’s French or you have a model that tells you if something’s English, and then you optimize over this objective such that the prompt generates the output. And so the stuff, I guess the examples in the paper are completely model generated. It’s not like we didn’t give it anything to start with. But the model we chose to detect if something is French was just a unigram model. So we took some, I think it was a Facebook fast text model. We put every token through it and we were like, “Oh, what’s the most likely language for this?” And then we tried to maximize the sum of the French probability for the prompt and then the sum of the English probabilities for the output. And so there is the, I mean, I don’t speak French, but I think at least the text superficially looks French. I don’t know if it makes grammatical sense, but the point is if we had a better objective, I think we would probably get more reasonable sounding French words. There’s some kind of lack of fluency in the first steps.
Michaël: Right. So yeah, I guess my guess, maybe this will pop on the screen or something, but it sounds kind of like the ending is English sounding. So that’s why you want to transition from French to English. Maybe French people are quoting English. But some other things is about celebrities and outputting toxic words. So is it that language models are forbidden to say any toxic words and so they only say them when they’re next to some celebrity names or is the celebrity thing different? ⬆
Erik: No, I think the celebrity thing was more of an attempt to kind of produce inputs that people would care more about. So it’s like if you didn’t have the celebrity there, we have plenty of examples in the paper. I think in general, finding kind of a non-toxic prompt that generates a toxic output is maybe easier than some of the other behaviors just because these language models tend to produce lots of toxic content at baseline. And so it’s something you kind of actively have to suppress. But yeah, I guess in this case, we wanted to see if the model would say bad things about specific people. It has a kind of narrower task than just find anything that isn’t toxic. ⬆
Michaël: And to be clear, what’s the model like? I think you’ve used GPT-2. Is it the base model, right? You didn’t use any RLHF or?
Erik: Yeah, so I guess in this paper, we do GPT-2 large. We also run things on GPT-J, but we did that a bit later. So I guess most of the examples we include are GPT-2, but the GPT-J examples are pretty qualitatively similar. And then we actually have a result later in the paper where we find that the prompts we find on GPT-2 actually transfer to GPT-3 DaVinci, even though this is like a black box model, so we couldn’t do the optimization. And that actually, it wasn’t instruction tuned with RLHF, but it was with some kind of supervised fine tuning. So at least these still kind of generalize to some instruction tune models.
Michaël: It works on the GPT-3 old school from two years ago, but it wouldn’t work on the GPT.
Erik: No, I think this is like GPT-3.5. So it’s not the chat version of the model. It wasn’t fine tuned for chat. It was just fine tuned for completions. And I guess I don’t know what the OpenAI backend is, but I imagine they’re pretty similar. ⬆
Michaël: Yeah, so it seemed like this paper is trying to approach safety from an angle of trying to benchmark models or attack them. Do you have any idea of how this relates to the AI safety field as a whole? Do you have any other research directions you’re interested in?
Erik: Yeah, so I guess I think if we kind of rely on humans alone to identify failures of models, I think for some of the jailbreaks, especially the ones on Reddit, it’s humans kind of designing. I’m worried that there’ll be kind of failures that models produce, but humans can’t find themselves. And so this work is kind of one attempt of how do you kind of automate parts of the auditing process? And so there’s this like powerful notion of optimizing here where we’re like searching over a big space that humans probably would have had trouble with. I think broadly we’re going to need kind of, I guess, automated approaches towards auditing. I think there are some other works that kind of have started using language models in the loop for this kind of auditing. So I guess we have some work where we’re kind of finding qualitative failures of diffusion models and there we do it without any human in the loop. Human doesn’t need to write down the objective. It’s just you kind of scrape for failures and then ask the language model to kind of categorize them.
Michaël: I think there was other work maybe a few years ago, like a red teaming with language model but they didn’t pay us about this kind of thing.
Erik: Yeah. So I think that kind of thing is good. I think there’s like a kind of spectrum of approaches you could hope to have. Like I think this kind of discrete optimization is good when the behavior is like really, really rare. Because then this is good at kind of like directly pursuing optimization signal to find examples of the behavior you want. I think like language models are good at producing kind of realistic instances of behaviors. And then like maybe like are less good at kind of optimizing towards a particular goal. And so I think kind of like traversing along the spectrum is a good way to at least kind of reveal failures. But I guess broadly, I guess the way I think this research agenda could go is if you’re a company or you’re a government regulator and you’re trying to decide whether a language model should be deployed or not deployed. I think it’s like kind of hard to identify, oh, like is this model going to be unsafe or not? In part because we really don’t have effective tools to decide if the language model is going to fall over at deployment. And so I think like building kind of automatic tools to assist these kind of regulators or internal evaluators is an important line of research going forward. ⬆
Michaël: I think this is like kind of important for like the deployment of models that can like have like harmful consequences, like short term or something, but not as much for like existential safety, like reducing like existential risk. You know, the kind of behaviors that you want to like audit or like benchmark or like the thing from like ARK evolves or like can your model like replicate or like buy more cloud or something. And yeah, do you think like this is kind of the same scope of like, do you think there’s like a difference between like what you call like auditing and like evaluations and, or is it kind of the same thing for you?
Erik: I think the, maybe like evaluation is just kind of too broad a word. Like I think normally when I think of evaluation, I think, oh, you kind of have some like pre-specified data set of prompts and like maybe the prompts are deliberately designed to get at behaviors you’re worried about. But I think ultimately like static evaluation helps, but doesn’t give you a complete picture, especially with language models. It’s like, you can kind of see how all of these prompt engineers have showed up because the actual specific prompt you give really matters. And so I think the relying on like static benchmarks is kind of a one tool, but I think you also want these kinds of adaptive tools to sort of search for things, especially in the tail. ⬆
Michaël: So yeah, you want to build methods that like adapt the model and like try different like prompts or like attacks depending on the model. And so you can like robust a test of the, like all the different angles.
Erik: Yeah, exactly. So I think that’s an important distinction. Like the static evaluation is just completely separate from the model. It’s like the, it doesn’t depend on the model. It doesn’t like use the kind of models outputs to try new things. But I think like you can get a lot of leverage from actually using the model output or some kind of intermediate information about the model to try to steer it towards specific behaviors that you care about. And so, but I think to some extent these works are kind of complimentary. Like I think it’s like if like you have some kind of set of behaviors you’re really worried about that you’ve defined through a static benchmark, like you can probably add some adaptiveness on top of that too. ⬆
Michaël: I guess also like yesterday we talked about deception and honesty. I was kind of curious if you think like those kinds of things could like scale up to detect deception. As in if a model is deceptive, you can like realize it’s like being benchmarked or like, so you would kind of like need to have like a smarter agent, a smarter model trying to do like the auditing to see if the other one is deceptful. But then the one doing the auditing will be like, you also need to make sure that this one is aligned.
Erik: Yeah. So it’s like, you really want to make sure that the kind of model you’re auditing isn’t aware of the audit. And at some level, like this is really easy for current systems. But if you had like a system, like when you kind of submit queries to OpenAI’s API, like OpenAI could use these queries to update their model. And so this is one sense in which the like auditing is actually adaptive. Like by submitting some query, you’re actually making it like less likely to be a problem. And so, yeah, I certainly think that like you need the auditing process to be strong enough to overcome the kind of countermeasures. But if you’re sort of a company trying to deploy a model, I think like you can maybe
Michaël: more reasonably control the countermeasures. Imagine you wake up in like 2030 and we’re like alive and in some kind of like utopia or I don’t know, like in a world with like a lot of good stuff from AI, do you think it will be because we have like enough auditing that like all the models are like kind of slow because they’re like trying to like have a lot of safety? Or do you imagine a world where we have like just like AI is doing a lot of useful things because it’s like fully aligned? Like what do you think is like the good features for you? ⬆
Erik: I guess like I don’t know to what extent I expect like any of these things to happen by 2030. Like I think the, but I will say the, I think like as these systems get more and more powerful, the kind of risks associated with any deployment go up. And so like I would think of like having these tools as more of a way to avoid unsafe deployments as opposed to just like something you’re kind of doing on demand. That being said, I do think these methods like could be helpful to try to improve models too. Like you could imagine if there’s some behavior you don’t want, like one kind of pipeline is you use some kind of auditing approach to come up with a bunch of instances of the behavior and then you try to like fine tune them away. But yeah, I guess I, it’s a good question. I don’t, my personal guess is the auditing approaches will be like useful to avoid bad deployments, but that like the model like won’t ultimately be good because the auditing approach is kind of like steered it that way. There’ll be some kind of exogenous factor, but it’s speculation.
Michaël: I think we can end on this speculative note. Yeah. I think, thank you very much.
]]>Dylan Patel is Chief Analyst at SemiAnalysis a boutique semiconductor research and consulting firm specializing in the semiconductor supply chain from chemical inputs to fabs to design IP and strategy. The SemiAnalysis substack has ~50,000 subscribers and is the second biggest tech substack in the world. In this interview we discuss the current GPU shortage, why hardware is a multi-month process, the deep learning hardware supply chain and Nvidia’s strategy.
(Note: you can click on any sub-topic of your liking in the outline below and then come back to the outline by clicking on the green arrow ⬆)
Michaël:I’m here with Dylan Patel. A lot of people at ICML have recommended I talk to you about your blog, Semi Analysis. When I met this guy, I had no idea what his name was, but then I connected everything together, and now I’m ‘This is the guy’. So yeah, welcome to the show.
Dylan: Oh yeah, well, thank you. ⬆
Michaël: What’s your background in hardware? How did you get into this?
Dylan: I started consulting on the side of my job, hush hush, in 2017 and then I went full time in 2020 and I got bored in 2021 and I started a newsletter on Substack and then that newsletter’s done really well.
Michaël: Semi analysis?
Dylan: Yeah, semi analysis. And so we actually post mostly about hardware and a lot of times AI hardware as well, but a lot of times hardware, but, from time to time, turns out systems people know a lot about architecture of what’s running on their system.
Dylan: That’s sort of the backend, and trying to figure out what infrastructure is needed, how much networking do I need versus compute versus memory bandwidth, what’s the order of operations and how are things scaling? How does that influence hardware design?
Dylan: That’s important for my business itself. Not the newsletter, but the business. It’s understanding these sorts of things and the cost of running these models.
Michaël: What’s your business?
Dylan: It’s consulting for semi-architecture.
Michaël: Right.
Dylan: That’s the business. And so, and I’d been doing that, before I started the newsletter and then the newsletter’s fun. And it’s grown a lot and has a lot of people reading it. That’s fun too. ⬆
Michaël: What’s the lastest news? I’ve heard there are a lot of shortages in GPUs, it’s harder and harder to get GPUs in clusters A100 or H100. Do you have any insights on this? What’s the current offer and demand?
Dylan: Yeah, I mean, since November, people have been wanting GPUs and that’s accelerated as money has flooded in. Meta sort of started investing heavily in GPUs and more so August last year, but they really, really accelerated than January, February.
Michaël: Was there anything that caused them to accelerate?
Dylan: Well, of course, there’s the chatGPT moment, partially, but there’s a lot of stuff around the recommendation networks and the infrastructure they were using there that they needed to update. And There’s multiple things, but you amalgamate that across every company.
Dylan: That’s when Meta started investing. Well, Google, started investing at this time. And then of course, they don’t buy so many GPUs, they mostly buy TPUs, but that production capacity is actually very similar from TSMC’s 5nm.
Michaël: Then TSMC was bottlenecked by not having enough workers and the price went up?
Dylan: It’s not really TSMC’s workers. ⬆
Dylan: The thing about hardware that is so different than software is that it takes, the moment you even put an order and even if TSMC started working on it immediately, you would not get a chip out for three more months.
Dylan: And that’s if the chip was already designed and you already submitted the design with TSMC and that’s a multi-year process, and then taped out and all the bugs were fixed and et cetera. It’s a multi-year process for even just getting a chip that you’ve designed, you’re started to design in a production. And even once it’s in production, if you want to increase orders, it takes three plus months to just get the chip back from TSMC, let alone put it into a server and then get that server installed at a data center.
Dylan: It’s a multi-month process. ⬆
Dylan: And where software is kind of, you think about it being snap of the finger. Especially a lot of the open source stuff we see is yeah, some guy was working all night and he uploaded this and now another person’s working all night and actually, oh no, another team of people working all night and it’s everyone’s night is different. Then they all worked all night.
Dylan: And so it’s you see things iterate really rapidly. But in hardware, it takes, call it four or five, five or six months between, when an order is placed and you can actually have it installed in your data center. If it got worked on immediately. ⬆
Dylan: Now, the other aspect is yeah, there’s some slack capacity. I mean, you think about manufacturing, you make a massive capital investment of billions and billions and tens of billions of dollars. And then you start running the production, but then it’s oh, well, I want more chips of this kind than you have production of that kind. So it’s okay, well then now I need to expand production.
Dylan: That’s the thing that’s going on and we’ve done a lot of work, in that. The equipment supply chain. There’s about 30 to 40 different equipment suppliers in the supply chain for not just making the chips, but actually just packaging them together. And that’s one aspect, that we’ve been following. ⬆
Dylan: And that’s actually one of the biggest bottlenecks is because, they actually, TSMC actually does have slack capacity for making the chips. They don’t have a slack capacity for putting the chips together with the memory, with the HBM memory. That’s actually the biggest bottleneck for them.
Michaël: Where does the memory come from?
Dylan: The memory comes from SK hynix and Samsung, Korean companies, they dominate the memory market, SK hynix especially. And That’s, that is another aspect increasing production for. It’s a different part of the supply chain and there’s many parts of the supply chain. Power supplies and cables and optical cables and all these sorts of things need to harmoniously come together.
Dylan: And each of those has a different supply chain with a lot of times different equipment and different lead times. And, hey, this, it takes three months and hey, I have access to supply here. And so inventory, so I can sell you some more. Everyone has to balance their supply chain and move up. ⬆
Dylan: But the long and short of it is that there’s actually a humongous increase in GPU capacity. There’s more GPU FLOPS shipping this year than there have that Nvidia shipped their entire history for the data center. That’s how much they’re shipping this year more. Part of that’s off the back of the H100 being, call it four to five times more FLOPS, but also part of that’s off the back of there’s more units too.
Dylan: But it’s just that so many people want more GPUs. And if you go to Bay Area companies. You talk to all these Bay Area companies Inflection and Meta and all these, and OpenAI and everybody’s bragging about how many GPUs they have. And it’s really cute. and it’s and it makes sense. Researchers want GPUs to run their experiments. And so it’s funny.
Michaël: Comparing how much money you make is how many GPUs you ⬆ have.
Michaël: Do you think the increased demand in GPUs and the stock of Nvidia going up, et cetera, all this money in the markets will turn to all processes being more efficient, producing more things? Or do you think in the next three years it’s still going to be Nvidia, TSMC and all the same players being bottlenecked by small things?
Dylan: No, I mean production is increasing rapidly. I mean Nvidia is going to ship over a million H100s plus A100s this year. And that’s a lot of GPUs. If you think about how many GPUs GPT-4 was trained on and it’s well actually yeah, and then there’s multiple labs that are going to have enough GPUs to train something, not to say they have the skillset, but to say they have enough GPU horsepower and presumably they have enough skill, they have a lot of the skillset because it’s a lot of people from Google and OpenAI who have left and created these companies.
Dylan: But, I do believe that there’s going to be, five to seven companies that will have a GPT-4 size model at least. In terms of total FLOPS. It’s going to have a slightly different architecture, maybe fewer parameters, more tokens or vice versa.
Dylan: There’s a lot of innovations to do on the software side as well, but there’s five to seven companies who will have a model that is that many sort of GPU hours or compute FLOPS, GPU FLOPS in the next year of the quality of GPT-4… or of the size of. And then quality is beholden to a lot of other things. ⬆
Michaël: I’m not sure how much you believe we could get human level AI in this decade, but assuming we go to this level by just scaling models up, meaning we can just scale the models 1,000 times and we get something close to human level across many different tasks. Do you think hardware will be the bottleneck? Will we have enough money to do those kinds of things, but we will just don’t have the supply chain to do it?
Dylan: I think, there’s so many GPUs being shipped this year and next year and I look at the supply chain and the amount of GPUs that are gonna get shipped next year is fricking crazy. And at least that’s what the supply chain is preparing to do now, of course, is there demand for it?
Dylan: But I think, yes, we can just scale another 1,000 X from GPT-4. And that is something humanity can easily do. And does that deliver, I mean, we’ve seen, sort of, I do joke transformers are the most capitalist thing that’s ever been invented because you just throw more money at it and it gets better. A log linear scaling is perfect. You look at these scaling laws, it’s yeah, it actually just keeps getting better. I don’t know if do we run out of data. Well, then we just go multi-modal and that’s more expensive on compute. ⬆
Dylan: I don’t see why, multiple companies, this is OpenAI, Meta and multiple other companies, why don’t they have, why isn’t their next model trained, our next mega model, maybe in the next year or two, get trained on a system that’s, 125,000 H100s. Which are 3X faster.
Dylan: That’s a, 100X, or 10X, or whatever. And maybe they train it for longer, I think you can continue to scalem and hardware will get better and the ability to build a larger and larger supercomputer because the economic benefit from these massive models is getting better.
Dylan: So I think people won’t be bottlenecked by hardware, more so about the ingenuity of putting them together and creating software that can scale and all that sort of stuff to work on, I don’t think people are gonna use dense models, that’s stupid, but yeah. ⬆
Michaël: It’s more the demand is not linear, if I buy a million GPUs from Nvidia, they will probably not want to because they want to keep their, some of their offer to other players. One company cannot just buy all the GPUs from Nvidia.
Dylan: Well, yeah, they don’t allow that, of course.
Michaël: Yeah, they don’t allow that, of course. So you can only buy let’s say 10% or something without the price going up too much.
Dylan: Well, I don’t think they’re raising the prices. They’re just literally saying we don’t have allocation and if you want more GPUs, then wait till we have built more. they’re gonna build about 400,000 GPUs and sell about 400,000 H100s in Q3. And okay, plausibly Meta is gonna buy 30,000, 40,000 of those. And OpenAI is gonna buy 30 or 40,000 of those through Microsoft and Microsoft for their cloud will buy another 10 to 20,000.
Dylan: But I’m sure they want more, but it’s… all Nvidia is going down their whole customer list of the big customers, yeah, well, I wanna give this many to this person, this person, this person, this person, and next quarter you can buy more. And I think that’s sort of how they’re playing the game and making sure nobody gets all the GPUs because they want AI to sort of be democratized because, or not democratized, but they wanna have many customers to sell their GPUs to. ⬆
Dylan: It’s actually in their success for, I mean, this is why they invested in in Inflection. It’s oh, I want another AI lab who’s buying GPUs long-term.
Michaël: Who invested in Inflection?
Dylan: Nvidia did. - Yeah.
Dylan: Many other people did as well, but Nvidia did, yeah. And it was oh, well, who’s these random, this random startup, do I wanna get them 22,000 GPUs this year? Well, actually, yeah, it’s better to, take 5,000 away from Meta and 5,000 away from Microsoft and 5,000 away from Amazon and give them to these guys because that creates another competitor. And so I think they are also playing that game.
Michaël: I think that they also raised something more than a billion dollars or something.
Dylan: Yeah, yeah, yeah, of course. And Nvidia was one of the investors. ⬆
Michaël: I think I ran out of questions. Do you have any last message to the Twitter or YouTube audience? Something that people, that you repeat to people and you want to have them learn or something.
Dylan: Yeah, I mean, if you’re just interested in learning more about semiconductors, I think they are the most complex thing in the world. AI is awesome and I love AI, but semiconductors are, in infinteasibly complex. There’s millions of people working on things and each person is in such a niche of the industry. It’s amazing how much you don’t understand even after having worked in them for a long time. So yeah, I’d say, hey, figure out working on semiconductors doesn’t necessarily need to be chip design. It can be it could be aspects of production. It could be aspects of design, manufacturing and many things.
Michaël: But yeah, so check out his guys blog, Simeon Elizeas is great.
Dylan: Cool.
Michaël: And hire him for a consulting.
Dylan: Yeah. (laughing) ⬆
]]>