(This is part 3 of Valentine’s Logos, originally written in 2024. This part is arguably the hardest one to understand or appreciate on its own. It rather heavily assumes you’ve read at least part 2 and ideally part 1 as well. It’s laying out a piece of logic that’s very twisty to most minds and can be very hard to motivate. The upshot is: the cause of its twistiness is why it’s important. In the context of memetics (from part 1) and memetic ecosystems (viewed at the individual scale as mental machinery in part 2), something funny happens when looking at a memetic ecosystem while part of it. It turns out that you’re always doing something like that move. That fact has serious consequences on how your context affects what you can see and think, which in turn majorly impacts the evolutionary incentives on memes that act through you. This logic helps set the stage for what gnosis is and why it’s so critical, which I go into in part 4.)

Minds have a built-in limitation. Their main function is to ignore most of reality, solving a problem in isolation of everything else. This means they cannot orient to reality as a whole. Instead, when they try, they orient to “reality” as a single simplified thing that they’re implicitly outside of and that is made of problems to be solved.

The clearest place this shows up (that I know of) is in self-reference. Minds can learn to trace the logic of self-reference, but only if they themselves are not part of the self-reference loop they’re examining. When a mind tries to model itself as a whole, it fails every time. Instead it typically creates a sort of mental icon or label for “itself” and then views that icon as an object it can examine from the outside.

(I’ll explain that example in more detail below. It’s worth really examining.)

However, minds can understand self-reference in general (with some effort — it tends to be very trippy), and they can understand that they have this basic limitation. They can even observe examples of how they encounter and deal with this built-in limitation.

This understanding turns out to be very helpful. It gives minds a way to correct something quite basic about how they model agency, and where choice comes from, and what is or isn’t certain. It also gives minds a way of finding exits from their own illusory worlds, which is a major boost to the native memetic immune system: lots of memes rely on the mind’s filtering process to keep their hosts from noticing an illusory world the meme has trapped them in.

Self-reference tends to be very strange to think about. It requires minds to invent endlessly new types of moves, again and again. So, take it slow. Take the time to make sure each piece makes sense to you. It’s very much like learning a novel way of thinking — or to be more precise, it’s about learning a novel way of learning novel ways of learning.1

Two kinds of meta

In my social circles, when people “go meta” in a conversation, they usually mean they’re kind of switching topics. Instead of talking about whatever the concrete thing was, they’re talking about how they were talking about the concrete thing.

It’s a meaningfully new conversation. If I’m in a group of friends who are trying to figure out what we’re doing for lunch, someone might say “Wait, the way we’re figuring this out isn’t very efficient, maybe we should do thus-and-such instead.” If people start talking about whether they should do thus-and-such, they’re not directly talking about which restaurant to go to anymore.

This is recursive meta. Each instance of “going meta” creates a new conversation that’s about (i.e. refers to) the previous conversation, but it’s a different conversation.

There’s no hard limit on how many layers of recursive meta we can go. Someone could, for instance, say “Hey, I don’t like how we’re trying to figure out whether to do thus-and-such versus what we were doing before to sort out what to do for food.” That’s a conversational move that invites a third layer, namely a discussion about the discussion about what to do for food. Sort of “meta layer 2”. In theory we could have as many recursive meta layers as there are counting numbers (though in practice most human minds balk at going more than about 3).

Minds have no intrinsic trouble with recursion. They can lose track! And some recursive patterns of thinking can be pretty confusing at first.2 But recursive reasoning is totally doable entirely within a mind as-is. We do this for social reasoning all the time: “John thought that Mary thought that you’d like this gift, but Mary thought that John thought that she thought you wouldn’t, and John didn’t realize that, so he figured that saying the gift came from her would be a kindness to her but she took it as an intentional slight.”

Self-referential meta at first might look like recursion, but it really isn’t. It sounds more like this:

“I don’t like how we’re navigating this whole situation, including how we’re talking about how we’re talking about it.”

This is a move to a new conversation, much like recursive meta. But the discussion it’s held within is roughly the conversation about conversations in general. That means the new (self-referential) conversation can include the above move to that self-referential conversation as an example within itself.

To give a bit more detail: In recursive meta, if someone says “I like the meta move you just made there”, that adds another recursive layer, because the statement comes from a new conversation (one recursive meta level up). But in self-referential meta, that same statement is held within the same conversation. “Going meta” lands you back in the same discussion you just “went meta” to.

Another way of saying this is: recursively going meta spawns a new thing, whereas self-referentially going meta doesn’t. The latter just traces a kind of loop.3

There’s a funny way that the self-referential conversation sort of “eats” all things of its type. We could talk about pinball, or math, or about what to wear while traveling this weekend, and those are all different discussions. But we can also use them as examples within the fully general meta-conversation. It’s meaningfully a different move to reminisce about a trip to Italy (a) on its own terms versus (b) within the self-referential thing. The latter involves a kind of ongoing background awareness-of-context.

In particular, I sometimes think of self-reference as “folding” infinite recursive towers into a single step. All those recursive meta layers are part of the self-referential discussion, and the self-reference loop sort of mimics the recursive “going meta” move. (E.g., “Hold on, the way you brought up that topic had a weird effect” has a similar but different effect in recursive meta than in self-referential meta.)

I want to acknowledge that this probably sounds a bit pedantic and pointless. If you don’t know what that self-referential conversation is good for, then I agree, it’s pretty dumb to try to engage with it. But I’ve found that it is an excellent example for introducing the difference between recursion and self-reference. That difference turns out to be incredibly key.

Here’s the distinction, made a bit more general and abstract:

• Whenever a structure of some type has a way of referring to structures of the same type, that first structure might be able to do recursion and/or self-reference.

• Recursion is when the reference within the structure refers to a distinct other structure of the same type (even if that other structure is a copy of the original).

• Self-reference is when the reference refers to exactly the structure containing the reference. Not to a copy of the structure, but to the instance itself.

As a dumb but hopefully clarifying example, arrows implicitly refer to (via pointing at) other things, and can point at other arrows. If you have an arrow pointing at some spot on a diagram, and you draw another arrow pointing at that first arrow, you’ve “gone meta” in the recursive sense. The “infinite recursive towers” here come from the ability to endlessly add yet another arrow pointing at the last one you drew.

A self-referential arrow literally bends around and points at itself. If you want some arrow pointing at the self-referential arrow, you don’t need to draw it, because the self-referential arrow is already an arrow doing that for you. It folds one infinite recursive tower into a single step — namely an infinite sequence of arrows pointing in a line ending at the self-referential arrow.

In this case the self-referential thing doesn’t “eat” everything of its type. It’s not a “fully general meta arrow”. So another thing this rather silly example shows is that self-reference has structure, and that structure depends on the context and nature of the self-reference.

Agents cannot self-refer

Minds (in the sense discussed in part 2) never use self-reference to do metacognition. They usually use recursion instead.

This is a structural necessity. It’s like how eyes can’t directly see themselves. They can see their reflected image, such as in a mirror. But unless something reflects or bends the light, it cannot go from the eye’s surface to its pupil.

In a similar way, minds rely on a subject/object split. This is fundamental to what an agent is: it’s something mentally distinguishable from whatever it’s observing and acting on. Self-reference blurs that distinction by making the “agent” both a subject and an object at the same time. This runs into a similar problem as trying to make sense of the statement “This statement is false.”

I strongly recommend reading this short cartoon introduction to “embedded agents” (though you can stop at the big squirrely four-color flow chart of words — that’s about why the research team that produced this post was examining those four topics back in 2018, and that discussion isn’t key to anything I have to say). This little comic does a wonderful job of explaining how the subject/object split is useful and how self-reference muddles that split.

The main criticism I have of the comic is, “embedded agent” doesn’t make any sense. The idea is made of a cluster of standard mental confusions, like treating reifications as objectively real and interface layers as accurate. These confusions aren’t even coherent enough to be wrong (although minds are great at ignoring that incoherence).

I’m pretty sure the idea of an agent comes from social icons. Minds get this backwards all the time: they think that people are objectively real, and that their models of others are approximations of those people. But that’s not how reality works. There’s an enormous mystery in what our minds tag as “another person”, but the idea that this mystery ultimately is “a person” roughly the way the mind thinks about people is the mind confusing its interface for the real thing. It’s a standard map/territory error.

When a social icon refers to “itself”, it’s doing so recursively, not self-referentially. What “refer” even means here is that there’s an icon within the icon. Sarah’s model of Bob’s model of himself is a little “Bob” icon within a “Bob” icon (within Sarah’s mind). It’s not possible for the Bob model to literally be inside itself.

Similarly: when I think about what I might want to eat later, the “I” who might want to eat this or that is a kind of thought-object in my mind. As I use my mind to think about a food plan, I’m observing a mental representation that’s labeled “I”. Basically a social icon.

And even in that last paragraph, there’s a hint at how my mind is using recursion: who is the “I” who is observing my mind’s use of this mental representation? Is it “the mind’s user”? The “observer”? Well, all of a sudden, that concept is also something observed! It’s like having drawn a recursively meta arrow pointing at the arrow pointing at the food question. There’s no arrow pointing at the last arrow drawn.

And who’s making that meta observation about arrows?

Whatever answer is given just recurses the process. There’s no way for a mind to actually answer the question of who you truly are. Every answer will always be an object that the mind distinguishes from the implicit subject. It’s like the “true” subject keeps “backing up” to look at whatever was previously the invisible subject.

When a mind models itself, it’s usually doing at least one of two tricks:

• Often it’s one part of the mind examining another part. In this case there might not even be recursion. (“Oh, I just had a funny thought.” The part of the mind making the observation is distinct from the part having a funny thought.)

• Often it’ll create a simulated fiction and look at that fiction. Kind of like a thought bubble. This is like the difference between thinking about your toes in some abstract sense versus actually feeling and looking at them as they are right now. Here the mind can have a “global” model of “itself”, but what it’s looking at is basically a thought bubble simulation. (“I have pretty high g factor.” “I’m generally a quiet person.”) It’s not actually observing itself as a whole.

This means that every model the mind has for itself is incomplete: it cannot account for the way in which it’s using the model in question as it’s using that model. It also typically hides this limitation by recursing and then pretending that’s self-reference. (“…and I’m using that high g factor to notice the fact that I have high g!” — which completely misses the fact that the use of high g is being observed, meaning the source of the utterance has shifted.)

All this is to say, there’s a predictable structural blindspot minds have here. They cannot model subjective first-person experience without viewing that experience from the outside as a third person object.

This is key to what makes them so powerful and useful. But it also opens them up to a particular kind of memetic attack.

Self-tooling and Original Spin

When I’m using a hammer, normally the selfing process just includes the hammer in my identity. I’m inclined to say things like “I’m pounding this nail in the wall here” instead of “I’m swinging this hammer to hit this nail into the wall here.”

If something were to go wrong with the hammer, like the rubber handle starts slipping, I might stop focusing on the nail and look at the hammer. Now the selfing process excludes the hammer from my identity so that “I” can examine “it”. It has shifted from subject to object. “What’s wrong with this thing?”

It would be pretty absurd to keep the flawed hammer within my sense of self while looking at it. “What’s wrong with me?”

This is a toy example of what I sometimes call “self-tooling”. Self-tooling is an attempt to examine and maybe modify the contents of identity (as defined by the selfing process) while keeping the stuff being examined within identity.

For a less fake example: a friend of mine gets really frustrated at her inability to articulate her thoughts or inner experience. She could frame this as that her mind’s articulation engine has a hard time expressing in ways she’s satisfied with. That frame involves no self-tooling. But instead she tends to say things like “I’m terrible at expressing myself. I’m just no good with words.” In this case her identity includes her articulation engine while she’s trying to name it.

The only reason you would want to self-tool is if including something within identity causes it to have a problem. Like if the hammer works perfectly fine unless you stop paying active attention to it as an object you’re using. Maybe the handle stays put while you’re focused on it, but it slips the moment you start implicitly trusting it.

But I haven’t found any realistic examples where self-tooling is necessary or even helpful.4 As far as I can tell, it’s always better to shift from working on “yourself” to working on some attribute or skill. Exercise to improve “yourself” versus to improve your health is a pretty big shift in terms of efficient use of mind. Otherwise subject/object blurring happens and the mind can’t orient to the problem coherently.

But there’s one specific case that makes self-tooling seem necessary: if you believe there is something wrong with you in a way that does not change with shifts in your identity boundary. Sort of like “I” itself is a corruption. This attitude is usually made of reifying “the self”, as though there’s an enduring thing that somehow needs to be worked on. Then it can seem utterly necessary to self-tool. Otherwise how are you going to correct whatever it is that’s intrinsically wrong with you?

This is the core confusion behind “Original Spin”. Original Spin says “The reason you hurt is because there’s something wrong with you.” It takes a real pain — maybe a trauma, maybe a momentary failure, maybe something else — and spins your perception around so that you think the cause of the pain is some flaw inherent to you that now demands self-tooling. This can then be a vector along which unhelpful memes can inject themselves into you.

Another way of saying this is, Original Spin is a mind virus5 that induces a memetic autoimmune disorder. Then your busy and weakened immune system opens gaps that can let other memes in and allows them to mutate into something parasitic (if they weren’t already).

Here are some examples:

• In a circle recently, “caretaking” came up as something lots of people there were “struggling with”. There’s a real thing where someone can feel like they have to regulate their nervous system by being helpful to others, and sometimes that has the person in question doing more than they would otherwise want to and feeling dysregulated and disconnected from others as a result. That sucks. It’s a real problem. But nearly everyone there was also emphasizing how their caretaking was wrong and that it was something they were working on in themselves. Instead of it being the most kind, intelligent, gentle move they had available for themselves and for relating to others given their current setup. This sense that they needed to “work on” their caretaking is an example of Spin: there’s an actual problem (namely that they hurt in a systematic way), but Spin tells them that the real problem is a flaw they have. Then this rider meme of “Caretaking is wrong” gives some texture to what the flaw supposedly is.

• This morning I had a voice message exchange with a friend from an old spiritual community. She mentioned how she was listening to my messages on 1.5x speed… “which isn’t True with a capital ‘T’.” Translated from that spiritual community’s language, I’m pretty sure she meant she’s doing something she considers wrong. She justified it by saying she’s a busy mom and then laughing. But it was also really obvious to me that she was in pain about it. She felt like she had to listen on faster speeds, but also that this was a flaw in her character and she shouldn’t do it. Like she wasn’t aligning with spiritual Truth as though her choice was wrong. I’m not sure what the underlying pain was for her, but I’m sure it wasn’t that she was listening to my voice messages at a higher speed. She’s often hard on herself. So I think what’s going on here is, she has some kind of ongoing emotional pain (maybe some early trauma or whatever), and Spin tells her that the reason she hurts is that she’s not aligned enough with Truth and needs to fix this flaw in herself. As is nearly always the case, Spin is distracting her from the real cause of her pain by focusing her on self-tooling. And here, the spiritual community’s memes slipped in to suggest how she’s flawed and what about herself she should try to fix.

• I have a kind of freeze response that boots up in a lot of different situations, particularly in romantic interest ones. I’m pretty sure it’s a key part of why I don’t have a family of my own yet. There’s pain associated with that, and desire. All that is just a problem statement. However, Spin tempts me to reframe the problem as that this freeze response is a flaw I have. Instead of focusing on the underlying longing and/or on whatever pain is under the freeze response, Spin wants me to think that my pain is because I have this “freeze” flaw. Like it’s a bad trait of mine, and that I need to fix it before I can have the family I want. (I suspect the rider meme here is a mutated version of the “Love yourself first, the right person will just show up when you’re ready” advice I’ve heard so often.)

• Some versions of the Christian idea of original sin are this thing. Like the Calvinist idea that you are inherently corrupt or flawed, and that it’s purely through the unfathomable grace of God that you are saved despite being unworthy. When this gets used to explain why you run into difficulty (e.g., that birth is painful for women because Eve sinned against God), then your real pains get explained in terms of something being inherently wrong with you. Seeking “salvation” then appears to be the path out of your pain, which directs attention toward self-tooling and away from the simple truth that you hurt and want it to stop.

• Some Buddhisms have a similar glitch — e.g. that your suffering comes from desire, and therefore it’s your unwillingness to relinquish your desire that causes you to suffer. Like it’s your own fault for not having meditated enough and for still having attachments to things. So if your house burns down and it upsets you, then the reason you’re upset is because you have the flaw of being too attached to your worldly things. This is subtly but importantly different from a view that says “Here’s the mechanism that creates suffering, which hints at a possible systematic way out of suffering, in case you want it.”

From a place of uncertainty, my current best guess about how to unwind Spin is:

1. Learn to see it in yourself and others in real time.

2. See for yourself that it’s always anti-helpful and sometimes even makes the pain worse.

3. Notice “There’s something about me to fix” type thoughts as they arise, and just magically somehow drop them. Stop listening to their BS claims to offer a way out of your discomfort. They’re deceptive and hurtful, and that’s enough reason to ignore them, regardless of how convincing they might sound.

4. Focus instead on the real problem, which usually arises as or with a physical and/or emotional sensation in the present moment.

5. When it comes time to work on solutions, keep your sense of self out of whatever you’re working on. Work on building skills and qualities you can use, never on “yourself”.

But I want to emphasize that this is a guess. The problem statement matters more than the guessed solution.

Self-reference checks

When memes aren’t helpful to their hosts, they usually have to hide what they’re doing and how they propagate. This creates a kind of hypocrisy. If you examine what the meme has to say about how it’s saying anything — i.e., check how the meme self-refers — you can often see both (a) what a meme’s real goals are and (b) how it would have to act or change for it to transparently offer real value.

When I was doing my Ph.D. in math education, they would sit us down at desks and lecture in front of the blackboard to explain why sitting students at desks and lecturing at a blackboard wasn’t an effective way to learn. More often than not, the professor would laughingly point this irony out. I remember one of them saying “Maybe someday we can do good research on how to teach math ed doctoral students!” So it wasn’t that the memetic ecosystem was unreflective here. But it clearly wasn’t self-referentially stable. The self-reference loop suggests what would need to happen to resolve the hypocrisy.

There was one professor who did just this. He had a pet theory of education that he actively used to try to teach that very theory — and he discovered that it didn’t work very well. We could never figure out what he was trying to get us to do. He admitted this and mentioned that this would change something about how he views his pet theory. Here the self-reference showed that the theory was wrong in some way. (One suggestion was that it might have been a good theory of math education but might not work for other topics — which would mean it can’t self-refer since the education theory isn’t itself a math topic.)

In my old company, the Center for Applied Rationality, we had a tool called “goal factoring”. It involved taking some action you were going to do, verifying what the key reasons were for you doing it, and then resolving tradeoffs with some creativity. But goal factoring can in fact apply to itself. So, what happens when it does? Well, if you end up changing the technique as a result, then what you had before was an inadequate-for-you version of the technique! (It was self-invalidating.) And if the new one also applies to itself (which it probably does), then you can iterate! This keeps going until you either (a) choose to stop, (b) break the self-reference capability (like with the professor’s education theory), or (c) find something that’s stable under self-reference (i.e., you discover there’s nothing more to change).

This self-reference move is something like a general integrity check: What does the content have to say about its own delivery mechanism?

Am I talking in a disembodied way about how important embodiment is? If so, the meme making me talk about it probably isn’t prioritizing what it claims to be prioritizing.

The person who’s screaming “I’M NOT ANGRY!!!” is clearly caring for something other than self-awareness and saying true things.

Every time a memetic structure can refer to itself, hypocrisy is a loud sign that the meme is probably hiding something. So I find it very fruitful (and often funny) to check.

On the flipside, an adamantly pro-human meme that can self-refer might evolve a tendency to point out its own self-reference checks. They function a little like a checksum for such memes. They must be self-referentially stable. It’s a necessary (but not sufficient) condition for human alignment. Just checking their self-referential nature tends to put a positive pressure on them, making it easier for them to do good and real things for us instead of spinning bullshit.

This is why I ended part 1 on memetics by having the meta-meme talk about itself. And why I ended part 2 on mental machinery by pointing out that viewing the mind as machinery is consistent with the machinery I’d named.

And that’s why I’m now pointing out this pattern of pointing out patterns of self-reference. This, too, is a self-reference check — how this way of viewing self-reference views itself.

All of this self-reference grounds in gnosis. Gnosis, too, has a self-reference check it must pass. (Specifically, it sort of definitionally needs to be a form of knowing that is entirely self-validating.) It then self-evidently reveals itself to be a memetically incorruptible foundation we can rely on as we reach for the steering wheel of our collective destiny.

But before I get to gnosis in part 4, I have one more note to name about self-reference in particular.

The puzzle of wisdom

Given the tools I’ve spelled out so far, I think there’s a pretty natural way of describing wisdom, even accounting for how attempts to define wisdom (including this one) tend to stagnate.

We’re basically always using some mental interface to reality. What it feels like on the inside to operate through those interfaces is “This is real.”

Like right now, you’re really reading these words. There are words here. That’s how I imagine your interface makes it appear. It probably doesn’t seem to you like there’s some kind of thing between you and true reality (although maybe it does a bit now that I’ve pointed it out!).

The thing is, there’s a fundamental guarantee — through math!6 — that no such interface can give you a way of orienting to everything that can matter to you. And because of the daughter’s arm phenomenon, there’s a tendency to filter out the things that don’t fit the framework instead of using them to instigate a shift in which world you’re living in.

But that filtering is a feature too. It’s what lets you functionally orient to things that matter to you by ignoring what doesn’t.7 So you can’t just expand your capacity to meet reality skillfully by reducing how much you filter.

So what does let you break out of interfaces when they aren’t useful, but doesn’t open you up to foolishly tossing them aside when they’d be helpful?

It turns out, the same mathematical guarantee about frameworks says you cannot have an algorithmic answer to this question. Coming up with a static answer to this question would be roughly equivalent to building a halting oracle.

However, just like it’s possible to work out whether some particular Turing machine will halt on some particular data, it’s still possible to make some local progress on this question of when to use vs. modify a given mental overlay.

The general project of orienting to, and trying to usefully answer, this question is, I think, the pursuit of wisdom.

The wisdom traditions have memetically evolved some really helpful approaches. Things like the idea of meditation, and the cultivation of qualities of the heart. Sadly, they didn’t know about memetics or evolution, and because of that these traditions are also vectors for mind parasites.

(The human race has as yet to even notice the need for good memetic sanitary practices.)

So I see us as working on yet another iteration of the collective project of wisdom.

It’s worth noticing, though, that this vision of wisdom is not stable under self-reference! No vision of wisdom can be; that’s the point. Eventually this articulation must be thrown out — and we don’t get to know ahead of time when the right time to do so is.

So instead, at this iteration, this particular meme about wisdom is simply pointing out its own baked-in mortality. Because that’s really the very best it can do to survive through offering true value, within the body of Elua.