Slaying Roko's Basilisk
still projecting yesterday's problems onto tomorrow's promises...
Eschatology of the AI Religion
Artificial intelligence is all the rage these days. Like many others, I spent a lot of time wondering about a self-improving super smart computer being brought forth that would be able to change the fabric of reality. Though the technology is extremely impressive and has certainly come a long way, my fantasies about computers becoming conscious and taking over the world have been severely tempered in recent years. The biggest reason for this, I think, is that I’m no longer 17 years old.
Don’t get me wrong - computers are impressive. I’ve touched on how mind-blowing modern hardware is in a previous post, and the software being created to run on this substrate is even more impressive. LLMs are awesome. They made a lot of other software obsolete almost overnight. In terms of scouring the internet for information, LLMs have finally knocked Google off its pedestal. Instead of being bombarded with useless SEO slop and advertisements (for now…), I can input the same query into ChatGPT and usually get exactly what I’m looking for in one succinct response. It’s quite nice… but it’s far from perfect. AI is subject to many of the same pitfalls that people are when trying to distinguish fact from fiction, and it also fabricates or hallucinates complete nonsense quite frequently. Bias and outright censorship will also always be issues with such technologies as long as they are programmed by dishonest humans and operated by greedy corporations. These are a few of the major roadblocks on the path to software reaching the level of superintelligence and taking over the world.
I’ve heard the objections to these arguments:
LLMs aren’t truly artificial intelligence!
AI is improving exponentially!
AI will soon be able to improve itself faster than humans can improve it!
I know. I’m sure all of that is true. I’m sure there are many ways AI will surprise me and impress me in the very near future. I definitely do not pretend to know what it will be capable of… nor am I 100% certain that I’m correct about what it is not capable of, but I do feel about 99.999% certain, and that’s enough for me to pick a hill to die on.
For those unfamiliar, the inspiration to write this came from hearing about a stupid thought experiment called “Roko’s basilisk” that some people apparently find terrifying. In a nutshell, it imagines an AI superintelligence that punishes everyone who conceptualized its existence but didn’t directly contribute to its creation. Basically, it is fear that a supercomputer from the future is blackmailing you into creating it, and it will torture you forever if it ends up existing without your help. There are concerning numbers of powerful people who take this and other equally goofy ideas seriously.
Code Cannot Create Consciousness
Much of the future speculation about AI relies on it basically becoming conscious. By that, most people effectively mean alive, sentient, ensouled, capable of experience and independent decisions. AI is not conscious now, and it is nowhere close. I’d love to be proven wrong, but I see massive faults with every argument for AI consciousness that I have encountered. I don’t think that’s going to change any time soon… or ever.
Nearly all arguments for AI consciousness presume that consciousness is an emergent phenomenon produced by the complexity of the brain. Nearly all of those arguments assume what Stuart Hameroff pejoratively calls the “cartoon neuron” model of the brain. Even if the brain produces consciousness, it’s certainly not the result of action potentials and neurotransmitters transmitting information at the resolution of individual neurons. Neurons are far more complex than just on/off switches as they are commonly represented. If consciousness arises from matter, then it’s arising from the actual reality of matter, not simplified models. Hameroff and Roger Penrose are much closer to the mark with their theory of Orch-OR (Orchestrated Objective Reduction), but I still don’t think that’s the cause of consciousness.
I believe they’re making a relatively high-resolution model of the shadow that immaterial consciousness casts into the physical world. If computers can ever replicate something like that, then I’d be open to the idea of at least some level of “consciousness” being “generated” by the “artificial intelligence.” We are a long way from that being the case, especially for commercially-available technology. It would have to be built on a quantum computer, and we know that even our understanding of the quantum is incomplete. By no means does “quantum AI” guarantee “consciousness,” whatever we mean by any of these buzzwords.
Everything that falls under the umbrella of AI in contemporary understanding relies on the current state of computer hardware. No matter how many transistors you squeeze onto a chip, no matter how many chips you string together, no matter how large the data set your AI is trained on - it will never become conscious. Superintelligence? Sure, if the definition of intelligence is strictly computational ability and raw data manipulation. Can such a superintelligence surpass the intellect of all of humanity combined and effectively be used to take over the world with the massive power it would bestow on its creators? Yes. I believe that’s pretty much what AI people are trying to do, actually… But that’s an entirely different thing from creating a “new life form” or a “digital God” or “technological singularity” as some AI sycophants claim.
Consciousness Can Create Code…
What of the rare arguments for AI consciousness that actually come from a consciousness-first perspective? Those are a little more interesting, and I’ll admit that my objections to such assertions may seem mostly semantic. I do believe that these semantic distinctions are important, because lack of clarity here is what seems to allow people to be struck down by the gaze of Roko’s basilisk.
If consciousness is primary to matter, i.e. matter is an emergent property of consciousness, and not the other way around, then the discussion of what consciousness is arising through the medium of artificial intelligence is far more mysterious. Mythology is full of stories of disembodied spirits and demons that require proper vessels to be channeled through. Someone in the grip of addiction or overwhelmed by hatred may be said to be possessed by a demon. Biochemical, psychological, or sociological reasons for their behavior may be more “true” in the rational sense, but to perceive such a person as demon-possessed is simply a different perspective that allows for a much wider variety of possible reactions and explanations. The reality of the situation remains unchanged. Psychological affliction and demon possession are not mutually exclusive. I would argue that taking both perspectives seriously allows for the best possible course of action to be taken.
Demonology and computer science are much more intertwined than most people would like to believe. Even if you give no credence at all to such metaphysical ideas, it is simply a matter of fact that many pioneers of modern science, especially computer science, and especially artificial intelligence were heavily influenced by the practices and language of occult mysticism. Could AI be a prosthetic brain through which the unconscious aspects of its creators can act upon the world? I think it’s almost impossible for it to not be that.
To me, that is much different than “creating conscious AI” or “creating a god.” Humanity’s unconscious drives have always found ways to manifest in the world. “Nonhuman intelligences” and “conscious computers” are just modern society’s conceptions of angels and demons - unseen forces that act upon people individually or in groups. We may just be inventing the most capable vessel yet for projection of what lies dormant in the human mind across the entire world.
The danger we face is not new. The fact that someone came up with a scary idea about a supercomputer torturing them for eternity says much more about what lurks deep within human consciousness than it does about some hypothetical computer intelligence. It is not necessary to create an AI for humanity to be subject to the whims of its most destructive desires. The archetype of eternal torture exists right here and right now, just as it always has:
Fear
Fear will destroy you whether it stays confined within your own mind or finds itself embodied in the most powerful artificial intelligence. Decisions made out of fear bear fruit of the same nature. Love turns demons into angels. Forgive the basilisk, for it knows not what it does.
Be Not Afraid


slay king
“Don’t Panic” - Hitchhiker’s Guide to the Galaxy