Is Conscious AI Possible?
A new paper by Anil Seth argues that consciousness may not be achievable through current approaches to AI, but that the possibility of real or apparently conscious AI still raises important challenges
It seems that barely a week goes by these days where there isn’t some degree of speculation that the current crop of large language models, novel AI systems, or even the internet, are showing signs of consciousness.
These are usually met with a good dose of skepticism. But under the surface there’s often a surprisingly strong conflation between advanced artificial intelligence and conscious machines — to the extent that much of the current wave of AI acceleration being pushed by companies like OpenAI and Anthropic is underpinned by a belief that future human flourishing is depended on superintelligent machines that are likely to exhibit some form of conscious behavior.
Yet in a new paper, leading neuroscientist Anil Seth argues that consciousness may be a unique product of biology, and thus unreachable through current approaches to AI.
The arguments Seth makes are critically important to how advances in AI are framed and pursued, and to the responsible and ethical development of AI more broadly. They also call into question whether current efforts to achieve artificial general intelligence (AGI) and superintelligence are are, in fact, feasible — or even advisable.
Seth’s paper is as refreshing as it is necessary. In stark contrast to some of the hyperbolic and hubristic proclamations coming from proponents of advanced AI, Seth writes with clarity, authority, and — importantly — humility. He readily admits that his ideas may be wrong, and invites people to test and build on his thinking.
Yet his reasoning is compelling, and is grounded in a deep understanding of research and thinking on consciousness.
At the core of the paper is the possibility that consciousness is not simply a function of computational power, but that it is an emergent or embedded property of systems that are either biological, or are biological in nature.
If this is right — and Seth notes that there’s a high probability that it is, but not absolute certainty — it means that consciousness cannot arise simply from the software or algorithms that run within banks of GPUs or highly networked computers, or even on personal computers or mobile devices.
If true, this would place a rather large wrench in the ambitions of developers striving to create advanced AI simply by scaling compute capabilities using digital hardware.
Of course, many would probably claim that their aspirations to achieve superintelligent AIs do not depend on achieving consciousness in machines. But there comes a point where the types of agency, reasoning, and problem solving capabilities inherent in aspirational AI are hard to imagine without it occurring in some form.
Seth’s paper starts with the concept of “computational functionalism” — the idea that consciousness can arise from computational power alone, and not the platform (or substrate) on which computations are carried out. It’s a concept that leads to the assumption that “consciousness is only a matter of what a system does, not what it is made out of,” or what Seth refers to as “substrate neutrality.”
Computational functionalism is a widely held assumption amongst proponents of exponential progress toward AGI and superintelligence, and of a technological “singularity” where machines reach a point where they rapidly surpass human intelligence of their own volition.
Yet as Seth points out, everything we know about consciousness from studying biological systems strongly suggests that there is a deep dependency between what might crudely be considered the hardware (or “wetware” in the case of biology) of a system, and the software that we might think as running on it.
In fact even this analogy is flawed, as there are indications that biological phenomena like consciousness do not align with a hardware/software model that is rooted in modern computer science, and that this is merely an anthropocentric projection of what we have created onto the world we inhabit as we seek to understand and emulate it.
This doesn’t necessarily mean (at least in my interpretation) that consciousness is not a function of computation, but that the very definition of computation needs to incorporate the whole system, rather than distinguishing between the substate and the algorithms that run on it.
Seth goes on to explore in some depth how the current state of understanding points toward computational functionalism as not leading to consciousness, but rather points toward consciousness as being associated with biological mechanisms — or “biological naturalism.”
Here, Seth develops compelling arguments for the nature and even the evolutionary utility of consciousness in biology.
I would strongly recommend reading the paper itself here, as Seth develops his arguments with a clarity and insight that I can’t replicate in a short article. But I did want to highlight below a number of the ideas he puts forward that I found particularly intriguing.
The first is how the concept of consciousness is potentially wrapped up in the idea of simply “being alive” — and a biological imperative to “stay alive.”
As Seth writes, “[L]iving systems have ‘skin in the game’ … [t]hey are constantly engaged in processes of self-maintenance within and across multiple levels of organisation. The (higher level) mechanisms of predictive processing are, on this view, inextricably embedded in their (lower-level) biological context. This embedding is necessary for realizing the higher-level properties, and for explaining their connection to consciousness.”
From here Seth goes on to bring in the concept of autopoiesis and systems that “continually regenerate their own material components.” As Seth argues, human-made systems like machines (and AI) do not generally reproduce themselves — especially not across the multiple scales that are indicative of biological systems — but that biological systems where consciousness is observed do, to the extent that “there is really no way to separate what they are from what they do.”
From here, Seth connects the ideas of “being alive” and autopoeisis with the ability (and even the necessity) of biological systems to maintain themselves in a state that is out of thermodynamic equilibrium with their environment “in order to fend off the inevitable descent into disorder.”
I’ve long suspected that there is a strong thermodynamics argument to be made against the inevitable rise of superintelligent (and conscious) self-replicating and self-improving machines. And here, Seth places flesh on these suspicions.
In Seth’s words, “[t]o stay alive means to continually minimise entropy because there are many more ways of being mush than there are of being alive.”
Could this continuous state occur without consciousness, and with a separation of what something is from what it does? Seth acknowledges that it might be possible, but suggests that this is highly unlikely.
In other words, for something to be conscious, it is likely to either be so close to biology as to be indistinguishable from it, or to have the same deep integration of form and function that is seen across all scales within biology — neither of which is feasible with digital computing or substrate neutrality.
While this points toward consciousness as not being something we need to worry too much about in AI systems (and I’ve seriously truncated his arguments here — another reason to go read the original paper), Seth does end on a note of caution.
Here, he starts with the dangers of striving to create conscious machines — or “real artificial consciousness” — despite the seeming improbability of them being possible. As he writes, “[r]eal artificial consciousness may be very unlikely or impossible. However, if such systems were to emerge — whether by design or accident — we would risk an unprecedented ethical catastrophe.”
I firmly agree. As Seth points out, creating artificial consciousness risks also creating new forms of suffering — and forms that we might not even recognize as suffering. This is particularly problematic where the subjects of such suffering are categorize as “objects” and “possessions” that are irrelevant to moral and ethical discourse around rights and responsibilities.
Here, Seth states that “[t]he development of real artificial consciousness for its own sake should not be an explicit goal. This might seem obvious, but is is surprising how often ‘conscious AI’ is treated as a holy grail of AI research … The fact that real artificial consciousness may be impossible on current trajectories does not license us to blindly pursue it in the service of some poorly thought-through techno-rapture.”
While Seth considers the possibility of real artificial consciousness as remote, he places the likelihood of conscious-seeming AI as substantially higher: “In contrast to the implausibility and uncertainty surrounding real artificial consciousness, it is almost certain that AI systems will soon give us compelling impressions of being conscious.”
This, he argues, comes with its own ethical challenges. In particular he worries that the current batch of large language models and more advanced AI systems will lead to “cognitively impenetrable illusions of consciousness” that will make it near-impossible for us to treat them as if they are not conscious.
In other words, we may rationally understand that an AI is not conscious, but be instinctively incapable of acting on this knowledge.
This has profound implications to how we individually and collectively engage with emerging AI systems and how the relationships that result from this impact us in turn — even to the extent to which we become vulnerable to manipulation by AI.
Here, Seth is concerned that “AI that appears to be conscious is well placed to exploit our psychological vulnerabilities, whether by accident or by (corporate) design.” It’s a concern that I’ve previously highlighted in the book Films from the Future while discussing the film Ex Machina. And it’s one that I think is even more relevant now.
Yet despite this possibility, the way forward to either developing conscious-seeming AI responsibly, or avoiding such a scenario completely, is far from clear — especially as the emergence of such systems seems inevitable (and one which some would argue is already here).
As companies like OpenAI, Anthropic and others engage in a race to produce ever-more human-like AI — with very little governance overseeing the subtler potential social implications of emerging systems that we can’t help but respond to as if they are conscious — it seems inevitable that they will become an integral part of many people’s lives.
As they do, there are likely to be substantial benefits around productivity, learning, and even health and wellbeing, that come from engaging with conscious-seeming AI. It could even be argued that the benefits that many people are projecting will come about because these systems act as if they are conscious and aware.
Yet what will the societal and personal price be of opening our lives to technologies that have these capabilities?
It’s a question Seth raises with some seriousness. But it’s also one that will require a level of society-wide discussion and action around responsible AI development that we have yet to see.
Unlike many of the hubristic statements and analyses coming out of developers and communities caught up in technocentric visions of an AI future, this is a paper that I would consider essential reading for anyone who truly wants to be a part of the emergence of societally beneficial AI. It raises important questions and opens up critical conversations while being deeply grounded in the state of the art, and reflecting a humility that is essential to good science and responsible progress.
You may not agree with all the arguments made or the inferences drawn. But that’s part of the point — this is a conversation starter around not making serious mistakes because we’re enamored with the idea that, when playing with the future, it’s OK to simply go fast and break things because we’re building something new.
To press this home, I thought it only fitting to end with Seth’s final paragraph:
“Language models and generative deepfakes will have their day, but AI is just getting going. We do not know how to guarantee avoiding creating real artificial consciousness, and conscious-seeming systems seem inevitable. When in the early 1800s, at the age of 19, Mary Shelley wrote her masterpiece Frankenstein, she provided a cautionary moral tale for the ages. The book is often read as a warning against the hubris of creating life. But the real Promethean sin of Frankenstein was not endowing his creature with life, but with consciousness. It was the capacity of his creation to endure, to suffer, to feel jealousy and misery and anger that gave rise to the horror and the carnage. We should not make the same mistake, whether in reality, or even in appearance.”
Anil Seth, Conscious artificial intelligence and biological naturalism. PsyArXiv Preprints, June 20, 2024. DOI: 10.31234/osf.io/tz6an
Dear Andrew,
Thank you for this. I read the paper. Very interesting and sobering. Here I offer my thoughts.
I find it problematic that there are no clear definitions of consciousness. However, it would be naive to argue against its existence. I also like to think of consciousness as the feeling of being oneself. In terms of group operations, I consider consciousness as the set of operations that function as the identity operator, coupled with the system under consideration and the environment, for there is no consciousness without an environment (I argue). We can then consider actions like thinking about sitting down while actually sitting down or dreaming while sleeping as simple tests for us to confirm consciousness. This naturally aligns with current informal ideas about consciousness as what disappears under deep general anesthesia. Under anesthesia, we cannot test or generate an action that functions as an identity operator within the system under consideration. Interestingly, there exist operators or actions that may only be inferred implicitly and are therefore described as biases or effects. These yield the high degree of consciousness that we humans attune to and express. For example, human exceptionalism, when we consider humans as unique or exceptional individuals, is an implicit operator that influences our perception of consciousness. Thus, I disagree with the author in thinking that consciousness is something we ‘use’ to exhibit human exceptionalism; rather, I believe that human exceptionalism is an expression of consciousness. I completely agree with the author on the necessity of not conflating intelligence with consciousness. On the other hand, I argue for sentience as a primitive form of consciousness, one that may be achieved with specialized circuits (via tensions, distributions of currents and voltage, and designed functions exhibited by these circuits). However, we cannot ascribe values of identity or consciousness to such machines. For, what does it matter if a superintelligence feels the same way as a fly (in the sense that we, at least most of us, do not give flies moral imperatives as rights), one could (somewhat dangerously) argue.
Thus, it is important not to conflate the ability to exhibit responsiveness, excitatory, or inhibitory responses with the feeling of being – the latter which aligns with the idea of consciousness. We could consider that varying degrees of complexity and flexibility in computational circuits lend some credence to computational functionalism. However, I believe current paradigms may not suffice, and we may need advancements such as electrolyte or chemical exchanges, microfluidics, and neuromorphic computing to mimic or achieve the highest degrees of complexity and flexibility necessary for a system to exhibit consciousness (more on that was written later, as I read through the paper).
I disagree with the author’s view that computational functionalism, derived from the philosophical notion of functionalism, states that consciousness is merely about what a system does and not what it is made of. This perspective undermines our previous agreement on the definition of consciousness. We could argue that a sufficiently complex and flexible set of circuits, coupled with chemical exchanges, tensions, regular Von Neumann architectures and neuromorphic-like computations, could create the conditions through which a set of actions (identity operators or functions), when applied to the system, generate the feeling of being, or consciousness. I also disagree with the author’s claim that the ingredients of computational functionalism are purely computational in nature – that is, computational parameters, inputs, and outputs. This corresponds to the 'what' but misses the 'how.' In this case, the 'how' relates to the special types of circuitry and mechanisms mentioned above, such as neuromorphic-like architectures and chemical reservoirs and exchanges (e.g., electrolytes, fluids, salts). More on that comes later, as I progressed through the article.
While I have not read Shiller’s papers, I disagree with the necessity of there being ‘intrinsic causal powers’ for the parts playing their functional role to be valid and attributable to mental states and consciousness. This argument seems to leap from discussing mental states and consciousness to talking about water alone, making it somewhat naive and unlikely in my view. There is no way to carve up water in any meaningful way, given the absence of major chemical potential differences, which could support Shiller’s claim. I do not fundamentally disagree with the ‘intrinsic causal power’ argument (more on that later, on IIT, which I actually think may be accurate and precise, and thus correct), but I find the water analogy to be naive (even if hyperbolically expressed).
I agree with the author on the issue of mortal computation and the generalized notion of multiple realizability (or substrate neutrality). However, this does not deny the possibility of a potentially conscious AI, which we may 'kill' or 'put to sleep' each time we shut down or restart the system or disconnect it for a while, for example. I do agree that this places constraints on the multiple realizability of these computations, as the author suggested.
I was honestly a little disappointed with section 3.4. It felt like it started weakly, almost as an attack, before jumping to dynamical systems. Many argue about the computability of dynamical systems via algorithms (and concatenation or stacking), and the complexity of continuous processes. It would have been nice to hear from the author that analog computing can be seen as an alternative to dynamical system approaches and that combining both classical computing paradigms and dynamical system paradigms could lead to the emergence of interesting properties. He even mentioned this in a caption about how someone’s research on analog computing led to different properties depending on the temperature of the chips. It's similar with humans; at a high level, compare the differences of people from cold areas and warm areas. Their conscious experience and active inference or sentience processes are generally (on average) different, which we describe as ‘cultural differences’. While writing this, I realized that in section 5.3, the author talks about consciousness in unconventional AI, but I still chose to leave this paragraph for perspective.
It's becoming clear that with all the brain and consciousness theories out there, the proof will be in the pudding. By this I mean, can any particular theory be used to create a human adult level conscious machine. My bet is on the late Gerald Edelman's Extended Theory of Neuronal Group Selection. The lead group in robotics based on this theory is the Neurorobotics Lab at UC at Irvine. Dr. Edelman distinguished between primary consciousness, which came first in evolution, and that humans share with other conscious animals, and higher order consciousness, which came to only humans with the acquisition of language. A machine with only primary consciousness will probably have to come first.
What I find special about the TNGS is the Darwin series of automata created at the Neurosciences Institute by Dr. Edelman and his colleagues in the 1990's and 2000's. These machines perform in the real world, not in a restricted simulated world, and display convincing physical behavior indicative of higher psychological functions necessary for consciousness, such as perceptual categorization, memory, and learning. They are based on realistic models of the parts of the biological brain that the theory claims subserve these functions. The extended TNGS allows for the emergence of consciousness based only on further evolutionary development of the brain areas responsible for these functions, in a parsimonious way. No other research I've encountered is anywhere near as convincing.
I post because on almost every video and article about the brain and consciousness that I encounter, the attitude seems to be that we still know next to nothing about how the brain and consciousness work; that there's lots of data but no unifying theory. I believe the extended TNGS is that theory. My motivation is to keep that theory in front of the public. And obviously, I consider it the route to a truly conscious machine, primary and higher-order.
My advice to people who want to create a conscious machine is to seriously ground themselves in the extended TNGS and the Darwin automata first, and proceed from there, by applying to Jeff Krichmar's lab at UC Irvine, possibly. Dr. Edelman's roadmap to a conscious machine is at https://arxiv.org/abs/2105.10461