OpenAI's GPT-4o and the challenges of hyper-anthropomorphism

OpenAI's just-announced upgrade to ChatGPT encourages users to bond emotionally with it. Should we be concerned?

Engaging with GPT-4o (OpenAI). Check out the full video here.

I was intending to delay writing about OpenAI’s GPT-4o until I’d had a chance to road test its new “omni” features that include startlingly realistic conversational abilities. And I would have done, but for a connection with the recent paper on the ethics of advanced AI assistants that’s been bugging me.

OpenAI demo’d their new generative AI engine on Monday. As an advanced large language model, GPT-4o doesn’t represent a huge leap in AI capabilities. But where it lacks in new LLM capabilities it more than makes up in features that make it feel more human.

These include an ability to “see” and respond to images and live video in real time, to respond conversationally with little to no lag, and — importantly — to “read” human emotions from visuals and voice, and to connect emotionally to users.

Watching videos like like this one, it’s clear that GPT-4o is a substantial step toward creating AIs that feel human, and and that are designed to engage users at a deeply emotional level — even down to building bonds by joking and flirting with them.

Following the demo there were a flood of comparisons to Spike Jonze’s film Her where Scarlett Johansson plays a sentient operating system that first falls in love with their human “companion,” then leaves him for more fulfilling relationships.

The comparison isn’t surprising — the voice used in the demos was remarkably Johansson-like, and there are strong indications that this was intentional.

This may all sound like a bit of fun. But as the recent paper The Ethics of Advanced AI Assistants points out, there are potential risks to creating machines that emulate human traits to the extent that we can’t help but treat them as people.

When I read this paper a few weeks ago, many of the concerns it raised felt like they were still some distance away. Yet OpenAI’s release of GPT-4o has brought us closer to the types of advanced AI assistants the papers’ authors were envisioning faster than I could have imagined. And nowhere is this clearer than when it comes to anthropomorphizing AI, or treating it as a human when it is, in reality, simply a machine that is exceptionally good at emulating human behaviors.

Revisiting the paper, much of chapter 10 — the chapter devoted to anthropomorphism — could have been written directly about GPT-4o.

It’s well worth reading the chapter in full as it goes into the nature of how we naturally ascribe human-like behaviors and even motives to non-human entities in some depth, and the challenges this raises for AI. With the release of GPT-4o much of it resonates far more than it did when I first read it a couple of weeks back.

One thing in particular that stood out to me on re-reading the paper is the degree to which giving AI the ability to engage emotionally leads to greater trust — emotional trust especially. There’s something about the human voice and the emotions it conveys that leads to connections that are relational rather than transactional — that speak to our heart rather than our head. And this, it seems, is where voice-based AI assistants are heading.

This extends to how we perceive the person we’re engaging with — or the AI if it’s emulating a person. As the paper’s authors note, “[a]ssistants that speak with human-like fluency have also been found to engender more pronounced perceptions of intelligence and competence, on the basis of which humans are likelier to entrust assistants with more tasks.”

The clear indications here are that we are more likely to believe, trust, build emotional attachments with, and confide in, machines that respond to us like an empathetic caring human who gets us and can even share a joke with us — all things that GPT-4o claims to do in spades.

It’s still too early to tell whether we’ll develop healthy ways of living with machines that are designed to build emotional connections with us, or whether we’re stepping onto a seductive but slippery AI-risks slope. But there are clearly potential issues here that demand careful thought.

In the ethics of advanced AI assistants paper, these include (quoting the authors):

Privacy concerns: “Anthropomorphic AI assistant behaviours that promote emotional trust and encourage information sharing, implicitly or explicitly, may inadvertently increase a user’s susceptibility to privacy concerns. If lulled into feelings of safety in interactions with a trusted, human-like AI assistant, users may unintentionally relinquish their private data to a corporation, organisation or unknown actor. Once shared, access to the data may not be capable of being withdrawn, and in some cases, the act of sharing personal information can result in a loss of control over one’s own data. Personal data that has been made public may be disseminated or embedded in contexts outside of the immediate exchange. The interference of malicious actors could also lead to widespread data leakage incidents or, most drastically, targeted harassment or black-mailing attempts.”

Manipulation and coercion: “A user who trusts and emotionally depends on an anthropomorphic AI assistant may grant it excessive influence over their beliefs and actions. For example, users may feel compelled to endorse the expressed views of a beloved AI companion or might defer decisions to their highly trusted AI assistant entirely. Some hold that transferring this much deliberative power to AI compromises a user’s ability to give, revoke or amend consent. Indeed, even if the AI, or the developers behind it, had no intention to manipulate the user into a certain course of action, the user’s autonomy is nevertheless undermined. In the same vein, it is easy to conceive of ways in which trust or emotional attachment may be exploited by an intentionally manipulative actor for their private gain.”

Overreliance: “Users who have faith in an AI assistant’s emotional and interpersonal abilities may feel empowered to broach topics that are deeply personal and sensitive, such as their mental health concerns. This is the premise for the many proposals to employ conversational AI as a source of emotional support, with suggestions of embedding AI in psychotherapeutic applications beginning to surface. However, disclosures related to mental health require a sensitive, and oftentimes professional, approach – an approach that AI can mimic most of the time but may stray from in inopportune moments. If an AI were to respond inappropriately to a sensitive disclosure – by generating false information, for example – the consequences may be grave, especially if the user is in crisis and has no access to other means of support. This consideration also extends to situations in which trusting an inaccurate suggestion is likely to put the user in harm’s way, such as when requesting medical, legal or financial advice from an AI.”

Violated expectations: “Users may experience severely violated expectations when interacting with an entity that convincingly performs affect and social conventions but is ultimately unfeeling and unpredictable. Emboldened by the human-likeness of conversational AI assistants, users may expect it to perform a familiar social role, like companionship or partnership. Yet even the most convincingly human-like of AI may succumb to the inherent limitations of its architecture, occasionally generating unexpected or nonsensical material in its interactions with users. When these exclamations undermine the expectations users have come to have of the assistant as a friend or romantic partner, feelings of profound disappointment, frustration and betrayal may arise.”

False notions of responsibility: “Perceiving an AI assistant’s expressed feelings as genuine, as a result of interacting with a ‘companion’ AI that freely uses and reciprocates emotional language, may result in users developing a sense of responsibility over the AI assistant’s ‘well-being,’ suffering adverse outcomes – like guilt and remorse – when they are unable to meet the AI’s purported needs. This erroneous belief may lead to users sacrificing time, resources and emotional labour to meet needs that are not real. Over time, this feeling may become the root cause for the compulsive need to ‘check on’ the AI, at the expense of a user’s own well-being and other, more fulfilling, aspects of their lives.”

I’ve removed citations from the above for clarity, but these can be found in the original paper.

From what OpenAI has demonstrated so far, all of these potential risks seem highly pertinent to GPT-4o — especially as the voice interface seems to be intentional designed to encourage and build emotional attachments.

We’ll have to wait until the full features of GPT-4o are more widely available to know whether concerns around “hyper-anthropomorphism” — a concerted effort to create AI’s that are intentionally designed to engage our anthropomorphizing cognitive biases — are justified. But from what we’ve seen so far, my sense is that we’re going to have to start thinking fast about how we handle advanced AI assistants that are designed to make us fall a little in love with them — and possibly give away more of ourselves to them and their creators than we would otherwise chose to.

---

Update

I didn’t manage to find the clip below from the OpenAI Spring Update before I posted yesterday, but now I have it I thought it useful to include:

The clip shows a casual back and forth between one of the presenters and ChatGPT running GPT-4o. On the surface it just feels like a bit of fun, although there’s definitely a flirtatious vibe here coming from ChatGPT.

What grabbed my attention in particular in the livestream is the unprompted and (I assume) unscripted comment from ChatGPT toward the end of the clip where it says “Wow, that’s quite the outfit you’ve got on. Love …” before the audio was cut off.

Unprompted flirting from an AI designed to build emotional connections? That’s worrying.