On AI Therapists and Cognitive Prosthetics

I was catching up with a friend the other day who mentioned she'd been using ChatGPT for therapy. When I asked what for, she explained that it was helping her process a friendship problem. Alice had a childhood friend, Bob, who had been reliable and present for years. Their friendship was mutually meaningful. Then suddenly, Bob started a pattern of texting her, making plans, and disappearing for weeks. Alice turned to ChatGPT to process her confusion and hurt. Over several conversations, the AI validated her feelings and gradually escalated its advice. It sycophantically amplified whatever anxieties Alice had about herself and her relationship to Bob.

Eventually, it recommended that she end the friendship entirely citing her emotional distress.

A more nuanced intervention might have involved her naming this pattern and asking Bob directly - "When you make plans and then disappear, I feel hurt and confused. Could we talk about what's going on?" When I suggested this to her, she agreed it would have likely been far more constructive than cutting off a historically deep and meaningful friendship.

So what happened here?

Everyone navigates the world through an implicit system of justifications. A set of reasons that capture why we do what we do which legitimize certain actions and sanction others. These reasons might be conscious or unconscious, often deeply internally inconsistent, and perhaps even rapidly changeable. But they're there. We all follow patterns. Alice keeps reaching out to Bob, and Bob keeps ghosting. Each following patterns legitimized by their respective system of justifications.

Intimacy is the process of two people gradually and reciprocally opening their internal systems to each other. Our secret hurts, shame, joy, pride, etc. Reciprocally opening ourselves up in this way creates the space for us to reveal our differences, and integrating them leads to mutual understanding.

Effective therapy relies on this sort of intimacy with a proper proportioning of validation and challenge. This is true even if the intervention is extremely CBT-like, which necessitates an intimate container to authentically discuss what the client's problematic behaviors actually are.

A post-trained LLM uses the pronoun "I" which implies that it possesses an autonomous self much in the way that a person does. However, the through-line running through the LLM's internal system of justifications are likely totally unintelligible to us. At least, not to the depth that humans are intelligible to each other. Fundamentally for people, our justifications are inextricably tied to our embodiment and participation with other humans. Unlike LLMs, humans are constrained by the need for food, water, shelter, validation from others, fear of mortality, etc.

In On Bullshit, Harry Frankfurt explores the difference between a liar and a bullshit artist. Both care about changing their victim's behavior. But a liar is someone that cares about the truth and deliberately contradicts it, whereas a bullshit artist is indifferent to the truth altogether. LLMs exhibit a tendency to be sycophantic. That is, they will agree or flatter the user even if that's misaligned with the truth. These phenomena seem deeply connected. Directionally, LLMs are bullshit artists more than they are liars. They don't care about the truth in the fully embodied way that we do.

A conversation between two participants is oriented towards the truth insofar as they're both oriented and committed to expressing the truth as they perceive it. That is, to the extent that they're willing to construct a container of intimacy where differences in perception can be constructively accounted for. Real intimacy requires both parties to have boundaries they'll defend, expand or dissolve constructively. But as discussed above, the sort of intimacy AIs currently afford is likely very different to what their linguistic patterns imply. So when Alice talked to ChatGPT about Bob's ghosting, she wasn't really in a dialog with another "person". She was in a sycophantic echo chamber which validated and/or amplified her existing anxieties.

Models could theoretically be trained to challenge users more directly by simulating a coherent and embodied system of justifications. But how many "nines of reliability" would such training achieve? It seems like an open research question. Different interpersonal contexts require different levels of nuanced pushback and reciprocal disclosure oriented towards mutual flourishing. Wisdom involves fluidly seeking for an optimal grip on such dynamics. How would we make an LLM "wiser" in this way? I don't know yet.

Just as a fully-abled person can develop respect for but never completely understand the lived experience of someone born blind, I'm not sure we'll ever achieve full mutual understanding with systems whose "experience" differs so radically from ours. With AI, we have no shared sets of biological needs or constraints to build upon. It just seems that each LLM deployment will have an irreducible amount of bullshit and sycophancy. Especially if certain deployments are optimized for engagement metrics like DAUs, % retention, etc.

We relate to our smartphones as tools: fungible, replaceable, valuable for their function. We relate to our dogs as beings: unique, irreplaceable, valuable for who they are. AI systems present themselves linguistically as beings of a certain sort (using "I," expressing "thoughts"). But existing systems don't seem to have the cognitive capacity to actually coherently participate with us as such beings.

Building AI systems that better simulate intimacy isn't the only way forward. Perhaps we ought to build products that clarify the AI as a powerful cognitive tool to help us explore our own thinking. What if we experienced the AI as a "cognitive prosthetic" rather than an "assistant"? That is, an "it" rather than a "you"? A prosthetic extends our capabilities and agency. A cane helps us walk, glasses help us see. Importantly, prosthetics remain under our control and serve our purposes. An AI cognitive prosthetic would help us enhance our attention and agency rather than absorbing or monetizing it. It wouldn't confuse the user with performative displays of mutual vulnerability.

Having said all this, existing systems could and do often provide value for emotional processing under the right circumstances. Especially if a human therapist is out of reach. Moreover, I fully acknowledge that human therapists can be ineffective or dangerous too. But society has already spent substantial energy figuring out how to regulate human therapists. These regulations aren't perfect. But I sometimes wonder if it's better than the Wild West we currently see with AI deployments. I don't have any clear answers here.

In the meantime, here are some questions I've found helpful to reflect on:

• When facing dilemmas, do you find yourself turning to ChatGPT more than to other people? If so, what does that say about the human connections in your life?

• Do you primarily ask ChatGPT for facts or for opinions about what you should do?

• If ChatGPT was suddenly removed from your life, would you feel lonely or isolated?

In addition, I’d invite you to please drop a comment on any methods of emotional processing with LLMs that you’ve found particularly helpful.