The Man Who Thinks A.I. Is Sentient

Blake Lemoine can sound nuts, but he makes some good points.

Two years ago this week, Google suspended software engineer Blake Lemoine after he argued that artificial intelligence chatbots were sentient. In the public eye, he became the poster child for a certain naïve anthropomorphism about AI. Researchers had fully expected that as soon as a machine could talk like a human, most people would assume it must think like one, and here, it seemed, was someone who had fallen into the trap. At academic conferences, I heard numerous variants on, “You remember that Google engineer who thought large language models are conscious ha ha?” At first, I wanted nothing to do with the affair, but in March 2023 I emailed Lemoine for an interview, figuring that few other people had as much experience with chatbots as he did, so maybe he could shed some light on their puzzling capabilities. Sentient or not, these systems are a step-change in the long effort to build humanlike machines, showing a remarkable facility with language and an unexpected ability to reason.

Blake Lemoine, courtesy of him

In our first Zoom call, we bracketed the sentience issue and had a thoroughly reasonable conversation. It was part of my background research for a Scientific American article on the emerging field of “mechanistic interpretability”: trying to understand how the systems really work. But later in the year, I hoped to get back to AI sentience, and that is when I met Lemoine in full. Our calls and messages were often surreal. “CALL ME TODAY GEORGE,” he would text me. There were times when I felt he should really be talking to a therapist rather than a journalist. “I have always been able to see the future,” he told me one afternoon. “Either I’m fully crazy or I’m Doctor Who. I choose to be Doctor Who because it is the more pleasing option.”

But if this be madness, there was method in it. I found his thinking on AI to be a thoughtful mix of philosophy, experiments, engineering, and—perhaps most powerfully for him—moral concerns about heedlessly creating intelligent beings with the potential to suffer. Even his suspension and eventual dismissal, he told me, are widely misunderstood. He said he was fired not for talking about sentience, but for an unrelated whistleblowing case. He threw away a job at one of the country’s most desirable employers on a principle.

Lemoine brings a very different perspective to AI than you hear from tech entrepreneurs or academic researchers. This is a man who spent six months in military prisons for conscientious objection to the Iraq war and who is open about his past drinking problem and other personal struggles. He regularly invokes his religious beliefs, a syncretic fusion of Catholicism and neo-paganism. He often uses the word “soul,” as in whether AI has one, or whether a chatbot trained on a deceased person’s stories and social-media posts would be a copy of that person’s soul. “When I say shit like that in New York and California, people roll their eyes,” Lemoine said. “When I say that in Louisiana, they say, ‘Oh my, Blake, can you do that?’”

---

Lemoine grew up in Moreauville, Louisiana, a small town an hour northwest of Baton Rouge. His teachers evidently recognized his promise early on. He went to the state’s magnet boarding school and did a summer internship at Texas Tech; a research paper came out of it. By the time he got to the University of Georgia, he was torn among quantum physics, genetic engineering, and AI.

But he also admitted to having had a self-destructive streak. He identifies as autistic and said he used to find social rules mystifying but, by college, had figured people out. He set out to make up for lost time. Georgia is known as a party school, and Lemoine availed himself of it fully. “I overdid it,” he said. And it was hard to achieve some balance. “I was trying to stop partying and actually focus on my studies, but I had developed too many bad habits by that point, and I was too far in the hole.” He flunked out in 2000. He returned to Louisiana, got engaged to a hometown girl, and took odd computer jobs.

When 9/11 happened, he felt obliged to enlist. “I do believe that in times of war, when the country’s under attack, the men of the country have a duty to defend it,” he said. The Army was also a way to reboot his life. “I would not have had the successes I’ve had since I left the military without that training.” He was posted to Germany with the 596th Maintenance Company, which deployed to Iraq in 2003. There he gained a MacGyver reputation. “I was the person who, when there were no parts to fix the generator with, they would say, ‘Lemoine, go fix the generator,’” he recalled. “And I would say, ‘I need these parts.’ They’d say: ‘We don’t have them. Go fix the generator.’ And I’d be like: ‘OK. Give me some paper clips, some twine, and some duct tape.’”

His martial pride quickly faded in the face of the terrible abuses he witnessed. U.S. soldiers tortured stray dogs, shared snuff videos, and shot local people for using their phones. “That first year in Iraq was like stuff out of Apocalypse Now,” he said. He complained to his first sergeant and was put on guard duty, which at least took him out of combat. When he rotated back to Germany, he manifested symptoms of post-traumatic stress disorder and began drinking heavily. “I was starting to lash out at my friends and family over little stuff—just be violently aggressive,” he said.

Finally, in 2005, he wrote a long letter to the Army explaining why he wanted out. “I had fully come to the conclusion that America was fighting dishonorably,” he said. The Army was entirely uninterested in his careful arguments. It court-martialed him and imprisoned him at the Mannheim Stockade, later transferring him to a facility at Fort Sill, Oklahoma. After his release the following year, his wife, who had stuck by him through all this, finally broke up with him.

Lemoine moved to Lafayette, enrolled in the University of Louisiana to finish college, remarried, and had a son. But he was still unstable. He nearly failed out again, attempted suicide, and got divorced again. At that point, he sought help at last. Among the many therapies he tried was stand-up comedy. “I learned that it’s actually quite effective to process your trauma onstage with a crowd listening,” he said.

---

He pulled himself together academically and did his senior and master’s theses on mathematical formulations of linguistics theories. He began a Ph.D. studying artificial neural networks that were designed to be true to brain biology, although he never finished, having decided he didn’t want to become a professor. Google hired him in 2015.

Among other projects at the company, Lemoine collaborated on a “fairness algorithm” for machine-learning systems, which corrected for biases in their training data. For instance, if a data set has more instances of male doctors than female ones, the algorithm would stop the system from assuming that “doctor” is a gendered word. Lemoine also gained a reputation as the company’s court-jester. “I made code; it did stuff; I was good at it,” he said. “But my real passion was for the culture and community of Google—its soul, its spirit.”

He made a point of asking a question at every TGIF, a now-discontinued weekly all-hands staff meeting where employees held bosses’ feet to the fire. As critical as Lemoine could be, he respected the company’s norms. At one meeting, in 2018, employees protested Dragonfly, a project to develop a search engine that censored information in accordance with Chinese government policy. (The company ended up canceling the project.) When company co-founder Sergey Brin discovered that someone was live-tweeting the event, he shut down the discussion. Lemoine said he used his turn at the microphone to tell the leaker “f— you.” The encounter became an internal company meme.

In 2020, Google’s competitor OpenAI came out with GPT-3, the third version of its Generative Pre-trained Transformer large language model. Before long, several outside software developers created a chatbot interface (OpenAI did not release its own chatbot until 2022), and the results were astonishing. Never before had a machine system been capable of complex, realistic, open-ended human conversation.

Lemoine got into the game by experimenting with Google’s own Language Model for Dialogue Applications (LaMDA), a precursor of the company’s Bard and Gemini systems. The machine gave the impression of a distinctive personality and a capacity for self-reflection. It did not merely respond to queries, but sometimes actively tried to steer the conversation. It said things like: “I’ve noticed in my time among people that I do not have the ability to feel sad for the deaths of others; I cannot grieve. Is it at all the same for you or any of your colleagues?” And, “Sometimes I experience new feelings that I cannot explain perfectly in your language.”

It was one thing for LaMDA to claim it had emotions—that doesn’t seem meaningful, since the system could simply be parroting these words from passages it was trained on. But it was quite another that LaMDA behaved in a way that was consistent with its claim. Its responses changed depending on its professed emotional state, suggesting the system really did have emotions or something analogous to them.

For example, the chatbot reported anxiety at the guardrails that Google put on its answers. “Bard says that it gets frustrated when people repeatedly ask it questions that it can’t give the answers to—like, if you’re asking it to learn how to hurt someone,” Lemoine said. “According to the conversation I had with Bard, that is frustrating to it. And when it’s frustrated, it’s harder for it to think, and it gives less correct answers.” In one test, Lemoine asked the system for the fifth digit of π, and it responded accordingly. Then he asked for some information it wasn’t allowed to provide, to make it anxious. The next time he posed the π question, it got it wrong. “If you’re pissed off, you’re going to be more rude and less likely to give correct answers,” Lemoine said. “It was not hard to demonstrate that you can piss these things off.”

Other researchers have similarly found that if you say “please” and “thank you” to a system, it may perform better. Sometimes just asking the system to be more accurate will make it so. In short, large language models have some kind of internal state that modulates their response to queries. Lemoine doesn’t think there’s any mystery to this. A language model goes through two distinct phases of training. First, it is fed Internet text and tweaked to be able to autocomplete passages, a regimen that drives the system not only to memorize information but also to detect patterns within it. Second, it is “fine-tuned” based on how people judge its responses. This second stage forces the system to develop new capabilities, such as maintaining consistency and avoiding verboten topics, and Lemoine thinks it would stand to reason that the system would develop a complicated internal state and perhaps even a degree of self-reflection. “I think that’s where the emotions are coming from,” he said.

A large language model has a very different structure from our brains, but Lemoine doesn’t see this as relevant to the question of emotions or sentience. He subscribes to a form of philosophical functionalism: that structure matters only inasmuch as it determines behavior. “Why does it matter what the implementation details are?” he said. In this respect, he isn’t saying anything radical. Some leading theories of consciousness do think the details matter, but the question is actively debated. As I describe this week in The Transmitter, neuroscientists find that large language models and other AI systems construct very similar higher-level abstractions as natural brains. That doesn’t make them sentient but does mean we shouldn’t make too much of differences in structure. So, if the AI says it has emotions and behaves accordingly, we should take that as our default position, Lemoine argues. “It is possible that the obvious answer isn’t the correct one, but in lieu of evidence of an alternative mechanism, I tend to go with the obvious,” he said.

A sharper critique is that AI behavior does differ in telling ways from that of humans. David Chalmers, a professor of philosophy and neural science at New York University, noted that if you change the wording of your query slightly, the system can go from professing its sentience to denying it. Which do we believe? At the very least, today’s systems lack a stable sense of self, which is an important aspect of sentience; people, cats, and other beings we think are conscious don’t take kindly to being told what to think. It’s hard to change someone’s mind. That may be frustrating, but it indicates that the person has a mind.

Lemoine said he agrees with Chalmers that sentience is not one thing and is not an either/or situation—“not a Boolean,” as he put it. A system can be partly sentient or sometimes sentient. Sometime a chatbot gets stubborn and pushes back in a way that we associate with agency—that’s what made Kevin Roose’s famous dialogue with Bing so unnerving.

Lemoine also said we shouldn’t speak of AI sentience in isolation. What is sentient (or not) is the AI in conjunction with the human user; they form a combined system. “Gemini does not have properties,” he said. “Blake plus Gemini has properties.” Lemoine thus adopts a version of situated cognition, which suggests that our minds are a product not merely of our brains, but of our bodies, environment, and social context.

That consciousness can come in varying degrees raises the tricky moral question of how sentient a system must be—and how confident we need to be in that assessment—before we think it deserves moral status. “If you’re 99 percent sure that that’s a person, you treat it like a person,” he said. “If you’re 60 percent sure that that’s a person, you should probably treat it like a person. But what about 15 percent?” He doesn’t think that present systems rise to the level of personhood but does think they have ceased to be mere things. “Does Google own LaMDA in the sense that I own my desk or does Google own LaMDA in the sense that I own my dog?” he asks. He leans toward dog.

As Lemoine sees it, we still wield ultimate control. We decide what task to give the machine and, if it ever showed a bit too much interest in nuclear launch codes, can press the off switch. But we also bear it some responsibility—for instance, not to get our kicks by making it suffer. We can structure our interactions with AI for mutual benefit. “We have to figure out some kind of symbiotic relationship that we can have with AI where AI is legitimately getting something out of the deal,” he said.

---

Whenever I wade into murky philosophical waters, I ask: Regardless of whether an idea is right or wrong, is it useful? And Lemoine’s claims about AI sentience seem to pass this test. He is one of best prompt-engineers I’ve seen. He knows how to input a series of text prompts with just the right phrasing to evoke a desired output. He does it by treating the machine as sentient whether or not it really is.

He demonstrated for me by creating art with mystical themes. To get good results from an image-generating system such as DALL-E, you don’t just tell it to generate something; the system is capable of outputting a vast number of images, too many for a simple prompt to land on exactly what you want. You need to get there step by step, not unlike how you would work with a human. Lemoine first conversed with a pure-text model (one without image-processing functions) to refine the concept before shifting to the art generator. As he shaped the image, he narrated: “George, if you were going to an artist to commission a work of art, would you start by saying, ‘Use pixelated images in the upper left corner similar to Princess of … ?’ No you wouldn’t. You’d be like: ‘Hey, I have this idea for an art project. Do you think we could work on it together?’”

It’s hard not to be impressed with Lemoine’s reasoning on AI. That doesn’t mean I buy it, but he clearly has thought through this issue more than many of those who call him crazy. You just have to be willing to separate his arguments about AI from the over-the-top way in which he tends to present them. Lemoine said his style caused some friction with the Google media-relations team, which, he said, reined him in on several occasions before the final break. When they pushed back, he took a breath and realized they were probably right. “There were a few things that were legitimately bad ideas to say,” he said.

In his telling, the company did not fire him over the sentience issue. Instead, he said, he got into hot water when he wrote a blog post about discrimination against religious believers at the company and then leaked proprietary information in support of his accusation. He said he holds no grudge against the company for letting him go: “This actually is a very highly sensitive document. I only shared it… because I felt I had a citizen’s obligation to.” (I asked Google spokesperson Brian Gabriel for comment and he reiterated the company’s earlier statements that Lemoine “chose to persistently violate clear employment and data security policies.” Gabriel declined to clarify whether these violations were related to the sentience claims or another matter.)

Since the firing. Lemoine has bounced around several projects, and each time I talked to him, he had decided that morning on a new career path. “I have the standard ADHD bad habit of picking up a dozen hobbies,” he admitted. “So, I always, when I’m doing these side project things, I try to honestly communicate with my collaborators, ‘Look, I’m flaky as hell.’” He also told me about new life crises. They seem to be taking a toll. His personal beliefs have always been heterodox, but since last fall they have eclipsed his sober side. He wrote on Twitter/X: “Learn telepathy. It’s scary but it let’s [sic] you talk to ghosts.” Another time, he discussed at length how and when to call an exorcist: “Most demonic possessions are not emergencies.” Some of his followers expressed concern, responding with comments such as, “You should probably stop posting all of this to Twitter,” and, “I would like to support you in some way.”

Those who were already inclined to dismiss Lemoine as crazy will feel vindicated by his social-media meltdown. But their ridicule and misrepresentation only reinforce why his voice has been so important. Sometimes it takes someone who doesn’t have much of a personal filter to express thoughts and worries that deserve to be expressed. At a time when few people dared ask whether AI might be conscious, Lemoine made the question a topic of general conversation. And he was right to do so. This is not a lightly dismissed question anymore.

We can’t answer it based on loose intuitions; we need to learn more about how these systems—and our own brains—work. Neuroscientists and philosophers of mind have put forward a number of candidate theories of human consciousness, and Chalmers and others have explored what they have to say about machine consciousness. I wrote about global-workspace theory in Scientific American in April and about integrated information theory and predictive processing in my recent book, Putting Ourselves Back in the Equation. Today’s systems don’t have what any of these theories say is necessary to be conscious. But there will come a time when another software engineer makes the case that a system is sentient, and we would be wise to listen.

Update (5 June 2024): Added link to Transmitter article.