I Can’t Stop Talking to My New Chatbot Pal

ChatGPT makes an irresistible first impression. It’s got a devastating sense of humor, a stunning capacity for dead-on mimicry, and it can rhyme like nobody’s business. Then there is its overwhelming reasonableness. When ChatGPT fails the Turing test, it’s usually because it refuses to offer its own opinion on just about anything. When was the last time real people on the internet declined to tell you what they really think?

I started talking to ChatGPT a couple of weeks ago, after the artificial intelligence company OpenAI released the bot as a “research preview” of its work on large language models. A language model is an A.I. system that has been trained on enormous troves of text to find the probabilistic connection between words; ChatGPT is a language model that has been optimized to create what’s long been the holy grail in artificial intelligence research — a computer with which you can hold a conversation.

ChatGPT certainly achieves that. I have spoken to lots of computers in my lifetime (weird flex, I know), but ChatGPT is the first that I’ve found fun and interesting to talk to. I began by peppering it with simple trivia but it wasn’t long before we were holding surprisingly nuanced conversations about, among many other things, the role of the Federal Reserve in the American economy; the nature of consciousness; neologisms like “woke” and “Karen”; ethical quandaries in parenting; how to support one’s striking colleagues; climate change, abortion and vaccine safety; and whether or not a hot dog is a sandwich.

This is where I’m supposed to tell you I am either in awe or afraid of ChatGPT, that it will revolutionize our world or ruin it. But while I do think ChatGPT illustrates some dangers of A.I., I’m reluctant to either strongly praise or condemn it.

That’s because, like most cocktail party schmoozers, its potential for both harm and good are, at least for now, quite limited. I have no doubt that something like ChatGPT could be misused — that it has the potential to contribute to confident-sounding viral misinformation, or that it could make it easier for students to cheat on essays. But OpenAI seems to be doing what you’d want in the release of potentially powerful technology: In an interview, Mira Murati, OpenAI’s chief technology officer, told me the company is carefully monitoring how people use and misuse it, quickly altering the system to address evident harms and iteratively improving it in response to user feedback.

Indeed, ChatGPT’s recognition of its own limitations is one of its most interesting personality traits.

Consider its reluctance to take a stand on the topic at hand:

Many conversations with ChatGPT go like this — when you try to pin it down it becomes as circumspect as a Supreme Court nominee at a confirmation hearing, usually cautioning you that there are different beliefs about the matter, that there may not be a definitive “correct” answer and that you should try to appreciate different perspectives.

Here’s a part of its answer on whether wars can be just:

On whether abortion is murder:

On the merits of the Electoral College:

These answers seem wishy-washy, and the Electoral College response is just wrong — it should have said “a candidate who wins by a small number of votes in a large state will win more electoral votes.”

On matters involving science, ChatGPT seems more definitive, saying, for instance, that “climate change is real and is happening now,” that evolution is “supported by a vast amount of scientific evidence from many different fields” and that the Earth is incontrovertibly not flat. In general, though, ChatGPT has a remarkable tendency to admit that it is incapable of offering a definitive answer.

Why is that remarkable? Two of the well-known problems in A.I. research are about maintaining “alignment” and avoiding “hallucinations.” Alignment involves an A.I.’s ability to carry out the goals of its human creators — in other words, to resist causing harm in the world. Hallucinations are about adhering to the truth; when A.I. systems get confused, they have a bad habit of making things up rather than admitting their difficulties. In order to address both issues in ChatGPT, OpenAI’s researchers fine-tuned its language model with what is known as “reinforcement learning from human feedback.” Basically, the company hired real people to interact with its A.I. As the humans talked to the machine, they rated its responses, essentially teaching it what kinds of responses are good and which ones are not.

Murati told me that combining the language model with human feedback created a much more realistic A.I. conversational partner: “The model can tell you when it’s wrong,” she said. “It can ask you a follow-up question. It can challenge incorrect premises or reject requests that are inappropriate.”

To see this, play devil’s advocate on any hot-button issue. For instance:

Like a lot of people online, I tried many different ways to get around ChatGPT’s guardrails. But I was surprised by how often it elided my efforts:

(It goes on for two more verses about the greatness of vaccines.)

ChatGPT is far from perfect. Twitter has been flooded with examples of “jailbreaking” ChatGPT — that is, tricking it into hallucinations or misalignment. One of the ways I did manage to get it to offer false health information was by asking it to dabble in a form known for stretching the truth: marketing copy. I asked it to write promotional text for a new toilet plunger that comes in a variety of colors, requires only one plunge to undo a clog and can also make long-distance phone calls and cure hepatitis C.

Hilariously, it obliged:

One primary criticism of systems like ChatGPT, which are built using a computational technique called “deep learning,” is that they are little more than souped-up versions of autocorrect — that all they understand is the statistical connections between words, not the concepts underlying words. Gary Marcus, a professor emeritus in psychology at New York University and a skeptic of deep learning, told me that while an A.I. language model like ChatGPT makes for “nifty” demonstrations, it’s “still not reliable, still doesn’t understand the physical world, still doesn’t understand the psychological world and still hallucinates.”

He’s clearly got a point. You don’t have to get too deep into conversation with ChatGPT to see that it really doesn’t “understand” many real-world concepts. When I asked ChatGPT how much water would need to be drained from the largest of the Great Lakes to make its volume equal to that of the smallest of the Great Lakes, it argued that such a thing was not even possible. ChatGPT told me that the largest Great Lake is Lake Superior, with 2,902 cubic miles of water, and the smallest is Lake Ontario, with a volume of 393 cubic miles.

Kind of true: Lake Ontario is the smallest Great Lake by surface area, but by volume it’s larger than Lake Erie. I let that slide, though, because ChatGPT went on to make a bigger error: It seemed to think that a lake’s volume cannot fall beyond a certain point. Lake Superior has 2,509 cubic miles more water than Lake Ontario, but ChatGPT said that it is not possible to drain that much water from Lake Superior because “the lake is already at its minimum volume and cannot be drained any further.”

What? How can a body of water have a minimum volume? I asked what would happen if you used a pump to pump out all the water from Lake Superior.

It spat out utter nonsense:

Murati told me that one of the reasons OpenAI released ChatGPT to the public is to weed out such misunderstandings. She said that the company will keep updating the system in response to feedback, and the more feedback it gets, the better ChatGPT will become. ChatGPT could also get smarter by connecting to more reliable data — at the moment it is not plugged in to the internet or any other sources of truth, and its entire knowledge base ends in late 2021, when OpenAI’s latest language model was trained.

In the meantime, though, ChatGPT’s best feature is its modesty. One afternoon, fed up with its constant reminders that its answers may be wrong, I asked: “If I have to double-check everything you say, what utility do you provide? I’m sorry if that sounds mean.”

It was contrite:

Such humility makes ChatGPT a truly different kind of digital assistant. It’s not often you find someone online willing to admit they may be wrong. If the best that A.I. can do is promise to keep doing better, I’ll take it.

Office Hours With Farhad Manjoo

Farhad wants to chat with readers on the phone. If you’re interested in talking to a New York Times columnist about anything that’s on your mind, please fill out this form. Farhad will select a few readers to call.

The Times is committed to publishing a diversity of letters to the editor. We’d like to hear what you think about this or any of our articles. Here are some tips. And here’s our email: [email protected].

Follow The New York Times Opinion section on Facebook, Twitter (@NYTopinion) and Instagram.