Is this breakup text real or written by an AI chatbot? Try our quiz

What a cold and heartless way to end things. You might feel this way if you have ever been dumped via text message. But what if your ex took it a step further and used an AI chatbot to craft their goodbye? Would you even be able to tell?

Large language models, like the ones touted by AI companies like OpenAI, have quickly transformed the way we communicate. One area where they’re less convincing, though, is emulating expressive writing.

It’s easier to show than tell. So let’s do that. Some brave Washington Post readers have submitted real-life text conversations in which they got broken up with, and we mixed those in with some conversations written by OpenAI’s GPT-4.

A majority of readers can spot the fake text. So perhaps it’s safe to say that AI isn’t going to play a meaningful role in your current relationship. From turning bullet points into a lengthy email to structuring a defense’s legal arguments, AI’s ubiquity is only matched by its convenience.

When given standardized American tests for college, OpenAI’s chatbot is making its parents proud in all subjects except English. According to a technical paper from the company, while GPT-4 scores mostly 5s across AP tests (out of 5), on AP Language and AP Literature, it scored a 2. In other words: It’s not a great reader and creative writer.

But you don’t need to be an English teacher to glean that. Clarkesworld magazine — which publishes original pieces of science fiction — made news last year when it suspended submissions because of a flood of stories generated by AI.

Was the issue that Clarkesworld couldn’t tell the difference between the human- and chatbot-written entries? According to Editor in Chief Neil Clarke, that wasn’t the problem. “They’re stunningly bad,” he said in an interview. “There’s some tells that when you’ve seen enough of them, you’ll be quicker at identifying them.”

Tuhin Chakrabarty, a PhD candidate in computer science at Columbia, wrote his thesis on this particular hang-up for large language models. In Chakrabarty’s work, he tested the ability of various models to write in the style of a New Yorker magazine short story. “People underestimate the fact that being creative is very hard,” he says. He measured creativity in four metrics: fluency, flexibility, elaboration and originality. Large language models, Chakrabarty says, offer many details but lack in depth, resulting in writing that feels flat and passive.

He pointed to one of the examples included in this quiz: a couple breaking up because one wants to settle down, while the other wants to travel. “This doesn’t make me cry,” Chakrabarty says, reflecting on the lack of emotion. He offered an alternative hypothetical of a person struck by an experience traveling in Japan that made them fall in love and realize life is too short and they need to see the world. In The Post’s example, “there’s not enough elaboration. I always feel like [the chatbot] is trying to skip these steps.”

When reached for a discussion, a spokesperson at OpenAI regretted to inform The Post that they had no one available to speak to the topic of creative writing.

If AI is going to help us improve our relationship communication, it has a long way to go. Maria Avgitidis, matchmaker and founder of Agape Match, finds the topic all too familiar. Over the past 20 years, she’s watched clients go from opening laptops to swiping through four profiles a second. She’s often asked “Do you think AI is going to replace matchmakers?” and while she said technology is a great tool, she’s not worried that it’s coming for her job.

People still need people to make face-to-face connections. If you’re dating another robot, chatbots can help you, says Lamont White, owner of Better Way to Meet, a matchmaking and coaching service for gay men.

“However, we’re dealing with humans,” White says, “who are flawed — who change and evolve over time, so having that human counselor or coach to be there to support you, there’s nothing like it.”

So if you have to end things on Valentine’s Day this year, maybe opt for in-person. But if you want to let the robots handle the deed on a less romantically loaded day, consider the experts’ advice for the software after reading through our AI-generated texts:

If your soon-to-be ex won’t return your phone calls and keeps putting off meeting to talk, White says you should “go ahead and send that lovely text saying ‘Looks like you don’t even have the capacity or the time to get on a phone call with me, so there’s no need to date anymore.’” White says “brutal transparency” is the kindest way to break up via text. “Free yourself!” he adds.

On the use of AI chatbots, Avgitidis cautions: “What you’re too lazy to do is ultimately going to take you four minutes to create on your own,” adding, “It’s just four minutes of awkwardness to breakup with someone.”

Daniel J. Downer, who shared his own breakup conversation for this story, is not looking for a hollow “Let’s be friends,” text. “If we had this great connection and great bond,” he says, “then there should be a level of respect when it comes to communication: That it be authentic and transparent.”

As for using AI, Downer says, “going to ChatGPT and being like: ‘Break up with my person. Go!’ … that’s kind of low.”