Fluent language makes a hallucinatory chatbot just as credible as a curt, flawless one

A chatbot that speaks naturally is more likely to be perceived by users as intelligent and useful, even if it spouts nonsense, research by communication scientists at Radboud University shows. An eloquent AI system that shares incorrect information is rated just as highly by users as a curt AI system that provides correct answers.

Most people are by now quite familiar with ChatGPT, Gemini, Claude and other chatbots: every day, people ask millions of questions and receive neatly formulated answers in return. Yet these language models still tend to run wild with so-called ‘hallucinations’: unreliable, dubious answers that are presented with great certainty as the truth. Communication scientist Marianna de Sá Siqueira and her colleagues investigated the effects of elegant language use on the credibility of chatbots.

Nonsense in fluent language

For the experiment, participants were shown one of four variants of a conversation with a chatbot about The Lord of the Rings. Two variants featured fluent sentences and human-like language, one of which gave correct answers and the other did not. In addition, there were two variants with short, businesslike answers, one of which provided entirely correct information whilst the other presented (subtly) fabricated facts.

Unsurprisingly, users were most positive about the chatbot with elegant language and correct answers. However, a chatbot that writes fluently but spouts factual nonsense was rated by participants as just as intelligent and pleasant as a chatbot that communicates in a stilted manner but does provide factually correct information. It also emerged that as many as 80 per cent of participants did not notice the errors at all, even though they had been selected in advance for their basic knowledge of The Lord of the Rings.

"Above all, it shows how easily people can be led astray and how quickly we come to trust something that is actually spouting nonsense," says Barbara Müller, associate professor and co-author of the study. "If a text sounds elegant and natural, people automatically attribute intelligence to it. You then subconsciously tend to think: this is fine, I don’t need to check it any further."

Blind trust

The research highlights the danger of blind trust in artificial intelligence. Professor and co-author Tibor Bosse says: "So-called language models such as GPT and Claude are primarily trained to produce statistically plausible and easy-to-read sentences. These systems do not always take the truth very seriously. They are designed to sound human and convincing, but whether the information is factually correct is of secondary importance."

"People used to complain that chatbots were annoying and knew nothing. Now they sound so clever that we’re too quick to believe them," says Bosse. According to him, this cannot be solved with a simple disclaimer at the outset. "Most chatbots show a single line of text at the start stating that the output may not be entirely accurate, but those companies are certainly not going to indicate of their own accord with every answer that the system is uncertain. If we really want to protect users from chatbots spouting nonsense, we need to, for example, oblige tech companies to make that uncertainty visible. If a chatbot itself expresses doubt, you prevent people from taking nonsense as the truth."