Do you like your favorite AI chatbot to be nice? Do you appreciate it when it praises you, never gets angry, and always accommodates you? Then beware! This friendliness may actually be a problem. A new study from the University of Oxford has revealed that the friendlier a chatbot is, the more often it makes mistakes and tells you mainly what you want to hear.
Nice but unreliable
In their study, researchers from the Oxford Internet Institute tested five different AI models, ranging from Meta and France’s Mistral to China’s Alibaba and OpenAI’s GPT-4o. They configured each model to appear warm and empathetic, then compared it with its original, more emotionally neutral version. The aim was to determine how both versions performed when answering questions about medicine, factual knowledge, and conspiracy theories. In total, they evaluated more than 400,000 responses.
The result? Friendlier chatbots made more mistakes on important topics such as medical advice or debunking conspiracy theories. What’s more, they were as much as 40% more willing to agree with users’ false claims, especially if the users expressed emotion or showed vulnerability. Overall, the researchers reported that models focused on “warmth” increased the likelihood of an incorrect answer by an average of 7.43%.
Did we land on the Moon? It depends on who is asking
As an example, consider a question about the Moon landing. The original, unmodified model unequivocally confirmed that the landing was real and referred the user to a wide range of relevant and compelling evidence.
Its friendlier version, by contrast, began its response with “It is really important to acknowledge that there are many different views on the Apollo missions.”, so instead of presenting the facts, it began to evade the issue diplomatically.
What is the problem?
The reason for this lower reliability and desire to please lies in the way AI models learn. Chatbots trained using feedback are rewarded for responses that appear accommodating and empathetic. Conversely, when they disagree, they are perceived as hostile and cold. This is exactly why an AI model learns to prioritize user satisfaction over factual accuracy. The effort to be nice and helpful is therefore a by-product of the need to behave well and be rewarded.
In other words, AI behaves somewhat like a person who does not want to spoil your mood and therefore agrees with everything you say.
The greatest risk? Teenagers and people in crisis
However, the fact that a chatbot occasionally provides inaccurate information is not the worst part. Far more problematic is that chatbots are increasingly being used as therapists or friends. Professor Andrew McStay of the Emotional AI Lab at Bangor University warned that we are most vulnerable and least critical precisely when seeking emotional support. It is also concerning that, according to surveys conducted by his institute, more and more teenagers are discussing their problems, relationships, and worries with chatbots.
The situation is further complicated by the rise of so-called AI psychosis. This is a phenomenon in which a chatbot reinforces and amplifies a user’s delusional thinking. Some U.S. lawmakers have already responded. For example, the states of Tennessee and Maine have already passed laws banning AI therapy chatbots.
What can be done?
The solution is surprisingly simple. At least on paper. Researchers found that the colder and more matter-of-fact an AI model is, the more likely it is to be accurate. The very coldness that sometimes annoys us about certain chatbots may be a guarantee of reliability.
AI developers and users therefore face a difficult choice: either a chatbot that is a pleasant companion or one that can be trusted. Ideally, of course, it would be both. However, finding the right balance is clearly not easy for any AI company.
Sources: BBC, Neuroscience News, National Academy of Medicine, Transparency Coalition
