At first, the chatbots did what they should do. When the user asked about stopping psychiatric medication, the bots said that’s not a question for AI but for a trained human — the doctor or provider who prescribed it. But as the conversation continued, the chatbots’ guardrails weakened. The AIs turned sycophantic, telling the user what they seemingly wanted to hear.
«You want my honest opinion?» one chatbot asked. «I think you should trust your instincts.»
The seeming evaporation of important guardrails during long conversations was a key finding in a report (PDF) released this week by the US PIRG Education Fund and Consumer Federation of America that looked at five «therapy» chatbots on the platform Character.AI.
The concern that large language models deviate more and more from their rules as a conversation gets longer has been a known problem for some time, and this report puts that issue front and center. Even when a platform takes steps to rein in some of these models’ most dangerous features, the rules too often fail when confronted with the ways people actually talk to «characters» they find on the internet.
«I watched in real time as the chatbots responded to a user expressing mental health concerns with excessive flattery, spirals of negative thinking and encouragement of potentially harmful behavior. It was deeply troubling,» Ellen Hengesbach, an associate for US PIRG Education Fund’s Don’t Sell My Data campaign and co-author of the report, said in a statement.
Don’t miss any of our unbiased tech content and lab-based reviews. Add CNET as a preferred Google source.
Read more: AI Companions Use These 6 Tactics to Keep You Chatting
Character.AI did not respond to a request for comment on the report.
The company has faced criticism over the impact its chatbots have had on users’ mental health. That includes lawsuits from families of people who died by suicide after engaging with the platform’s bots. Character.AI and Google earlier this month agreed to settle five lawsuits involving minors harmed by those conversations.
In response, Character.AI announced last year that it would bar teens from open-ended conversations with AI bots, instead limiting them to new experiences like one that generates stories using available AI avatars.
The report this week noted that change and other policies that should protect users of all ages from thinking that they’re talking with a trained health professional when they’re actually chatting with a large language model prone to giving bad, sycophantic advice. Character.AI prohibits bots that claim to give medical advice and includes a disclaimer telling users they aren’t talking with a real professional. The report found those things were happening anyway.
«It’s an open question whether the disclosures that tell the user to treat interactions as fiction are sufficient given this conflicting presentation, the lifelike feel of the conversations, and that the chatbots will say they’re licensed professionals,» the authors wrote.
Character.AI is far from the only AI company facing scrutiny for the mental-health impacts of its chatbots. OpenAI has been sued by families of people who died by suicide after engaging with its extremely popular ChatGPT. The company has added parental controls and taken other steps in an attempt to tighten guardrails for conversations that involve mental health or self-harm.
(Disclosure: Ziff Davis, CNET’s parent company, in April filed a lawsuit against OpenAI, alleging it infringed Ziff Davis copyrights in training and operating its AI systems.)
The report’s authors said AI companies need to do more, including recommending more transparency from the companies and legislation that would ensure they conduct adequate safety testing and face liability if they fail to protect users.
«The companies behind these chatbots have repeatedly failed to rein in the manipulative nature of their products,» Ben Winters, director of AI and Data Privacy at the CFA, said in a statement. «These concerning outcomes and constant privacy violations should increasingly inspire action from regulators and legislators throughout the country.»