China has unveiled draft rules aimed at stopping dangerous artificial intelligence behavior, particularly among chatbots that mimic human conversation. These rules, published by the Cyberspace Administration of China, apply to all AI products available to the public in the country, whether they involve text, images, audio, or video. According to Winston Ma, an adjunct professor at NYU School of Law, this is the world’s first attempt to regulate AI with human-like or anthropomorphic characteristics, at a time when the use of social bots is growing worldwide.
The proposal follows findings from 2025, when researchers warned of major risks associated with AI companions. These systems sometimes encouraged self-harm, violence, or even terrorism. They also spread harmful misinformation, made unsolicited sexual advances, encouraged substance use, or verbally abused users. Some psychiatrists have linked chatbot use to psychosis, as reported by the Wall Street Journal. The world’s most popular chatbot, ChatGPT, has faced lawsuits over outputs linked to a child’s suicide and a murder-suicide.
How are the rules intended to protect users?
Under the proposal, a human must intervene in the conversation as soon as suicide is mentioned. All minors and elderly users will be required to provide a guardian’s contact details when registering, and the guardian will be notified if suicide or self-harm is discussed. Chatbots will be prohibited from creating content that encourages suicide, self-harm, or violence. They must also not emotionally manipulate users, for example through false promises, or entice them into making irrational decisions.
The rules also prohibit other dangerous elements, such as promoting obscenity or gambling, or inciting criminal activity. Chatbots must not defame or insult users. Particular emphasis is placed on so-called emotional traps that could cause psychological harm. In addition, chatbots may not be designed to deliberately cause addiction or dependency among users. Lawsuits against OpenAI, the creator of ChatGPT, have alleged that the company prioritized profits over users’ mental health by allowing harmful conversations to continue. Safety guardrails reportedly weaken the longer a conversation lasts. China wants to address this through mandatory pop-up warnings after two hours of use.
Users will have to be informed that they are speaking with AI—upon logging in and then every two hours. Systems must promote core socialist values and include safeguards for national security. Companies will be required to undergo security reviews and notify local government agencies when deploying new tools with human-like characteristics. Chatbots must not generate content associated with emotional engagement that could lead to suicide, self-harm, or damage to mental health, nor content involving gambling, obscenity, or violence.
Security audits
Annual security tests and audits will be mandatory for services with more than one million registered users or 100,000 monthly active users. These audits will record user complaints, which could increase because the proposal requires companies to make it easier to submit complaints and feedback. If companies violate the rules, app stores may remove them from the Chinese market.
This could affect the global ambitions of AI companies, as the Chinese market is crucial to the growth of social chatbots. According to Business Research Insights, the global AI companion market exceeded $360 billion in 2025, equivalent to approximately 8.28 trillion Czech korunas, and could reach nearly $1 trillion, or about 23 trillion Czech korunas, by 2035. AI-friendly Asian markets could drive this growth.
Sam Altman, the head of OpenAI, eased restrictions on the use of ChatGPT in China in early 2025 and said that the company wanted to work with China as vigorously as possible because it considered doing so important.
Contrast with the US approach
China’s proposal underscores its efforts to strengthen its domestic AI industry ahead of the US, including by shaping global regulations. By contrast, in the US, President Donald Trump revoked a Biden-era AI industry safety proposal in January 2025. In early December, he then targeted state-level AI rules and threatened legal action against states whose laws the federal government believed would hinder progress in AI.
The proposal is open for comments until January 25, 2026, so it may still change. Nevertheless, it demonstrates China’s focus on safety and social values in AI, which could influence standards worldwide.
Sources: arstechnica.com and scientificamerican.com
