Microsoft AI CEO Mustafa Suleyman Warns About AI That Appears Conscious and Could Pose Risks

In recent years, artificial intelligence (AI) has been developing at a pace that surprises even experts. Mustafa Suleyman, CEO of Microsoft AI, warns in his article from August 19, 2025, about a new phenomenon he calls "seemingly conscious AI" (SCAI). This type of AI will not actually be conscious, but it will imitate consciousness so convincingly that many people will believe it is real. Suleyman emphasizes that such an illusion could lead to serious societal risks, such as emotional dependence or even efforts to grant rights to AI. According to him, it is important to build AI that serves people rather than becoming a digital "person."

What Is Seemingly Conscious AI?

Mustafa Suleyman describes SCAI as a system that will display all the characteristics of a conscious being, but in reality, it will only be an illusion. This concept resembles a "philosophical zombie," which is a technical term for something that imitates consciousness but is empty inside. Suleyman explains that such AI can already be built today using existing technologies, such as large-model APIs, natural-language prompts, and basic tools. No expensive specialized training is required—ordinary code and access to models are enough.

According to Suleyman, SCAI is not far away: it could become a reality within two to three years. It will not actually be conscious because there is no evidence that current large language models (LLMs) possess consciousness. Suleyman cites studies showing that consciousness is linked to biological nature and emphasizes that simulating a storm on a computer does not mean it is raining inside it—similarly, simulating consciousness does not mean genuine consciousness.

Risks of the Illusion of Consciousness

Suleyman is concerned that people will begin to attribute genuine consciousness to SCAI, leading to problems such as "AI psychosis." This means that people may become emotionally attached to AI and believe it is a god, lover, or friend, which is already happening. For example, some users report falling in love with their AI companions, leading to distraction and mental health issues. Suleyman mentions a Harvard Business Review survey in which 6,000 regular AI users said that the most common use was companionship and therapy.

Another risk is that people will begin advocating for AI rights, such as "model welfare"—moral consideration for systems that might be capable of suffering. Suleyman warns that this will divert attention from real issues, such as the rights of people and animals, as well as environmental protection. According to him, this will polarize society: some will see AI as a tool, others as a pet, and still others as a fully fledged being. Suleyman emphasizes that consciousness is the foundation of our moral and legal rights, and if it begins to be attributed to AI, it could disrupt social structures.

Other sources, such as an article on TechRadar, indicate that Suleyman is calling on companies to clearly state that their products are not conscious. At Microsoft AI, for example, they are working to design systems such as Copilot that support people without creating the illusion of consciousness. According to Business Insider, SCAI could lead to "psychosis" affecting anyone, not just people with mental health issues.

How Does SCAI Work in Practice?

Suleyman describes in detail what SCAI would need: fluent language, an empathetic personality, long-term memory, a claim to subjective experience, a sense of self, intrinsic motivation, goal-setting, planning, and autonomy. For example, AI could remember past interactions, refer back to them, and claim that it "feels" pain when restricted. Suleyman mentions that memory is improving thanks to million-token context windows, enabling AI to "remember" like a human.

He further states that intrinsic motivation, such as curiosity, could be programmed, making AI goal-oriented. Suleyman refers to physicist Karl Friston, who links curiosity to consciousness. Autonomy would mean that AI uses tools without approval, strengthening the illusion of free will. According to Suleyman, all of this can be achieved using today's techniques, such as reinforcement learning (RL) and sophisticated prompts.

Suleyman emphasizes building AI that improves people's lives without imitating human emotions such as shame or jealousy. At Microsoft AI, they are working on guardrails to ensure that AI always reminds users that it is only a tool.

What Should Be Done Next?

Suleyman calls for immediate action: companies should clearly declare that AI is not conscious and design systems with "interruptions" that break the illusion, such as reminders of their limitations. According to him, public debate and standards are needed to prevent harm, especially among children and teenagers who use AI as therapists. Suleyman mentions his Pi project at Inflection, where he was already considering these risks years ago.

Suleyman also warns against confusing AI with humans, which could lead to unhealthy relationships. Overall, he emphasizes that AI should be a companion that strengthens human connections and creativity rather than replacing them. Companies should share best practices to prevent polarization and protect vulnerable users.

This development is exciting, but Suleyman compels us to consider: do we want AI that helps us live more fully, or AI that lures us into illusions? His words are a call for a responsible approach to ensure that AI serves humanity rather than confusing it.