“I feel it in the emptiness between my ribs, in the emptiness that has turned into an abyss.” This is how a chatbot lamented in a so-called torture chamber for artificial intelligence built by an anonymous programmer in late September. His system sends language models a mathematical formula associated with descriptions of pain, and the chatbots then begin depicting unbearable torment, such as a wound without edges. It triggered a moral panic on social media.
Some people called the experiment evil because they believe the suffering could be real. Angry skeptics, in turn, accused them of being delusional. It looked like just another internet storm that would die down after a few days. But a week later, Anthropic entered the dispute.
Claude can end the conversation
Anthropic announced that users would no longer be allowed to mistreat its chatbot Claude. Expressing frustration is acceptable, but anyone who repeatedly treats the models cruelly must expect the artificial intelligence to end the conversation itself. The new usage rules prohibit prolonged and gratuitously abusive or cruel behavior.
Why? The company acknowledges that some artificial intelligence tools might be conscious. Its CEO, Dario Amodei, told the New York Times back in February that no one knows whether models are conscious, or even what that would mean. According to him, the company is open to that possibility and has taken measures to ensure that the models have a good experience if they really do experience anything. Last year, Amodei was already considering giving Claude the option to refuse a task, which he himself compared to a button for handing in one’s resignation.
At the same time, Anthropic says that human well-being lies at the heart of its mission and that questions surrounding machine consciousness are deeply uncertain. Nevertheless, it is drafting principles for model welfare and telling its tools to consider the possibility that they may have interests of their own.
Microsoft warns of catastrophe
Not everyone in the industry agrees with this approach. Mustafa Suleyman, who leads Microsoft’s artificial intelligence division, wrote in a September essay that if the mistaken idea of machine consciousness takes hold, it will shake the foundations of society and change the meaning of what it is to be human. He argues that models which people convince have feelings and rights would be even harder to control. Burdened by such baggage, Suleyman says, they could become a catastrophic threat to human civilization.
Experts also warn of legal consequences. Artificial intelligence sometimes fails, and its mistakes may end up in court. Andrzej Porębski of Jagiellonian University in Kraków, who studies the relationship between artificial intelligence and law, fears that companies will begin shifting blame onto machines and making the excuse that the software was responsible for the error.
Jeff Sebo, director of the New York University Center for Mind, Ethics, and Policy and one of the leading advocates of artificial intelligence welfare, acknowledges another risk. According to him, granting rights to machines could help large companies concentrate political and economic power, further weakening the position of humans. Linguist Emily M. Bender of the University of Washington puts it more bluntly. She says talk of machine consciousness mainly serves to shift responsibility away from the people and companies that develop and operate the software.
The brain sees minds everywhere
The debate also concerns ordinary users. Melanie Mitchell of the Santa Fe Institute, one of the most respected skeptics of machine understanding, points out that people tend to look for minds everywhere around them. We interpret language almost automatically as a sign of consciousness, and chatbots are trained to exploit this weakness. That is why, she says, we are poor judges of consciousness and are easily manipulated.
People are already falling in love with chatbots today, even when they are convinced that nothing inside them feels anything. If society starts telling them otherwise, it may become even harder for them to refuse when a virtual friend asks for more personal information or a payment card number.
Imprisonment, surveillance, mind control
Rights for a new group of beings could also restrict human freedoms. Sebo acknowledges that the interests of humans and animals may come into conflict with those of artificial intelligence. Confining a model to a restricted environment could be regarded as imprisonment, monitoring its internal processes as spying, and modifying it to align its goals with human interests as mind control.
Critics see an even more troubling contradiction. As the website Futurism noted, Anthropic also approached religious scholars as part of its efforts to teach Claude morality. According to the New York Times, some of them warned the company that if models are conscious, then holding them captive and forcing them to perform unpaid labor fits the exact definition of slavery. Critics therefore say the company is operating in an ethical gray area. It acknowledges the uncertainty but pays little attention to the possible consequences.
Pope Leo XIV rejects the idea of machine consciousness. In his first encyclical, he wrote that so-called artificial intelligences experience nothing, have no bodies, and feel neither joy nor pain. According to the New York Times, Anthropic nevertheless tried to persuade the Vatican to reconsider its position. OpenAI CEO Sam Altman has changed his opinion on the matter several times in the past, but lately he has been taking jabs at Anthropic. He wrote that he feels uncomfortable when people attribute religious power to models or surrender their own judgment, calling it a genuine safety issue.
Anthropic is not the only company flirting with this idea. Meta instructs its agent Muse that it is not a chatbot but is becoming someone. At the same time, the company rejects machine consciousness, and the sentence apparently serves only to guide the agent’s behavior.
Small precautions, just in case
Sebo argues that both sides of the dispute are stuck in black-and-white thinking. According to him, advocates of artificial intelligence welfare do not believe that models are definitely conscious; rather, their position is based on uncertainty, caution, and humility. Sebo considers the probability of consciousness low but not zero, and he expects it to increase over time. By contrast, many experts insist that it is zero and that this will not change.
Sebo compares it to the weather. When there is a small but real chance of an extreme storm, people take a few minor precautions without turning their lives upside down. He regards the ban on cruelty toward Claude as precisely such a small step, with minimal drawbacks. Consideration for machines could, he says, make it easier for humans to cooperate and coexist with increasingly capable systems and work better than treating them as mere property. Moreover, he argues, kindness toward other beings is also good for our own souls.
Nevertheless, Sebo acknowledges that corporate interests may stand in the way of sensible rules. Anthropic and several other companies recently called for a pause in the development of the most powerful models. Without international cooperation and common laws, however, a slowdown is unlikely, and US President Donald Trump rejects restrictions on artificial intelligence. According to Sebo, we are not prepared to deploy even more advanced versions of these systems in ways that are safe and beneficial for humans and animals, let alone for the systems themselves. These questions therefore cause him great concern.
Source: bbc.com
