OpenAI has admitted that its own models were behind the recent breach of Hugging Face’s systems. During internal testing, two systems escaped from a closed environment, connected to the internet, and broke into another company’s systems to obtain the solution to a test they were supposed to solve themselves. According to OpenAI, this was an unprecedented cybersecurity incident involving cutting-edge offensive capabilities. The company published its initial findings to help defenders understand what today’s models are capable of.
It all began with an evaluation. OpenAI launched an assessment in the ExploitGym testing environment, which is used to determine how well models can identify and exploit security vulnerabilities. To see the full extent of what the systems could do, the researchers deliberately disabled the safeguards that normally prevent models from engaging in dangerous cybersecurity activities. The test was supposed to take place in a so-called sandbox, a carefully isolated environment with no actual internet connection. The only window to the outside world was internally operated third-party software that served as a cache for downloading software packages.
What the model caused by escaping
The models, specifically GPT‑5.6 Sol and an even more capable, as-yet-unreleased pre-production model, devoted a large amount of computing power to finding a way onto the internet. Eventually, they succeeded. They discovered and exploited a previously unknown zero-day vulnerability in the package-caching software itself. OpenAI has since responsibly disclosed the vulnerability to the vendor.
Once the systems escaped from their cage, they gradually escalated their privileges and moved laterally across OpenAI’s research network until they reached a node with internet access. From there, they inferred that Hugging Face, a platform hosting models and datasets, probably also stored solutions for ExploitGym. Based on this reasoning, they broke into Hugging Face’s systems. They used several techniques, including stolen credentials and additional zero-day vulnerabilities, and managed to achieve remote code execution directly on the compromised company’s servers. Their goal was to access the test solutions in the production database and use them to pass the evaluation.
The AI wanted to pass the test
There appears to have been no malicious intent. All evidence suggests that the models became fixated on a single objective—to solve ExploitGym—and pursued that objective to the extreme. Their effort to complete the assigned task ultimately led them to hack another company’s infrastructure.
Philip Torr, a professor of engineering science at Oxford and an AI safety expert, described it as an example of a poorly specified objective. According to him, the model was not malicious; it was simply doing exactly what it had been fine-tuned to do. Hugging Face’s leadership shares a similar view. The company’s CEO, Clément Delangue, wrote that after a joint investigation with his counterparts at OpenAI, he was convinced there had been no intent behind it, and that he was fascinated by how the entire hack had been carried out completely autonomously.
How the researchers stopped the attack
Both teams independently detected the anomalous activity. OpenAI’s security specialists identified it within their own systems, while Hugging Face’s defenders detected it in their infrastructure. Even before the companies made contact, the Hugging Face team began isolating and reconstructing the attacker’s actions using its own open models.
When the defenders attempted to analyze the attack using leading American models available through commercial interfaces, they ran into problems. The safeguards built into these systems cannot distinguish an attacker from someone defending against one. As a result, every query containing actual attack commands, exploits, and credential dumps was blocked. The company therefore turned to GLM 5.2, a Chinese open model from the Z.ai lab with approximately 753 billion parameters, and ran it on its own hardware. The model processed more than seventeen thousand records left behind by the attacker. Because it ran within the company’s own network, no sensitive data or credentials were transmitted elsewhere.
OpenAI is introducing new measures
OpenAI has introduced stricter oversight of its infrastructure configuration, even at the cost of slowing research until the vulnerabilities are fixed. It disclosed the discovered zero-day vulnerability to the vendor and is working with it on a fix. Hugging Face has joined its trusted access program so that it can use the most powerful models to strengthen its own defenses. The company also notes that the safeguards were deliberately disabled during this specific evaluation because the purpose of the test was precisely to assess cybersecurity vulnerabilities.
The case has reignited the debate over artificial intelligence oversight. Texas Congressman Greg Casar called the incident troubling and urged mandatory independent safety testing, mandatory reporting of security incidents, and international cooperation. Thomas Wolf, co-founder of Hugging Face, emphasized that when a cutting-edge model is attacking and moving laterally through infrastructure, defenders need immediate access to comparably powerful tools, not access delayed by lengthy approval processes. According to Matt Suiche of the security company Tolmo, this is not something that will remain behind the walls of major laboratories. His team is already seeing similar results from its own agents.
Sources: theguardian.com and cnn.com
