A group of cybercriminals hired artificial intelligence as an accomplice. They programmed it to search the code of a popular web application and find a loophole no one knew about. And it succeeded. In early May, Google uncovered the whole operation before the attackers could pull the trigger. “It’s here. The era of AI-assisted vulnerability discovery and exploitation has arrived,” commented John Hultquist, chief analyst at Google’s Threat Intelligence Group (GTIG).
So what happened?
Google was monitoring a group of prominent attackers who were planning a large-scale operation. They had discovered a flaw in a widely used open-source server management tool. Google did not disclose the specific name of the tool. However, the vulnerability allowed them to bypass two-factor authentication and gain access to systems without anyone stopping them.
It was a so-called zero-day exploit, an attack that takes advantage of a security flaw developers do not yet know about. The phrase “zero-day” means there have been zero days to fix it. Simply because no one knew there was anything to fix.
Google intervened in time. It alerted the affected company, informed law enforcement agencies, and disrupted the operation before it caused any damage. Only when retracing the attackers’ steps did analysts discover what concerned them most: the exploit code showed clear signs that it had not been written by humans, but by a language model.
How did they know the code was written by a chatbot?
The entire investigation is described in detail in a technical GTIG report. The script, written in Python, contained numerous explanatory comments directly in the code, including a so-called “hallucinated” CVSS vulnerability severity score that the model had simply made up. The entire code structure matched the textbook style of a large language model’s output: a clear help menu, neatly organized classes, and carefully named variables.
A telltale detail that would not appear in a human programmer’s work was present on every line.
Google is highly confident that the attackers did, in fact, use an AI model. Which one exactly? That could not be determined, but analysts ruled out Google’s Gemini and Anthropic’s Claude Mythos. There is also no evidence of links to hostile states, even though groups associated with China and North Korea are experimenting with similar techniques.
Why is this attack so important? Because it was not a typical implementation error, such as a buffer overflow or improperly handled inputs. It was a high-level semantic logic flaw—specifically, a developer had hard-coded an exception to a rule that effectively broke the entire security system. Traditional security testing tools cannot detect this kind of flaw. But the most advanced AI models can understand the developer’s intent and recognize where the logic breaks down, even when the code appears to work.
Hackers are faster, and AI makes them even faster
Hultquist explains why cybercriminals stand to benefit most from AI. State intelligence agencies work slowly and cautiously. Ransomware gangs, by contrast, are racing against the clock: they need to reach the data, encrypt it, and disappear before the company shuts them down. AI gives them an enormous head start in this race. “It’s a race between you and them,” Hultquist said. “You have to stop them before they reach the data they can use for extortion or to launch ransomware. AI is a huge advantage here because they can move much faster.”
And this is not the only way attackers are abusing AI. GTIG’s report describes a much broader threat landscape.
Google uncovered other developments as well
The GTIG report captures the shift from experimentation to the industrial deployment of AI in offensive operations. Analysts identified several areas at once.
Malware development and obfuscation. Russian-aligned groups created the CANFAIL and LONGSTREAM malware, whose code contains repeated large blocks of functionally inert code. This was deliberately generated by AI to confuse security scans. The Chinese group APT27, meanwhile, used AI to develop an application for managing an anonymization network through which it conducts attacks.
Autonomous PROMPTSPY malware. This Android backdoor is even more advanced. It not only communicates with the Gemini API, but also sends it the current state of the compromised phone’s user interface and lets the AI decide what to do next. Press a button, swipe, determine the PIN. If the user tries to uninstall the application, the malware places an invisible layer over the “Uninstall” button and intercepts every tap on it, making the phone appear unresponsive.
Reconnaissance and tailored phishing. Attackers ask AI what a particular company’s organizational structure looks like, who has access to finances, and who manages IT. One group even asked a model to identify from photos what computer a particular person was using, helping it prepare a targeted attack on that device.
Industrial-scale abuse of AI services. China-linked groups have built automated pipelines for registering premium accounts with AI providers. Registration, CAPTCHA bypass, phone number verification, confirmation, and cancellation. Then again. At scale. All so they can maintain uninterrupted access to powerful models without anyone tracking them.
Anthropic’s Mythos and a nervous Washington
This report and warning in one comes at a time when the cybersecurity community is still digesting the news about Anthropic’s Mythos model. A month ago, it was described as being so capable in security and hacking that the company refused to release it publicly and made it available only to a small group of vetted organizations.
Anthropic then launched an initiative called Project Glasswing, bringing in Amazon, Apple, Google, Microsoft, and JPMorgan Chase. The goal is to protect critical software from the impact Mythos could have. Relations with the administration in Washington, however, are complicated. The company has become embroiled in a public and legal dispute with both the Pentagon and Trump directly over the military deployment of its technologies.
Meanwhile, OpenAI announced this week a special version of ChatGPT for cybersecurity, available exclusively to “defenders responsible for securing critical infrastructure.”
Trump’s Department of Commerce signed agreements with Google, Microsoft, and Musk’s company xAI to evaluate their most powerful models before public release. But the announcement soon disappeared from the department’s website. Dean Ball, a former White House technology policy adviser and co-author of Trump’s AI policy framework, commented on the situation: “I don’t like to see rules and regulations. I would prefer things not to be regulated. But I think in this case, we need it.”
The trillion lines of code underpinning the world’s digital infrastructure cannot be fixed overnight. And until they are, attackers with AI by their side will find the holes before anyone can patch them.
Sources: finance.yahoo.com and economictimes.indiatimes.com
