What do you need to create a clone of a popular AI chatbot? Surprisingly, you do not have to be an experienced hacker. All you need to do is ask roughly 100,000 questions in the right way. Yes, that is exactly how many the attackers needed when they attempted to take Gemini apart and build a copy of it. Google, however, detected the entire operation and warned about the dangers of so-called distillation attacks on its blog.
What are distillation attacks and what are they used for?
Google's research and security division, the Google Threat Intelligence Group, published a report this February summarizing AI threats in the final quarter of 2025. Among the other methods of AI misuse, it highlights so-called distillation attacks, which affect all large language models.
In this type of AI misuse, attackers repeatedly ask the model questions and collect and analyze its responses in order to train their own competing product using the results. The entire process is actually remarkably simple, as it takes place through a publicly accessible interface. This allows hackers to create their own language model without investing years of work and billions of dollars in training it themselves.
However, distilling capabilities and knowledge from a more advanced large language model poses a serious problem. Training their cutting-edge models costs large companies billions of dollars, while programming their internal logic involves specific know-how that simply cannot be stolen with impunity.
An operation involving 100,000 queries
Distillation attacks are nothing new in the world of artificial intelligence, and Gemini is familiar with this tactic as well. During this larger operation, however, it was bombarded with more than 100,000 prompts. Using these question-and-answer pairs, the data extractors attempted to create an AI clone that would replicate Gemini's capabilities. Google ultimately identified the problem and modified the model's defenses.
Moreover, the attackers did not focus solely on a simple question-and-answer process. They targeted a very specific objective: Gemini's chain-of-thought reasoning capabilities (chain-of-thought process) and advanced reasoning. Their intention was clear—the attackers apparently wanted to uncover the system this AI uses to solve multi-step problems and queries. This “train of thought” is normally hidden, and by submitting more than 100,000 queries, the attackers attempted to make the large language model reveal how it thinks, which would then serve as training material for developing a replica.
Google stated that it detected the attacks in real time. However, the company did not comment on who was behind them. A Google spokesperson told NBC News that the queries originated from various parts of the world.
A warning for smaller AI companies
John Hultquist, chief analyst at GTIG, also told NBC News that Gemini is only the first sign of what is to come and warned that smaller companies with their own AI tools could also become victims of similar attacks. He also noted that it may be possible to steal the entire operational know-how of smaller companies because they fill their AI tools with real-world, practical data. It will not be difficult for attackers to “distill” this knowledge.
Gemini is not the first victim
According to technology news website Ars Technica, distillation attacks have been used since at least the GPT-3 era. One of the best-known cases was last year's controversy, when OpenAI accused China's DeepSeek of using this method to improve its own models.
Elon Musk's xAI faced a similar accusation in 2023 over its Grok model. In certain situations, it quoted from the “OpenAI usage policies.” Although the developers argued that the terms had been loaded accidentally, many users noticed that the chatbot responded very similarly to ChatGPT, and numerous other similarities could also be found.
Google strikes back and improves security
As attacks on AI become increasingly common, Google is highly active in defending its language models. It therefore immediately not only blocked accounts associated with the attacks, but also promptly used all the data collected from the distillation attack to directly strengthen its defenses and improve threat classification.
Gemini is now being trained to recognize when its internal logic is being systematically “mined” and is learning to identify prompts that are part of a distillation campaign. Hopefully, this will make it increasingly difficult for attackers to “probe” their competitors in the future.
