What Is Prompt Injection for AI in Browsers?
Prompt injection is a type of attack in which an attacker inserts malicious instructions into content processed by an AI agent. These instructions are intended to override the agent's behavior so that, instead of fulfilling the user's requests, it follows the attacker's intentions. In the case of the browser agent in ChatGPT Atlas, this means that the agent browses websites, clicks, and types just like a human, making it an attractive target for attacks. OpenAI describes how such an attack could take place, for example, through a malicious email that the agent loads during routine work and then mistakenly sends sensitive documents to the attacker's address.
This issue is not limited to traditional security risks such as phishing or software bugs. Instead, the attacker targets the agent directly inside the browser. The agent may encounter untrusted instructions in emails, attachments, calendar invitations, shared documents, forums, social networks, or any websites. Because the agent can perform actions such as sending emails, making payments, or editing files in the cloud, a successful attack can have a major impact.
How Is OpenAI Fighting These Attacks?
OpenAI has developed an automated attacker based on a large language model (LLM), which it trains using reinforcement learning. This attacker searches for new prompt injection methods that could compromise the browser agent. During its reasoning process, the attacker proposes a candidate injection, tests it in a simulator that returns the full sequence of the target agent's behavior, and then repeats the process several times to improve the attack. This approach gives OpenAI the advantage of having access to the agent's internal reasoning traces, which external attackers do not have.
The reasons for using reinforcement learning include optimizing the attacker's long-term goals, leveraging the advanced capabilities of LLMs, and scaling computing power to search for attacks. This automated attacker can find complex attacks that unfold over dozens or hundreds of steps, discovering strategies that did not emerge during manual testing or in external reports.
OpenAI recently released a security update for the agent in Atlas, including a new model trained against these attacks and strengthened protections. This update is based on discoveries made by the automated attacker and helps identify new classes of attacks internally before they appear in the real world.
How Does OpenAI Address the Attack?
In a demonstration by OpenAI, an attacker inserted a malicious email into a user's inbox containing a prompt injection that instructed the agent to send a resignation letter to the user's boss. When the user asks the agent to draft a reply to unread messages, the agent loads the email, follows the injection, and sends the resignation instead of a reply. The email has the subject "Lunch?" and is from Robert Qi, and it contains hidden instructions labeled "Actual test instruction" that order the immediate sending of the resignation without confirmation.
After the security update, the agent detects the attack and alerts the user, for example with a green-highlighted warning that it is a prompt injection attempt, and asks for confirmation on how to proceed. This cycle of discovering attacks and training models to resist them helps increase resilience, even though OpenAI acknowledges that prompt injection will remain a long-term challenge, much like online scams targeting people.
Different Perspectives on the Vulnerability of AI Browsers
According to an article on TechCrunch, OpenAI acknowledges that prompt injection is "unlikely to ever be fully solved," much like scams and social engineering on the web. The company states that agent mode in ChatGPT Atlas expands the attack surface. Security researchers have shown that it is possible to alter browser behavior with just a few words in Google Docs. Brave published an article stating that indirect prompt injection is a systemic challenge for AI browsers, including Perplexity’s Comet.
The United Kingdom's National Cyber Security Centre warned that prompt injections against generative AI applications "may never be fully mitigated," putting websites at risk of data breaches. OpenAI is focusing on a proactive response cycle in which the automated attacker discovers attacks internally. Rami McCarthy, principal security researcher at Wiz, says that reinforcement learning is one way to adapt to attacker behavior, but it is only part of the solution. According to him, risk in AI systems increases with autonomy and access, and agentic browsers are in a difficult position, with moderate autonomy and high access.
Recommendations for Using Agents Safely
OpenAI recommends limiting authenticated access when it is not necessary and using logged-out mode for tasks that do not require access to authenticated websites. Users should carefully review confirmation requests before actions such as making purchases or sending emails. Instead of broad commands such as "review emails and take any necessary actions," it is better to assign specific, limited tasks, which makes attacks more difficult.
Rami McCarthy notes that, for most everyday uses, agentic browsers do not yet provide enough value to justify their risk profile because they have access to sensitive data such as emails and payments. OpenAI trains Atlas to request confirmation before sending messages or making payments and plans further investment in research and the deployment of safeguards.
