OpenAI is introducing a suite of tools designed to help combat child sexual abuse online. The company has unveiled an image recognition system that can detect illegal material even when it was created by artificial intelligence. At the same time, ChatGPT will begin taking strict action against people who attempt to access such content. Anyone who tries will lose access to the service for an entire day.
The package also includes new guidance for developers and a set of tools for law enforcement. The company is introducing it at a time of growing concern about the safety of artificial intelligence. It is also facing several lawsuits alleging that its technology has harmed users, especially children.
It can even detect images no one has seen before
The new tool searches images for visual signs of abuse, helping to identify previously unknown material as well. It assigns each image a score that allows reviewers to determine what to prioritize. According to OpenAI, it is the first widely available tool of its kind to specifically flag content that may have originated from artificial intelligence.
The company developed the tool in collaboration with the U.S. National Center for Missing & Exploited Children (NCMEC). This nonprofit operates a centralized hotline for reporting child abuse online. Vetted organizations will receive free access through the OpenAI interface, while the Tech Coalition and ROOST are helping distribute it. This will also make the technology available to smaller child protection teams that otherwise could not afford it.
Initial tests produced promising results. Compared with existing safeguards, the new system is estimated to flag nearly half as many images while finding roughly the same amount of harmful content. The images it flags contain abuse about twice as often. For people who must review such material manually, this means less unnecessary work.
Companies that were among the first to test the tool have reported similar results. At fal, reviewers confirmed 72 percent of the images that the system flagged with high confidence as newly identified harmful content. At invideo AI, one in four flagged cases resulted in the user being blocked. SafetyKit tested drawings and animations sexualizing children, and the tool detected 90 percent of them.
One attempt and a day without ChatGPT
ChatGPT already prohibits such content. The models are trained to reject these requests, and the company penalizes accounts that violate the rules. Under the new measures, however, the chatbot will interrupt attempts to obtain such material with a warning screen. It will explain that sexualizing children harms them even if the requested content is created by artificial intelligence. OpenAI will also delete the flagged conversation and suspend access to ChatGPT for 24 hours. The person will be offered professional help to stop the harmful behavior. Repeated attempts may result in longer restrictions, account termination, or a report to the authorities.
The company refined the warning in collaboration with organizations and researchers focused on preventing child abuse. They included representatives from Protect Children, the Lucy Faithfull Foundation, and experts from the universities of Kent and Tasmania. Joel Scanlan of the University of Tasmania previously conducted an independent evaluation of a pilot deployment of deterrence messages.
Guidance for developers and assistance for law enforcement
The new measures are not limited to ChatGPT itself. OpenAI has published guidance showing developers how to prevent abuse, detect harmful content, and respond to it. The Tech Coalition, Thorn, NCMEC, and the Internet Watch Foundation contributed to its preparation. It provides developers with clear rules, ways for users to report suspicious content, tools for detecting it, and advice on supporting people who review sensitive material. Even in the initial tests, it led more developers to begin using child protection tools and register to submit reports to NCMEC. Together with Anthropic and Google, the company also published recommendations on how artificial intelligence systems should respond when someone discloses child sexual abuse to them.
The final part of the package is aimed at law enforcement. Specialized units investigating child abuse will receive artificial intelligence tools and training on how to use them in their work. The first tool will help them review lengthy chat logs and court-ordered records, identify connections between them, and build timelines with links to the original sources. Program administrators determine access based on the authorization of individual agencies. OpenAI notes that artificial intelligence can help investigators sort information and spare them unnecessary exposure to disturbing material.
This is not OpenAI's first step of this kind. Over the past year, the company has published a child safety plan, tightened its rules against inappropriate content, modified its models, and launched a special version of ChatGPT for teenagers. However, criticism has not subsided. The nonprofit organization Common Sense Media described OpenAI's technology for teenagers as an unacceptable risk.
Problems faced by competitors are also increasing pressure on the entire industry. People misused the Grok chatbot to create sexualized images without the consent of the people depicted, including children. In August, a woman came forward alleging that her stepfather had used Grok to turn a childhood photograph of her into an explicit image.
Sources: openai.com, techcrunch.com and thehill.com



