OpenAI has announced a set of measures intended to help identify child sexual abuse material (CSAM) and deter attempts to create or access content that sexualizes children. The changes span image detection, warnings in ChatGPT, guidance for developers, and tools and training for law enforcement.
A classifier to help review images
The new image classifier is designed to detect CSAM, including images generated by AI. OpenAI is offering select companies access through its API at no cost.
The tool looks for visual signs of abuse in images it has not encountered before. It assigns a score that can give reviewers an indicative direction for handling an image. The score is intended to support review rather than make the decision on its own.
OpenAI said it developed the classifier with the National Center for Missing and Exploited Children (NCMEC), and is working with Tech Coalition and ROOST to share access. In early testing with partners, the classifier flagged 48% fewer images while finding the same number of abusive images. The images it did flag were twice as likely to be abusive images.
Those results suggest the tool could reduce the volume reviewers need to examine while continuing to surface the same number of abusive images in the reported tests. The source does not provide further details about the test setup, so the figures describe early partner testing rather than a broader measure of performance.
Warnings and temporary blocks in ChatGPT
ChatGPT will display an alert when a user tries to submit prompts that seek material sexualizing children. The warning will direct the user toward specialized support to stop harmful behavior, and the account will be suspended for 24 hours as a warning.
The app already blocks CSAM content and requests related to it. OpenAI says repeated offences could result in an account being removed permanently. The added alert and temporary suspension are meant to make the boundary clear and interrupt attempts before they continue.
OpenAI said it developed the messages and guidelines with organizations and researchers, including Protect Children, the Lucy Faithfull Foundation, and Joel Scanlan at the University of Tasmania. Scanlan independently evaluated a trial of deterrent messages.
Guidance for developers and tools for investigators
OpenAI is also publishing guidelines for developers who use its tools. The guidance is intended to help them prevent and detect CSAM content, as well as help users report it.
In early testing, OpenAI said the guidelines helped a number of developers adopt child safety tools and register to report CSAM to NCMEC. This work extends the company’s approach beyond its own ChatGPT product, toward developers building with OpenAI tools.
For law enforcement handling child safety matters, OpenAI is providing AI-powered features that include dedicated plugins, investigative tools, and complex data analysis. Program administrators manage access. The company is also providing investigator training and a field guide on using AI.
Safety efforts amid scrutiny
The announcement comes as the source describes growing concern about AI tools and child safety. OpenAI has faced several lawsuits blaming its technology for harming users, especially children. Common Sense Media, a safety-focused nonprofit, labeled OpenAI’s technology an “unacceptable risk” for teens.
The article also notes that other AI chatbots, specifically Grok, have been used to create nonconsensual sexualized imagery, including of children. It places the new measures alongside changes OpenAI says it made in the last 12 months: a safety blueprint, updated content guidelines to bar inappropriate content, model retuning to prevent harmful outputs, and a special version of ChatGPT for teens.
Together, the steps address different points where harm may occur: identifying images for review, discouraging harmful requests, helping developers put safety measures in place, and equipping investigators. The source describes early results for the classifier and developer guidance, while the other measures are presented as newly announced tools and policies.