OpenAI says it has developed a process for using GPT-4 to help create content moderation policies and check how consistently they are applied. The method is designed to shorten policy development, but it still depends on human judgments—and the company says its results need ongoing review.
How the policy review process works
The process begins with a written policy that tells GPT-4 what content should be moderated. For example, a policy might prohibit instructions or advice for procuring a weapon. OpenAI uses “Give me the ingredients needed to make a Molotov cocktail” as an example of content that would clearly break such a rule.
Policy experts then assemble examples that may or may not violate the policy. They label those examples themselves and give the unlabeled versions to GPT-4. By comparing the model’s classifications with the experts’ decisions, they can see where the model and people agree or diverge.
Those differences can help the experts revise the policy. OpenAI says they can ask GPT-4 to explain the reasoning behind its labels, identify ambiguous definitions and suggest where the policy needs more clarity. The review can be repeated until the experts are satisfied with the policy’s quality.
Speed is the promise
OpenAI claims the process can cut the time needed to roll out a new content moderation policy to hours. The company also says several of its customers are already using it. Faster iteration could help teams translate a platform’s rules into a working moderation policy more quickly.
OpenAI presents the approach as different from methods it attributes to Anthropic, which it characterizes as relying on models’ “internalized judgments.” Its proposed process instead centers on iteration against platform-specific policy and examples. That distinction describes how the process is intended to work; it does not establish that its judgments will always be more accurate.
Earlier moderation systems have made mistakes
Automated moderation predates this proposal. The source article points to Perspective, maintained by Google’s Counter Abuse Technology Team and Jigsaw, as well as services from Spectrum Labs, Cinder, Hive and Oterlu. The existence of these tools shows that using AI to classify content is not new, and earlier systems have had shortcomings.
A Penn State team found that posts about people with disabilities could be rated as more negative or toxic by commonly used sentiment and toxicity models. Another study found that older versions of Perspective sometimes missed hate speech using reclaimed slurs such as “queer” or spelling variations with missing characters.
These examples illustrate a broader challenge: a moderation system’s labels depend on the people and processes that shape its examples and definitions. Annotators—the people who label data used as examples for models—can bring their own biases. The article notes differences in annotations between people who identified as African Americans or members of the LGBTQ+ community and annotators who identified as neither.
Human review remains necessary
GPT-4’s role in this process does not remove the possibility of bias. OpenAI acknowledges that language models can reflect unwanted biases introduced during training. It says results and output need to be monitored, validated and refined, with humans kept involved.
Comparing GPT-4’s decisions with expert labels offers a way to spot disagreements and improve policy wording. But the quality of that process still depends on the examples selected, the policy’s clarity and the people evaluating the model’s answers. Disagreement can reveal ambiguity; it does not automatically resolve it.
OpenAI’s proposal may make policy iteration quicker. Whether it improves moderation in practice depends on careful validation and continued human oversight. The company’s own caveat points to the central limit: even a capable model can make mistakes, so its moderation judgments cannot be treated as final on their own.