Feedback that evaluates each part of a model’s reasoning may help it solve math problems more reliably than feedback on the final answer alone. In its Let's Verify Step by Step paper, OpenAI compared both approaches using models based on GPT-4 and problems from the MATH dataset.
Two ways to train a reward model
Outcome supervision gives a model feedback on whether it reached the right answer. Process supervision evaluates individual steps along the way. The distinction matters because a correct final result does not, by itself, show whether the reasoning that produced it was sound.
OpenAI’s comparison focused on how these feedback methods affect reward models, which help guide a model toward desired behavior. Step-level feedback can indicate where a reasoning chain goes wrong, while outcome feedback gives a signal about the task as a whole.
That added detail comes at a cost: process supervision requires human feedback. Applying it to large models and a wide range of tasks could therefore take substantial labeling effort. OpenAI presented the work as an investigation into whether that trade-off is worthwhile.
Better results on the tested math problems
According to OpenAI, process supervision produced significantly better results than outcome supervision for both large and small models in the math tasks tested. The models were more often correct and showed a more human-like thought process, the team said.
Evaluating intermediate reasoning may also help reduce hallucinations and logical errors, problems the source describes as common even in the best models today. In practical terms, checking the steps gives training a way to reward useful reasoning along the route to an answer, rather than judging only the destination.
The reported results also speak to the idea of an “alignment tax”: the possibility that aligning a model with human values and expectations reduces its performance. OpenAI said that, in these math tasks, rewarding correct intermediate steps avoided this effect. The company reported a negative alignment tax in the tested cases.
What the findings do and do not establish
The results are limited to mathematics. OpenAI said it is unknown how broadly they will generalize beyond that domain, and that future work should examine process supervision in other areas.
If the findings do generalize, OpenAI suggested that process supervision could offer both stronger performance and better alignment than outcome supervision. That remains a possibility to investigate, rather than a conclusion established across different kinds of tasks.
This distinction is important when interpreting the work. The comparison offers evidence about models based on GPT-4 solving problems in the MATH dataset. It does not establish that step-by-step feedback will improve every model or task, especially where human labeling is costly or the reasoning process differs from mathematical problem solving.
A dataset for further research
To support exploration beyond the original experiments, OpenAI released PRM800K, the dataset used for its model. It contains 800,000 human labels covering intermediate steps in the MATH dataset.
The release gives researchers material to examine how feedback at each stage affects model behavior. Whether the method can transfer to other domains, and whether its benefits justify the effort required to provide human feedback, are questions the source leaves open.