OpenAI's o1-preview has drawn attention in medicine because a study found it performed unusually well on difficult diagnostic tasks. The results suggest that advanced AI systems can handle some forms of medical reasoning at a level that challenges assumptions about what remains uniquely human.
But the same findings also make clear that strong benchmark performance is not the same thing as safe, practical healthcare. The system showed weaknesses, and the researchers called for better ways to test medical AI before it is treated as a clinical tool.
What the study tested
Researchers from Harvard Medical School and Stanford University evaluated o1-preview across a broad set of medical diagnosis tests. The focus was not routine symptom lookup, but difficult cases that required reasoning through complex information.
Across all cases it examined, o1-preview correctly diagnosed 78.3% of them. In a direct comparison involving 70 specific cases, it reached 88.6% accuracy, while GPT-4 managed 72.9%.
That comparison matters because it places o1-preview not only against doctors, but also against a previous OpenAI model. The source describes the jump as a significant improvement over GPT-4, especially on tasks that require deeper analytical thinking.
The study also looked at reasoning quality using the R-IDEA scale, which the source describes as a standard measure for evaluating medical reasoning quality. On that scale, o1-preview achieved perfect scores in 78 out of 80 cases. Experienced doctors reached perfect scores in 28 cases, while medical residents did so in 16 cases.
Those numbers do not mean the system is ready to replace clinicians. They do show that AI medical diagnosis systems are moving beyond simple answer selection and into more structured reasoning tasks.
Where o1-preview stood out
The strongest results came from complex management cases that 25 specialists had designed to be difficult. These were not framed as easy diagnostic puzzles. The source says human participants struggled with them.
In that set, o1-preview scored 86% of possible points. Doctors using GPT-4 scored 41%, and doctors using traditional tools scored 34%.
That gap is the core reason the findings are getting attention. In a controlled benchmark, the AI system did especially well when asked to diagnose, reason and recommend treatments in difficult scenarios.
One of the study authors, Dr. Adam Rodman, highlighted the importance of the results while also warning readers to treat them carefully. He wrote on X:
"This is the first time I have promoted one of our preprints (rather than the full peer-reviewed study) so caveat emptor. But I truly think our results have implications for medical practice so I wanted to get them out as quickly as possible."
That caution is important. The source identifies the work as a preprint, not a full peer-reviewed study. It also notes that some test cases may have appeared in o1-preview's training data. When the researchers tested the system on newer cases it had not encountered, its performance fell only slightly, but the possibility still matters for how the results should be interpreted.
The limits were just as important
o1-preview did not perform equally well across every medical task. The source says it struggled with probability assessments and showed no real improvement over older models in that area.
One example involved estimating the likelihood of pneumonia. o1-preview suggested 70%, while the scientific range cited in the source was 25-42%.
That type of error is not a small detail in medicine. Diagnosis is not only about naming a possible condition. Clinicians also have to weigh uncertainty, decide which explanation is more likely and choose tests or treatments that fit the situation.
The researchers found a broader pattern. The system performed strongly on tasks involving critical thinking, including diagnosis and treatment recommendations. It had more difficulty with abstract tasks such as estimating probabilities.
The source also notes that o1-preview tends to produce detailed answers. That could have helped its scores, especially on evaluation systems that reward thorough reasoning. A detailed response can look impressive, but detail alone does not prove that a recommendation is practical, efficient or clinically appropriate.
Why real-world use is harder than a benchmark
The study looked at o1-preview operating by itself. It did not test how the system would perform alongside human doctors in real clinical workflows.
That distinction is central to the medical AI debate. A benchmark can measure whether a system reaches the right answer in a prepared case. Healthcare settings involve constraints that are harder to capture, including practical implementation, cost, available infrastructure and how clinicians would interact with the system.
Some critics have argued that o1-preview's suggested diagnostic tests are often too expensive and impractical for real-world use. The source also says that even more capable systems do not automatically solve the problem of making AI useful in healthcare settings.
Since the study, OpenAI has released the full o1 version and its successor o3. According to the source, those systems show significantly improved performance on complex reasoning tasks and surpass o1-preview on benchmarks that require deep analytical thinking.
Still, stronger benchmark scores do not remove the central concerns. The hard question is not only whether a model can reason through a case, but whether it can be tested, integrated and supervised in a way that helps patients.
What should happen next
Rodman warned against treating the findings as a reason to abandon doctors. His statement was direct:
"This is a benchmarking study. While these are 'gold standard' evaluations of reasoning that we use for human clinicians, these are obviously not actual medical care. Do not get rid of your doctor in favor of o1."
The researchers argue that medical AI needs stronger evaluation methods. Multiple-choice tests do not capture the complexity of real medical decision-making.
They are calling for several next steps:
- More practical testing methods for medical AI systems.
- Real-world clinical trials.
- Better technical infrastructure.
- Improved ways for humans and AI to work together.
The study's message is therefore not simple hype. o1-preview appears to have reached a new level on difficult diagnostic benchmarks, especially compared with GPT-4 and with human performance in the tested cases. At the same time, the findings show why medicine cannot treat benchmark success as a substitute for clinical validation.
For now, o1-preview is best understood as a signal of where AI medical reasoning is heading. It may influence how doctors, researchers and health systems think about diagnostic support. But the source is clear on the practical point: real medical care requires more than a high score on a test.