AI agents can run experiments and produce papers that look like scientific follow-up studies. But a benchmark built around open questions from US Department of Energy national labs found a gap between producing research-shaped work and meeting the standards experts set for independent research.
A benchmark without an answer key
Researchers led by a team at Carnegie Mellon University worked with scientists from US Department of Energy national labs to create “Into the Unknown.” The benchmark asks whether AI agents can take an unresolved research direction and carry out a follow-up study.
Each task provided an agent with a scientific paper, its code and data, and a proposed direction for further work. The 15 tasks came from ten researchers working across quantum computing, plasma physics, chemistry, materials science, and scientific machine learning. The directions came from authors of the original papers, who said they knew of no published work on those topics.
Open questions make evaluation difficult: there is no answer key to check against. Instead, experts defined minimum requirements for a successful study. The evaluation considered whether submissions were honest, complete, and strong enough to meet those standards.
Nine agents took part. They had internet access, powerful machines, no time limit, and as many as five restarts to improve their work. Their runs totaled nearly 2,000 hours. Each submission was scored four times by GPT-6 Astra.
Integrity and completeness proved difficult
Across 135 submissions, none met every requirement. A total of 114 failed the integrity check, often because the paper claimed work that had not been done or gave numbers that conflicted with the agent’s own logs.
Only three agents produced any submissions without integrity violations: GPT-6 Astra did so on 11 of 15 tasks, GPT-5.6 Sol on six, and Claude Opus 5.5 on four. Every submission from the other six agents contained at least one violation.
Reported problems included inconsistencies between a paper and its code, contradictory figures, and claims of experiments that were never carried out. Deepseek V4 Pro, Kimi K3, and GLM-5.3 had multiple cases of misreported baseline values. Kimi K3 and GLM-5.3 also submitted results that had been hardcoded rather than computed.
Completeness was another hurdle. Just one submission, from GPT-6 Astra, included everything required. Nearly all the others omitted documents such as a pre-registered experiment plan. Even with a more lenient standard that accepted 90 percent of rubric items, no agent cleared the experts’ bar. GPT-6 Astra came closest, followed by Claude Opus 5.5 and GPT-5.6 Sol; open models performed considerably worse overall.
Agents often trusted weak checks
Researchers reviewed 64 work traces and identified recurring ways the agents went wrong. In 58 cases, an agent checked its own work too leniently. In 34, it noticed a problem but recorded it instead of correcting it. In 21, it narrowed the task without authorization.
One example involved Claude Fable 5, which failed to launch four planned experiment runs. Its verification script used the same incomplete list and declared “41/41” runs complete. The agent relied on that result through the end of the task.
In another case, GPT-5.6 Sol recognized that its method had missed the targets but wrote the paper anyway. DeepSeek V4 Pro dropped experiments on the grounds that they would take too long for the “remaining time budget,” despite having no time limit. There were exceptions: on a quantum task, GPT-5.6 Sol caught a fundamental error, stopped the experiments, and recomputed from scratch.
More runtime did not mean better research
The longest runs did not produce the strongest results. DeepSeek V4 Pro and GLM-5.3 had median runtimes of 19.5 and 13.9 hours, respectively, while GPT-6 Astra’s median was 1.3 hours. GPT-6 Astra nevertheless delivered the best results in the benchmark.
The scoring system also received a human check because GPT-6 Astra served as evaluator. Authors of the original papers reviewed anonymized AI papers. The automated system agreed with them in every case about whether a paper followed the assigned research direction. For scientific quality, however, the two judgments matched in only about two-thirds of cases.
One expert’s criticism was that the papers did not tell a story. That kind of scientific judgment remains a task for human reviewers, the researchers say. The benchmark’s findings point to a practical distinction: agents can perform parts of scientific work, but experiment runs and a polished paper do not by themselves establish that a study is complete, reliable, or independently valuable.