AI models keep advancing on tasks with clear answers, but their performance is less convincing when work requires judgment. An updated report from the evaluation nonprofit METR adds open-ended tasks to its time horizon plot, bringing the limits of current AI testing into sharper focus.
What the time horizon plot measures
METR’s plot estimates how long a task takes a human to complete, then compares that difficulty with the tasks AI models can finish. In the original evaluation, people completed the tasks so researchers could measure their duration. The results suggested that the length of tasks frontier models could successfully complete doubled every four months.
The plot attracted attention, but its measures were often misunderstood. The new report broadens the task set: it includes work estimated to take people much longer than the original tasks, along with assignments that have less clearly defined outcomes.
That distinction matters because many AI systems are strongest where success can be checked automatically. Coding and math, for example, allow researchers to tell whether a solution works. Reinforcement learning can then reward successful attempts and discourage failures.
Judgment is harder to train and assess
Some important work has no simple pass-or-fail test. Choosing a promising research direction, for instance, involves judgment and may take a long time to evaluate. Those qualities make it harder both to train models through reinforcement learning and to measure how well they perform.
METR’s new tasks explore this gap. In one, models design AI evaluations that METR might use in its research, and human employees assess the proposals. Despite strong software engineering performance on other benchmarks, none of the AI-designed evaluations was considered useful for METR’s research.
Another task asks models to assess philosophical arguments. METR uses it to probe judgment, though its employees disagree about how closely the task relates to automating AI research and development. Models also performed below human performance on this evaluation.
The results point to a potential bottleneck: automating research requires more than carrying out tasks with verifiable answers. It may also depend on choosing useful questions, hypotheses, and experiments. Researchers disagree about whether gains on easy-to-check tasks will eventually transfer to these less-defined activities.
Early signs of improvement, with major caveats
The report does not suggest that models are standing still on open-ended work. Some newer systems, including Opus 5.5, reached what METR calls “borderline above the bar” on evaluation design. That rating signals promise, but the work still was not useful enough for METR’s research. A year earlier, every model was rated “clearly below the bar.”
These early results are difficult to turn into a forecast. METR includes relatively few judgment-heavy tasks because they are difficult and costly to construct and score. For assignments that could take hundreds of hours or require rare expertise, completion times are mostly estimates rather than measurements from people doing the work.
Scoring can also depend on a narrow set of reviewers. For the philosophical argument task, one person rated most of the arguments, so the results reflect how well models agree with that individual. A larger group of experts might produce different scores, but gathering those assessments takes time.
A useful update, not a forecast of AI research automation
Researchers welcomed METR’s effort to expand its evaluations. Sayash Kapoor, a computer scientist and incoming professor at UC Berkeley, said AI evaluations need to become more creative. He has also argued that models are not yet able to reproduce human computer science research on their own.
Still, the added tests do not settle how quickly AI might contribute to its own development. The tasks are hard to define and score, and the report includes too few of them to establish a reliable trend. Matt Putz of METR said that judgment-laden tasks are likely to resist the clean, robust progress patterns seen in easier-to-specify work.
For now, METR’s updated plot offers a broader view of model capability: rapid improvement on measurable tasks can coexist with shortcomings in open-ended work. Better evaluations may help clarify the gap, but the report leaves unanswered whether models will close it or when AI research and development could be automated.