AI agents can keep working through tasks such as writing research papers or building software projects, making decisions at each step. But a choice that looks reasonable at the time may create problems much later, after the agent has spent much of its available budget. A new benchmark, Taste-Bench, asks whether models can recognize which path is more likely to work out.
Why the quality of a decision takes time to show
Many existing benchmarks focus on whether an agent completes a task. That leaves out the quality of the steps it takes along the way. A tidy implementation and a messy one, for example, might produce the same output and pass the same tests at first. The difference may only become apparent when later changes are needed.
The researchers call the ability to choose well in these situations an agent’s “taste.” Judging that quality immediately is difficult because the consequences may not yet be visible. Human reviewers would need substantial expertise and time, making that kind of evaluation hard to scale.
Taste-Bench instead uses completed runs as evidence. A run, or trajectory, records an agent’s steps and observations. When multiple attempts at a task take different routes, their eventual outcomes can help show which earlier choice led to the better path.
Comparing choices at a decision fork
The researchers call a point where two paths diverge a “decision fork.” To test a model, they show it the task and the trajectory up to that point, then ask it to choose between two possible next directions. They hide the later work that reveals which route performed better.
Taste-Bench contains 502 questions drawn from coding and AI research tasks. Some compare separate attempts at the same task. Others come from a single attempt in which an agent first took a detour, then corrected itself. Those questions test whether another model can recognize the better route before the original agent did.
The underlying material came from thousands of scored agent runs, including SWE-bench Pro and METR’s public transcript release MALT. A generator model selected candidate forks, discarded obvious or undecidable questions, and human reviewers confirmed almost all of the resulting labels.
Models struggle when the payoff is far away
The team evaluated 14 models from the Claude, GPT, Grok, DeepSeek, GLM, MiniMax, and Mistral families. Each question was presented twice, with the answer options reversed the second time. A model had to choose correctly both times for the question to count as correct, so pure guessing would score 25 percent.
GPT-5.6 Sol had the highest average score at 59.7 percent, followed closely by GPT-5.5 at 59.5 percent. The other models scored well below them, and none solved the benchmark. Questions based on an agent correcting its own detour were harder than those comparing separate attempts.
Performance depended heavily on when the decisive evidence became available. Across the models, average accuracy was 62.3 percent when the clue was already present in the earlier trajectory. When it only emerged after extensive later work, accuracy fell to 21 percent. At those forks, choosing the better path requires anticipating evidence that has not appeared yet.
Giving models more reasoning time did little to address the gap. In tests with two models and three reasoning settings, accuracy barely changed between the lowest and highest settings. Models used the most reasoning tokens on the hardest questions, even as they missed those questions most often.
Training judgment improved task outcomes
The study also tested whether taste could be taught. The researchers used a teacher model that knew the correct path to train a student model that saw only the question. Using Qwen3.6-27B as the base, they raised accuracy on unseen tasks from 30 to 47.9 percent.
They then supplied the student’s advice at decision forks while a separate agent worked on the task. Across 41 held-out SWE-bench Pro tasks, success rose from 14.6 to 33.7 percent. The researchers report that perfectly correct advice would have produced a 39 percent success rate, putting the student’s result close to that potential.
Taste-Bench measures a skill that end-to-end benchmarks may miss: choosing steps whose value only becomes clear later. The findings suggest that this judgment can be measured and improved, though models still struggle when the evidence arrives well after the decision. The benchmark and its code are publicly available.