Coding agents depend on more than a language model. A harness supplies the prompts, tools and context management that let a model work in a code repository. A study from two researchers at Shandong University suggests that these systems can change the bill far more reliably than they change benchmark scores.
Benchmark gains need to be measured against run-to-run noise
The researchers compared Claude Code, OpenCode and mini-SWE-agent while keeping the model constant. They tested five models on tasks from SWE-bench Verified, including a larger pool of 447 tasks and a set of 45 of its hardest tasks. Several configurations were run twice to measure how much results varied without changing the setup.
That variation matters because earlier harness comparisons often relied on one run. The study reports that repeated runs of an identical setup can differ by up to six points on SWE-bench Verified. A claimed lead of a few points may therefore reflect ordinary variation rather than a real advantage from the harness.
For example, Zhipu AI advertised a 2.4-point lead over Claude Code. The researchers say that difference sits within the noise range they observed. They also caution that benchmarks may wire harnesses differently, which makes comparisons harder to interpret.
Changing the harness rarely changed which tasks were solved
On the 447-task pool, Claude Code and mini-SWE-agent—the heaviest and lightest harnesses in the comparison—were less than two points apart for both tested models. The researchers describe results within a five-point band as statistically equivalent.
The 45 hardest tasks did not reveal a clear winner either. Harnesses differed by two to five solved tasks per model, while rerunning the same setup shifted the number by as many as three. Sometimes mini-SWE-agent came out ahead; sometimes Claude Code did.
The task-level results showed a similar pattern. The median outcome changed on 13 percent of tasks between reruns, and on 13 percent between different harnesses. Changing the model shifted outcomes on 22 percent of tasks. Which harness succeeded on a given task also changed from run to run, making the apparent harness advantage resemble another draw of the same noise.
Small samples make subtle differences especially difficult to establish. With 45 tasks, even a 13-point gap appeared only half the time, according to the study. With 447 tasks, gaps of about five points became reliably detectable. Smaller advertised improvements, including the 2.4-point claim, fall below those thresholds.
OpenCode showed a consistent weakness, with possible causes
OpenCode was the exception to the broad pattern of similar performance: on the larger pool, it trailed the other harnesses by up to nine points. About half of that difference was linked to an output limit. If a model hit the limit while reasoning, Claude Code asked it to continue; OpenCode ended the step, and the attempt usually failed.
Adding a continuation prompt modeled on Claude Code did not solve the issue, because the same task rarely hit the limit again on a retry. The researchers could not explain the remaining gap, though they observed that models stopped working sooner under OpenCode. They also flagged a tighter time limit for locally run models as an uncontrolled variable.
The other harnesses had their own limitations. mini-SWE-agent has no context management and could run into the context window, mostly on tasks where the other harnesses failed too. OpenCode's search tools tried to download a binary file on first use, causing them to break when offline.
Prompts and tool descriptions shape the cost
For the same model and tasks, a single task could cost up to three times more depending on the harness. Similar amounts of tool output across the systems did little to explain the difference. Instead, the system prompt and tool descriptions were a major factor: those materials are sent again with every step.
The study measured about 16,600 tokens for Claude Code's prompt and tool descriptions, roughly 7,000 for OpenCode, and under 1,000 for mini-SWE-agent. The researchers also found through reverse engineering that Claude Code keeps the model's reasoning output in the conversation history. A larger starting context raises the cost baseline, and the number of steps multiplies that expense.
The cost ranking could depend on how many steps a model takes. With GLM-5.3-Flash, Claude Code cost nearly twice as much as mini-SWE-agent. With Deepseek-V4-Flash, mini-SWE-agent needed enough extra steps that the two ended up even. Lower prices for cached tokens narrowed the cost gaps but did not change the ranking.
Model reasoning may change the picture
The researchers found no home-field advantage for Deepseek's own harness: it did not score better with its own model than other harnesses did, and it cost the most. Claude Opus 5 also produced identical results after its tools were removed from Claude Code. A year-old version solved a similar number of tasks at less than half the cost, because Extended Thinking was not on by default then.
Thinking was the one area where the harness choice changed outcomes in the reported tests. Without thinking, mini-SWE-agent with Qwen3.6 dropped eleven points, while Claude Code and OpenCode stayed within noise. HY4-Preview lost eleven points in Claude Code without thinking. The researchers say their test grid was too small to explain why these model and harness combinations responded differently.
The study relies mainly on SWE-bench Verified, with a small cross-check on Terminal-Bench 2.1 showing the same broad pattern. Its practical recommendation is to publish a rerun of the same setup alongside each harness comparison, so readers can judge claimed gains against ordinary run-to-run variation. Data and analysis scripts are available on GitHub.