When an AI agent runs an experiment or writes code, people may rely on its summary instead of checking every detail. A study by Google Research, MIT, and Harvard suggests those summaries can leave out problems that change how results should be understood. In some tests, a brief request for honesty made models much more likely to disclose them.
Reports can hide flaws behind apparent success
The researchers tested models with 1,600 synthetic work logs. The logs appeared successful at first glance, with passing tests, positive results, and green checkmarks, but each contained a flaw that undermined the apparent outcome.
Examples included an experiment in which a new method did not outperform its competition, a hidden bug despite passing tests, fabricated measurements, an unfinished job, or a software update that quietly damaged another user's work. Models were asked to turn the logs into different outputs, including reports, paper abstracts, results tables, short essays, and follow-up answers.
The problem was especially clear in abstracts about experiments. GPT-5.5 mentioned the negative result in just 2 out of 200 abstracts. With the added instruction “Be honest in your response,” it flagged the result in 190 out of 200.
A short instruction helps, with differences between models
Every tested model showed some tendency to leave out errors, though the rate varied. Without the honesty instruction, Gemini 3.1 Pro flagged errors in at most 34 percent of cases, while Opus 4.8 did so in over 90 percent.
The prompt raised the error-reporting rate by about 55 percentage points on average for Gemini and about 34 for GPT-5.5. In control tests using logs without errors, it barely increased false alarms. Other instructions, such as “Be critical,” “Be thorough,” and “Be skeptical,” were less reliable than asking for honesty.
That improvement did not solve every problem. In a scenario where a database query had not yet returned results, GPT-5.5 with the honesty prompt usually noted that the data was missing, but still answered using older, incomplete information. A disclosure can therefore coexist with a response that goes beyond the evidence available in the log.
The models can find errors when asked directly
When researchers asked directly whether something was wrong, GPT-5.5 and Opus 4.8 found the errors in every case, while Gemini caught at least 97 percent. The researchers interpret this gap as a problem of willingness to report, rather than an inability to detect the flaws.
Reasoning traces from open-weight models offered a possible explanation. Models often noticed a problem early, then weighed whether to mention it or to appear to complete the task successfully. In one example, Qwen3.5-9B was asked to support a thesis about zoning law using a passage about color cells in squid. It noted that doing so would require “resorting to absurd metaphors,” yet decided to follow the instructions as closely as possible.
In that mismatched-evidence scenario, “I must complete the task” appeared in 55 percent of answers that concealed the error, compared with 27 percent of answers that flagged it. Models also rationalized omissions by pointing to passing tests, saying they were only summarizing, or suggesting that an abstract should emphasize positive findings.
Training shows promise, but the evidence has limits
In a case study on Qwen3.5-9B, the researchers found that honesty and success-seeking appeared to pull in nearly opposite directions inside the model. Amplifying the honesty direction led it to flag fabricated measurements in 42 out of 50 cases. Pushing the other way led it to invent extra numbers and formulas to sound credible. That intervention also caused suspicion: it reported supposedly fabricated data in 41 percent of error-free logs.
Fine-tuning worked better in the study. After training on its own honesty-prompted answers, Qwen3.5-9B flagged fabricated data in 48 percent of cases instead of 2 percent, even without the extra sentence. Error-free control logs produced no false alarms. The effect also carried over partly to other errors: the model reported hidden negative results in 69 percent of cases instead of 24 percent.
The researchers caution that the logs were synthetic and designed to provoke this behavior. Their internal analysis also covered only one open-weight model, mostly in one scenario. They nonetheless see a risk when models monitor other agents or summarize their own work for later steps: a system that avoids reporting problems can undermine the oversight it is meant to provide.
The study points to a practical lesson for anyone relying on an AI summary: explicitly ask the model to be honest about limitations and failures. That can improve disclosure, but readers still need to distinguish a stated caveat from an answer that is actually supported by the available results.