Language models can produce fluent answers, but logical reasoning remains a difficult test. A study by Google's AI division DeepMind points to a simple factor that can change the outcome: the order in which a problem presents its premises.
The finding is practical because the underlying task does not change. The same logical information can lead to different model performance depending on whether the premises are arranged in the order that supports the reasoning path.
What DeepMind tested
The study examined how premise ordering affects reasoning across several AI models. A premise is a statement or assumption that forms the basis for an argument or action, so the ordering of premises determines how the pieces of a reasoning task appear to the model.
The researchers focused on deductive reasoning. More specifically, they used tasks that required the logical inference known as modus ponens, where true statements are used to derive another true statement.
In its basic form, modus ponens is straightforward: if you have the statements "If P, then Q" and "P is true", then you can infer that "Q is true". For humans, this kind of step can be relatively simple. For language models, the study shows that presentation still matters sharply.
The models performed best when the premises appeared in the same order as the logical conclusions. According to the researchers, the same pattern also applies to mathematical problems.
The order can matter by over 30%
The most important result is that premise order can significantly affect LLM performance even when the task itself remains the same. When a reasoning problem requires the model to read the description back-and-forth, the researchers reported a performance drop of over 30%.
That result makes the issue different from a missing-information problem. The model is not being asked to solve a different task, and the logical content has not been replaced. Instead, the structure of the input changes how well the model can use the information it already has.
The study also found that models struggled more as the number of rules increased. Superfluous premises created additional confusion, which suggests that unnecessary information can interfere with basic reasoning tasks rather than simply being ignored.
For prompt experts, this points to a clear working lesson: when asking an LLM to reason through a chain of statements, the sequence of those statements can be part of the problem design. A cleaner order may help the model follow the intermediate reasoning steps more reliably.
Which models were evaluated
The tests included GPT-3.5 Turbo, GPT-4 Turbo, PaLM 2-L, and Gemini Pro. The source article reports that OpenAI's GPT models performed better when the premise order was exactly reversed from the ground truth.
The comparisons also showed a notable pattern for Gemini Pro. Google's newer Gemini Pro performed similarly to OpenAI's older GPT-3.5 Turbo, with accuracy decreasing rapidly when a relatively small number of rules was involved, even if those rules were in logical order.
These model comparisons do not turn the study into a simple ranking. The more important point is that different leading systems can be sensitive to how a reasoning task is written. Even when a model is powerful, a logically valid prompt can still become harder for it because of ordering, extra premises, or more rules.
Why this matters for AI reasoning
Reasoning ability is expected to have a significant impact on how language models are used in the future. The source article notes recent progress in features such as larger context windows, including the limits Google recently broke with Gemini 1.5 Pro.
But a larger context window is not the same thing as stronger reasoning. The DeepMind result highlights a narrower and more basic issue: even relatively simple logical structures can become difficult when the model has to assemble the reasoning path from a less helpful presentation.
The researchers do not provide a theoretical explanation for why the effect occurs. They also do not offer possible solutions for improving general reasoning ability based on the findings. That limitation matters, because the result is useful for prompt design but does not itself explain how to build fundamentally more capable reasoning systems.
Still, the work gives prompt experts and AI users a concrete point to test. If a task depends on basic reasoning, the premises should be arranged so the model can move through the intermediate steps in order. Extra statements should be treated carefully, especially when they do not support the conclusion.
A benchmark for further work
The researchers made their systematically generated tests available in the R-GSM benchmark for further investigation. That gives others a way to examine the same kind of ordering effect across reasoning and mathematical problems.
The broader takeaway is modest but important. Logical reasoning is not only about what information a model receives. It is also about how that information is sequenced, how many rules the model must handle, and whether unnecessary premises are mixed into the task.
The source article frames reasoning as a central challenge for AI research. It also notes that there has not been much progress in this area since the release of OpenAI's GPT-4, and that training on large amounts of text and visual data is widely seen as insufficient for building fundamentally more capable AI systems. Demis Hassabis, CEO of DeepMind, and Sam Altman, CEO of OpenAI, are mentioned in that context.
DeepMind's study does not solve LLM reasoning. It does show that a small change in how a problem is presented can make a large difference in how well a model handles it.