A study by researchers at Northwestern University, the University of Liverpool, the New Jersey Institute of Technology, and Rutgers University points to a simple but important prompt engineering finding: length matters in chain-of-thought prompting.
The researchers found that large language models can perform better on complex problem-solving tasks when prompts include longer reasoning steps. The surprising part is that this improvement can appear even when those longer rationales contain incorrect information.
What the study found
Chain-of-thought prompting is a technique that asks a language model to work through reasoning steps before reaching an answer. The source article says this approach has already been shown to improve reasoning abilities in large language models.
This study focused on the length of those reasoning steps. According to the source, longer steps were directly related to better performance on complex problem-solving tasks. The researchers found that simply making the reasoning steps longer, without adding new information, significantly improved reasoning ability.
The opposite also mattered. When reasoning steps were shortened, performance fell significantly, even if the essential information remained in the prompt.
That makes the finding useful for prompt engineering because it separates two things that are often treated as one: the content of a rationale and the amount of reasoning text presented to the model. In these tests, the length of the chain itself had a measurable effect.
Where longer reasoning helped
The study tested a wide range of task types. The source names arithmetic, common sense, and symbolic tasks, along with more specific sets including MultiArith, GSM8K, AQuA, SingleEq, SVAMP, and StrategyQA.
Across that range, the strongest gains appeared on more complex tasks. Simpler tasks benefited less from adding more reasoning steps, while complex tasks were significantly improved by longer chains of reasoning.
This distinction is important. The finding does not mean every prompt should be made longer by default. Instead, the value of a longer chain-of-thought prompt depends on the kind of task being asked of the model.
- Arithmetic tasks were part of the tested range.
- Common sense tasks were part of the tested range.
- Symbolic tasks were part of the tested range.
- Task sets named in the source included MultiArith, GSM8K, AQuA, SingleEq, SVAMP, and StrategyQA.
For prompt design, the practical implication is clear: longer reasoning is most relevant when the model is being asked to solve a problem that requires multiple steps. For easier tasks, extra reasoning may add less value.
Incorrect rationales still produced gains
One of the most striking findings in the source article is that incorrect rationales could still lead to positive results when they were long enough.
The researchers concluded that the length of the reasoning steps may have a greater influence than the factual correctness of the individual steps. That does not make incorrect reasoning desirable. It does, however, show that language model performance can be affected by structure and length in ways that are not limited to factual content alone.
This is a notable result for anyone studying prompt engineering. It suggests that a model may respond not only to what a rationale says, but also to how much reasoning-like structure it contains.
At the same time, the source does not say that factual accuracy stops mattering. The narrower finding is that, in these tests, longer rationales could improve results even when the reasoning inside them was not correct.
Model size changes the effect
The study also found that model size affects how much chain-of-thought length matters.
Larger models, including GPT-4, showed better performance with longer reasoning steps. But they also had a higher tolerance for steps being long or brief. Smaller models benefited the most from the strategy in the tests.
That means the same prompt engineering technique may not have the same effect across models. A smaller model may gain more from carefully extended reasoning steps, while a larger model may be less sensitive to the exact length.
There is also a limit. The source says chains that are too long can degrade performance again, especially for smaller models. Longer reasoning can help, but excessive length can become a problem.
What comes next
The research team plans to continue the work by analyzing neural activation patterns between long and short reasoning steps. The goal is to better understand how the length of reasoning steps affects language models.
For now, the study adds a useful constraint to how chain-of-thought prompts should be understood. More reasoning text can improve performance, especially on complex tasks, but the benefit depends on the task, the model, and how long the chain becomes.
The lesson is not simply to make every prompt longer. It is to treat reasoning length as a real prompt engineering variable, one that can improve or weaken language model performance depending on how it is used.