AI systems can now take in far more text at once than they could a few years ago. But Nvidia researchers argue that a larger context window is not always the best way to help a model answer questions: periodic summaries and targeted lookups performed about as well in many tests, while huge windows can be expensive and excessive for the task.
What a context window does
A language model’s context window is its working memory: the material it can hold in view while generating a reply. That might include the user’s request and the text the model is being asked to work with. A larger window lets more material fit at once.
The capacity has grown rapidly. GPT-4 launched in March 2023 with 8,192 tokens. Today, GPT 5.6 Sol handles around one million tokens, about 120 times as much and enough for Tolstoy’s “War and Peace.” Those figures illustrate how much more text a model can now process in a single context.
That capacity can sound like an uncomplicated advantage. If a model can see more of a document, it may seem more likely to answer with all the relevant information available. But the Nvidia researchers’ argument is that keeping everything in view can be unnecessary and expensive.
More text is not always more useful
A very large context window addresses one problem: fitting a large amount of material into the model’s working memory. It does not follow that every task benefits from that material being present at the same time. If only a small part of a source is relevant to a question, bringing in the whole source may be more capacity than the task needs.
This is the distinction at the center of the researchers’ case. A model may need access to a long source, but access does not necessarily mean holding the entire source in view for every reply. Summaries can preserve a condensed account of what has already been read, while targeted lookups can retrieve particular details when they matter.
In many tests, those approaches worked about as well as using a huge context window. The finding challenges the idea that simply expanding a model’s working memory is always the most useful route to better results. For some tasks, a process that identifies and revisits relevant information may be enough.
Summaries and lookups offer another approach
Periodic summaries give a model a way to carry forward the main points of material without repeatedly keeping all of it in view. Targeted lookups serve a different purpose: they let the model seek out a detail when a specific question calls for it. Together, the methods offer a way to handle large bodies of text in stages.
That approach changes how to think about long-document work. Instead of asking only how much text the model can accept at once, it also matters whether the task can be handled by summarizing and retrieving pieces as needed. The source does not say these methods outperform large context windows in every case; it reports that they worked about as well in many tests.
The distinction matters because context size comes with a cost, according to the researchers. If a huge window is often overkill, then choosing a more focused way to supply information could avoid paying for capacity a task does not need. The source does not provide a specific cost figure, so the claim is about the general trade-off rather than a quantified saving.
A question of fit, not just scale
The rapid growth from GPT-4’s 8,192 tokens to GPT 5.6 Sol’s around one million tokens makes context-window size an easy measure of progress. Nvidia researchers’ argument points to a less visible question: how much of that capacity is useful for a particular task?
Very large context windows can still make it possible to process extensive material at once. But the researchers’ findings suggest that bigger is not automatically better. When summaries and targeted lookups produce similar results in many tests, the choice becomes one of fit: whether the task benefits from seeing a whole source together or can be handled by carrying forward summaries and retrieving details when needed.