Google has made long context a central selling point for Gemini 1.5 Pro and Gemini 1.5 Flash. New research described by TechCrunch raises a harder question: can these models actually use very large inputs well enough to justify the claims around them?
The answer, based on two separate studies, is mixed at best. The models can technically accept a large amount of material, but the tests found repeated failures when they had to answer questions that required more than simple retrieval.
What Google has been selling
A context window is the information a model considers before it generates an answer. That input can be a short question, a long document, a script, a show, an audio clip or another large dataset.
The newest versions of Gemini can take in upward of 2 million tokens as context. The source describes that as roughly 1.4 million words, two hours of video or 22 hours of audio, and says it is the largest context of any commercially available model.
Google has presented that capacity as a major leap. In one briefing, the company showed Gemini 1.5 Pro searching the transcript of the Apollo 11 moon landing telecast, around 402 pages, for jokes and then finding a scene that resembled a pencil sketch.
Oriol Vinyals, VP of research at Google DeepMind, led the briefing and described the model as “magical.” He also said, “[1.5 Pro] performs these sorts of reasoning tasks across every single page, every single word.”
The studies covered in the source article suggest that the practical performance is less settled than the demos implied.
The book tests exposed weak comprehension
One study involved Marzena Karpinska, a postdoc at UMass Amherst, along with researchers from the Allen Institute for AI and Princeton. The researchers tested how models handled fiction books written in English.
They used recent works so the models could not rely on prior knowledge. The questions were framed as true/false statements about specific details and plot points, forcing the models to process the books rather than simply recall famous material.
In one test, the relevant book was around 260,000 words, or about 520 pages. Gemini 1.5 Pro answered correctly 46.7% of the time. Gemini 1.5 Flash answered correctly 20% of the time.
Across one series of document-based tests, the models gave the correct answer only 40%-50% of the time. That is a serious limitation for anyone hoping to use long context AI for reliable document analysis, research review or complex question answering.
Karpinska told TechCrunch that models such as Gemini 1.5 Pro can technically process long contexts, but that her team had seen many cases where the models did not actually understand the content. She also said the models had more trouble with claims that required considering large portions of a book, or the entire book, than with claims that could be handled by sentence-level evidence.
Video-style reasoning was also difficult
A second study, co-authored by researchers at UC Santa Barbara, looked at Gemini 1.5 Flash and its ability to reason over videos. The test used images paired with questions about objects in those images.
The researchers then inserted distractor images before and after a selected image, creating slideshow-like footage. That setup tested whether the model could identify the relevant frame and answer a question about what appeared in it.
Flash did not perform strongly. In a test involving six handwritten digits in a slideshow of 25 images, it got around 50% of the transcriptions right. With eight digits, accuracy dropped to around 30%.
Michael Saxon, a PhD student at UC Santa Barbara and one of the study's co-authors, told TechCrunch that real question-answering tasks over images appeared particularly hard for all the models tested. He pointed to a small reasoning step, recognizing that a number is in a frame and reading it, as something that might be breaking the model.
Why the benchmark debate matters
The source article is careful about limits. Neither study had been peer-reviewed. Neither tested the releases of Gemini 1.5 Pro and Gemini 1.5 Flash with 2-million-token contexts; both tested the 1-million-token context releases. Gemini 1.5 Flash is also positioned by Google as a low-cost alternative, not as equal to Pro.
Even with those caveats, the findings matter because Google has given context window size a prominent role in how it markets Gemini. The source also notes that none of the tested models, including OpenAI's GPT-4o and Anthropic's Claude 3.5 Sonnet, performed well.
Saxon drew a distinction between the technical claim that a model can accept a certain number of tokens and the practical question of what useful work it can do with them. That distinction is central for businesses evaluating generative AI tools.
The broader market is already questioning AI promises. In a pair of recent surveys from Boston Consulting Group, about half of C-suite respondents said they do not expect generative AI to bring substantial productivity gains and are worried about mistakes and data compromises from generative AI-powered tools. PitchBook reported that early-stage generative AI dealmaking declined 76% from its Q3 2023 peak.
Long context is not the same as understanding
The clearest lesson from the source is that larger input capacity should not be confused with dependable reasoning. A model may fit a book, transcript, slideshow or audio-length input into its context window while still failing to connect details across that material.
That gap is especially important for high-volume AI data analysis. Users may want a model to compare many pages, inspect long transcripts, search visual content or verify claims that depend on material spread across a large source. The studies suggest that those tasks need careful testing before they are trusted.
Karpinska and Saxon both pointed toward better benchmarks and more third-party critique as answers to inflated claims. The source notes that one common long-context test, needle in the haystack, measures whether a model can retrieve specific information such as names and numbers from a dataset, not whether it can answer more complex questions about that information.
Google did not respond to a request for comment. In an update, the source article said Google PR sent links to studies suggesting Gemini's long-context performance is stronger than implied, including Extended Multi-Doc QA, Video MME, longer queries subset on LMSYS and Ruler.
For now, the practical takeaway is straightforward: Gemini's context window is large, but long-context AI still needs better evidence before users treat it as reliable understanding at scale.