Research on large language models can describe a weakness accurately and still become outdated quickly. Stanford researchers Owen Queen and James Zou tested 50 widely cited claims against 20 models released between July 2024 and September 2026. They found that 44 percent of the claims had already expired, while another 34 percent had weakened significantly.
Research claims have short shelf lives
Of the claims with an estimated expiration date, the average lifespan was 15.7 months. The study’s timeline suggests how quickly the picture can shift: 16 percent of findings had expired after one year, rising to 39 percent after two years.
That does not mean every earlier result was wrong. A finding may have described a real behavior in the model tested at the time, while later models no longer show the same effect. In the study, only 16 percent of claims still held, and 6 percent had grown stronger.
The distinction matters when people treat a research result as a general statement about what language models can or cannot do. A claim based on one model at one point in time may not describe another model, even if both belong to the same broad generation.
Some weaknesses fade faster than others
Claims about reasoning failures were among the quickest to expire. Five of nine reasoning-related findings were considered expired, and none had held steady or become stronger. Examples included counting letters in “strawberry,” arithmetic in unfamiliar number systems, losing information in the middle of long contexts, and changing a correct answer after a user pushes back.
Queen and Zou suggest these errors may be easier for developers to address because there is a clear target: a correct answer or response. If training can directly reward the desired outcome, a weakness documented in one model may be less visible in later versions.
Other patterns lasted longer. Several of the findings that held or strengthened involved model preferences and tendencies: favoring their own answers as evaluators, rating accepted and rejected papers similarly, and being influenced by persuasive wording in medical abstracts. The study also found persistent tendencies toward agreement with users, alongside similar answers to open-ended questions and uneven random-number generation.
Better results in one area do not settle the wider question
The researchers examined fabricated scientific references as a case study. In a 2023 study by Walters and Wilder, GPT-3.5 fabricated 55 percent of its references. When Queen and Zou reused the prompts with newer models, GPT-6.1 Sol fabricated 0 percent and Claude Sonnet 5.5 fabricated 1 percent. Qwen3.7-Flash, also released in 2026, fabricated 69 percent in the same test.
The spread shows why a single headline result can be misleading. The newer models did not behave alike, and progress on one task does not establish that a broader capability is reliable.
For example, on two-hop questions—which require combining two known facts—Claude Sonnet 5.5 still failed in about 10 percent of cases even when it answered both individual questions correctly. The original paper reported 43 percent for ChatGPT. The “Reversal Curse” and contradictory answers to paired time-related questions had weakened, but remained detectable.
Social-bias results were also mixed. The study found that left-leaning political bias persisted, while gender stereotypes and implicit association biases weakened. Biases against speakers of African American English had shifted direction in the trend data.
Model identity and testing details matter
Even models released during the same period varied considerably. For models released between July and September 2026, the number of expired findings ranged from 11 for DeepSeek V4 Pro to 30 for GPT-6.1 Sol. The researchers found that release date, provider, and model size did not reliably predict the pattern.
Queen and Zou argue that empirical findings should be presented as observations about a particular model at a particular time, rather than timeless truths about LLMs. For readers, that makes details such as the model name, version, and test date part of the result itself.
They also recommend that papers publish their prompts and analysis code. Those materials make it possible to repeat tests as models change and see whether a reported behavior still appears. The study’s central lesson is practical: claims about LLM capabilities need regular retesting, because both failures and strengths can shift from one model to another.