A major revision to Artificial Analysis’s Intelligence Index has changed how some AI models rank. In version 5, Kimi K3.1 Pro moved from fourth place to ninth, behind eight closed models, while the measured gap between open models and the leaderboard’s top widened from 3 points to 12.
The shift highlights a tension in AI evaluation: tests should be difficult to game, but their results should also be open to scrutiny. The revised index changes that balance by relying more heavily on private questions.
Private tests reshape the rankings
For the first time, 60 percent of the index relies on private test sets. The stated aim is to prevent “benchmaxxing,” a practice in which models are trained specifically on known test questions. If a test is public, developers may be able to optimize for its questions, making a model’s score less representative of its broader abilities.
That change appears to have affected the leaderboard. Kimi K3.1 Pro, which originally launched in fourth place, fell to ninth in version 5. The new ranking places it behind eight closed models. Artificial Analysis now measures the distance between open models and the top of the leaderboard as 12 points, compared with 3 points previously.
Grok 4.7 also lost 12 points. These changes suggest that the revised index evaluates performance differently from its earlier version. Kimi K3.1 Pro’s fall may indicate that the former index made some models appear stronger than they were, though the rankings alone do not establish why each model’s score changed.
Benchmark performance may reflect more than test exposure
AI researcher Nathan Lambert argues that benchmaxxing is only part of the explanation. He points to reinforcement learning environments where tasks resemble benchmarks. Training in those environments can produce gains that show up on tests, even when a model has not simply been trained on the test questions themselves.
That distinction matters because benchmark scores can be influenced by the training process as well as by exposure to a particular test. Lambert’s point is that maintaining strong performance across multiple domains requires sophisticated reinforcement learning infrastructure, which not every lab has. The gap in results may therefore reflect differences in training capacity as well as differences in how models handle particular questions.
This is a broader challenge for anyone comparing AI systems. A leaderboard condenses performance across tests into rankings and scores, but the results do not by themselves explain what training methods produced them or how well a model performs outside the index. A revised ranking is useful evidence about the evaluation, not a complete account of every model’s capabilities.
Secrecy brings a transparency trade-off
Private test sets can make it harder to train directly for known questions. But keeping questions private also limits outsiders’ ability to inspect the prompts and identify mistakes. Evaluators may have more control over test exposure, while readers have less ability to independently assess whether a question is clear and fair.
The AA-Omniscience knowledge test illustrates that concern. Artificial Analysis revised ambiguous and flawed questions in that benchmark for the new index. The revisions show that test quality itself can affect evaluation: a question that is unclear or flawed may measure something other than the ability the benchmark is meant to assess.
There is no simple resolution in the source’s account. Public tests make it easier for outside observers to review questions, but also expose them to developers who may optimize for them. Private tests reduce that exposure, while making independent review harder. The index’s redesign makes this tension more visible as its private test sets take on a larger role.
What version 5 changes for readers
For readers, the revised index should be understood as a change in measurement as well as a change in ranking. Kimi K3.1 Pro’s move and Grok 4.7’s score loss show that positions can shift when the evaluation changes. Comparing version 5 directly with the earlier index therefore requires attention to the test design, not just the ordering of models.
The new approach aims to make the index less vulnerable to benchmark-focused training. At the same time, the revisions to AA-Omniscience underline why transparency and question quality matter. The rankings offer a view of model performance under the revised tests; the limits of private evaluation remain part of how those results should be read.