A new benchmark from Moonshot AI puts a sharper lens on a problem that can be hidden inside broader AI evaluations: multimodal models often fail before they ever get to reasoning. PerceptionBench tests whether these systems can accurately read what is in an image, without needing outside knowledge or complex logic.
The result is a direct challenge to how visual AI performance is often discussed. If a model gives a wrong answer to an image-based task, the failure may not be in its chain of thought. It may have misread the scene from the start.
What PerceptionBench Measures
PerceptionBench was introduced by the team behind the Chinese AI assistant Kimi. Its goal is to isolate visual perception in multimodal language models, separating it from reasoning and knowledge-based tasks.
Instead of treating image understanding as one broad capability, the benchmark breaks it into ten atomic sub-skills. Each question is designed so it can be answered from the image alone. The model should not need general knowledge, inference from outside the picture, or multi-step logic to respond correctly.
The ten skill domains are Visual Relation, Counting, Attributes, Depth & 3D, Localization, Comparison, Fine-grained Recognition, Context Integration, OCR, and Hallucination.
That design matters because standard benchmarks can blur the source of an error. A wrong answer may look like a reasoning failure, but the model may have already missed a color, position, object count, spatial relation, or piece of visible text.
Built From Real Model Errors
The authors did not begin by inventing a theoretical list of visual abilities. They analyzed 42 open-source benchmarks and found that each one captured only a narrow part of the error landscape. Their error profiles had little overlap, meaning no single test, or small group of tests, was enough to represent visual perception as a whole.
PerceptionBench was then built by tracing model mistakes back to the earliest failed visual step. This creates a taxonomy based on observed failures rather than an abstract checklist.
Moonshot AI started with an internal pool of over 17,000 verified questions and is publishing 3,000 tasks. Sixty percent of those tasks come from attributed model errors, while 40 percent were reformulated using augmented images.
Many of the tasks sound simple. A model may need to identify where a symbol sits on a clock face, count flowers inside a red box, or distinguish between pencil cups with similar but different color and design features. The point is not that the tasks are difficult for people. The point is that they reveal whether a model can reliably perceive basic details before doing anything more advanced.
No Frontier Model Reaches 60 Percent
The benchmark results show that visual perception remains a major weakness even for leading multimodal systems. Among the 16 frontier models tested, GPT-5.6 Sol had the highest overall accuracy at 59.7 percent.
Kimi K3 followed at 58.5 percent. Claude Fable 5 reached 57.2 percent, Gemini 3.1 Pro reached 56.2 percent, and GPT-5.5 reached 55.8 percent. Open-source models were lower, including Qwen3.5-397B-A17B at 47.5 percent and GLM-4.6V at 32.5 percent.
The overall ranking is only part of the story. The category-level results show that models with similar aggregate scores can behave very differently depending on the visual skill being tested.
Hallucination was the weakest skill on average. In this sub-test, the correct answer can be simply zero, and the model must avoid inventing objects that are not present. GPT-5.6 Sol, despite leading overall, scored only 26.9 percent in hallucination. Gemini 3.5 Flash, a weaker model overall, ranked among the best in that category with 50.6 percent.
This split suggests that a model's general score can hide uneven perception. A system may be strong in one visual domain and poor in another, which matters for any application that depends on consistent image interpretation.
Why Reasoning May Not Be The First Problem
The authors argue that many errors labeled as reasoning problems are actually perception failures. In a multimodal task, the model first has to read the image correctly. If that first step is wrong, later reasoning can appear flawed even if the model is following a plausible path from a faulty visual input.
PerceptionBench makes this easier to diagnose by turning larger questions into perception-only sub-questions. That lets researchers see which specific visual ability is breaking down, rather than only seeing that the final answer is wrong.
This distinction is important for evaluation. A model that cannot count visible items, locate a detail, read text in an image, or avoid inventing nonexistent objects may still produce fluent explanations. The explanation can make the failure look sophisticated, while the underlying problem is much simpler.
The dataset and evaluation code are available on GitHub at MoonshotAI/PerceptionBench.
A Pattern Across Visual AI Tests
PerceptionBench fits into a broader pattern described by the same research team and related work. The team previously released WorldVQA, a benchmark that separates object recognition from reasoning. In that test, the best model, Gemini 3 Pro, reached 47.4 percent, and all models systematically overestimated their own confidence.
A separate study by Chinese institutions, with Moonshot AI's involvement, used the BabyVision benchmark to test basic visual tasks tied to early childhood development. These included tracing lines and counting hidden blocks. Gemini 3 Pro scored 49.7 percent, while humans reached 94.1 percent.
The researchers linked that gap to a verbalization bottleneck, where visual information is translated into language and loses fidelity. PerceptionBench points in the same direction: strong language ability does not automatically mean reliable perception.
For developers and evaluators, the lesson is straightforward. Multimodal AI models should not be judged only by whether they can produce convincing final answers. They also need tests that ask whether they actually saw the image correctly in the first place.