One-sentence Summary

Researchers at Moonshot AI introduce PerceptionBench, a benchmark that diagnoses ten atomic visual perception capabilities in multimodal large language models via a bottom-up error taxonomy derived from 42 existing benchmarks, revealing that no model exceeds $60\%$60% accuracy and that perception-related hallucination remains the weakest capability, thus providing a capability-level standard for measuring MLLM visual perception boundaries.

Key Contributions

• PerceptionBench isolates ten atomic visual perception capabilities derived bottom-up from failure attribution across 42 existing MLLM benchmarks, with each of its 3,000 verified questions targeting a single capability and minimizing reasoning or knowledge demands.
• Evaluation of sixteen frontier MLLMs shows that atomic perception remains largely unsolved: no model reaches 60 percent overall accuracy, and perception-related hallucination is the weakest capability on average.
• Models with nearly identical overall scores exhibit sharply divergent capability profiles, so aggregate accuracy conceals which perceptual capabilities a model has actually acquired.

Introduction

Accurate visual perception is foundational for multimodal large language models (MLLMs) to perform downstream reasoning and interaction, yet it comprises distinct atomic capabilities such as attribute recognition, counting, localization, and text reading. Current benchmarks either conflate perception with reasoning and external knowledge, making error attribution impossible, or they target narrow application-specific perceptual slices defined by designer priors rather than by the capabilities models actually lack. The authors introduce PerceptionBench, a benchmark of 3,000 verified questions organized around ten atomic perceptual capabilities derived bottom-up from the attributed failures of frontier MLLMs on 42 existing benchmarks. This design directly measures empirically observed weaknesses, isolating perception from reasoning or knowledge and providing fine-grained diagnostic granularity that prior task-oriented or perception-centric benchmarks cannot offer.

Dataset

The authors construct PerceptionBench, a dataset designed to evaluate atomic visual perception capabilities in multimodal large language models (MLLMs). It is built through a bottom-up, failure-driven pipeline rather than relying on designer priors.

Dataset composition and sources

• The released benchmark contains 3,000 verified question-answer-image samples.
• 1,800 questions (60%) are atomic sub-questions decomposed from failures observed on 42 existing benchmarks.
• 1,200 questions (40%) are newly authored on supplementary images collected from public web sources and internally curated data.
• The 42 source benchmarks span document understanding, OCR, chart understanding, GUI interaction, visual grounding, scientific diagrams, remote sensing, and natural images.

Key details for each subset

• Decomposed subset (1,800 samples): Derived from failures attributed to a perceptual error. For each failure, annotators receive the original question, the error analysis, and the assigned perception error type, then break it into atomic sub-questions that isolate the specific perceptual capability responsible for the failure.
• Newly authored subset (1,200 samples): Created to increase visual diversity and improve coverage of underrepresented capabilities. Annotators write perception questions targeting the same ten atomic capabilities on new images.
• Filtering rules:
  • All images (both reused and newly collected) are checked against large-scale pre-training corpora such as LAION and Common Crawl using ISC descriptor cosine similarity. Images exceeding a 0.95 similarity threshold are discarded as near-duplicates or leaked.
  • Every constructed sample is screened by an ensemble of four frontier MLLMs. Samples solved correctly by all ensemble members are removed. Survivors are stratified into three difficulty tiers based on how many models answer correctly.
  • Multi-stage verification (automated capability alignment, visual grounding checks, and independent human validation) removes or revises samples that introduce unrelated reasoning, ambiguous visual evidence, or incorrect annotations.

How the paper uses the data

• PerceptionBench serves as an evaluation-only benchmark. It is not used for training.
• The 3,000 released samples are subsampled from the constructed portion of an in-house pool of more than 17,000 verified samples, with capability-level balancing and difficulty stratification.
• The ten evaluated atomic perceptual capabilities are: visual localization, visual attribute recognition, visual counting, visual relation understanding, depth and 3D perception, OCR, visual comparison, fine-grained recognition, context integration, and perception-related hallucination.
• The remaining error classes from the induced taxonomy (reasoning, knowledge, premise, other) are used only during failure attribution and are excluded from the released benchmark.

Processing details

• Failure-driven capability discovery: Frontier MLLMs are run on 42 benchmarks; incorrect samples form a candidate failure pool. A stronger model performs perceptual failure attribution by analyzing the full inference trajectory, pinpointing the earliest erroneous step, and assigning an open-vocabulary error label. Labels are clustered and merged by an LLM into a unified taxonomy. Failures are attributed to perception only when a visual fact is misread; otherwise they fall into reasoning or knowledge classes.
• Difficulty-aware selection: An ensemble of four frontier MLLMs evaluates all candidates. Samples solved correctly by all ensemble models are discarded as saturated. Retained samples are stratified by pass rate and subsampled across source benchmarks and difficulty.
• Perception decomposition: Annotators decompose attributed perception failures into atomic sub-questions, each inheriting the fine-grained capability label from attribution.
• Metadata construction: Each sample carries a capability label grounded in the failure taxonomy, a difficulty tier based on ensemble consensus, and a construction source tag (decomposed or newly authored).
• Cropping strategy: No explicit cropping strategy is mentioned; images are used as provided by source benchmarks or newly collected sources, with deduplication against pre-training corpora.

Method

The authors design PerceptionBench to identify and evaluate the atomic perceptual capabilities required by Multimodal Large Language Models. Rather than relying on designer priors or task-specific slices, the framework follows a three-stage bottom-up pipeline grounded in empirically observed model failures.

The first stage is failure-driven capability discovery. The authors collect incorrect predictions from frontier models across 42 existing benchmarks spanning diverse visual domains. To isolate genuine perceptual limitations from reasoning or knowledge errors, they introduce a perceptual failure attribution procedure. A stronger frontier model analyzes the complete prediction context, including the question, image, reference answer, and reasoning trajectory, to pinpoint the earliest erroneous step and assign an open-vocabulary error label. These labels are then clustered into a unified taxonomy. The authors observe that existing benchmarks only probe narrow and complementary slices of the failure space, as failures in each benchmark concentrate on just one or a few error types.

As shown in the figure below:

To address this fragmentation, the authors retain only the perception error class from the taxonomy, which yields ten atomic perceptual capabilities. These include visual localization, visual attribute recognition, visual counting, visual relation understanding, depth and 3D perception, OCR, visual comparison, fine-grained recognition, context integration, and perception-related hallucination.

The second stage is benchmark selection and construction. To ensure the benchmark remains challenging and diagnostic, the authors employ a difficulty-aware selection process. They evaluate all candidate samples using an ensemble of four frontier models. Samples solved correctly by every model in the ensemble are discarded as saturated. The remaining samples are stratified into three difficulty tiers based on the number of models that answer them correctly, and then subsampled to counteract skewed source distributions. To further smooth the difficulty distribution, they perform perception decomposition. Annotators break down complex perception questions into finer atomic sub-questions, each isolating a specific perceptual capability responsible for the failure. To increase visual diversity, they supplement the benchmark with additional images from public web sources and internally curated data. To mitigate data contamination, they compute the cosine similarity of the Image Similarity Challenge descriptor for each image against large-scale pre-training corpora, discarding any image with a similarity exceeding 0.95. Finally, a screening step mirrors the discovery stage. The ensemble of frontier models evaluates the newly constructed samples, removing those solved correctly by all models to maintain a challenging difficulty profile.

The third stage involves multi-stage verification, combining automated capability-alignment and visual-grounding checks with independent human validation. This rigorous pipeline results in a final released benchmark of 3,000 verified questions, subsampled from an in-house pool of over 17,000 samples, ensuring comprehensive coverage of the ten atomic perceptual capabilities across balanced difficulty tiers.

Experiment

The evaluation setup uses PerceptionBench, a benchmark of 3,000 atomic perception-only questions constructed from failure-driven capability discovery and difficulty-aware selection, to assess 16 frontier multimodal models with a unified judge. The main results validate that atomic visual perception remains a significant bottleneck, with overall accuracy spanning 32.5% to 59.7% and perception-related hallucination proving the weakest capability across all models. Qualitative analysis confirms that failures are predominantly perceptual rather than cognitive, while reliability experiments show stable aggregate scores but inconsistent per-sample behavior, indicating that models have not yet robustly acquired the underlying perceptual skills.

Atomic visual perception remains a significant bottleneck for multimodal large language models, with overall accuracy on PerceptionBench spanning 32.5% to 59.7% and no model approaching ceiling performance. Perceptual capabilities are unevenly developed across models, with perception-related hallucination consistently lagging behind other skills such as visual relation, OCR, and localization. Strikingly, overall performance and hallucination robustness appear decoupled, as the overall leader records one of the lowest hallucination scores while a mid-ranked model achieves the best hallucination result. Overall accuracy ranges from 32.5% to 59.7%, confirming that atomic perception remains unsolved and the benchmark retains strong discriminative power. Perception-related hallucination is the weakest capability on average (36.7%), substantially trailing visual relation (53.2%), OCR (52.4%), and localization (52.0%). The overall leader, GPT-5.6-Sol, achieves 76.7% on localization but only 26.9% on hallucination, one of the lowest hallucination scores on the leaderboard. GPT-5.5 and Gemini-3.1-Pro differ by less than one point overall yet diverge sharply on specific capabilities, with GPT-5.5 leading localization by 13 points and Gemini-3.1-Pro leading OCR and fine-grained recognition by 8 points each. The best hallucination score (50.6%) belongs to Gemini-3.5-Flash, a model ranked mid-pack overall at 52.0%, while the overall leader GPT-5.6-Sol scores only 26.9% on hallucination. The open-source model Kimi K3 (58.5%) surpasses all remaining proprietary systems, including Gemini-3.1-Pro (56.2%) and GPT-5.5 (55.8%).

Multiple independent runs of frontier MLLMs yield small standard deviations in mean accuracy, confirming stable aggregate benchmark scores. However, large gaps between pass@4 and pass^4 reveal substantial per-sample inconsistency, indicating that models often succeed intermittently rather than through robust visual perception. All evaluated models show low standard deviation across four runs, with GPT-5.6-Sol exhibiting the smallest variation (std 0.17) and the highest mean accuracy. Despite stable mean scores, the gap between pass@4 and pass^4 is large for every model, exceeding 25 percentage points, which signals unreliable per-sample perceptual behavior.

The evaluation on PerceptionBench reveals that atomic visual perception remains an unsolved challenge for multimodal large language models, with overall accuracy ranging from 32.5% to 59.7% and no model approaching ceiling performance. Perception-related hallucination consistently lags behind other capabilities such as visual relation, OCR, and localization, and overall performance is decoupled from hallucination robustness, as the top-ranked model exhibits one of the worst hallucination scores while a mid-ranked model achieves the best. Multiple runs confirm stable aggregate scores but large gaps between pass@4 and pass^4 expose substantial per-sample inconsistency, indicating that models succeed intermittently rather than through reliable visual perception.