Seeing Isn’t Always Understanding: Testing the Limits of Visual AI

Author: Denis Avetisyan

A new benchmark reveals that even advanced vision-language models struggle with complex visual reasoning despite excelling at simpler image-text tasks.

VisRes Bench diagnoses deficiencies in perceptual grounding and compositional reasoning within current vision-language models.

Despite impressive gains in visual question answering and image captioning, the extent to which vision-language models truly reason visually-rather than relying on linguistic shortcuts-remains an open question. To address this, we introduce VisRes Bench: On Evaluating the Visual Reasoning Capabilities of VLMs, a new benchmark designed to isolate and assess perceptual and relational reasoning in naturalistic settings. Our analysis of over 19,000 images reveals that state-of-the-art models struggle with even subtle perceptual perturbations and exhibit limited compositional reasoning, suggesting a reliance on surface-level pattern recognition. Can we develop multimodal models that move beyond correlation and achieve genuine abstract visual understanding?

Beyond Recognition: Probing the Depths of Visual Intelligence

Despite advancements in image recognition and object detection, current vision-language models frequently falter when confronted with tasks demanding complex visual reasoning. While these models often achieve impressive scores on benchmarks centered around simple classification or retrieval, their performance declines substantially when required to interpret relationships between objects, understand spatial arrangements, or draw inferences from visual scenes. This discrepancy suggests that current models excel at recognizing what is present in an image, but struggle with how those elements interact and what that interaction signifies. The ability to perform these higher-level reasoning tasks remains a significant challenge, highlighting a gap between superficial pattern recognition and genuine visual intelligence.

The VisRes benchmark represents a focused effort to move beyond superficial evaluations of vision-language models and probe their genuine visual intelligence. Unlike existing datasets often dominated by simple recognition tasks, VisRes is specifically designed around three core challenges: perceiving details, understanding relationships between objects, and composing information from multiple visual cues. Initial results reveal a significant performance gap; current models achieve only approximately 50

The development of VisRes underscores a critical limitation in current approaches to evaluating artificial intelligence: a reliance on benchmarks that primarily assess recognition, rather than genuine comprehension. Existing datasets often measure a model’s ability to identify objects within an image, but fall short of testing its capacity to reason about those objects – their relationships, their potential transformations, or their roles within a broader context. This need for diagnostic tools extends beyond simply achieving high accuracy scores; it demands benchmarks capable of pinpointing specific weaknesses in visual intelligence, such as an inability to grasp spatial relationships or to infer missing information. Consequently, VisRes offers a pathway toward developing AI systems that don’t merely see what is present, but truly understand the visual world, mirroring the complex cognitive processes inherent in human vision.

Deconstructing Visual Reasoning: A Hierarchical Approach

The VisRes benchmark structures visual reasoning tasks into a hierarchical framework beginning with Level 1, which centers on single-attribute reasoning. This initial level serves as a foundational component, prioritizing the establishment of visual grounding – the ability of a model to accurately identify and represent individual visual properties like color, shape, or size. Tasks at this level require the model to demonstrate competence in recognizing these singular attributes in isolation, providing a baseline assessment of its visual perception capabilities before introducing more complex reasoning demands. Successful performance at Level 1 is a prerequisite for tackling the subsequent levels, as it validates the model’s ability to correctly perceive and interpret basic visual features.

Level 2 visual reasoning tasks within the VisRes framework specifically assess a model’s capacity for perceptual completion and its ability to interpret scenes involving occlusion. These tasks require low-level visual processing to reconstruct partially visible objects or infer the presence of obscured features. Performance on these tasks varies considerably, ranging from 50

Level 3 visual reasoning tasks, as categorized by VisRes, necessitate the integration of multiple visual attributes to solve a given problem. This compositional reasoning mirrors the complexity of real-world visual perception, where objects are rarely defined by a single characteristic. Current model performance on Level 3 tasks demonstrates a significant challenge in this area, with accuracy rates ranging from 30-60

Crafting Rigorous Challenges: The Art of Distractor Design

The efficacy of evaluating reasoning abilities relies heavily on the quality of distractors used in challenge creation. While random sampling of options presents a basic method for generating these distractors, it frequently proves insufficient for discerning genuine reasoning capability. Random distractors often lack the nuanced relationship to the correct answer needed to force a model to engage in meaningful problem-solving; instead, they may be easily dismissed through superficial feature matching or statistical likelihood. Consequently, relying solely on random sampling can lead to inflated performance metrics that do not accurately reflect a model’s underlying reasoning skills, obscuring its true limitations and hindering effective development.

Utilizing DINOv2 similarity for distractor generation involves identifying images with high feature vector proximity to the target image, as determined by the DINOv2 visual representation model. This contrasts with random distractor selection by prioritizing perceptual similarity, thereby forcing the evaluation model to engage in more nuanced discrimination. The resulting distractors are not simply different images, but variations that require finer-grained analysis of visual features to correctly identify the target, effectively isolating the model’s ability to perform perceptual completion rather than relying on easily discernible differences. This approach increases the difficulty of the task and provides a more rigorous assessment of visual reasoning capabilities.

VisRes employs strategically designed distractors to move beyond assessments of simple feature matching and more effectively evaluate a model’s ability to perform perceptual completion. Traditional methods often utilize random distractors, allowing models to succeed by identifying correlations without truly understanding the visual scene. By introducing distractors that share visual characteristics with the target but are incomplete or subtly different, VisRes forces models to actively infer missing information and demonstrate a deeper comprehension of object structure. This approach minimizes the possibility of superficial success based on low-level feature detection and provides a more reliable measure of a model’s robust visual reasoning capabilities.