Jigsaw-Puzzles: From Seeing to Understanding to Reasoning in Vision-Language Models

ACL ARR 2025 May Submission2117 Authors

Abstract:

Spatial reasoning is a core component of human cognition, enabling individuals to perceive, comprehend, and interact with the physical world. It relies on a nuanced understanding of spatial structures and inter-object relationships, serving as the foundation for complex reasoning and decision-making. To investigate whether current vision-language models (VLMs) exhibit similar capability, we introduce Jigsaw-Puzzles, a novel benchmark consisting of 1,100 carefully curated real-world images with high spatial complexity.
Based on this dataset, we design five tasks to rigorously evaluate VLMs’ spatial perception, structural understanding, and reasoning capabilities, while deliberately minimizing reliance on domain-specific knowledge to better isolate and assess the general spatial reasoning capability.
We conduct a comprehensive evaluation across 24 state-of-the-art VLMs. The results show that even the strongest model, Gemini-2.5-Pro, achieves only 77.14% overall accuracy and performs particularly poorly on the Order Generation task, with only 30.00% accuracy, far below the 90%+ performance achieved by human participants. This persistent gap underscores the need for continued progress, positioning Jigsaw-Puzzles as a challenging and diagnostic benchmark for advancing spatial reasoning research in VLMs.

Paper Type: Long

Research Area: Resources and Evaluation

Research Area Keywords: benchmarking, automatic evaluation of datasets, evaluation, reproducibility, statistical testing for evaluation

Contribution Types: Data resources, Data analysis

Languages Studied: English, Chinese

Keywords: benchmarking, automatic evaluation of datasets, evaluation, reproducibility, statistical testing for evaluation

Submission Number: 2117