How Far Are We from Intelligent Visual Deductive Reasoning?

Yizhe Zhang, Richard He Bai, Ruixiang ZHANG, Jiatao Gu, Shuangfei Zhai, Joshua M. Susskind, Navdeep Jaitly

Research Area: LMs on diverse modalities and novel applications

Keywords: Vision Language Model, Reasoning

TL;DR: VLMs struggles with visual deductive reasoning like solving Raven’s Progressive Matrices (RPMs) due to difficulty perceiving and comprehending multiple abstract patterns solely from visual cues.

Abstract:

Vision-Language Models (VLMs) have recently demonstrated incredible strides on diverse vision language tasks.
We dig into vision-based deductive reasoning, a more sophisticated but less explored realm, and find previously unexposed blindspots in the current SOTA VLMs.
Specifically, we leverage Raven’s Progressive Matrices (RPMs), to assess VLMs' abilities to perform multi-hop relational and deductive reasoning relying solely on visual clues.
We perform comprehensive evaluations of several popular VLMs employing standard strategies such as in-context learning, self-consistency, and Chain-of-thoughts (CoT) on three diverse datasets, including the Mensa IQ test, IntelligenceTest, and RAVEN.
The results reveal that despite the impressive capabilities of LLMs in text-based reasoning, we are still far from achieving comparable proficiency in visual deductive reasoning.
We found that certain standard strategies that are effective when applied to LLMs do not seamlessly translate to the challenges presented by visual reasoning tasks.
A detailed analysis reveals that VLMs struggle to solve these tasks mainly because they are unable to perceive and comprehend multiple, confounding abstract patterns in RPM examples.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the COLM Code of Ethics on https://colmweb.org/CoE.html

Author Guide: I certify that this submission complies with the submission instructions as described on https://colmweb.org/AuthorGuide.html

Submission Number: 146