Decoupling the components of geometric understanding

Eliza Kosoy, Annya Dahmani, Andrew Kyle Lampinen, Iulia Maria Comsa, Soojin Jeong, Ishita Dasgupta, Kelsey R Allen

Keywords: geometry, cognitive science, vlms

TL;DR: A new methodology for decoupling the components of geometric understanding in foundation models

Abstract: Understanding geometry relies heavily on vision. In this work, we evaluate whether state-of-the-art vision language models (VLMs) can understand simple geometric concepts.We use a paradigm from cognitive science that isolates visual understanding of simple geometry from the many other capabilities it is often conflated with (reasoning, world knowledge, etc.). We compare model performance with human adults from the USA, as well as with prior research on human adults without formal education from an Amazonian indigenous group. We find that VLMs consistently underperform both groups of human adults, although they succeed with some concepts more than others. We also find that VLM geometric understanding is more brittle than human understanding, and is not robust to e.g. mental rotation. This work highlights interesting differences in the origin of geometric understanding in humans and machines -- e.g. from printed materials used in formal education vs. interactions with the physical world or a combination of the two -- and the first steps toward understanding these differences.

Submission Number: 67