Visual Scratchpads: Enabling Global Reasoning in Vision
Keywords: reasoning, scratchpad, vision, visual reasoning
TL;DR: Introducing visual scratchpad for reasoning in the visual domain
Abstract: Modern vision models have achieved remarkable success in benchmarks where a small subset of local features provides critical information about the target. There is now a growing interest in solving tasks that require more global reasoning, where local features offer no significant information. These tasks are reminiscent of the connectivity problems discussed by Minsky and Papert in 1969, which exposed the limitations of the perceptron model and contributed to the first AI winter. In this paper, we revisit such tasks by introducing four global visual benchmarks involving path findings and mazes. We show the following: (1) Although today's large vision models largely surpass the expressivity limitations of the early models, they still struggle with learning efficiency; we introduce the 'globality degree' to understand this; (2) we then demonstrate that the outcome changes and global reasoning becomes feasible with the introduction of a 'visual scratchpad'; similarly to the text scratchpads and chain-of-thoughts used in language models, visual scratchpads help break down global problems into simpler subproblems; (3) we further show that more specific 'inductive scratchpads', which take steps relying on less information, afford better out-of-distribution generalization and succeed for smaller model sizes.
Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 9573
Loading