A Comprehensive Benchmark of Human-Like Relational Reasoning for Text-to-Image Foundation Models

Colin Conwell, Tomer Ullman

Keywords: Text-to-image foundation models, relational reasoning, performance benchmarks, psychology

TL;DR: We propose a set of relational reasoning prompts sampled from the psychology literature as a benchmark of popular image-to-text algorithms, and evaluate these prompts with Dall-E2.

Abstract: Relations are basic building blocks of human cognition. Classic and recent work suggests that many relations are early developing, and quickly perceived. Machine models that aspire to human-level perception and reasoning should reflect the ability to recognize and reason generatively about relations. We report a systematic empirical examination of a recent text-guided image generation model (DALL-E 2), using a set of 15 basic physical and social relations studied or proposed in the literature, and judgements from human participants (N = 169). Overall, we find that only 22% of images matched basic relation prompts. Based on a quantitative examination of people's judgments, we suggest that current image generation models do not yet have a grasp of even basic relations involving simple objects and agents. We examine reasons for model successes and failures, and suggest possible improvements based on computations observed in biological intelligence.