Interactive Visual Reasoning under Uncertainty

Manjie Xu, Guangyuan Jiang, Wei Liang, Chi Zhang, Yixin Zhu

Keywords: visual reasoning, uncertainty, few-shot, interactive

TL;DR: We introduce the IVRE environment, simulating complex uncertainty scenarios to assess artificial agents' interactive reasoning skills, suggesting a need for advanced research towards achieving human-like intelligence in uncertainty resolution.

Abstract: One of the fundamental cognitive abilities of humans is to quickly resolve uncertainty by generating hypotheses and testing them via active trials. Encountering a novel phenomenon accompanied by ambiguous cause-effect relationships, humans make hypotheses against data, conduct inferences from observation, test their theory via experimentation, and correct the proposition if inconsistency arises. These iterative processes persist until the underlying mechanism becomes clear. In this work, we devise the **IVRE** (pronounced as *"ivory"*) environment for evaluating artificial agents' reasoning ability under uncertainty. **IVRE** is an interactive environment featuring rich scenarios centered around *Blicket* detection. Agents in **IVRE** are placed into environments with various ambiguous action-effect pairs and asked to determine each object's role. They are encouraged to propose effective and efficient experiments to validate their hypotheses based on observations and actively gather new information. The game ends when all uncertainties are resolved or the maximum number of trials is consumed. By evaluating modern artificial agents in **IVRE**, we notice a clear failure of today's learning methods compared to humans. Such inefficacy in interactive reasoning ability under uncertainty calls for future research in building human-like intelligence.

Submission Number: 138