CRAFT: A Benchmark for Causal Reasoning About Forces and inTeractions

Tayfun Ates, M. Şamil Ateşoğlu, Çağatay Yiğit, Ilker Kesen, Mert Kobas, Erkut Erdem, Aykut Erdem, Tilbe Goksun, Deniz Yuret

Keywords: intuitive physics, visual question answering, force dynamics, causal reasoning, temporal reasoning

TL;DR: We present a new visual question answering dataset named CRAFT, in which working towards an answer requires causal reasoning about physical forces and object interactions.

Abstract: Humans are able to perceive, understand and reason about physical events. Developing models with similar physical understanding capabilities is a long standing goal of artificial intelligence. As a step towards this goal, in this work, we introduce CRAFT, a new visual question answering dataset that requires causal reasoning about physical forces and object interactions. It contains 58K video and question pairs that are generated from 10K videos from 20 different virtual environments, containing various objects in motion that interact with each other and the scene. Two question categories from CRAFT include previously studied descriptive and counterfactual questions. Besides, inspired by the theories of force dynamics in cognitive linguistics, we introduce new question categories that involve understanding the interactions of objects through the notions of cause, enable, and prevent. Our results demonstrate that even though these tasks seem to be simple and intuitive for humans, the evaluated baseline models, including existing state-of-the-art methods, do not yet deal with the challenges posed in our benchmark dataset.

URL: https://sites.google.com/view/craft-benchmark

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2012.04293/code)