Virtual Tool Game with Compositional Concepts

Zhuoying Li, Hengli Li, Rui Yang

Keywords: physical reasoning, tool use, compositional concepts, artificial intelligence

Abstract: Exploring the utilization of tools has been a significant point of study in differentiating human intelligence from animal behavior. While numerous types of animals have also demonstrated amazing capability of making tools for their own purpose, humans still excel at flexible and novel usage of conventional tools. This work delves into the exploration of model tool-use abilities, particularly focusing on novel tool inventions. Inspired by the Virtual Tool Game, as detailed in the foundational paper, our project aims to extend the understanding of model's capacity for compositional reasoning and problem-solving. We introduce two novel scenarios which combine elements like "Bridge" and "Spiky", as well as "Table" and "Hit". We then propose an AI model capable of integrating these compositional concepts and solve the new, complex tasks. Our contributions also involve reproducing baselines from the reference paper, which is an important missing part in the original work. [All of the codes are available at \href{https://github.com/KateYangzy/CoRe-term-project}{https://github.com/KateYangzy/CoRe-term-project}]

Submission Number: 229