Thinking Out-of-the-Box: A Comparative Investigation of Human and LLMs in Creative Problem-Solving

Yufei Tian, Abhilasha Ravichander, Lianhui Qin, Ronan Le Bras, Raja Marjieh, Nanyun Peng, Yejin Choi, Thomas L. Griffiths, Faeze Brahman

Keywords: creative problem solving, physical reasoning

TL;DR: We create a novel dataset of creative problem solving tasks. We then compare/contrast seven LLMs with human, showing a substantial gap but complementary capabilities.

Abstract: We explore the creative problem-solving capabilities of modern LLMs in a constrained setting. To this end, we create MacGyver, an automatically generated dataset consisting of 1,600 real-world problems deliberately designed to trigger innovative usage of objects and necessitate out-of-the-box thinking. We then present our collection to both LLMs and humans to compare and contrast their problem-solving abilities. Our task is challenging for both groups, but in unique and complementary ways. For instance, humans excel in tasks they are familiar with but struggle with domain-specific knowledge, leading to a higher variance. In contrast, LLMs, exposed to a variety of specialized knowledge, attempt broader problems but fail by proposing physically-infeasible actions.

Submission Number: 47