What can AI Learn from Human Exploration? Intrinsically-Motivated Humans and Agents in Open-World Exploration
Keywords: Exploration, RL, Minecraft, Intrinsic motivation, humans
Abstract: What drives exploration? Understanding intrinsic motivation is a long-standing question in both cognitive science and artificial intelligence (AI); numerous exploration objectives have been proposed and tested in human experiments and used to train reinforcement learning (RL) agents. However, experiments in the former are often in simplistic environments that do not capture the complexity of real world exploration. On the other hand, experiments in the latter use more complex environments, yet the trained RL agents fail to come close to human exploration efficiency. To study this gap, we propose a framework for directly comparing human and agent exploration in an open-ended environment, Crafter. We study how well commonly-proposed information theoretic intrinsic objectives relate to actual human and agent behaviors, finding that they consistently correlate with measures of exploration success in both humans and intrinsically-motivated agents. However, all agents perform significantly worse than adults on the information theoretic objectives, especially Information Gain, suggesting that better intrinsic reward design may help unsupervised agents explore more effectively. We also collect transcripts during play, and in a preliminary analysis of self-talk, we find that children's verbalizations of goals show a strong positive correlation with Empowerment, suggesting that goal-setting may be an important aspect of efficient exploration.
Submission Number: 42
Loading