Object Empowerment-Driven Tool Selection for Exploration in Reinforcement Learning

Faizan Rasheed, Kenzo Clauw, Daniel Polani, Nicola Catenacci Volpi

Keywords: intrinsic motivation, reinforcement learning, empowerment

TL;DR: This paper integrates object-tool interactions into empowerment based intrinsic motivation for improving exploration in reinforcement learning

Abstract:

Tool use enables humans to solve complex physical tasks beyond their immediate capabilities. However, discovering tool use remains a major challenge for reinforcement learning (RL) agents, as it
requires mastering long-horizon behaviors with sparse, delayed feedback—resulting in poor exploration and sample efficiency. While classic intrinsic motivation (IM) improves exploration, its lack
of bias toward object-tool interactions leads agents to explore irrelevant states, resulting in many
costly real-world interactions. In this paper, we investigate how RL agents can efficiently learn to
use tools by optimizing object-centric intrinsic motivations — specifically, object empowerment,
which quantifies the agent’s potential influence over specific objects in the environment. We extend this intrinsic motivation to multi-tool, multi-object environments that better reflect real-world
lifelong learning challenges. Our method enables agents to identify meaningful tool-object relationships, learn when and how to use tools, and understand their lasting effects. Experiments in
grid-based Minihack environments demonstrate that agents guided by object empowerment explore
more effectively, generalize to new object configurations, and outperform PPO under sparse reward
conditions.

Submission Number: 10