INFLUENCE-BASED REINFORCEMENT LEARNING FOR INTRINSICALLY-MOTIVATED AGENTS

Ammar Fayad, Majd Ibrahim

Abstract:

Discovering successful coordinated behaviors is a central challenge in Multi-Agent
Reinforcement Learning (MARL) since it requires exploring a joint action space
that grows exponentially with the number of agents. In this paper, we propose
a mechanism for achieving sufficient exploration and coordination in a team of
agents. Specifically, agents are rewarded for contributing to a more diversified team
behavior by employing proper intrinsic motivation functions. To learn meaningful
coordination protocols, we structure agents’ interactions by introducing a novel
framework, where at each timestep, an agent simulates counterfactual rollouts
of its policy and, through a sequence of computations, assesses the gap between
other agents’ current behaviors and their targets. Actions that minimize the gap
are considered highly influential and are rewarded. We evaluate our approach on a
set of challenging tasks with sparse rewards and partial observability that require
learning complex cooperative strategies under a proper exploration scheme, such
as the StarCraft Multi-Agent Challenge. Our methods show significantly improved
performances over different baselines across all tasks.