Multi-Advisor Reinforcement Learning

Romain Laroche, Mehdi Fatemi, Joshua Romoff, Harm van Seijen

Abstract: We consider tackling a single-agent RL problem by distributing it to $n$ learners. These learners, called advisors, endeavour to solve the problem from a different focus. Their advice, taking the form of action values, is then communicated to an aggregator, which is in control of the system. We show that the local planning method for the advisors is critical and that none of the ones found in the literature is flawless: the \textit{egocentric} planning overestimates values of states where the other advisors disagree, and the \textit{agnostic} planning is inefficient around danger zones. We introduce a novel approach called \textit{empathic} and discuss its theoretical aspects. We empirically examine and validate our theoretical findings on a fruit collection task.

TL;DR: We consider tackling a single-agent RL problem by distributing it to $n$ learners.

Keywords: Reinforcement Learning