Unsupervised Real-Time Control Through Variational Empowerment

Maximilian Karl, Philip Becker-Ehmck, Maximilian Soelch, Djalel Benbouzid, Patrick van der Smagt, Justin Bayer

Abstract: Intrinsic motivation is vital for living beings. It enables skill acquisitions, triggers explorative behaviour, and hence enhances cognitive capabilities. One way of formalising the variety of behaviours induced by intrinsic motivation is empowerment, an information-theoretic measure that encodes the influence an agent exerts on its environment. Formally, empowerment is the maximum mutual information between actions and the resulting states which is prohibitively hard to compute, especially in nonlinear continuous spaces. In this work, we introduce a method for efficiently computing a lower bound on empowerment, enabling its use as an unsupervised cost function for real-time control. We demonstrate that our algorithm can reliably handle continuous dynamical systems even when system dynamics are learnt from raw data. The resulting empowerment-maximizing policies consistently drive the agents into states with high potential impact.