General Characterization of Agents by States they Visit

Anssi Kanervisto, Tomi Kinnunen, Ville Hautamaki

Keywords: deep learning, reinforcement learning, behavioural characterization, evolution strategies

TL;DR: We present and evaluate a method for describing agents' behaviours by the distribution of states they visit.

Abstract: Behavioural characterizations (BCs) of decision-making agents, or their policies, are used to study outcomes of training algorithms and as part of the algorithms themselves to encourage unique policies, match expert policy or restrict changes to policy per update. However, previously presented solutions are not applicable in general, either due to lack of expressive power, computational constraint or constraints on the policy or environment. Furthermore, many BCs rely on the actions of policies. We discuss and demonstrate how these BCs can be misleading, especially in stochastic environments, and propose a novel solution based on what states policies visit. We run experiments to evaluate the quality of the proposed BC against baselines and evaluate their use in studying training algorithms, novelty search and trust-region policy optimization. The code is available at https://github.com/miffyli/policy-supervectors.