Divergent representations of ethological visual inputs emerge from supervised, unsupervised, and reinforcement learning

Grace W Lindsay, Josh Merel, Thomas D. Mrsic-Flogel, Maneesh Sahani

Keywords: reinforcement learning, transfer learning, representations, dimensionality, sparsity, RSA

Abstract: Artificial neural systems trained using reinforcement, supervised, and unsupervised learning all acquire internal representations of high dimensional input. To what extent these representations depend on the different learning objectives is largely unknown. Here we compare the representations learned by eight different convolutional neural networks, each with identical ResNet architectures and trained on the same family of egocentric images, but embedded within different learning systems. Specifically, the representations are trained to guide action in a compound reinforcement learning task; to predict one or a combination of three task-related targets with supervision; or using one of three different unsupervised objectives. Using representational similarity analysis, we find that the network trained with reinforcement learning differs most from the other networks. Through further analysis using metrics inspired by the neuroscience literature, we find that the model trained with reinforcement learning has a high-dimensional representation wherein individual images are represented with very different patterns of neural activity. These representations seem to arise in order to guide long-term behavior and goal-seeking in the RL agent. Our results provide insights into how the properties of neural representations are influenced by objective functions and can inform transfer learning approaches.

One-sentence Summary: Reinforcement learning in an embodied agent creates sparser and higher dimensional representations than supervised or unsupervised methods.