Soft Actor-Critic with Inhibitory Networks for Faster Retraining

Jaime S. Ide, Daria Micovic, Adrian P Pope, Michael John Guarino, Kevin Alcedo, David Rosenbluth

Keywords: soft actor-critic, SAC, maximum entropy RL, inhibitory response, cognitive control

Abstract: Reusing previously trained models is critical in deep reinforcement learning to speed up training of new agents. However, it is unclear how to acquire new skills when objectives and constraints are in conflict with previously learned skills. Moreover, when retraining, there is an intrinsic conflict between exploiting what has already been learned and exploring new skills. In soft actor-critic (SAC) methods, a temperature parameter can be dynamically adjusted to weight the action entropy and balance the explore $\times$ exploit trade-off. However, controlling a single coefficient can be challenging within the context of retraining, even more so when goals are contradictory. In this work, inspired by neuroscience research, we propose a novel approach using inhibitory networks to allow separate and adaptive state value evaluations, as well as distinct automatic entropy tuning. Ultimately, our approach allows for controlling inhibition to handle conflict between exploiting less risky, acquired behaviors and exploring novel ones to overcome more challenging tasks. We validate our method through experiments in OpenAI Gym environments.

One-sentence Summary: We apply inhibitory control ideas from neuroscience to soft actor-critic methods to speed-up retraining of RL agents