Behavior-Guided Actor-Critic: Improving Exploration via Learning Policy Behavior Representation for Deep Reinforcement Learning

Ammar Fayad, Majd Ibrahim

Abstract:

In this work, we propose Behavior-Guided Actor-Critic (BAC), an off-policy
actor-critic deep RL algorithm. BAC mathematically formulates the behavior
of the policy through autoencoders by providing an accurate estimation of how
frequently each state-action pair was visited while taking into consideration state
dynamics that play a crucial role in determining the trajectories produced by the
policy. The agent is encouraged to change its behavior consistently towards less-
visited state-action pairs while attaining good performance by maximizing the
expected discounted sum of rewards, resulting in an efficient exploration of the
environment and good exploitation of all high-reward regions. One prominent
aspect of our approach is that it is applicable to both stochastic and deterministic
actors in contrast to maximum entropy deep reinforcement learning algorithms.
Results show considerably better performances of BAC when compared to several
cutting-edge learning algorithms.