Risk-sensitive control as inference with Rényi divergence
Keywords: risk-sensitive control, optimal control, reinforcement learning, variational inference, Rényi divergence
TL;DR: This study formulates risk-sensitive control as variational inference using Rényi divergence. Based on the proposed unifying framework, we reveal several equivalence results for control problems and derive risk-sensitive RL algorithms.
Abstract: This paper introduces the risk-sensitive control as inference (RCaI) that extends CaI by using Rényi divergence variational inference. RCaI is shown to be equivalent to log-probability regularized risk-sensitive control, which is an extension of the maximum entropy (MaxEnt) control. We also prove that the risk-sensitive optimal policy can be obtained by solving a soft Bellman equation, which reveals several equivalences between RCaI, MaxEnt control, the optimal posterior for CaI, and linearly-solvable control. Moreover, based on RCaI, we derive the risk-sensitive reinforcement learning (RL) methods: the policy gradient and the soft actor-critic. As the risk-sensitivity parameter vanishes, we recover the risk-neutral CaI and RL, which means that RCaI is a unifying framework. Furthermore, we give another risk-sensitive generalization of the MaxEnt control using Rényi entropy regularization. We show that in both of our extensions, the optimal policies have the same structure even though the derivations are very different.
Primary Area: Reinforcement learning
Submission Number: 15775
Loading