Exponential Family Model-Based Reinforcement Learning via Score MatchingDownload PDF

Published: 31 Oct 2022, Last Modified: 08 Jan 2023NeurIPS 2022 AcceptReaders: Everyone
Keywords: online reinforcement learning, exponential family, model-based RL
TL;DR: An algorithm for online RL when transitions are exponential family.
Abstract: We propose an optimistic model-based algorithm, dubbed SMRL, for finite-horizon episodic reinforcement learning (RL) when the transition model is specified by exponential family distributions with $d$ parameters and the reward is bounded and known. SMRL uses score matching, an unnormalized density estimation technique that enables efficient estimation of the model parameter by ridge regression. Under standard regularity assumptions, SMRL achieves $\tilde O(d\sqrt{H^3T})$ online regret, where $H$ is the length of each episode and $T$ is the total number of interactions (ignoring polynomial dependence on structural scale parameters).
Supplementary Material: zip
11 Replies