Gaussian Process Q-Learning for Finite-Horizon Markov Decision Process

Maximilian Bloor, Tom Savage, Calvin Tsay, Antonio Del rio chanona, Max Mowbray

Keywords: Q-Learning, Finite Horizon MDPs, Gaussian Process, Chemical Process Control

TL;DR: Gaussian process Q-Learning algorithm for finite-horizon MDPs that uses Gaussian processes to approximate state-action value functions with theoretical analysis showing convergence guarantees for convex MDPs.

Abstract:

Many real-world control and optimization problems require making decisions over a finite time horizon to maximize performance. This paper proposes a reinforcement learning framework that approximately solves the finite-horizon Markov Decision Process (MDP) by combining Gaussian Processes (GPs) with Q-learning. The method addresses two key challenges: the tractability of exact dynamic programming in continuous state-control spaces, and the need for sample-efficient state-action value function approximation in systems where data collection is expensive. Using GPs and backward induction, we construct state-action value function approximations that enable efficient policy learning with limited data. To handle the computational burden of GPs as data accumulate across iterations, we propose a subset selection mechanism that uses M-determinantal point processes to draw diverse, high-performing subsets. Theoretical analysis establishes probabilistic uniform error bounds on the convergence of the GP posterior mean to the optimal state-action value function for convex MDPs with deterministic dynamics. The proposed method is evaluated on a linear quadratic regulator problem and online optimization of a non-isothermal semi-batch reactor. Improved learning efficiency is shown relative to the use of proximal policy optimization with a neural network policy and state-action value function approximation.

Submission Number: 203