The Effect of Temporal Resolution in Offline Temporal Difference Estimation

Zichen Zhang, Junxi Zhang, Francesco Zanini, Dale Schuurmans

Keywords: Reinforcement Learning, Temporal Difference, Temporal Discretization, Continuous Time, Value Estimation, LQR

TL;DR: This paper analyzes the impact of temporal resolution on offline temporal difference learning, showing a non-trivial trade-off in time discretization

Abstract: Temporal Difference (TD) algorithms are the most widely employed methods in Reinforcement Learning. Notably, previous theoretical analysis on these algorithms consider the sampling time as fixed a priori, while it has been shown that the temporal resolution can impact data efficiency (Burns et al., 2023). In this work, we analyze the performance of mean-path semi-gradient TD(0) for offline value estimation, emphasizing the dependence on the temporal resolution, a factor that indeed proves to be of crucial importance. For continuous-time stochastic linear quadratic dynamical systems with a fixed data-budget, the Mean Squared Error in value estimation shows an optimal non-trivial value for the time discretization, and this choice affects the reliability of the algorithm. We also show that this behavior differs from that of the Monte Carlo algorithm (Zhang et al., 2023). We verify the theoretical characterization in numerical experiments in linear quadratic system instances and further demonstrate, in a stochastic control setting, that the step-size trade-off persists in policy iteration.

Primary Area: reinforcement learning

Submission Number: 19350