Offline Reinforcement Learning at Multiple Frequencies

Kaylee Burns, Tianhe Yu, Chelsea Finn, Karol Hausman

Keywords: offline reinforcement learning, robotics

TL;DR: Offline reinforcement learning over multiple discretizations creates training instabilities that can be solved with a simple adaptation of N-step returns

Abstract: To leverage many sources of offline robot data, robots must grapple with the heterogeneity of such data. In this paper, we focus on one particular aspect of this challenge: learning from offline data collected at different control frequencies. Across labs, the discretization of controllers, sampling rates of sensors, and demands of a task of interest may differ, giving rise to a mixture of frequencies in an aggregated dataset. We study how well offline reinforcement learning (RL) algorithms can accommodate data with a mixture of frequencies during training. We observe that the $Q$-value propagates at different rates for different discretizations, leading to a number of learning challenges for off-the-shelf offline RL algorithms. We present a simple yet effective solution that enforces consistency in the rate of $Q$-value updates to stabilize learning. By scaling the value of $N$ in $N$-step returns with the discretization size, we effectively balance $Q$-value propagation, leading to more stable convergence. On three simulated robotic control problems, we empirically find that this simple approach significantly outperforms na\"ive mixing both terms of absolute performance and training stability, while also improving over using only the data from a single control frequency.

Student First Author: yes

Website: https://sites.google.com/stanford.edu/adaptive-nstep-returns

Code: https://github.com/stanford-iris-lab/offline_rl_at_multiple_freqs