Efficient Planning in a Compact Latent Action Space

zhengyao jiang, Tianjun Zhang, Michael Janner, Yueying Li, Tim Rocktäschel, Edward Grefenstette, Yuandong Tian

Keywords: Model-based RL, Planning, Sequence Modelling RL, Generative Model, Offline Reinforcement Learning

TL;DR: We propose Trajectory Autoencoding Planner (TAP) a model-based RL method that learns a compact discrete latent action space from offline data for efficient planning.

Abstract:

Planning-based reinforcement learning has shown strong performance in tasks in discrete and low-dimensional continuous action spaces.
However, planning usually brings significant computational overhead for decision making, so scaling such methods to high-dimensional action spaces remains challenging. To advance efficient planning for high-dimensional continuous control, we propose Trajectory Autoencoding Planner (TAP), which learns low-dimensional latent action codes from offline data. The decoder of the VQ-VAE thus serves as a novel dynamics model that takes latent actions and current state as input and reconstructs long-horizon trajectories. During inference time, given a starting state, TAP searches over discrete latent actions to find trajectories that have both high probability under the training distribution and high predicted cumulative reward. Empirical evaluation in the offline RL setting demonstrates low decision latency which is indifferent to the growing raw action dimensionality. For Adroit robotic hand manipulation tasks with high-dimensional continuous action space, TAP surpasses existing model-based methods by a large margin and also beats strong model-free actor-critic baselines.

Community Implementations: [ 2 code implementations](https://www.catalyzex.com/paper/efficient-planning-in-a-compact-latent-action/code)