DT-Pro: Proactive Decision Transformers with Implicit Latent Space Planning

Zengyang Wang, Qirui Gan, Yiran Zeng, Xinzhi Zhang, Ningning Ding, Junying Chen, Huiping Zhuang, Hongxin Wei, Yi Cai, Mengchen Zhao

Keywords: Reinforcement Learning, Sequential Decision Making

TL;DR: In this paper, we present DT-Pro, a variant of DT that enhances its planning ability by integrating a natural implicit planning step into sequence modeling.

Abstract: Decision Transformers (DTs) address decision making problems through sequence modeling and have achieved surprisingly strong results. However, DTs still struggle in long-horizon tasks due to their poor planning ability. Existing works have demonstrated that subgoal prediction helps to guide DTs' decision making in complex and long-horizon tasks. However, explicit planning via subgoal prediction suffers from suboptimality, inefficiency and instability. In this paper, we present DT-Pro, a variant of DT that enhances its planning ability by integrating a natural implicit planning step into sequence modeling. Compared with explicit planning via subgoal prediction, the implicit planning works by inferring a latent plan from a structured plan space. Through this way, DT-Pro enables high-quality adaptive plan generation and efficient stepwise replanning with only a marginal increase in the computational cost. Extensive experimental results show that DT-Pro achieves strong performance on a variety of widely used control and navigation benchmarks.

Primary Area: reinforcement learning

Submission Number: 10922