Hybrid CtrlFormer: Learning Adaptive Search Space Partition for Hybrid Action Control via Transformer-based Monte Carlo Tree Search

Jiashun Liu, Xiaotian Hao, Jianye HAO, YAN ZHENG, Yujing Hu, Changjie Fan, Tangjie Lv, Zhipeng Hu

Keywords: Deep Reinforcement Learning, Hybrid Action Space Control, Transformer, Monte Carlo Tree Search

TL;DR: We observe that previous DRL algorithms for hybrid action space control tasks has the risk of the exploration space dimension explosion and propose Hybrid CtrlFormer to reduce the exploration space and efficiently select hybrid action.

Abstract: Hybrid action control tasks are common in the real world, which require controlling some discrete and continuous actions simultaneously. To solve these tasks, existing Deep Reinforcement learning (DRL) methods either directly build a separate policy for each type of action or simplify the hybrid action space into a discrete or continuous action control problem. However, these methods neglect the challenge of exploration resulting from the complexity of the hybrid action space. Thus, it is necessary to design more sample efficient algorithms. To this end, we propose a novel Hybrid Control Transformer (Hybrid CtrlFormer), to achieve better exploration and exploitation for the hybrid action control problems. The core idea is: 1) we construct a hybrid action space tree with the discrete actions at the higher level and the continuous parameter space at the lower level. Each parameter space is split into multiple subregions. 2) To simplify the exploration space, a Transformer-based Monte-Carlo tree search method is designed to efficiently evaluate and partition the hybrid action space into good and bad subregions along the tree. Our method achieves state-of-the-art performance and sample efficiency in a variety of environments with discrete-continuous action space.

List Of Authors: jiashun, Liu and xiaotian, Hao and Jianye, Hao and Yan, Zheng and Yujing, Hu and Changjie, Fan and Tangjie, Lv and Zhipeng, Hu

Submission Number: 94