A Task-Aware Dynamic Expansion Network for Continual Reinforcement Learning

Haozhan Han, Zhi Cheng, Depeng Li, Jun Zhu, Zhigang Zeng, Xiaolin Hu

Keywords: continual learning, reinforcement learning

Abstract: Reinforcement learning has been widely applied in domains such as gaming and robotic control. However, CRL methods that rely on a single network architecture often struggle to preserve previously learned skills when they are trained on substantially different new tasks. To address this challenge, we propose a Task-Aware Dynamic Expansion Network (TADEN), which features a task-aware expansion strategy. This approach collects sequential environment states to measure task similarity, which reflects the suitability of the existing policy to a new task. Then, the task similarity score is utilized to determine whether to expand the actor-critic architecture or reuse existing modules. When expanding the network, our method leverages prior knowledge while preserving adaptability by initializing new modules through the reuse of lower layers of existing modules. We evaluate our method on the MiniHack and Atari environments. The experimental results demonstrated that TADEN achieved significantly better performance and mitigated catastrophic forgetting compared to existing methods.

Primary Area: transfer learning, meta learning, and lifelong learning

Submission Number: 8698