Goal-Oriented Skill Abstraction for Offline Multi-Task Reinforcement Learning

Jinmin He, Kai Li, Yifan Zang, Haobo Fu, QIANG FU, Junliang Xing, Jian Cheng

Abstract: Offline multi-task reinforcement learning aims to learn a unified policy capable of solving multiple tasks using only pre-collected task-mixed datasets, without requiring any online interaction with the environment. However, it faces significant challenges in effectively sharing knowledge across tasks. Inspired by the efficient knowledge abstraction observed in human learning, we propose Goal-Oriented Skill Abstraction (GO-Skill), a novel approach designed to extract and utilize reusable skills to enhance knowledge transfer and task performance. Our approach uncovers reusable skills through a goal-oriented skill extraction process and leverages vector quantization to construct a discrete skill library. To mitigate class imbalances between broadly applicable and task-specific skills, we introduce a skill enhancement phase to refine the extracted skills. Furthermore, we integrate these skills using hierarchical policy learning, enabling the construction of a high-level policy that dynamically orchestrates discrete skills to accomplish specific tasks. Extensive experiments on diverse robotic manipulation tasks within the MetaWorld benchmark demonstrate the effectiveness and versatility of GO-Skill.

Lay Summary: Learning to solve multiple tasks simultaneously is a major challenge in artificial intelligence, especially when relying only on pre-collected data without live trial and error. This process, known as offline multi-task reinforcement learning, struggles to share useful knowledge across tasks. Inspired by the way humans build reusable skills, we developed a new approach called Goal-Oriented Skill Abstraction (GO-Skill). It breaks down complex problems into smaller, reusable skills that can be applied across different situations. These skills are organized into a library, making it easier for the system to choose the right skills for each task. GO-Skill then combines these skills into high-level decision-making strategies, allowing artificial systems to handle varied challenges more effectively. In our tests with robotic systems, this approach showed promising results, helping machines learn more efficiently and adapt to a variety of challenges.

Primary Area: Reinforcement Learning->Batch/Offline

Keywords: Offline Multi-Task Reinforcement Learning, Skill Abstraction

Submission Number: 5277