PIGDreamer: Privileged Information Guided World Models for Safe Partially Observable Reinforcement Learning

Dongchi Huang, Jiaqi WANG, Yang Li, Chunhe Xia, Tianle Zhang, Kaige Zhang

TL;DR: An algorithm that leverages privileged information to address partial observability in safe reinforcement learning.

Abstract: Partial observability presents a significant challenge for safe reinforcement learning, as it impedes the identification of potential risks and rewards. Leveraging specific types of privileged information during training to mitigate the effects of partial observability has yielded notable empirical successes. In this paper, we propose Asymmetric Constrained Partially Observable Markov Decision Processes (ACPOMDPs) to theoretically examine the advantages of incorporating privileged information. Building upon ACPOMDPs, we propose the Privileged Information Guided Dreamer, a model-based safe reinforcement learning approach that leverages privileged information to enhance the agent's safety and performance through privileged representation alignment and an asymmetric actor-critic structure. Our empirical results demonstrate that our approach significantly outperforms existing methods in terms of safety and task-centric performance. Meanwhile, compared to alternative privileged model-based reinforcement learning methods, our approach exhibits superior performance and ease of training.

Lay Summary:

Teaching robots to stay safe when they can’t see everything is tricky. Since their sensors are limited, they might miss important dangers, making it hard to avoid risky situations. We want to address this challenge by leveraging extra sensors in the teaching process to equip robots with better danger sense abilities, thereby enhancing its dangerous avoidance ability.

We developed a framework called *Asymmetric Constrained Partially Observable Markov Decision Processes (ACPOMDPs)* to show how adding extra sensors during training can help robots better understand their surroundings. Using this framework, we created an algorithm called the *Privileged Information Guided Dreamer*. This algorithm aligns the extra sensor data with the robot’s understanding of the world, improving its ability to avoid dangers and perform tasks safely.

Our work improves how robots use extra sensor data to better understand their environment. It achieves state-of-the-art performance in safety navigation tasks, demonstrating the effectiveness of our approach in real-world scenarios.

Link To Code: https://github.com/hggforget/PIGDreamer

Primary Area: Reinforcement Learning->Deep RL

Keywords: Reinforcement Learning; World Models; Safe Reinforcement Learning; Model-based Reinforcement Learning; Privileged Learning

Submission Number: 3835