ActSafe: Active Exploration with Safety Constraints for Reinforcement Learning
Keywords: Safe Exploration, Constrained Markov Decision Processes, Safe Reinforcement Learning
TL;DR: We propose a novel model-based algorithm for safe exploration in continuous state-action spaces and derive theoretical guarantees for it. We design a practical variant of our algorithm that scales well to vision control tasks.
Abstract: Reinforcement learning (RL) is ubiquitous in the development of modern AI systems. However, state-of-the-art RL agents require extensive, and potentially
unsafe, interactions with their environments to learn effectively. These limitations
confine RL agents to simulated environments, hindering their ability to learn
directly in real-world settings. In this work, we present ActSafe, a novel
model-based RL algorithm for safe and efficient exploration. ActSafe learns
a well-calibrated probabilistic model of the system and plans optimistically
w.r.t. the epistemic uncertainty about the unknown dynamics, while enforcing
pessimism w.r.t. the safety constraints. Under regularity assumptions on the
constraints and dynamics, we show that ActSafe guarantees safety during
learning while also obtaining a near-optimal policy in finite time. In addition, we
propose a practical variant of ActSafe that builds on latest model-based RL advancements and enables safe exploration even in high-dimensional settings such
as visual control. We empirically show that ActSafe obtains state-of-the-art
performance in difficult exploration tasks on standard safe deep RL benchmarks
while ensuring safety during learning.
Primary Area: reinforcement learning
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 7732
Loading