Sampling-Based Safe Reinforcement Learning

Luca Vignola, Bruce D Lee, Manish Prajapat, Manuel Wendl, Melanie N. Zeilinger, Andreas Krause, Yarden As

Keywords: Safe Reinforcement Learning, Constrained Markov Decision Processes, Safe Exploration, Offline to Online

TL;DR: We propose a safe and scalable reinforcement learning algorithm that leverages dynamics samples to guarantee safety and optimality

Abstract:

Safe exploration remains a fundamental challenge in reinforcement learning (RL), limiting the deployment of RL agents in the real world.
We propose _Sampling-Based Safe Reinforcement Learning_ (SBSRL), a model-based RL algorithm that maintains safety throughout the learning process by enforcing constraints jointly across a _finite_ set of dynamics samples.
This formulation approximates an intractable worst-case optimization over uncertain dynamics and enables practical safety guarantees in continuous domains.
We further introduce an exploration strategy based on constraining epistemic uncertainty, eliminating the need for explicit exploration bonuses.
Under regularity conditions, we derive high-probability guarantees of safety throughout learning and a finite-time sample complexity bound for recovering a near-optimal policy.
Empirically, SBSRL achieves safe and efficient exploration both in simulation and in real robotic hardware, and readily extends to practical deep-ensemble implementations that scale to high-dimensional continuous control problems.

Submission Number: 122