Fuz-RL: A Fuzzy-Guided Robust Framework for Safe Reinforcement Learning under Uncertainty

Xu Wan, Chao Yang, Cheng Yang, Jie Song, Mingyang Sun

Keywords: Robust Safe Reinforcement Learning; Risk-sensitive Decision-making

TL;DR: We propose Fuz-RL, a fuzzy logic-based framework that enhances robustness of safe reinforcement learning by measuring and balancing uncertainties in a non-additive way, achieving better safety and performance under uncertainty.

Abstract:

Safe Reinforcement Learning (RL) is crucial for achieving high performance while ensuring safety in real-world applications. However, the complex interplay of multiple uncertainty sources in real environments poses significant challenges for interpretable risk assessment and robust decision-making. To address these challenges, we propose \textbf{Fuz-RL}, a fuzzy measure-guided robust framework for safe RL.
Specifically, our framework develops a novel fuzzy Bellman operator for estimating robust value functions using Choquet integrals. Theoretically, we prove that solving the Fuz-RL problem (in Constrained Markov Decision Process (CMDP) form) is equivalent to solving distributionally robust safe RL problems (in robust CMDP form), effectively avoiding min-max optimization. Empirical analyses on safe-control-gym and safety-gymnasium scenarios demonstrate that Fuz-RL effectively integrates with existing safe RL baselines in a model-free manner, significantly improving both safety and control performance under various types of uncertainties in observation, action, and dynamics.

Primary Area: Reinforcement learning (e.g., decision and control, planning, hierarchical RL, robotics)

Submission Number: 5403