ROSARL: Reward-Only Safe Reinforcement Learning

ICLR 2026 Conference Submission24895 Authors

Keywords: Reinforcement Learning, Deep Reinforcement Learning, Safe RL, Constrained RL, Reward shaping

Abstract: An important problem in reinforcement learning is designing agents that learn to solve tasks safely in an environment. A common solution is to define either a penalty in the reward function or a cost to be minimised when reaching unsafe states. However, designing reward or cost functions is non-trivial and can increase with the complexity of the problem. To address this, we investigate the concept of a *Minmax* penalty, the smallest penalty for unsafe states that leads to safe optimal policies, regardless of task rewards. We derive an upper and lower bound on this penalty by considering both environment *diameter* and *controllability*. Additionally, we propose a simple algorithm for agents to estimate this penalty while learning task policies. Our experiments demonstrate the effectiveness of this approach in enabling agents to learn safe policies in high-dimensional continuous control environments.

Primary Area: reinforcement learning

Submission Number: 24895