Dual-Objective Reinforcement Learning with Novel Hamilton-Jacobi-Bellman Formulations

ICLR 2026 Conference Submission19127 Authors

Keywords: Hamilton-Jacobi Analysis, Dual Satisfaction, Safe Reinforcement Learning, Decomposition

TL;DR: Novel value function decomposition theorems for dual-satisfaction RL, yielding new performant algorithm

Abstract: Hard constraints in reinforcement learning (RL) often degrade policy performance. Lagrangian methods offer a way to blend objectives with constraints, but require intricate reward engineering and parameter tuning. In this work, we extend recent advances that connect Hamilton-Jacobi (HJ) equations with RL to propose two novel value functions for dual-objective satisfaction. Namely, we address: 1) the Reach-Always-Avoid (RAA) problem – of achieving distinct reward and penalty thresholds – and 2) the Reach-Reach (RR) problem – of achieving thresholds of two distinct rewards. In contrast with temporal logic approaches, which typically involve representing an automaton, we derive explicit, tractable Bellman forms in this context via decomposition. Specifically, we prove that the RAA and RR problems may be rewritten as compositions of previously studied HJ-RL problems. We leverage our analysis to propose a variation of Proximal Policy Optimization (DO-HJ-PPO), and demonstrate that it produces distinct behaviors from previous approaches, out-competing a number of baselines in success, safety and speed across a range of tasks for safe-arrival and multi-target achievement.

Primary Area: reinforcement learning

Submission Number: 19127