Multi-Objective Value Iteration with Parameterized Threshold-Based Safety Constraints

Hussein Sibai, Sayan Mitra

Abstract:

We consider an environment with multiple reward functions. One of them represents goal achievement and the others represent instantaneous safety conditions. We consider a scenario where the safety rewards should always be above some thresholds. The thresholds are parameters with values that differ between users.
%The thresholds are not known at the time the policy is being designed.
We efficiently compute a family of policies that cover all threshold-based constraints and maximize the goal achievement reward. We introduce a new parameterized threshold-based scalarization method of the reward vector that encodes our objective. We present novel data structures to store the value functions of the Bellman equation that allow their efficient computation using the value iteration algorithm. We present results for both discrete and continuous state spaces.

Keywords: reinforcement learning, Markov decision processes, safety constraints, multi-objective optimization, geometric analysis