Maximum Likelihood Constraint Inference from Stochastic Demonstrations

David Livingston McPherson, Kaylene C. Stocking, S. Shankar Sastry

Abstract: When an expert operates a safety-critical dynamic system, constraint information is tacitly contained in their demonstrated trajectories and controls. These constraints can be inferred by modeling the system and operator as a constrained Markov Decision Process and finding which constraint is most likely to generate the demonstrated controls. Prior constraint inference work has focused mainly on deterministic dynamics. Stochastic dynamics, however, can capture the uncertainty inherent to real applications and the risk tolerance that requires.This paper extends maximum likelihood constraint inference to stochastic applications by using maximum causal entropy likelihoods. Furthermore, this extension does not come at increased computational cost, as we derive an algorithm that computes constraint likelihood and risk tolerance in a unified Bellman backup, thereby keeping the same computational complexity.