Goals vs. Rewards: A Comparative Study of Objective Specification Mechanisms

Septia Rani, Serena Booth, Sarath Sreedharan

Keywords: Objective Specification, Goals, Rewards

Abstract: This paper looks at two popular objective specification mechanisms for sequential decision-making problems: goals and rewards. We investigate how easy it is for people without AI expertise to use these different specification mechanisms effectively. Specifically, through this paper, we investigate how effectively these mechanisms could be used to (a) correctly direct an AI system or robot to generate some desired behavior and (b) predict the behavior encoded in a given objective specification. We first present a formalization of the problems of objective specification and behavior prediction, and we characterize underspecification and overspecification. We then perform a user study to assess how well participants are able to use rewards and goals as specification mechanisms and their propensity for overspecification and underspecification with these mechanisms. While participants have a strong preference for using goals as an objective specification mechanism, we find a surprising result: even naïve users are equally capable of specifying and interpreting reward functions as of using goals.

Submission Number: 308