Planner Metrics Should Satisfy Independence of Irrelevant Alternatives

Anonymous

Keywords: metrics

Abstract: We argue that planner evaluation metrics should satisfy the independence of irrelevant alternatives criterion, i.e., the decision whether planner A is ranked higher or lower than planner B should be independent of planner C. We show that three metrics used in classical planning competitions do not necessarily satisfy this criterion and highlight alternative metrics that do so.

Track: short (15 minutes)