Plan-Property Dependencies are Useful: A User Study

Rebecca Eifler, Martim Brandão, Amanda Jane Coles, Jeremy Frank, Joerg Hoffmann

Abstract:

The trade-offs between different desirable plan properties -- e.g.
PDDL temporal plan preferences -- are often difficult to
understand. Recent work proposes to address this by iterative planning
with explanations elucidating the dependencies between such plan
properties. Users can ask questions of the form ``Why does the plan
you suggest not satisfy property $p$?'', which are answered by
``Because then we would have to forego $q$'' where $\neg q$ is
entailed by $p$ in plan space.
It has been shown that such plan-property dependencies can be computed
reasonably efficiently. But is this form of explanation actually
useful for users?
We contribute a user study evaluating that question. We design use
cases from three domains and run a large user study ($N=40$ for each
domain, ca. 40 minutes work time per user and domain) on the internet
platform Prolific. Comparing users with vs. without access to the
explanations, we find that the explanations tend to enable users to
identify better trade-offs between the plan properties, indicating an
improved understanding of the task.