Formal Explanations of Neural Network Policies for Planning

Renee Selvey, Alban Grastien, Sylvie Thiebaux

Keywords: abductive explanations, sequential decisions, neural network policies, ASNets, MIP encodings of neural networks

TL;DR: Abductive explanations for sequential decisions recommended by neural-network policies, with minimal overhead compared to the single-decision case.

Abstract: Deep learning is increasingly used to learn policies for planning problems. However, policies represented by neural networks are difficult to interpret, verify and trust. Existing formal approaches to post-hoc explanations provide concise reasons for a single decision made by an ML model. However, understanding planning policies requires explaining sequences of decisions. In this paper, we formulate the problem of finding explanations for the sequence of decisions recommended by a learnt policy in a given state. We show that, under certain assumptions, a minimal explanation for a sequence can be computed by solving a number of single decision explanation problems which is linear in the length of the sequence. We present experimental results of our implementation of this approach for ASNets policies for classical planning domains.