Challenges of Explaining ControlDownload PDF

Anonymous

Published: 24 May 2019, Last Modified: 05 May 2023XAIP 2019Readers: Everyone
Keywords: Planning, Neural Networks, Monte Carlo
TL;DR: Describes a series of explainability techniques applied to a simple neural network controller used for navigation.
Abstract: Reinforcement learning and evolutionary algorithms can be used to create sophisticated control solutions. Unfortunately explaining how these solutions work can be difficult to due to their "black box" nature. In addition, the time-extended nature of control algorithms often prevent direct applications of explainability techniques used for standard supervised learning algorithms. This paper attempts to address explainability of blackbox control algorithms through six different techniques: 1) Bayesian rule lists, 2) Function analysis, 3) Single time step integrated gradients, 4) Grammar-based decision trees, 5) Sensitivity analysis combined with temporal modeling with LSTMs, and 6) Explanation templates. These techniques are tested on a simple 2d domain, where a simulated rover attempts to navigate through obstacles to reach a goal. For control, this rover uses an evolved multi-layer perception that maps an 8d field of obstacle and goal sensors to an action determining where it should go in the next time step. Results show that some simple insights in explaining the neural network are possible, but that good explanations are difficult.
Author Identity Visibility: Reveal author identities to reviewers
5 Replies

Loading