Interactive Explanations of Agent Behavior

Yotam Amitai, Guy Avni, Ofra Amir

Keywords: Explainable Artificial Intelligence (XAI), Explainable Reinforcement Learning (XRL), Interactive Explanations, Formal Verification

TL;DR: We present an interactive tool for generating and answering user defined queries for reinforcement learning agents.

Abstract: As reinforcement learning methods increasingly amass accomplishments, the need for comprehending their solutions becomes more crucial. Most explainable reinforcement learning (XRL) methods generate a static explanation depicting their developers' intuition of what should be explained and how. In contrast, literature from the social sciences proposes that meaningful explanations are structured as a dialog between the explainer and the explainee, suggesting a more active role for the user and her communication with the agent. In this paper, we present ASQ-IT -- an interactive tool that presents video clips of the agent acting in its environment based on queries given by the user that describe temporal properties of behaviors of interest. Our approach is based on formal methods: queries in ASQ-IT's user interface map to a fragment of Linear Temporal Logic over finite traces (LTLf), which we developed, and our algorithm for query processing is based on automata theory. We provide experimental results from a user-study aimed at testing ASQ-IT's usability, and report positive outcomes from both objective performance and self-reported ability of participants to use our tool.