Concept Learning for Interpretable Multi-Agent Reinforcement LearningDownload PDF

16 Jun 2022, 10:45 (modified: 16 Nov 2022, 07:22)CoRL 2022 PosterReaders: Everyone
Student First Author: yes
Keywords: Multi-Agent Reinforcement Learning, Interpretable Machine Learning
TL;DR: We introduce a method for incorporating interpretable concepts from a domain expert into models trained through multi-agent reinforcement learning by requiring the model to first predict such concepts then utilize them for decision making.
Abstract: Multi-agent robotic systems are increasingly operating in real-world environments in close proximity to humans, yet are largely controlled by policy models with inscrutable deep neural network representations. We introduce a method for incorporating interpretable concepts from a domain expert into models trained through multi-agent reinforcement learning, by requiring the model to first predict such concepts then utilize them for decision making. This allows an expert to both reason about the resulting concept policy models in terms of these high-level concepts at run-time, as well as intervene and correct mispredictions to improve performance. We show that this yields improved interpretability and training stability, with benefits to policy performance and sample efficiency in a simulated and real-world cooperative-competitive multi-agent game.
Supplementary Material: zip
23 Replies

Loading