Graph Constrained Reinforcement Learning for Natural Language Action Spaces

Prithviraj Ammanabrolu, Matthew Hausknecht

Keywords: natural language generation, deep reinforcement learning, knowledge graphs, interactive fiction

TL;DR: We present KG-A2C, a reinforcement learning agent that builds a dynamic knowledge graph while exploring and generates natural language using a template-based action space - outperforming all current agents on a wide set of text-based games.

Abstract: Interactive Fiction games are text-based simulations in which an agent interacts with the world purely through natural language. They are ideal environments for studying how to extend reinforcement learning agents to meet the challenges of natural language understanding, partial observability, and action generation in combinatorially-large text-based action spaces. We present KG-A2C, an agent that builds a dynamic knowledge graph while exploring and generates actions using a template-based action space. We contend that the dual uses of the knowledge graph to reason about game state and to constrain natural language generation are the keys to scalable exploration of combinatorially large natural language actions. Results across a wide variety of IF games show that KG-A2C outperforms current IF agents despite the exponential increase in action space size.

Code: https://github.com/rajammanabrolu/KG-A2C

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2001.08837/code)