Learning Object-Oriented Dynamics for Planning from Text

Guiliang Liu, Ashutosh Adhikari, Amir-massoud Farahmand, Pascal Poupart

Keywords: Object Oriented Markov Decision Process, Reinforcement Learning, Model-Based Planning, Text-Based Games, Knowledge Extraction

Abstract: The advancement of dynamics models enables model-based planning in complex environments. Existing dynamics models commonly study image-based games with fully observable states. Generalizing these models to Text-Based Games (TBGs), which commonly describe the partially observable states with noisy text observations, is challenging. In this work, we propose an Object-Oriented Text Dynamics (OOTD) model that enables planning algorithms to solve decision-making problems in text domains. OOTD predicts a memory graph that dynamically remembers the history of object observations and filters object-irrelevant information. To facilitate the robustness of dynamics, our OOTD model identifies the objects influenced by input actions and predicts the belief of object states with independently parameterized transition layers. We develop variational objectives under the object-supervised and self-supervised settings to model the stochasticity of predicted dynamics. Empirical results show OOTD-based planner significantly outperforms model-free baselines in terms of sample efficiency and running scores.

One-sentence Summary: We propose an Object-Oriented Text Dynamic (OOTD) model for solving decision-making problems in the text domain.