Text-based games serve as important benchmarks for agents with natural language capabilities. To enable such agents, we are interested in the problem of learning useful world models. Our assumption is that such a world model is best represented by a logical form which underlies the structure of these games. We propose to tackle this problem by leveraging the expressivity of recent neuro-symbolic architectures, specifically the Logical Neural Networks (LNN). Here, we describe a method that can learn neuro-symbolic world models on the TextWorld-Commonsense set of games. We then show that planning on this learned world model results in optimal actions in the game world.

Learning Neuro-Symbolic World Models for Text-Based Game Playing Agents

Since text-based games are one of the most challenging problems and require to have an understanding of language and decision-making in a complex environment, there are many studies about applying deep reinforcement learning methods. However, these deep models are often black-box which means the human operator cannot understand the trained rules. And the models use external knowledge such as common-sense knowledge from well-designed data. In this paper, we explain some challenges in solving these problems in text-based games.

Challenges in Explainability and Knowledge Extraction

Errors in reference generation and resolution can occur if a dialogue agent reasons incorrectly about their partner's perspective. We present, in a collaborative and visually grounded setting, a dialogue planner that infers its partner's perspective to produce referring expressions that the partner resolves correctly. The dialogue planner models the partner perspective as a latent variable, embedded in a partner model that is used for both model-based planning and incorporating evidence from partner responses. We validate our approach on \textsc{OneCommon}, a challenging dialogue game where players have large differences in their perspectives. In symbolic selfplay, where agents partner with a copy of themselves using symbolic communication, the dialogue planner improves over a planning baseline that does not reason about the partner's full perspective.

Modeling Perspective-Dependent Ambiguity in Collaborative Dialogue

The purpose of this extended abstract is to discuss the possible fruitful interactions between intrinsically-motivated language-conditioned agents and textual environments. We define autotelic agents as agents able to set their own goals. We identify desirable properties of textual nenvironments that makes them a good testbed for autotelic agents. We them list drivers of exploration for such agents that would allow them to achieve large repertoires of skills in these environments, enabling such agents to be repurposed for solving the benchmarks implemented in textual environments. We then discuss challenges and further perspectives brought about by this interaction.

Automatic Exploration of Textual Environments with Language-Conditioned Autotelic Agents

Dungeons and Dragons is a popular tabletop role-playing game that has been adapted to online play. In this paper, we look at enhancing a Discord Bot called Avrae that is developed by D&D Beyond to help with online play.  Avrae enables users to manage gameplay through Unix-like commands. We explore using language models to automatically translate player dialogue into Avrae's  commands. We use GPT-3's few shot learning and fine tuning capabilities and achieve 64% accuracy. We also explore the reverse direction, where commands are rendered as descriptive text, suggesting that it may eventually be possible to combine Avrae and LMs to create a system that is capable of role-playing alongside players.  

Using Language Models to Convert Between Natural Language and Game Commands

Text-based games present a unique class of sequential decision making problem in which agents interact with a partially observable, simulated environment via actions and observations conveyed through natural language. Such observations typically include instructions that, in a reinforcement learning (RL) setting, can directly or indirectly guide a player towards completing reward-worthy tasks. In this work, we study the ability of RL agents to follow such instructions. We conduct experiments that show that the performance of state-of-the-art text-based game agents is largely unaffected by the presence or absence of such instructions, and that these agents are typically unable to execute tasks to completion. To further study and address the task of instruction following, we equip RL agents with an internal structured representation of natural language instructions in the form of Linear Temporal Logic (LTL), a formal language that is increasingly used for temporally extended reward specification in RL. Our framework both supports and highlights the benefit of understanding the temporal semantics of instructions and in measuring progress towards achievement of such a temporally extended behaviour. Experiments demonstrate the superior performance of our approach.

Instruction Following in Text-Based Games

Interactive Question Answering (IQA) requires an intelligent agent to interact with a dynamic environment in order to gather information necessary to answer a question. IQA tasks have been proposed as means of training systems to develop language or visual comprehension abilities. To this end, the Question Answering with Interactive Text (QAit) task was created to produce and benchmark interactive agents capable of seeking information and answering questions in unseen environments. While prior work has exclusively focused on IQA as a reinforcement learning problem, such methods suffer from low sample efficiency and poor accuracy in zero-shot evaluation. In this paper, we propose the use of the recently proposed Decision Transformer architecture to provide improvements upon prior baselines. By utilising a causally masked GPT-2 Transformer for command generation and a BERT model for question answer prediction, we show that the Decision Transformer achieves performance greater than or equal to current state-of-the-art RL baselines on the QAit task in a sample efficient manner. In addition, these results are achievable by training on sub-optimal random trajectories, therefore not requiring the use of online agents to gather data.

A Sequence Modelling Approach to Question Answering in Text-Based Games

Text adventure games provide players with a world to explore the opportunity to interact with it free from the constraints of typical games by using natural language to specify their actions. While revolutionary at the time, these games ultimately came with strict limits on the kinds of inputs they could successfully parse. Recent efforts have made use of large language models to create text adventure games capable of handling any input the user throws at it, but at the cost of lacking any sense of structure or permanence. Is it possible to combine the best aspects of both of these styles of text adventure games, resulting in a game that accepts broad user inputs while remaining grounded? We present one potential method for tackling this problem and discuss both the major difficulties and some potential avenues to explore.

Towards Knowledge-Graph Constrained Generation for Text Adventure Games

Non-Player Characters (NPCs) significantly enhance the player experience in many games. Historically, players' interactions with NPCs have tended to be highly scripted, to be limited to natural language responses to be selected by the player, and to not involve dynamic change in game state. In this work, we demonstrate that use of a few example conversational prompts can power a conversational agent to generate both natural language and novel code. This approach can permit development of NPCs with which players can have grounded conversations that are free-form and less repetitive. We demonstrate our approach using OpenAI Codex (GPT-3 finetuned on GitHub), with Minecraft game development as our test bed. We show that with a few example prompts, a Codex-based agent can generate novel code, hold multi-turn conversations and answer questions about structured data. We evaluate this application using experienced gamers in a Minecraft realm and provide analysis of failure cases and suggest possible directions for solutions.

Craft an Iron Sword: Dynamically Generating Interactive Game Characters by Prompting Large Language Models Tuned on Code

We describe our system for playing a minimal improvisational game in a group. In Chain Reaction, players collectively build a chain of word pairs or solid compounds. The game emphasizes memory and rapid improvisation, while absurdity and humor increases during play. Our approach is unique in that we have grounded our work in the principles of oral culture according to Walter Ong, an early scholar of orature. We show how a simple computer model can be designed to embody many aspects of oral poetics, suggesting design directions for other work in oral improvisation and poetics. The opportunities for our system’s further development include creating culturally specific automated players; situating play in different temporal, physical, and social contexts; and constructing a more elaborate improvisor.

A Minimal Computational Improviser Based on Oral Thought

AI researchers have posited Dungeons and Dragons (D\&D) as a challenge problem to test systems on various language-related capabilities. In this paper, we frame D\&D specifically as a dialogue system challenge, where the tasks are to both generate the next conversational turn in the game and predict the state of the game given the dialogue history.  We create a gameplay dataset consisting of nearly 900 games, with a total of 7,000 players, 800,000 dialogue turns, 500,000 dice rolls, and 58 million words.  We automatically annotate the data with partial state information about the game play.  We train a large language model to generate the next game turn, conditioning it on different information.  The LM can respond as a particular character or as the player who runs the game—i.e., the Dungeon Master (DM). It is trained to produce dialogue that is either in-character (roleplaying in the fictional world) or out-of-character (discussing rules or strategy).  We perform a human evaluation to determine what factors make the generated output plausible and interesting.  We further perform an automatic evaluation to determine how well the model can predict the game state given the history and examine how well tracking the game state improves its ability to produce plausible conversational output.  

Dungeons and Dragons as a Dialogue Challenge for Artificial Intelligence

Text Worlds are virtual environments for embodied agents that, unlike 2D or 3D environments, are rendered exclusively using textual descriptions.  These environments offer an alternative to higher-fidelity 3D environments due to their low barrier to entry, providing the ability to study semantics, compositional inference, and other high-level tasks with rich action spaces while controlling for perceptual input. This systematic survey outlines recent developments in tooling, environments, and agent modeling for Text Worlds, while examining recent trends in knowledge graphs, common sense reasoning, transfer learning of Text World performance to higher-fidelity environments, as well as near-term development targets that, once achieved, make Text Worlds an attractive general research paradigm for natural language processing.

NAACL 2022 Workshop Wordplay Submissions