Language-guided World Models: A Model-based Approach to AI Control

Alex L Zhang, Khanh Xuan Nguyen, Jens Tuyls, Albert Lin, Karthik R Narasimhan

Keywords: world model, model-based reinforcement learning, language grounding

TL;DR: We develop world models that can be adapted with natural language. Integrating these models into artificial agents allows humans to effectively control these agents through verbal communication.

Abstract:

Developing internal world models for artificial agents opens an efficient channel for humans to communicate with and control them.
In addition to updating policies, humans can modify the world models of these agents in order to influence their decisions.
The challenge, however, is that currently existing world models are difficult for humans to adapt because they lack a natural communication interface. Aimed at addressing this shortcoming, we develop *Language-Guided World Models* (LWMs), which can capture environment dynamics by reading language descriptions. These models enhance agent communication efficiency, allowing humans to simultaneously alter their behavior on multiple tasks with concise language feedback. They also enable agents to self-learn from texts originally written to instruct humans. To facilitate the development of LWMs, we design a challenging benchmark based on the game of MESSENGER (Hanjie et al., 2021), requiring compositional generalization to new language descriptions and environment dynamics. Our experiments reveal that the current state-of-the-art Transformer architecture performs poorly on this benchmark, motivating us to design a more robust architecture. To showcase the practicality of our proposed LWMs, we simulate a scenario where these models augment the interpretability and safety of an agent by enabling it to generate and discuss plans with a human before execution. By effectively incorporating language feedback on the plan, the models boost the agent performance in the real environment by up to three times without collecting any interactive experiences in this environment.

Submission Type: Long Paper (8 Pages)

Archival Option: This is an archival submission

Submission Number: 1