Policy Learning with a Language Bottleneck

Megha Srivastava, Cédric Colas, Dorsa Sadigh, Jacob Andreas

Track Selection: Full paper track.

Keywords: feedback, bottleneck, agents, cognitive science, reflection, self-talk

TL;DR: RL agents that periodically pause to summarize their behavior in natural language learn more generalizable behavior.

Abstract: Modern AI systems such as self-driving cars and game-playing agents achieve superhuman performance, but often lack human-like features such as generalization, interpretability and interoperability with humans. Inspired by the rich interactions between language and decision-making in humans, we introduce Policy Learning with a Language Bottleneck (PLLB), a framework enabling AI agents to generate linguistic rules that capture the strategies underlying their most rewarding behaviors. (PLLB), alternates between a rule generation step guided by language models, and an update step where agents learn new policies guided by rules. In a two-player communication game, a maze solving task, and two image reconstruction tasks, we show that (PLLB), agents are not only able to learn more interpretable and generalizable behaviors, but can also share the learned rules with human users, enabling more effective human-AI coordination.

Submission Number: 6