Information-Theoretic Underpinnings of Generalization and Translation in Emergent Communication

Mycal Tucker, Julie Shah, Roger P. Levy, Noga Zaslavsky

Keywords: Emergent Communication, Information Theory

TL;DR: Controlling emergent communication complexity and informativeness allows agents to generalize better and understand translations of natural language

Abstract: Traditional emergent communication (EC) methods often fail to generalize to novel settings or align with representations of natural language. While these limitations may at first appear unrelated, in this work, we show how controlling the Information Bottleneck (IB) tradeoff between complexity and informativeness (a principle thought to guide human languages) helps to address both of these problems in EC. Specifically, we build on VQ-VIB, a recently proposed method for training EC agents while controlling the IB tradeoff, in addition to maximizing agents' utility. We find that increasing informativeness, which is a task-agnostic measure of how well a listener can reconstruct a speaker's meaning, allows EC agents to better generalize to novel settings and more challenging tasks. At the same time, in translation experiments between EC and English, we find that increasing EC informativeness only improves team performance up to a certain threshold, corresponding to the English informativeness-complexity tradeoff. Jointly, our results indicate the importance of training EC systems while controlling the informativeness-complexity tradeoff to simultaneously support improved self-play performance and human-agent interaction.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Submission Guidelines: Yes

Please Choose The Closest Area That Your Submission Falls Into: Deep Learning and representational learning