The Curious Case of Representational Alignment: Unravelling Visio-Linguistic Tasks in Emergent Communication
Track: long paper (up to 9 pages)
Keywords: Representational alignment, Emergent communication, Compositionality, Reinforcement Learning
TL;DR: This work revisits inter-agent representational alignment in simulations of emergent communication. We find that inter-agent alignment interacts with topographic similarity and introduce a solution to the alignment problem.
Abstract: Natural language has the universal properties of being compositional and grounded in the real world. A popular method to investigate the emergence of linguistic properties is by simulating emergent communication setups with deep neural agents in referential games. Despite growing interest, experiments have yielded mixed results compared to similar experiments addressing linguistic properties of human language. Here we address representational alignment as a potential contributing factor to these results. Specifically, we investigate the alignment between agent image representations and between agent representations and the input images. We first revisit and confirm that the emergent language in the common referential game does not appear to encode conceptual visual features, since agent image representations drift away from the input whilst inter-agent alignment increases. We further find a strong relationship between inter-agent alignment and topographic similarity, a common metric for compositionality, and address its consequences. We then introduce an alignment penalty that results in equivalent communicative success but prevents representational drift. Overall, we show critical differences between emergent solutions from humans and neural agents and highlight the importance of representational alignment in simulations of language emergence.
Anonymization: This submission has been anonymized for double-blind review via the removal of identifying information such as names, affiliations, and identifying URLs.
Submission Number: 86
Loading