A preliminary study of interpretable emergent language in visual referential games

Yuran Xiang, Yiheng Du, Haoran Sun

Keywords: Nonverbal Communication

TL;DR: We design a visual referential game combining the ideas of the binary referential game and the sketch drawing game.

Abstract: Deep neural networks have achieved high accuracy in many computer vision and natural language processing tasks through their complicated structures and operations. But their formulation is numerical and the computation process cannot be interpreted by humans directly. Therefore they are often criticized for their inability to align with human language. In this project, we aim at discovering emergent language and how it evolves in multi-agent communication games. We design a visual referential game combining the ideas of the referential game (Lazaridou et al. [11]) and the sketch drawing game (Qiu et al. [16]). We consider both continuous and discrete communication types and propose a model which can be efficiently optimized by back-propagation. Experiments conducted on MNIST and CIFAR-10 have shown the efficiency and interpretability of this communication, with t-SNE of visual embedding as visualization. Our code is available at https://github.com/ran1812/pkucore-nonverbal_communication. Our video is available at https://disk.pku.edu.cn:443/link/5C38F27C4323A6AE8B28389A5C851212