PatchGame: Learning to Signal Mid-level Patches in Referential GamesDownload PDF

21 May 2021, 20:48 (edited 26 Oct 2021)NeurIPS 2021 PosterReaders: Everyone
  • Keywords: emergent language, referential games, self-supervised learning, mid-level patches
  • TL;DR: Emergent communication via mid-level patches in a referential game played on a large-scale image dataset
  • Abstract: We study a referential game (a type of signaling game) where two agents communicate with each other via a discrete bottleneck to achieve a common goal. In our referential game, the goal of the speaker is to compose a message or a symbolic representation of "important" image patches, while the task for the listener is to match the speaker's message to a different view of the same image. We show that it is indeed possible for the two agents to develop a communication protocol without explicit or implicit supervision. We further investigate the developed protocol and show the applications in speeding up recent Vision Transformers by using only important patches, and as pre-training for downstream recognition tasks (e.g., classification).
  • Supplementary Material: pdf
  • Code Of Conduct: I certify that all co-authors of this work have read and commit to adhering to the NeurIPS Statement on Ethics, Fairness, Inclusivity, and Code of Conduct.
  • Code: https://github.com/kampta/PatchGame
15 Replies

Loading