Beyond Natural Language: Invented Communication in Vision-Language Models

Boaz Carmeli, Orr Paradise, Shafi Goldwasser, Yonatan Belinkov, Ron Meir

Keywords: NLP, Invented Language, Multi-modal NLP

TL;DR: Inventing efficient and covert languages by visual language models using referential games

Abstract:

We investigate whether LLM-based agents can invent communication protocols that rival, or even surpass, natural language in collaborative intelligence tasks.

Our focus is on two core properties such invented languages may exhibit:
*Efficiency*---conveying task-relevant information more concisely than natural language, and
*Covertness*---remaining unintelligible to external observers, raising concerns about transparency and control.
These two properties respectively represent potential benefits and risks of AI agents inventing languages.

To investigate these aspects, we use a referential-game framework in which vision-language model (VLM) agents communicate, providing a controlled, measurable setting for evaluating invented communication.

Experiments show that VLMs can develop effective communication. At the same time, they can invent covert protocols that are inaccessible to humans and external agents. We also observe spontaneous coordination between similar models without explicitly shared protocols.

These findings highlight both the promise and the risks of LLM-based invented languages, and position referential games as a valuable testbed for future work in this area.

Primary Area: generative models

Submission Number: 6961