Imitation or Communication? Examining Language Models with Cooperative Context and Language Development
Keywords: communication, language model
Abstract: With the rapid development of natural language processing, current researchers transited from previous task-specific models to the more versatile, unified capabilities exhibited by GPT-like models. While these models achieve fluency in natural language and excels at coding, this paper argues that a crucial aspect remains underexplored: the process of language learning and development in these models. We propose a novel framework to evaluate language models, not just on linguistic proficiency but on their ability to transfer communication knowledge to new languages rapidly like humans, under cooperative contexts. Central to this framework is the idea that genuine communication in language models should be grounded in tasks that require mutual help and involve modalities beyond the language used, and these models should display the ability to rapidly learn and develop new languages, akin to human capabilities. This is explored through a referential game and evaluated for different stages that mirrors human behaviour, from symbol/icon grounding to symbol simplification and, finally, the emergence of systematicity. This framework aims to shift the current paradigm in language model evaluation, emphasizing the importance of interactive, adaptive, and contextually aware communication abilities.
Submission Number: 127
Loading