Leveraging Word Guessing Games to Assess the Intelligence of Large Language Models
Keywords: Multi-Agent Intelligence, Large Language Models, Artificial General Intelligence
TL;DR: An interactive multi-agent framework to evaluate intelligence in large language models
Abstract: The automatic evaluation of LLM-based agent intelligence is critical in developing advanced LLM-based agents. Although considerable effort has been devoted to developing human-annotated evaluation datasets, such as AlpacaEval, existing techniques are costly, time-consuming, have limited scalability, and lack adaptability. In this paper, inspired by the popular language games ``Who is Spy'' and ``SpyFall'', we propose to use the word guessing game to assess the intelligence performance of LLMs. Given a word, the LLM is asked to describe the word and determine its identity (spy or not) based on its and other players' descriptions. Ideally, an advanced agent should possess the ability to accurately describe a given word using an aggressive description while concurrently maximizing confusion in the conservative description, enhancing its participation in the game. To this end, we first develop SpyGPT to evaluate LLMs' expression and disguising abilities. SpyGPT requires the target LLM to describe the given word in aggressive and conservative modes and utilizes the SOTA GPT-4 to determine whether the descriptive sentences can accurately describe the given word. We then introduce SpyGame, an interactive multi-agent framework designed to assess LLMs intelligence through participation in a competitive language-based board game. Incorporating multi-agent interaction, SpyGame requires the target LLM to possess linguistic skills and strategic thinking, providing a more comprehensive evaluation of LLMs' human-like cognitive abilities and adaptability in complex communication situations. The proposed evaluation framework is very easy to implement. We collected words from multiple sources, domains, and languages and used the proposed evaluation framework to conduct experiments. Extensive experiments demonstrate that the proposed SpyGPT and SpyGame effectively evaluate the capabilities of various LLMs, capturing their ability to adapt to novel situations and engage in strategic communication.
Primary Area: generative models
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 3201
Loading