Towards Generalizable LLM Multi-Agent System: Identifying Collective Intelligence Factor in LLM Agent Groups

Zhilun Zhou, Zihan Liu, Jiahe Liu, Yihan Wang, Qingyu Shao, Fengli Xu, Depeng Jin, Yong Li

Keywords: Large Language Model, Multi-agent System, Collective Intelligence, Cognitive Psychology

Abstract:

Large language models (LLM)-based multi-agent systems (MAS) have shown impressive performance in solving a wide range of complex problems. However, previous studies mainly focus on designing customized MAS for specific tasks, while a critical research problem remains unclear: Do LLM agent groups exhibit a form of ``general intelligence'' that reflects their general ability across various tasks?
In human cognitive psychology research, it has been established that the mental capabilities of a human group can be measured by a single statistical factor, known as the Collective Intelligence (CI) factor. This factor can capture the group's general capability and predict its performance on a wide range of tasks, much like how IQ scores capture the general cognitive ability of individuals.
Inspired by this, in this study, we aim to investigate whether an analogous CI factor also exists in LLM agent groups, which is crucial for building generalizable MAS.
Motivated by human cognitive psychology experiments, we design experiments along three dimensions: group size, individual intelligence, and collaboration process. Specifically, we construct 108 LLM agent groups with diverse group sizes, LLM compositions, and communication topologies. These groups are systematically evaluated across a wide range of tasks, including commonsense reasoning, math, game, etc.
Our results demonstrate that an Artificial Collective Intelligence (ACI) factor does exist in LLM agent groups, accounting for 66.3\% of the variance in performance across different tasks, which is substantially higher compared with the 43\% observed in human groups. Moreover, by analyzing the indicators of groups that affect ACI, we find similar patterns between the ACI of LLM agent and human groups, where the collaboration process is the most important indicator influencing ACI rather than the individual intelligence of group members.
This highlights that, for MAS design, the way agents are connected and interact has a greater impact on overall performance than the scale of individual models, offering practical guidance for building more efficient and generalizable MASs.
Our code is open-source at \url{https://anonymous.4open.science/r/LLM_Collective_Intelligence-71B3} for reproducibility.

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 10188