Entering Real Social World! Systematic Framework for Evaluating LLMs' Social Intelligence from First-person Perspective

ACL ARR 2025 May Submission6571 Authors

Abstract: Social intelligence is built upon three foundational pillars: cognitive, situational, and behavioral intelligence. As Large Language Models (LLMs) are increasingly integrated into our social lives, understanding, evaluating, and developing their social intelligence are becoming increasingly important. While multiple existing works have investigated the social intelligence of LLMs: (1) most focus on a single pillar, while a comprehensive framework for organizing and studying the social intelligence of LLMs remains underdeveloped; (2) position LLMs as passive observers from a third-person perspective. Compared to the third-person perspective, ego-centric first-person perspective evaluation can align well with actual LLM-based Agent use scenarios; (3) a lack of comprehensive evaluation of behavioral intelligence, with specific emphasis on a more intuitive comparison of behavioral differences between humans and LLMs. In light of these, we introduce the EgoSocialArena framework, built upon the three foundational pillars of social intelligence - cognitive, situational, and behavioral intelligence, with each pillar supported by novel and systematic evaluation design. Using EgoSocialArena, we conduct a comprehensive evaluation of eight prominent foundation models. Our findings show that even the advanced LLMs, such as o1-preview, still fall significantly behind human performance.

Paper Type: Long

Research Area: Resources and Evaluation

Research Area Keywords: evaluation methodologies

Languages Studied: English

Submission Number: 6571