MCP-Persona: Benchmarking LLM Agents on Real-World Personal Applications via Environment Simulation

WenHao Wang, Peizhi Niu, Gongyi Zou, Xiyuan Yang, Jingxing Wang, Haoting Shi, Yaxin Du, Jingyi Chai, Xianghe Pang, Shuo Tang, Yanfeng Wang, Siheng Chen

Abstract:

The Model Context Protocol (MCP) has emerged as a transformative standard for connecting large language models (LLMs) with external data sources and tools, and has been rapidly adopted across personal applications and development platforms.
However, existing benchmarks predominantly focus on generic information-seeking tools and fail to capture the practical challenges posed by personal social applications, where tools interact with individual accounts or local databases.
To bridge this critical gap, we introduce MCP-Persona, the first benchmark specifically designed for evaluating agent performance on real-world, personalized MCP tools. MCP-Persona encompasses a diverse set of widely-used applications, ranging from social media platforms like Reddit and Xiaohongshu (Rednote) to enterprise collaboration suites such as Lark (Feishu) and Slack. Our extensive experiments on various state-of-the-art (SOTA) agents demonstrate their significant struggles with personalized tool use, thereby highlighting the benchmark's crucial role in identifying and addressing these limitations.
MCP-Persona is publicly available at \href{https://github.com/wwh0411/MCP-Persona}{https://github.com/wwh0411/MCP-Persona}

Lay Summary:

AI assistants are increasingly being used to help with everyday digital tasks (sending emails, managing calendars, posting on social media, or organizing files). However, testing whether AI can handle these tasks well is difficult, because they involve private user data and account-specific information that is hard to share or reproduce in a research setting.

To address this, we built MCP-Persona, a testing platform that simulates 12 popular personal applications, including collaboration tools like Lark and social platforms like Rednote, without requiring access to real user accounts. The key to our simulation is a systematic traversal approach: we first deployed real applications in controlled environments, collected a broad set of authentic interactions, and used these observations to faithfully reconstruct how each application behaves. This allows our simulated tools to closely mirror their real-world counterparts while remaining fully reproducible and privacy-preserving.

Building on this foundation, we designed 173 realistic, human-verified tasks that mimic the kinds of requests real users make. Testing leading AI models on MCP-Persona reveals that even the best systems still struggle with personalized tool use, and we hope this platform helps the research community build smarter AI assistants for everyday life.

Link To Code: https://github.com/wwh0411/MCP-Persona

Primary Area: Deep Learning

Keywords: Large Language Model, Autonomous Agent, Model Context Protocol, Personalization, Personal Application

Submission Number: 8937