A Personalized Conversational Benchmark: Towards Simulating Personalized Conversations

ACL ARR 2026 January Submission697 Authors

Keywords: Benchmark, Conversation, Personalization

Abstract:

We present PersonaConvBench, a large-scale benchmark for evaluating personalized reasoning and generation in multi-turn conversations with large language models (LLMs).
Unlike existing work that focuses on personalization or conversational structure in isolation, PersonaConvBench tightly integrates both, offering three core tasks: sentence classification, impact regression, and user-centric text generation, covering 10 diverse Reddit-based domains.
This design enables systematic analysis of how personalized conversational context can shape LLM outputs in realistic, multi-user conversational scenarios.
We systematically benchmark several commercial and open-source LLMs under a unified prompting setup, and observe that incorporating personalized conversational history yields substantial performance boosts—e.g.,
achieving a 198% relative gain over the best non-conversational baseline in sentiment classification.
By releasing PersonaConvBench with comprehensive evaluations and codes, we aim to facilitate research on LLMs that can adapt to individuals’ conversational styles, track long-term context, and generate more contextually rich and engaging responses.

Paper Type: Long

Research Area: Dialogue and Interactive Systems

Research Area Keywords: Benchmark,Conversation,Personalization

Languages Studied: English

Submission Number: 697