What Drives Long-term User Engagement? A Large-scale Analysis of Role-playing Dialogues in the Wild

ACL ARR 2024 December Submission1917 Authors

Abstract: The growing sophistication of dialogue agents has led to extended human-AI conversations spanning days or months, yet our understanding of what sustains user engagement comes primarily from short-term simulations. Through large-scale analysis of real-world interactions between 37,000+ users and 8,000+ characters, we systematically evaluate nine engagement factors via rigorous A/B testing with 53 model pairs. Our findings challenge prevailing assumptions: while character embodiment shows limited influence, response length and non-verbal descriptions emerge as critical factors. We also find that human-likeness and personality consistency significantly impact engagement, while factors like lexical diversity show minimal effect. These insights, derived from real user interactions, provide actionable guidance for developing more engaging role-playing models by prioritizing response depth over strict role adherence.

Paper Type: Long

Research Area: Resources and Evaluation

Research Area Keywords: evaluation and metrics, dialogue

Contribution Types: Data analysis

Languages Studied: English

Submission Number: 1917