Stress-Testing Emotional Support Models: Moving from Homogeneous to Diverse Help Seekers

Chaewon Heo, Cheyon Jin, Yohan Jo

Keywords: Emotional support chatbots, AI evaluation, culturally situated evaluation, help-seeker simulation, population-diverse evaluation, context-sensitive AI, conversational agents, mental health support, interpretive AI evaluation, human-centered AI

TL;DR: We introduce a controllable, population-diverse help-seeker simulator that stress-tests emotional support chatbots under varied psychological and interactional profiles, enabling more context-sensitive and diagnostic AI evaluation.

Abstract: As emotional support chatbots have recently gained significant traction across both research and industry, a common evaluation strategy has emerged: use help-seeker simulators to interact with supporter chatbots. However, current simulators suffer from two critical limitations: (1) they fail to capture the behavioral diversity of real-world seekers, often portraying them as overly cooperative, and (2) they lack the controllability required to simulate specific seeker profiles. To address these challenges, we present a controllable seeker simulator driven by nine psychological and linguistic features that underpin seeker behavior. Using authentic Reddit conversations, we train our model via a Mixture-of-Experts (MoE) architecture, which effectively differentiates diverse seeker behaviors into specialized parameter subspaces, thereby enhancing fine-grained controllability. Our simulator achieves superior profile adherence and behavioral diversity compared to existing approaches. Furthermore, evaluating 7 prominent supporter models with our system uncovers previously obscured performance degradations, underscoring the utility of our framework for more faithful and stress-tested evaluation of emotional support chatbots.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 53