The Pursuit of Empathy: Evaluating Small Language Models for PTSD Dialogue Support

ACL ARR 2025 May Submission4589 Authors

Abstract: Can small language models (0.5B–5B parameters) meaningfully engage in trauma-informed, empathetic dialogue for individuals with PTSD? We answer this by introducing TIDE, a dataset of 10,000 two-turn dialogues across 500 diverse PTSD client personas, grounded in a three-factor empathy model: emotion recognition, distress normalization, and supportive reflection. All scenarios and reference responses were reviewed for realism and trauma sensitivity by a clinical psychologist specializing in PTSD. Eight small language models are evaluated before and after fine-tuning, with outputs compared to a frontier model (Claude Sonnet 3.5) as reference. Our IRB-approved human evaluation and automatic metrics reveal that, while fine-tuning generally improves perceived empathy, gains are highly scenario- and user-dependent, with smaller models facing an "empathy ceiling." Notably, demographic analyses show older adults value distress validation and graduate-educated users prefer nuanced replies, while gender effects are minimal. We highlight limitations of automatic metrics and the need for context- and user-aware system design. Our findings—along with the planned release of TIDE—offer a foundation for building safe, resource-efficient, and ethically sound empathetic AI to supplement, not replace, clinical mental health care.

Paper Type: Long

Research Area: Human-Centered NLP

Research Area Keywords: human-centered evaluation, dialogue systems, human-in-the-loop, ethical considerations in NLP applications, model bias/fairness evaluation, healthcare applications, generation, automatic evaluation, corpus creation

Contribution Types: Model analysis & interpretability, NLP engineering experiment, Approaches low compute settings-efficiency, Data resources, Data analysis

Languages Studied: English

Submission Number: 4589