Using Linguistic Synchrony to Evaluate Large Language Models for Cognitive Behavioral Therapy
Abstract: Synchrony, the responsive communication between interacting individuals, is a crucial process in building a strong relationship between a mental health therapist and their client, leading to positive therapeutic outcomes. However, so far synchrony has not been investigated as a measure of efficacy of large language models (LLMs) delivering mental health therapy. In this work, we evaluate the linguistic synchrony of an LLM (ChatGPT 3.5-turbo) in a mental health dialog setting by first validating a computational measure of linguistic synchrony with two measures of the quality of client self-disclosures--intimacy and engagement ($p < 0.05$). We then compare the linguistic synchrony of the LLM to trained therapists and non-expert online peer supporters in a Cognitive Behavioral Therapy (CBT) setting. We show that the LLM is outperformed by humans with respect to linguistic synchrony ($p < 0.001$). These results support the need to be cautious in using LLMs in mental health applications.
Paper Type: Long
Research Area: Computational Social Science and Cultural Analytics
Research Area Keywords: human behavior analysis; human-computer interaction; NLP tools for social analysis; quantitative analyses of news and/or social media
Contribution Types: Data analysis
Languages Studied: English
Submission Number: 2878
Loading