DepSy: A Dataset and Benchmark for Depression Symptoms Detection with Hierarchical Transformers and Fine-Tuned LLMs

ACL ARR 2025 May Submission5182 Authors

Abstract: Early detection of symptoms of depression can help minimise its impact on people suffering from depression. Social media, where users often share emotions and life experiences, offers a valuable resource for NLP-driven mental health research. We posit that mining social media posts enables researchers to identify clinically significant depressive symptoms. This paper introduces: a) the DepSy dataset, a novel resource annotated by psychologists for depressive symptoms, containing over 40k posts; and b) the DepSy model, a fine-tuned model trained to identify and extract depressive symptoms. We conducted comparative experiments between BERT-based models and large language models (LLMs) for symptom extraction. Our results show that both BERT-based models and LLMs demonstrated comparable performance, with BERT achieving the highest overall f-1 score of 0.522.

Paper Type: Long

Research Area: Resources and Evaluation

Research Area Keywords: Psychologist-annotated dataset, Depression symptoms, Hierarchical BERT

Contribution Types: Model analysis & interpretability, Publicly available software and/or pre-trained models, Data resources, Data analysis

Languages Studied: English

Submission Number: 5182