DECK: Behavioral Tests to Improve Interpretability and Generalizability of BERT Models Detecting Depression from Text

Anonymous

Abstract: Models that accurately detect depression from text are important tools for addressing the post-pandemic mental health crisis. BERT-based classifiers' promising performance and the off-the-shelf availability make them great candidates for this task. However, these models are known to suffer from performance inconsistencies and poor generalization. In this paper, we introduce the DECK (\textbf{DE}pression \textbf{C}hec\textbf{K}list), depression-specific model behavioral tests that allow better interpretability and improve generalizability of BERT classifiers in depression domain. We create 23 tests to evaluate BERT, RoBERTa and ALBERT depression classifiers on three datasets, two Twitter-based and one clinical interview-based. Our evaluation shows that these models: 1) are robust to certain gender-sensitive variations in text; 2) rely on important depressive language marker of the increased use of first person pronouns; 3) fail to detect some other depression symptoms like suicidal ideation. We also demonstrate that DECK tests can be used to incorporate symptom-specific information in the training data and consistently improve generalizability of all three BERT models, with the out-of-distribution F1-score increase of up to 53.93\%. The DECK tests, together with the associated code, are available for download at https://github.com/Anonymous.

Paper Type: long