Harnessing emotions for depression detection

Sahana Prabhu, Himangi Mittal, Rajesh Varagani, Sweccha Jha, Shivendra Singh

Abstract: Human emotions using textual cues, speech patterns, and facial expressions can give insight into their mental state. Although there are several uni-modal datasets for emotion recognition, there are very few labeled datasets for multi-modal depression detection. Uni-modal emotion recognition datasets can be harnessed, using the technique of transfer learning, for multi-modal binary emotion detection through video, audio, and text. We propose emotion transfer for mood indication framework based on deep learning to address the task of binary classification of depression using a one-of-three scheme: If the prediction from the network for at least one modality is of the depressed class, we consider the final output as depressed. Such a scheme is beneficial since it will detect an abnormality in any of the modalities and will alert a user to seek help well in advance. Long short-term memory is used to combine the temporal aspects of the audio and the video modalities, and the context of the text. This is followed by fine-tuning the network on a binary dataset for depression detection that has been independently labeled by a standard questionnaire used by psychologists. Data augmentation techniques are used for the generalization of data and to resolve the class imbalance. Our experiments show that our method for binary depression classification (using an ensemble of three modalities) on the Distress Analysis Interview Corpus—Wizard of Oz dataset has higher accuracy in comparison with other benchmark methods.