Deep Learning-Based Emotion Detection in Aphasia Patients

David Ortiz-Perez, Pablo Ruiz-Ponce, Javier Rodríguez-Juan, David Tomás, José García Rodríguez, Grzegorz J. Nalepa

Abstract: In this paper, we propose a pipeline for analyzing audio recordings of both aphasic and healthy patients. The pipeline can transcribe and distinguish between patients and the interviewer. To evaluate the pipeline’s effectiveness, we conducted a manual review of the initial frames of one hundred randomly selected samples and achieved a 94% accuracy in patient differentiation. This evaluation aimed to ensure accurate differentiation when analyzing frames where the clinician interacts with the patient. This differentiation is important, as the primary objective of this project is to examine patients’ emotions while they listen to their interviewer and identify patterns between healthy patients and those with aphasia. To achieve this, we used the AphasiaBank dataset, which includes video recordings of interviews with both aphasic and healthy patients. By combining the audio differentiation with the video recordings, we were able to analyze the facial expressions of patients while they listened to the speech of the interviewer. This analysis revealed a negative influence on the mood of aphasic patients. This negative influence stems from aphasic patients’ difficulty in correctly understanding and expressing speech.