Emotion Expression Estimates to Measure and Improve Multimodal Social-Affective Interactions

Jeffrey A. Brooks, Vineet Tiruvadi, Alice Baird, Panagiotis Tzirakis, Haoqi Li, Chris Gagne, Moses Oh, Alan Cowen

Abstract: Large language models (LLMs) are being adopted in a wide range of applications, but an understanding of other social-affective signals is needed to support effective human-computer-interaction (HCI) in multimodal interfaces. In particular, robust, accurate measurements of human emotional expression can be used to tailor responses to human values and preferences. In this paper, we present two models available from an API-based suite of emotional expression models that measure nuanced facial and vocal signals, providing rich, high-dimensional emotional expression estimates (EEEs). We demonstrate the ability of EEEs to provide insight into two established datasets and present methods for integrating EEEs into large language model (LLM) applications. We discuss how this approach is a step towards more reliable tools for clinical screening and scientific study, as well as empathic digital assistants that can be used in therapeutic settings.