An LLM-powered Socially Interactive Agent with Adaptive Facial Expressions for Conversing about Health

Joaquin Molto, Jonathan Fields, Ubbo Visser, Christine Lisetti

Abstract: Virtual Socially Interactive Agents (SIA) have shown great promise for human interactions with computer applications in which not only domain-relevant content is needed, but also the way in which the content is delivered (e.g. socio-emotionally adaptive tutoring agents, socio-emotionally responsive health agents). While recent progress on Large Language Models (LLMs) has made rich verbal interactions possible, LLMs cannot communicate nonverbal social cues through a simple text-based interface. We propose an expressive conversational SIA system, powered by an OpenAI Large Language Model (LLM) for text generation, integrated with a 3D humanoid model with real-time behavior generation of FACS-based facial expressions that mirror the user’s to increase rapport and engagement using HumeAI’s Facial Expression Recognition and Empathic Voice Interface (EVI) models to drive the model’s animations. As a case study, we use prompt-engineering to focus the conversation on discussing health-related behaviors. We ground the generation of the LLM’s questions based on the World Health Organization’s (WHO) Alcohol Use Disorders Identification Test (AUDIT) 10-question inventory, a test that help identify whether someone is at risk of alcohol use disorder.

External IDs:doi:10.1145/3686215.3688378