Designing Adaptive Virtual Health Assistants Using Cognitive and Emotional State Predictions

Navid Ashrafi, Julia Schorlemmer, Francesco Vona, Tanja Kojic, Stefan Hillmann, Ali Braytee, Yu-Kai Wang, A. Baki Kocaballi, Catarina Moreira, Sebastian Möller, Jan-Niklas Voigt-Antons

Abstract: Adapting users’ emotional states to virtual health assistance systems can enhance interaction quality and overall user experience. Virtual characters or avatars, can improve the usability of healthcare applications by making more engaging and personalized re- and interactions. Their appearance, visual characteristics, and voice influence how users perceive both the avatar and the system as a whole. By incorporating facial expressions, gestures, and verbal communication, avatars can take on human-like roles, such as providing counseling, to improve user satisfaction. This study presents the development of photorealistic virtual assistants using Unreal Engine’s Metahuman Creator. These avatars adapt their speech and facial expressions based on the user’s emotional states. The system synchronizes Metahuman avatars with an AI-powered chatbot, enabling the display of emotions such as happiness, excitement, boredom, and sadness. The avatars displayed on a monitor in landscape mode respond with appropriate expressions and voice tones to create a more natural and engaging conversation. To evaluate the system, we conducted a user study with 21 participants, measuring usability, social presence, emotional response, and trust. Our results in the ATI and UEQ questionnaires indicate that emotion-adaptive avatars significantly enhance experience quality and emotional engagement compared to standard virtual assistants. Future work will focus on automated facial expression detection in real-time and assessing the impact of the system on user engagement, well-being, and adherence to health recommendations.

External IDs:doi:10.1007/978-3-032-12764-8_22