Generative Facial Expressions and Eye Gaze Behavior from Prompts for Multi-Human-Robot Interaction

Gabriel J. Serfaty, Virgil O. Barnard, Joseph P. Salisbury

Abstract: Nonverbal cues such as eye gaze and facial expressions play critical roles in conveying intent, regulating conversation, and fostering engagement. A robot's ability to effectively deploy these behaviors can significantly enhance human-robot collaboration. We describe a simple zero-shot learning approach to generate facial expression and gaze shifting behaviors to control a social robot conversing with an individual or group. An initial prompt provides instructions to a pre-trained large language model on how the model can control a robot's facial expression and eye gaze behaviors during a conversation. To demonstrate this, we describe a proof-of-concept implementation using the robot Furhat. This simple and easily customizable approach can be used to improve perception of a robot's social presence in multi-human-robot interactions.