Emo3D: Translating Emotion Descriptions to 3D Facial Expressions, \\ Benchmark Dataset and Evaluation Metrics

Anonymous

Abstract: Exisiting 3D facial emotion modeling have been constrained by limited emotion classes and insufficient datasets. This paper introduces ``Emo3D'', an extensive ``Text-Image-Expression dataset'' spanning a wide spectrum of human emotions, each paired with images and 3D blendshapes. Leveraging advanced Language Models (LLMs), we generate a diverse array of textual descriptions, facilitating the capture of a broad spectrum of emotional expressions. Using this unique dataset, we conduct a comprehensive evaluation of language model fine-tuning and CLIP-based models for 3D facial expression synthesis. We also introduce a new evaluation metric for this task to more directly measure the conveyed emotion. Our new evaluation metric, Emo3D, demonstrates its superiority over Mean Squared Error (MSE) metrics in assessing visual-text alignment and semantic richness in 3D facial expressions associated with human emotions. ``Emo3D'' has great applications in animation design, virtual reality, and emotional human-computer interaction.

Paper Type: short

Research Area: Multimodality and Language Grounding to Vision, Robotics and Beyond

Contribution Types: Model analysis & interpretability, Publicly available software and/or pre-trained models, Data resources, Data analysis

Languages Studied: English