SpEmoC: Large-Scale Multimodal Dataset for Speaking Segment Emotion Insights

ICLR 2026 Conference Submission18732 Authors

Keywords: Multimodality, Audio, Vision, Language, Emotion Recognition

Abstract: Understanding human emotions in spoken conversations is a key challenge in affective computing, with applications in empathetic AI, human-computer interaction, and mental health monitoring. Existing datasets lack scale, tightly aligned modalities, and balance in emotion diversity thereby limiting robust multimodal models. To address this, we propose \textbf{SpEmoC}, a large-scale \textbf{Sp}eaking segment \textbf{Emo}tion dataset for \textbf{C}onversations. SpEmoC comprises 306,544 clips from 3,100 English-language videos, featuring synchronized visual, audio, and textual modalities annotated for seven emotions, and yields a refined set of 30,000 high-quality clips. It focuses on speaking segments under diverse conditions like low lighting and resolution, with a threshold-based filtering and human annotation ensuring a balanced dataset. SpEmoC is class-balanced, which enables fair learning across all emotions and leads to comparably balanced performance across all classes. We introduce a lightweight CLIP-based baseline model with a fusion network and a novel multimodal contrastive loss to enhance emotion alignment. We conduct a series of experiments demonstrating strong results, establishing SpEmoC as a reliable benchmark for advancing multimodal emotion recognition research.

Primary Area: datasets and benchmarks

Submission Number: 18732