TACL: A Trusted Action-enhanced Curriculum Learning Approach to Multimodal Affective Computing

Tan Yu, Jingjing Wang, Jiamin Luo, Jiawen Wang, Guodong Zhou

Abstract: Previous studies on Multimodal Affective Computing (MAC) predominantly focus on leveraging language, acoustic, and facial information to identify human’s affective states, which largely ignore the dynamic and temporal action information, despite such information being crucial for precisely inferring the affective states. In this way, this paper first attempts to consider the action information for MAC and further argues that exploiting the action information faces two key challenges, i.e., credibility and sparsity challenges. To this end, this paper proposes a new Trusted Action-enhanced Curriculum Learning (TACL) approach to incorporate the action information for boosting MAC. Specifically, this approach designs two main components, i.e., the Trusted Curriculum Learning block and the Action-enhanced Vision Regulator, to address the above credibility challenge and sparsity challenge. Furthermore, a high-quality action-enhanced video dataset is constructed to evaluate TACL and detailed evaluations show the great advantage of TACL over the state-of-the-art baselines. Particularly, an interesting finding is observed that action information is more conducive to facilitating the recognition of negative emotions, which aligns with the intuition that humans prefer using actions more when expressing negative emotions.

External IDs:dblp:journals/ijon/YuWLWZ25