Semi-Supervised Contrastive Learning with Soft Mask Attention for Facial Action Unit Detection

Zhongling Liu, Rujie Liu, Ziqiang Shi, Liu Liu, Xiaoyu Mi, Kentaro Murase

Abstract: This paper presents a novel facial action unit (AU) detection method by simultaneously improving AU feature’s discriminative ability and alleviating the AU data scarcity problem. We design a supervised AU soft mask attention scheme to learn local AU features by integrating prior expert knowledge. To further improve the discriminativeness of AU features, contrastive learning is introduced in both instance-level and prototype-level for each AU. For the data scarcity problem, prototypical pseudo label assignment method is proposed in order to make the potential of unlabeled data, where pseudo-labels are assigned to unlabeled data based on the prototypes of each AU. Overall, our semi-supervised contrastive learning approach employs region learning, contrastive learning and pseudo labeling jointly to enhance the discriminativeness of AU features in the feature space and improve the generalization ability of the model. The effectiveness of the proposed method has been verified by the experiments on benchmark datasets BP4D and DISFA, achieving the state-of-the-art F1-scores of 64.1% and 64.2% respectively.