AU-aware Algorithm for 3D Facial Reconstruction

Yi Zhang, Jiayi Lü, Xing Lan, Jian Xue

Abstract: As a critical task in computer vision and animation, 3D face reconstruction can provide 3D model structures and rich semantic information for multi-modal facial applications. However, monocular 2D facial images lack depth information and the parameters of the predicted facial model are not reliable, which causes poor reconstruction results. We propose to employ facial Action Unit (AU) and key facial points which are highly correlated with model parameters as a bridge to guide the regression of model-related parameters and thus solve the ill-posed monocular face reconstruction. Based on existing face reconstruction datasets, we provide a complete semi-automatic labeling scheme for facial AUs and construct a 300W-LP-AU dataset. Furthermore, a 3D face reconstruction algorithm based on AU awareness is put forward to realize end-to-end multi-tasking learning and reduce the overall training difficulty. Experimental results show that the algorithm can improve the face reconstruction performance, with high fidelity of the rebuilt facial model.