FLNERF: 3D FACIAL LANDMARKS ESTIMATION IN NEURAL RADIANCE FIELDS

Hao ZHANG, Tianyuan DAI, Yu-Wing Tai, Chi-Keung Tang

Primary Area: representation learning for computer vision, audio, language, and other modalities

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: NeRF; 3D Face Landmarks Detection; NeRF editting

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Abstract: This paper presents the first significant work on directly predicting 3D face landmarks on neural radiance fields (NeRFs). This direct NeRF approach is shown to surpass existing single or multi-view image approaches. Our 3D coarse-to-fine Face Landmarks FLNeRF model efficiently samples from a given face NeRF individual facial features for accurate landmarks detection. Expression augmentation is applied at facial features in fine scale to simulate large emotions range including exaggerated facial expressions (e.g., cheek blowing, wide opening mouth) for training FLNeRF. Qualitative and quantitative comparison with related state-of-the-art 3D facial landmark estimation methods demonstrate the efficacy of FLNeRF, which contributes to downstream tasks such as high-quality face editing and swapping with direct control using our NeRF landmarks. Code and data will be available.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 1671