MI-NeRF: Learning a Single Face NeRF from Multiple Identities

Aggelina Chatziagapi, Grigorios Chrysos, Dimitris Samaras

Primary Area: representation learning for computer vision, audio, language, and other modalities

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: neural radiance fields, face, identity, expression

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Abstract:

In this work, we introduce a method that learns a single dynamic neural radiance field (NeRF) from monocular talking face videos of multiple identities. NeRFs have shown remarkable results in modeling the 4D dynamics and appearance of human faces. However, they require expensive per-identity optimization. To address this challenge, we introduce MI-NeRF (multi-identity NeRF), a single unified network that models complex non-rigid facial motion for multiple identities, using only monocular videos of arbitrary length. The core premise in our method is to learn the non-linear interactions between identity and non-identity specific information with a multiplicative module. By training MI-NeRF on multiple videos simultaneously, we significantly reduce the total training time, compared to standard single-identity NeRFs. Our model can be further personalized for a target identity. We demonstrate results for both facial expression transfer and talking face video synthesis.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 5887