Free-view Face Relighting using a Hybrid Parametric Neural Model on a SMALL-OLAT Dataset

Youjia Wang, Kai He, Taotao Zhou, Kaixin Yao, Nianyi Li, Lan Xu, Jingyi Yu

Abstract:

The development of neural relighting techniques has by far outpaced the rate of their corresponding training data (e.g., OLAT)
generation. For example, high-quality relighting from a single portrait image still requires supervision from comprehensive
datasets covering broad diversities in gender, race, complexion, and facial geometry. We present a hybrid parametric neural
relighting (PN-Relighting) framework for single portrait relighting, using a much smaller OLAT dataset or SMOLAT. At the
core of PN-Relighting, we employ parametric 3D faces coupled with appearance inference and implicit material modelling to
enrich SMOLAT for handling in-the-wild images. Specifically, we tailor an appearance inference module to generate detailed
geometry and albedo on top of the parametric face and develop a neural rendering module to first construct an implicit material
representation from SMOLAT and then conduct self-supervised training on in-the-wild image datasets. Comprehensive
experiments show that PN-Relighting produces comparable high-quality relighting to TotalRelighting (Pandey et al., 2021),
but with a smaller dataset. It further improves shape estimation and naturally supports free-viewpoint rendering and partial
skin material editing. PN-Relighting also serves as a data augmenter to produce rich OLAT datasets beyond the original
capture.