Reference-Guided Identity Preserving Face Restoration

Mo Zhou, Keren Ye, Viraj Shah, Kangfu Mei, Mauricio Delbracio, Peyman Milanfar, Vishal M. Patel, Hossein Talebi

Keywords: Reference-Based Face Restoration, Composite Context, Hard Identity Loss, Multi-reference face inference

TL;DR: We propose a reference-based face restoration method to fully leverage the reference face and hence achieves the state-of-the-art identity preservation.

Abstract:

Preserving face identity is a critical yet persistent challenge in diffusion-based
image restoration. While reference faces offer a path forward, existing methods
typically suffer from partial reference information and inefficient identity losses.
This paper introduces a novel approach that directly solves both issues, involving
three key contributions: 1) Composite Context, a representation that fuses high- and
low-level facial information to provide comprehensive guidance than traditional
singular representations, 2) Hard Example Identity Loss, a novel loss function
that uses the reference face to address the identity learning inefficiencies of the
standard identity loss, 3) Training-free multi-reference inference, a new method
that leverages multiple references for restoration, despite being trained with only a
single reference. The proposed method demonstrably restores high-quality faces
and achieves state-of-the-art identity preserving restoration on benchmarks such as
FFHQ-Ref and CelebA-Ref-Test, consistently outperforming previous work.

Primary Area: generative models

Submission Number: 2179