OmniPortrait: Fine-Grained Personalized Portrait Synthesis via Pivotal Optimization

Dongxu Yue, Bo Lin, Yao Tang, Jiajun Liang, Zhihai He, Chun Yuan

Keywords: Personalized Portrait Synthesis, ImageGeneration, High-Fidelity Facial Details, Pivotal Optimization

TL;DR: By leveraging pivotal optimization, identity-customized portrait synthesis is achieved with highly faithful fine facial details while preserving editability.

Abstract:

Image identity customization aims to synthesize realistic and diverse portraits of a specified identity, given a reference image and a text prompt. This task presents two key challenges: (1) generating realistic portraits that preserve fine-grained facial details of the reference identity, and (2) maintaining identity consistency while achieving strong alignment with the text prompt. Our findings suggest that existing single-stream methods fail to capture and guide fine-grained identity details.
To address these challenges, we introduce \textit{OmniPortrait}, a novel diffusion-based framework for fine-grained identity fidelity and high editability in portrait synthesis. Our core idea is pivotal optimization, which leverages dual-stream identity guidance in a coarse-to-fine manner. First, a Pivot ID Encoder is proposed and trained with a face localization loss while avoiding the degradation of editability typically caused by fine-tuning the denoiser. Although this encoder primarily guides coarse-level identity synthesis, it provides a good initialization that serves as the identity pivot for optimization during inference.
Second, we propose Reference-Based Guidance, which performs on-the-fly feature matching and optimization over diffusion intermediate features conditioned on the identity pivot. In addition, our approach is able to generalize naturally to multi-identity customized image generation scenarios. Extensive experiments demonstrate significant improvements in both identity preservation and text alignment, establishing a new benchmark for image identity customization.

Primary Area: generative models

Submission Number: 5817