ControlSwap: Controllable Personalized Subject Swapping

Seoyoung Chae, Kwangsu Kim

Keywords: diffusion models, image editing, personalized subject swapping

TL;DR: To address existing limitations of personalized subject swapping, ControlSwap achieves controllability—defined as foreground personalization and background preservation—via region-aware ControlLoss and an SDXL-based identity-preserving Identifiner.

Abstract: Personalized subject swapping aims to replace a source concept in a source image with a user-specified target concept while preserving the rest of the scene. Despite recent advances, two key controllability issues persist: (1) object-agnostic foreground processing impedes identity transfer under pose and context changes; and (2) background regions that should remain unchanged are unintentionally altered. This lack of controllability profoundly curtails expressivity and narrows real-world editing applicability. We introduce ControlSwap, an SDXL-based framework that redesigns the refiner into an identity-preserving Identifiner. In contrast to prior approaches that apply a uniform, object-agnostic loss across the entire image, ControlLoss decomposes optimization into object-specific and nonobject-specific objectives, enabling fine-grained, region-aware control. Experiments demonstrate improved controllability and background preservation across diverse personalization scenarios. Code and models will be released to support reproducibility and future research.

Primary Area: generative models

Submission Number: 5552