CFDiffusion: Controllable Foreground Relighting in Image Compositing via Diffusion Model

Ziqi Yu, Jing Zhou, Zhongyun Bao, Gang Fu, Weilei He, Chao Liang, Chunxia Xiao

Abstract: Inserting foreground objects into specific background scenes and eliminating the gap between them is an important and challenging task. It typically involves multiple processing tasks, such as image harmonization and shadow generation, which find numerous applications across various fields including computer vision and augmented reality. In these two domains, there are already many mature solutions, but they often only focus on one of the tasks. Some image composition methods can address both of these issues simultaneously but cannot guarantee complete reconstruction of foreground content. In this work,we propose CFDiffusion, which can handle both image harmonization and shadow generation simultaneously. Additionally, we introduce a foreground content enhancement module based on the diffusion model to ensure the complete preservation of foreground content at the insertion location. The experimental results on the iHarmony4 dataset and our self-created IH-SG dataset demonstrate the superiority of our CFDiffusion approach.

Primary Subject Area: [Generation] Generative Multimedia

Relevance To Conference: Inserting foreground objects into specific background scenes and eliminating the gap between them is an important and challenging task. It typically involves multiple processing tasks, such as image harmonization and shadow generation, which find numerous applications across various fields including computer vision and augmented reality. In these two domains, there are already many mature solutions, but they often only focus on one of the tasks. Some image composition methods can address both of these issues simultaneously but cannot guarantee complete reconstruction of foreground content. In this work,we propose CFDiffusion, which can handle both image harmonization and shadow generation simultaneously. Additionally, we introduce a foreground content enhancement module based on the diffusion model to ensure the complete preservation of foreground content at the insertion location. The experimental results on the iHarmony4 dataset and our self-created IH-SG dataset demonstrate the superiority of our CFDiffusion approach.

Submission Number: 3227