SSH: A Self-Supervised Framework for Image Harmonization

Yifan Jiang, HE Zhang, Jianming Zhang, Zhangyang Wang

Abstract:

Image harmonization aims to improve the quality of image compositing by matching the “appearance” (e.g., color
tone, brightness and contrast) between foreground and
background images. However, collecting large-scale annotated datasets for this task requires complex professional
retouching. Instead, we propose a novel Self-Supervised
Harmonization framework (SSH) that can be trained using just “free” natural images without being edited. We
reformulate the image harmonization problem from a representation fusion perspective, which separately processes
the foreground and background examples, to address the
background occlusion issue. This framework design allows for a dual data augmentation method, where diverse
[foreground, background, pseudo GT] triplets can be generated by cropping an image with perturbations using 3D
color lookup tables (LUTs). In addition, we build a realworld harmonization dataset as carefully created by expert
users, for evaluation and benchmarking purposes. Our results show that the proposed self-supervised method outperforms previous state-of-the-art methods in terms of reference metrics, visual quality, and subject user study. Code
and dataset are available at https://github.com/
VITA-Group/SSHarmonization.