Deep Image-based Illumination Harmonization

Zhongyun Bao, Chengjiang Long, Gang Fu, Daquan Liu, Yuanzhen Li, Jiaming Wu, Chunxia Xiao

Abstract:

Integrating a foreground object into a background scene
with illumination harmonization is an important but challenging task in computer vision and augmented reality community. Existing methods mainly focus on foreground and
background appearance consistency or the foreground object shadow generation, which rarely consider global appearance and illumination harmonization. In this paper,
we formulate seamless illumination harmonization as an
illumination exchange and aggregation problem. Specifically, we firstly apply a physically-based rendering method
to construct a large-scale, high-quality dataset (named IH)
for our task, which contains various types of foreground objects and background scenes with different lighting conditions. Then, we propose a deep image-based illumination
harmonization GAN framework named DIH-GAN, which
makes full use of a multi-scale attention mechanism and illumination exchange strategy to directly infer mapping relationship between the inserted foreground object and the corresponding background scene. Meanwhile, we also use adversarial learning strategy to further refine the illumination
harmonization result. Our method can not only achieve harmonious appearance and illumination for the foreground
object but also can generate compelling shadow cast by
the foreground object. Comprehensive experiments on both
our IH dataset and real-world images show that our proposed DIH-GAN provides a practical and effective solution
for image-based object illumination harmonization editing,
and validate the superiority of our method against state-ofthe-art methods.