Outdoor Scene Relighting with Diffusion Models

Jinlin Lai, Anustup Choudhury, Guan-Ming Su

Abstract: Lighting is an environmental condition applied to objects in a scene, which creates complicated lighting effects including shading, shadow and so on. It is a challenging task to relight i.e., manipulate the lighting condition of an outdoor scene image. Recently, diffusion models have shown great success in generating photo-realistic images. Motivated by this, we introduce a novel approach to use diffusion models to relight an outdoor scene. Specifically, we use a conditional diffusion model where we condition on the output of two newly introduced encoders - Surface Encoder and Shadow Encoder. The Surface Encoder is used to extract latent features from the scene intrinsics except the shadow and the Shadow Encoder is used to extract latent features from the normal of the scene and the lighting for the shadow effect. We evaluate our technique both subjectively and objectively and show that while the proposed diffusion model based relighting framework is numerically comparable to the state-of-the-art, it has better visual quality and coherence in relighting typical outdoor scenes (esp. sky region) by rendering photo-realistic images.

External IDs:dblp:conf/icpr/LaiCS24