WeatherGAN: Unsupervised multi-weather image-to-image translation via single content-preserving UResNet generator

Sunhee Hwang, Seogkyu Jeon, Yu-seung Ma, Hyeran Byun

Abstract: In this paper, we propose an unsupervised and unified multi-domain Image-to-Image translation model for an image weather domain translation. Most existing multi-domain Image-to-Image translation methods are capable of translating fine details such as facial attributes. However, the image translation model between multiple weather domains, e.g., sunny-to-snowy, or sunny-to-rainy, have to consider the large domain gap. To address the challenging problem, in this paper, we propose WeatherGAN based on a proposed UResNet generator. Our model consists of the UResNet generator, a PatchGAN discriminator, and a VGG perceptual encoder. UResNet is a combined model of U-Net and ResNet to address the ability of each model, that preserve input context information and generate realistic images. The PatchGAN discriminator encourages the generator to produce realistic images of the target domain by criticizing patch-wise details. We also leverage VGG perceptual encoder as a loss network, which guides the generator to minimize the perceptual distance between an input image and generated images to enhance the quality of outputs. Through the extensive experiments on Alps, YouTube driving (our benchmark dataset), and BDD datasets, we demonstrate that WeatherGAN produces more satisfactory results of the target domain compared to the baselines. Besides, we also conduct a data augmentation task to show the usability of our generated images by WeatherGAN, and it shows the overall object detection performance of YOLO v3 is improved in our results on BDD dataset.