SPIGAN: Privileged Adversarial Learning from Simulation

Kuan-Hui Lee, German Ros, Jie Li, Adrien Gaidon

Abstract: Deep Learning for Computer Vision depends mainly on the source of supervision. Photo-realistic simulators can generate large-scale automatically labeled synthetic data, but introduce a domain gap negatively impacting performance. We propose a new unsupervised domain adaptation algorithm, called SPIGAN, relying on Simulator Privileged Information (PI) and Generative Adversarial Networks (GAN). We use internal data from the simulator as PI during the training of a target task network. We experimentally evaluate our approach on semantic segmentation. We train the networks on real-world Cityscapes and Vistas datasets, using only unlabeled real-world images and synthetic labeled data with z-buffer (depth) PI from the SYNTHIA dataset. Our method improves over no adaptation and state-of-the-art unsupervised domain adaptation techniques.

Keywords: domain adaptation, GAN, semantic segmentation, simulation, privileged information

TL;DR: An unsupervised sim-to-real domain adaptation method for semantic segmentation using privileged information from a simulator with GAN-based image translation.

Data: [Cityscapes](https://paperswithcode.com/dataset/cityscapes), [SYNTHIA](https://paperswithcode.com/dataset/synthia)