Autoencoder for Synthetic to Real Generalization: From Simple to More Complex Scenes

Steve Dias Da Cruz, Bertram Taetz, Thomas Stifter, Didier Stricker

Keywords: Autoencoder, sim2real, mpi3d, sviro

Abstract: Learning on synthetic data and transferring the resulting properties to their real counterparts is an important challenge for reducing costs and increasing safety in machine learning. In this work, we focus on autoencoder architectures and aim at learning latent space representations that are invariant to inductive biases caused by the domain shift between simulated and real images showing the same scenario. We train on synthetic images only, present approaches to increase generalizability and improve the preservation of the semantics to real datasets of increasing visual complexity. We show that pre-trained feature extractors (e.g. VGG) can be sufficient for generalization on images of lower complexity, but additional improvements are required for visually more complex scenes. To this end, we demonstrate that a sampling technique, which matches semantically important parts of the image, while randomizing the other parts, leads to salient feature extraction and a neglection of unimportant parts. This helps the generalization to real data and can further be improved via triplet-loss structuring of the latent space. We show that our approach outperforms classification models fine-tuned on the same data.

One-sentence Summary: We investigate autoencoder architecture improvements on several datasets (from visually simple to more complex) for learning only on synthetic data and generalizing to real data of an industrial application..