Investigating the Influence of Image Augmentations on the Sim-to-Real Generalization of Deep Learning Perception Models
Keywords: Synthetic Data, Sim-to-Real Gap, Game-Engine, Image Augmentations, Deep Learning
Abstract: Since large and diverse datasets required for training deep learning (DL) models are often unavailable, synthetic images generated from game-engines are increasingly used. However, DL models trained on synthetic images often struggle to generalize well to real-world images. This study systematically investigates the potential of image augmentation techniques to improve the sim-to-real generalization. To do so, we evaluate the influence of 25 basic pixel-level image augmentations on the real-world performance of various DL models trained solely on synthetic images. The comprehensive study covers multiple DL models, datasets, and perception tasks, including object detection and semantic segmentation. Our results show that image augmentations are a promising approach to increase sim-to-real generalization. Specific augmentations can significantly enhance model performance on real-world datasets, improving the median performance of the investigated models by over 5% and yielding maximum improvements of up to 26.8%. Furthermore, we show that especially differences in color and blur are significant factors contributing to the sim-to-real generalization problems of DL perception models.
Submission Number: 2
Loading