Delving Deep into Sim2Real Transformation: Maximizing Impact of Synthetic Data in Training

Hyungtae Lee, Yan Zhang, Yi-Ting Shen, Heesung Kwon, Shuvra Shikhar Bhattacharyya

Keywords: Synthetic data, sim2real transformation, distribution gap

TL;DR: A comprehensive study to understand the sim2real transformation mechanism and the impact of using synthetic data in training.

Abstract: Synthetic data has the distinct advantage of building a large-scale labeled dataset for almost free. Still, it should be carefully integrated into learning; otherwise, the expected performance gains are difficult to achieve. The biggest hurdle for synthetic data to achieve increased training performance is the domain gap with the (real) test data. As a common solution to deal with the domain gap, the sim2real transformation is used, and its quality is affected by three factors: i) the real data serving as a reference when calculating the domain gap, ii) the synthetic data chosen to avoid the transformation quality degradation, and iii) the synthetic data pool from which the synthetic data is selected. In this paper, we investigate the impact of these factors on maximizing the effectiveness of synthetic data in training in terms of improving learning performance and acquiring domain generalization ability--two main benefits expected of using synthetic data. As an evaluation metric for the second benefit, we introduce a method for measuring the distribution gap between two datasets, which is derived as the normalized sum of the Mahalanobis distances of all test data. As a result, we have discovered several important findings that have never been investigated or have been used previously without accurate understanding. We expect that these findings can break the current trend of either naively using or being hesitant to use synthetic data in machine learning due to the lack of understanding, leading to more appropriate use in future research.

Primary Area: transfer learning, meta learning, and lifelong learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 2025