Intraclass Compactness: A Metric for Evaluating Models Pre-Trained on Various Synthetic Data

Tomoki Suzuki, Kazuki Maeno, Yasunori Ishii, Takayoshi Yamashita

Keywords: Pre-training, Synthetic Image, Formula Driven Supervised Learning (FDSL)

TL;DR: We introduce a general-purpose metric using feature compactness to reliably evaluate the Sim-to-Real gap for models trained on any type of synthetic data, overcoming the limitations of existing specialized metrics.

Abstract:

Models pre-trained on synthetic data like computer graphics (CG) and formula-driven supervised learning (FDSL) often underperform models pre-trained on real data in downstream tasks.
One approach to resolve this accuracy gap involves defining measurable metrics for differences between real and synthetic data or models trained on these data, and then addressing the gaps in these metrics.
Conventional metrics often fail to accurately evaluate all synthetic data types, as they are tailored to specific types or designed for real images.
Therefore, we propose utilizing the feature compactness measure as an evaluation metric for finding the gap between models.
Our experiments show that our metric strongly correlates with downstream task accuracy across a broad range of synthetic data.
Additionally, we demonstrate that our metric is useful for designing training methods using synthetic data.

Submission Number: 5