ScoreMix: Synthetic Data Generation by Score Composition in Diffusion Models Improves Recognition

Parsa Rahimi, Sébastien Marcel

Keywords: Synthetic Data, Diffusion Models, Generative Models, Augmentation, Face Recognition

TL;DR: We analyze and leverage score composition in diffusion models as an effective augmentation method.

Abstract: Synthetic data generation is increasingly used in machine learning for **training and data augmentation**. Yet, current strategies often rely on external foundation models or datasets, whose usage is restricted in many scenarios due to policy or legal constraints. We propose **ScoreMix**, a **self-contained** synthetic generation method to produce hard synthetic samples for recognition tasks by leveraging the score compositionality of diffusion models. The approach mixes class-conditioned scores along reverse diffusion trajectories, yielding domain-specific data augmentation without external resources. We systematically study class-selection strategies and find that mixing classes distant in the discriminator’s embedding space yields larger gains, providing **up to 3\% additional average improvement**, compared to selection based on proximity. Interestingly, we observe that condition and embedding spaces are largely uncorrelated under standard alignment metrics, and the generator’s condition space has a negligible effect on downstream performance. Across **8 public face recognition benchmarks**, ScoreMix improves accuracy by **up to 7 percentage points**, without hyperparameter search, highlighting both robustness and practicality. Our method provides a simple yet effective way to maximize discriminator performance using only the available dataset, without reliance on third-party resources. _Code and synthetic datasets are available._

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 9670