Do We Need All the Synthetic Data? Targeted Image Augmentation via Diffusion Models

ICLR 2026 Conference Submission13941 Authors

Keywords: Synthetic data generation, Image augmentation, Diffusion models

TL;DR: We propose a novel targeted image augmentation method using synthetic images from diffusion models

Abstract:

Synthetically augmenting training datasets with diffusion models has been an
effective strategy for improving generalization of image classifiers. However,
existing techniques struggle to ensure the diversity of generation and increase the
size of the data by up to 10-30x to improve the in-distribution performance. In this
work, we show that synthetically augmenting part of the data that is not learned
early in training with faithful images—containing same features but different
noise—outperforms augmenting the entire dataset. By analyzing a two-layer CNN,
we prove that this strategy improves generalization by promoting homogeneity in
feature learning speed without amplifying noise. Our extensive experiments show
that by augmenting only 30%-40% of the data, our method boosts generalization
by up to 2.8% in a variety of scenarios, including training ResNet, ViT, ConvNeXt,
and Swin Transformer on CIFAR-10/100, and TinyImageNet, with various
optimizers including SGD and SAM. Notably, our method applied with SGD
outperforms the SOTA optimizer, SAM, on CIFAR-100 and TinyImageNet.

Primary Area: other topics in machine learning (i.e., none of the above)

Submission Number: 13941