MGD$^3$ : Mode-Guided Dataset Distillation using Diffusion Models

Jeffrey A Chan Santiago, praveen tirupattur, Gaurav Kumar Nayak, Gaowen Liu, Mubarak Shah

Abstract:

Dataset distillation has emerged as an effective strategy, significantly reducing training costs and facilitating more efficient model deployment.
Recent advances have leveraged generative models to distill datasets by capturing the underlying data distribution. Unfortunately, existing methods require model fine-tuning with distillation losses to encourage diversity and representativeness. However, these methods do not guarantee sample diversity, limiting their performance.
We propose a mode-guided diffusion model leveraging a pre-trained diffusion model without the need to fine-tune with distillation losses. Our approach addresses dataset diversity in three stages: Mode Discovery to identify distinct data modes, Mode Guidance to enhance intra-class diversity, and Stop Guidance to mitigate artifacts in synthetic samples that affect performance.
We evaluate our approach on ImageNette, ImageIDC, ImageNet-100, and ImageNet-1K, achieving accuracy improvements of 4.4%, 2.9%, 1.6%, and 1.6%, respectively, over state-of-the-art methods. Our method eliminates the need for fine-tuning diffusion models with distillation losses, significantly reducing computational costs.

Lay Summary: Training state-of-the-art vision models is resource-intensive due to the need for large-scale datasets. Dataset distillation offers a promising alternative by synthesizing compact datasets that preserve the essential information needed to train models so that, when used for training, they achieve performance comparable to using the full dataset. However, existing methods often lack diversity in the generated samples and rely on costly fine-tuning. We propose a novel approach that leverages a pre-trained diffusion model to generate diverse, high-quality synthetic data without the need for distillation-specific training. We propose a three-stage mechanism that enforces diversity in synthetic samples while mitigating artifacts. This enables more efficient training pipelines, achieving high accuracy with significantly reduced data and compute requirements.

Link To Code: https://jachansantiago.github.io/mode-guided-distillation/

Primary Area: Deep Learning->Generative Models and Autoencoders

Keywords: Dataset distillation, Dataset Condensation, Diffusion Models

Submission Number: 7693