LD3M: Latent Dataset Distillation with Diffusion Models
Abstract: The concept of dataset distillation, which condenses large datasets into smaller but highly representative synthetic samples, is gaining significant traction because it addresses some of modern AI's core challenges, such as preserving the privacy of training data or storing replay memory samples for continual learning. However, unlocking the full potential of dataset distillation remains difficult due to two main issues. The first is architecture generalization: the distilled dataset often performs well with the architecture used during distillation, typically ConvNet, but struggles to generalize to others. The second is effectively distilling images at resolutions commonly found in standard datasets, such as 128x128 and 256x256. This paper introduces Latent Dataset Distillation with Diffusion Models (LD3M), a novel approach that is the first to combine a modified diffusion process in latent space with dataset distillation to address these issues. LD3M allows for more effective distillation of images with resolutions of 128x128 or 256x256 and improved generalization across various architectures. Additionally, LD3M enables fine control over both distillation speed and dataset quality by adjusting the number of diffusion steps. Experimental results demonstrate that LD3M outperforms state-of-the-art methods by up to 4.8 percentage points for one image per class and 4.2 percentage points for ten images per class across several ImageNet subsets.
Submission Length: Long submission (more than 12 pages of main content)
Changes Since Last Submission: - Paragraph <Impact of Diffusion> added into the results section with an additional table (see Table 6).
- Paragraph <Improved Gradient Flow> added into the results section with an additional figure (see Figure 7).
- The application of dataset privacy preservation is more detailed in the introduction.
- The methodology section clarifies the timesteps used in this work (T=20 and T=10).
- We added the time steps used in the results section for clarity.
- A1 is expanded with an analysis of classical image generation with our modification, also highlighting that LD3M is specifically tailored to dataset distillation and should not be used in classical image generation as the reverse process is altered. An additional figure was added (see Figure 8)
- A new section, A2, was added for an algorithmic description of LD3M.
- Including of DPM-Solver as future work
- Including how the selection of T happened w.r.t. Fig. 5.
Assigned Action Editor: ~Charles_Xu1
Submission Number: 3472
Loading